Lightning at its core is a SaaS development platform. It contains all the core features any application requires to launch and be successful. Lightning consists of 3 major pieces: Object Store, Analytics and Users. What makes this offering unique is that it takes those 3 major pieces and creates a solid and interconnected foundation. This means the system has an in depth knowledge of users and the objects users interact with for analytics, objects can be catered around user profiles and their activities, or having a deep understanding of the user based on their activity. In the age of big data and contextual experiences, Lightning stands out as a simple to use out of the box solution.
The object store is an application’s database. This contains all the data for the application, anything from items that a social game may sell in their virtual store to a user interface for your mobile application. The idea behind object store is that it would be capable of storing any type of data to remove the need of building, scaling and maintaining a database. To many developers doing those things can either be daunting or impossible for them to do. What happens when you need to update your content? Many apps these days do not have a backend because developers either didn’t know how to launch it or couldn’t find the resources to do so. Not only does Lightning offer a simple API for storing, retrieving and modifying objects, it has a nice CMS to do it.
The object store supported schema. Most of us who know and work with SQL can relate to this functionality. The schema for object type defined how all objects who adopt that schema will store its data and operate. This is the equivalent to creating a table, all objects created are rows within the tables. This meant if we have a news article the schema would contain columns for title, author, date written, excerpt, content, etc.
Object properties (columns) can be stored in any data format with support for attachments. Currently supported data types include:
- varchar (255 chars or less)
- text (more than 255 chars)
Actions are Lightning’s biggest core differentiator and I’ve done a terrible job describing it in the past. The reason this is such an important part of Lightning and objects is that actions determine how users can interact with every single object in the Object Store. This makes analytics powerful and very detailed all while understanding how objects are related one to another. Whenever users read an article, this will execute the read action for an article object. When you have multiple articles that can be read, you suddenly have the capability of combining all articles with the same actions to generate insights on what types or articles users read, how many they read, etc. But then you also begin to understand which kinds of users read articles as well.
The idea behind actions is that developers won’t need to implement a whole new analytics system, the actions themselves trigger the analytics from within. If you add rules to your action then the rules can either be met or they will fail. Each result will be logged against the analytics for your viewing pleasure. For rules, you can implement anything you’d like. However, for the sake of providing an example, still using articles, say you want to limit users to read 5 articles until they buy a subscription, you can enforce those rules directly through the actions associated to objects.
Due to the great deal of context that comes out of every activity generated by the user, analytics are thus powerful to let anyone know what is going on in their product. All activity is tracked to the millisecond allowing reports on activity to use any range of activity and update at any interval. If you want a real time report that updates every second or a daily report that updates daily, both will be possible.
Lightning’s technology has evolved a lot over the years. This was due to ever growing demands to scale and host GB or even TB of data. Of course when building out the system, I was looking to solve many problems of which no one solution could do out of the box. Frameworks weren’t possible, they were not fast enough nor would they be able to scale well enough to numbers Lightning was looking to handle.
On the version that is currently running the stack looks something like this:
- Riverbed Stingray Load Balancers
- Solaris nodes for web processing
- C for connecting nodes and keeping each machine up to date
- CentOS nodes for frontend database
- Java server
- CentOS nodes for the warehouse database
- Powered by Vertica
The stack is performance built and optimized. At our peak, we were handling over 70k transactions per second. The databases were tested to support over 1m requests per second. As many of us know, PHP has a lot of great things about it, unfortunately it falls short when trying to scale due to the way the code is executed. I had to work a lot towards building a system that was mostly working from cache and performed limited processing.
What is next?
Lightning on it’s current stack has gotten me very far, but there are still a lot of limitations that I would like to work through. Some of which include the need for Socket support. Sockets are something I’ve wanted to add for a very long time. The other limitations are around the API. I wanted to deploy a full RESTFUL API and one that also had support for versioning.
The next version of Lightning is being built on Node.js. This would essentially replace the PHP portion of the existing stack with this new one. I am also updating the Java portion to put out a new Object Store. Many of the updates to the Object Store are tweaking bits and pieces to make it more user friendly.
With socket support, any updates made in Object Store will now have the ability to be pushed to every connected device. This opens up a world of possibilities including the ability to now cache content indefinitely. This also means that applications will also not become out of date when changes are made, as they can be sent to the device in real time. Additionally it means that notifications and other multiplayer features are now immediately possible as devices will be alerted when something affecting the user has been updated.