It is Sunday night, CBC is going well and no server hiccups at all, so I’d take a bit of time to post some stuff and benchmarks we’ve hit with Lightning. Lightning is the name we are calling our new platform. Not only does it sound better, it also works with a few other products that are coming out that support Lightning. Lightning is a name that has meaning for the goals we are looking to accomplish.
I haven’t posted in a while. I know. Its been quite an exhaustive past few months, but I’m still alive and kicking. I’ll post a very short summary of what I’ve been up to since March.
Social Game Universe and the Lightning Platform are now powering a live CBC TV show called Over the Rainbow. Over the Rainbow is a production which asks the audience to pick their favorite Dorothy. During the show, viewers get to follow along with their computer or mobile device, interact with the TV show and vote at the end of the episode. On the 8th episode, one Dorothy will be cast to Andrew Lloyd Webber’s production of the Wizard of Oz. This project was a true test to Lightning’s scalability and prowlness, achieving higher amounts of users per minute than we’ve achieved with Dirty Dancing, running on less servers. The average API call executed in 30ms (0.003 seconds) which really isn’t too shabby when you are seeing such a high number of users, a number I am hoping I can disclose in the near future!
[EDIT: For some odd reason, WordPress decided it wanted to delete the rest of this post… so here’s to writing it again]
Haven’t you ever told yourself while in school : “I won’t ever use this again, why am I learning this?”. Well I do all the time, being that I am fresh out of university, those are still thoughts that continue to come to me. However, recently I’ve discovered that I am beginning to use concepts learned in school in order to adapt them to real world scenarios. More importantly, I am using algorithms. There are two algorithms I am using primarily these days, that revolve around Tree Structures and Statistics (as in that really confusing class you need to take when enrolled in Computer Science).
Dirty Dancing’s launch on September 13th 2011 has been quite exciting. Since the launch, we’ve witnessed over 1.1m eyeballs. Scaling had its own set of complications, of which we all managed to resolve in a relatively good order. In the first days of launching, we achieved some interesting numbers, perhaps not comparable to what companies like Zynga achieve, but great nonetheless. We’ve been consistently growing since the launch, averaging over 20k installs per day. Over a million wall posts made and invites sent. However, the stats of the game and interactions aren’t what I’d particularily like to focus on, rather, I’d like to look into the challenges we’ve encourtered.
For launch, our original analysis of the servers indicated we could support at least 100k DAU. Such seemed to be reasonable based on the dozens of tests made. However, what we had issues with the most were bursts of traffic. Some bursts of traffic had enough traffic that would equate to around 600k DAU. As such, it lead to some rather interesting effects on the servers, making it hard for them to stay up or chugging along at a normal rate. This was our biggest problem since the launch of the game, which we resolved by setting up a proper database setup to properly replicate and manage the load over multiple nodes.
On October 1st, Facebook turned on forced HTTPS support. We use Zeus Traffic to balance our traffic, which made this migration simple, by sending the HTTPS traffic to it, decrypting it, sending it back to the web nodes and then back to the client encrypted again. This meant that no changes were made to our server side code to support HTTPS. We did have to change some things on the client side to support it, but it was a rather simple set of commands, which replaced all instances of HTTP with HTTPS when in secure mode.
Since the launch, our databases have registered over 15b queries and stored over 25GB worth of data. Such has caused some rethinking about how data would be stored, how indexes would be generated, and how we run some of our queries throughout the platform. In some cases, we have managed to cut 80% of stored data size (including indexes) and managed to cut the number of queries by 75%. These have been particularily interesting challenges, as it would have to be done in such a way as to not effect current players and limit downtime.
Although the game has been running smoothly for the past few weeks, the problems are now ones that are to be addressed with data. Data is always something I’ve loved and could never really get enough of, but the problems for me have always been finding a properly indexable way of storing and querying this data. Such a goal was accomplished by our platform and has given us more than enough data on our users, now the problem lies in making software to analyse this data in order to make reacting more efficient.
I’ve gotta say, the past few weeks have been tough, but exciting all the way through. The questions we’ve asked ourselves were how to be innovative, but fun at the same time. Often that is a mix that doesn’t go well together, which has lead to a lot of work massaging the user experience and making the game fun.
I still can’t quite tell what this game is, but I can assure you, it will all be known in a matter of days as we await the final word from our partners. I can’t wait to blog and write about this game. By far one of our greatest games, the technology under the hood is very unique and the game mechanics are ones you’ve never seen in a social game before.
In the meantime, enjoy an image of squirrels being annoying around some golfers attempting to make their putt.
Despite starting this post yesterday, figured I’d give a quick update, since the picture I last posted was a day 5 beard. Surprisingly has progressed quite a bit since then! Don’t worry, a more detailed update will be given when we reach closer to launch.
Our latest project at Social Game Universe has been extremely exciting. Excited because we are innovating and creating something totally unique and fun in a social game. I look forward to announcing what this project is, in the meantime, I will talk about this growth I now have on my face.
Unfortunately I was blessed with unusually fast growing facial hair. As you can see in the picture, that is only after 3 days. You can only imagine what it is now at 6 days. If you cannot imagine it, pay attention to future posts.
As much as I hate this beard, it is my commitment to the project and the team behind it. This thing is irritating, itchy and I can’t stop getting food pieces stuck in it! I want to get this shaved off fast, any delays will result in my irritation increasing.
Needless to say, the date is set and I will do anything within my capabilities to ensure no delays!
Sometimes I like to think of the past as a good example of how much I have learned. Scaling is a particular domain that I have learned a lot over the past year. 2 years ago, I had always assumed that scaling merely consisted of upgrading servers. The concept of putting servers together to work side by side was baffling to me.
I do like to poke some fun at my past, only because today I am proud of my accomplishments, proud to say I have scaled a product that can easily handle hundreds of thousands / millions of users with a very small pool of servers.
However, the point of this post is not so much to poke fun at myself, but to teach some of the places I have made mistakes and how to get around them. Of course, without giving some of my secrets or the key to making extremely responsive APIs, like igapi.
Continue reading…
I must admit, building Facebook apps is certainly not what I had envisioned I would be doing some 5-10 years ago. What surprises me even more is how much such a platform has evolved. Nowadays, it seems that you need a degree in social behavior, be a marketing genius, along with a PHD in software engineering. Nothing is as easy as it was, but I do like challenges.
Back when the platform just came out, I could bang an app out within a day, send some invites to friends and put links up on various websites, next thing you know I had thousands of people in my app. One at it’s peak reaching 800,000 Daily Active Users. Whatever happened to those days? I miss them! Getting those kinds of numbers on new apps will likely run you a few hundreds of thousands of dollars. It is a shame, because now the smaller guys have absolutely no chance to compete with the bigger guys.
The complexities and the level of detail required to make a game or app function are also astonishing. Thankfully for me, I grew up with the platform and consider myself to know many of the dos and don’ts (there are a lot). Apps and Games that you want to be successful require the love one mother would give to any child (Happy Mother’s Day Mom). You want success? Well you need to focus on user messaging, social behavior, providing value to your content and giving continuity to your game. Don’t try to silo those 4, because there is a LOT of overlap between them. Continue reading…
More and more, igapi is getting press. Earlier this month, Canadian Innovation Exchange has nomiated Social Game Universe as one of Canada’s most innovative companies, with igapi.
The Canadian Innovation Exchange (CIX) revealed the final slate of nominees that will compete to be included among Canada’s Hottest Innovation Companies. The annual awards are handed out in three categories: Information and communication technologies, clean technology and digital media. A total of 25 companies received nominations this year.
“Congratulations to all our nominees for showing the best Canada has to offer the world,” said Rick Nathan, co-chair of the Canadian Innovation Exchange.
Each of the finalists will make a live presentation in front of a panel consisting of leaders from major corporations, entrepreneurs and investors. Winners will be announced at the Canadian Innovation Exchange, due to take place on December 7 at the MaRS Discover Centre in Toronto.
igapi was recently showcased on IGN as the first playable cross-promotion toolbar:
TORONTO, Canada – Social Game Universe, pioneers of the concept of inter-connected social games, today announced the launch of the Inter-Game API (IGAPI), a solution for game developers to drive traffic by letting users “play” through a cross-promotion bar.
IGAPI uses the power of the social graph to let players interact with their friends across games and apps without leaving the game they are currently playing. Rather than seeing a typical banner ad or cross promotion, players see IGAPI “Action Cards” that they can play on their friends in other apps. Action Cards can vary from the simple (e.g., “Give film to your friend Nathon to help him grow his studio in Hollywood Tycoon”) to the more zany and bizarre (e.g., “Release a horde of zombies on your friend Jen’s horror set!”). IGAPI’s actions are limited only by the imagination and creativity of the developers looking to attract attention to their games.