Data driven design is a rather rare term that gets tossed around these days. I often find those suggesting they are making decisions based on data are doing so on a limited set of data, unable to show the entire picture. As I mentioned in my last post, data may not have been something I could easily query or find the answers to all my problems, at least not until we’ve launched our latest platform. Having this platform has transformed the way I think and react to events happening within games or applications.
It is great to have a large amount of reports, often a good set of pre-defined reports will cover the majority of questions regarding the general health of your application, game or website. However, digging deeper and finding the answers necessary to making the correct changes require a unique set of questions that will help fetch data to analyse different results or falloff points. That’s to say, in order to be successful with a data driven design model, you need someone capable of asking the questions, a system to produce the answers, and someone capable of analysing the answers.
Asking the right questions I’d say is likely one of the toughest parts of this design model. Often a question like “why is my revenue down? Are users not buying items?” is far from an adequate enough question. You need to ask questions like, what levels are my core spenders in?; what is the break down of my daily active users?; how well do new items sell?; what are the top selling items?; and so on. I’ll use this example throughout this post, as it is a good question, often difficult to answer.
Common problems found in trying to figure out problems with revenue is that the first place one looks is the item sales performance. Many times, the breakdown of the top selling items seems to be relatively the same as a high spending day, but simply less users spending. Some may mistake this as a sign of someone being bored of items and order more items to be developed. However, a proper analysis would be to look at the break down of spending users by level as well as a break down of daily active users by level and compare those on high selling days versus lower selling days. Other places to look at is the game economy (a currency imbalance in the economy?), break down of spenders by source, look at the user’s inventory amount, see if the users have room to build. These breakdowns would identify problems.
The system for finding answers is crucial. A proper system will allow an analyst the ability to find the answers he needs without asking developers to run queries on data. As such, it makes the model more efficient and begins to set clear boundaries for what an analyst, a developer, a designer, etc. should do. A system capable of fetching data, processing it and graphing it in a clear concise way will likely lead to finding out the proper answers and making fewer judgement calls. Judgement calls, in my experience, are often guesses (including educated guesses) on why certain events exist. This isn’t a shameful plug for our platform, but ideals of any designer or analyst or executive who wants to know answers.
Analysing data retrieved can often be tricky. Sometimes, one will be required to process through events (like bugs or marketing events) to see if the data is part of a trend or whether something affected the data. The analyst will also need to identify whether the data they now have is useful or whether they need to dig deeper. Taking my previous question of “why is my revenue dropping”, we found that core spenders were often within a range of levels in the game, keeping those spenders happy will result in higher sales. The users need to have more buying options, room to build those items, space in their inventory to buy the items, incentives to buy them (cool looking) and of course money to buy them items (must be priced properly). The data would show whether those are addressed, by ensuring there are no sharp drop-offs in the break down of daily active users by level, seeing something like that would often mean the users are missing something at that level that makes them want to continue paying or even spending.
Data driven design, I think, is relatively new and underused. Larger companies use them and the results are clear. But looking at smaller or medium sized companies, it is clear this simply isn’t done. In many cases, likely due to the lack of staff / time or expertise. This is something I hope everyone could do or use, without someone with a PHD in stats or without needing a team of developers a year to build.