If your database has export capabilities, use it. Now!

It should be to no one’s surprise that ETL is a process that should go out the door. If you read my two prior posts, you will see how newer databases that employ export functionality provide far better ways to capture data and send it data warehouses.

The difference in data quality is stark.  Through an ETL process for capturing data and loading it into databases, you have to work through several sources, some of which you may never have the data you need. Sometimes it feels like you are a magician for making data appear.  Then you have the export process which sends all data, that you choose, in as raw as a form as possible for the data gurus to play around with it an mold it into terrific stories.

Continue reading…

Extracting, Transforming, Loading data into a warehouse database

The ETL process is a common practice in nearly all companies that have data that want to analyse or repurpose. Often are times that products cannot be maintained due to resource constraints, that the developers working on the product aren’t aware of the goals from those working with warehouse data, or the cost to update the product is too high to replace ETL with a far better process as I described in my last post.

I’m not in any way going to say that ETL is bad, because unfortunately for many, that process is required and there is no other way to get the data.  Some of my greatest data challenges came out of ETL processes earlier on before developing a full platform capable of capturing and sending the data to the warehouse in a straightforward and easy to use format.

Continue reading…

Contextual Series: User Engagement

In the second part of the contextual series, I will be continuing from where I left off last time with User Onboarding. I’ll be covering how to make your application relevant after the user has gotten past their initial experience.  The goal is to engage with the user in a way that will retain them and have them spend money.

The first few minutes of the user trying your product is crucial. In that short period of time they will decide whether they will uninstall it and move on or keep trying it out. But the next 30 minutes are also just as important. Like a drug, you want the user hooked to your product, you want them to feel as though they are dependant of it.  The most common way products get you addicted is through social engineering by getting you to engage with people you know. While it is something I would recommend each product would do, the product should be able to stand on it’s own even if the user has no friends.

Let’s assume that your product is capable of tracking a lot of data, including basic user profiles to narrow down their demographics and activity history. The data can be used to improve your analytics, determine business logic, and can be used to feed into your contextual engines.

Continue reading…

Data driven design

Data driven design is a rather rare term that gets tossed around these days.  I often find those suggesting they are making decisions based on data are doing so on a limited set of data, unable to show the entire picture.  As I mentioned in my last post, data may not have been something I could easily query or find the answers to all my problems, at least not until we’ve launched our latest platform.  Having this platform has transformed the way I think and react to events happening within games or applications.

It is great to have a large amount of reports, often a good set of pre-defined reports will cover the majority of questions regarding the general health of your application, game or website.  However, digging deeper and finding the answers necessary to making the correct changes require a unique set of questions that will help fetch data to analyse different results or falloff points.  That’s to say, in order to be successful with a data driven design model, you need someone capable of asking the questions, a system to produce the answers, and someone capable of analysing the answers.

Asking the right questions I’d say is likely one of the toughest parts of this design model.  Often a question like “why is my revenue down? Are users not buying items?” is far from an adequate enough question.  You need to ask questions like, what levels are my core spenders in?; what is the break down of my daily active users?; how well do new items sell?; what are the top selling items?; and so on.  I’ll use this example throughout this post, as it is a good question, often difficult to answer.

Common problems found in trying to figure out problems with revenue is that the first place one looks is the item sales performance.  Many times, the breakdown of the top selling items seems to be relatively the same as a high spending day, but simply less users spending.  Some may mistake this as a sign of someone being bored of items and order more items to be developed.  However, a proper analysis would be to look at the break down of spending users by level as well as a break down of daily active users by level and compare those on high selling days versus lower selling days.  Other places to look at is the game economy (a currency imbalance in the economy?), break down of spenders by source, look at the user’s inventory amount, see if the users have room to build. These breakdowns would identify problems.

The system for finding answers is crucial.  A proper system will allow an analyst the ability to find the answers he needs without asking developers to run queries on data.  As such, it makes the model more efficient and begins to set clear boundaries for what an analyst, a developer, a designer, etc. should do.  A system capable of fetching data, processing it and graphing it in a clear concise way will likely lead to finding out the proper answers and making fewer judgement calls.  Judgement calls, in my experience, are often guesses (including educated guesses) on why certain events exist.  This isn’t a shameful plug for our platform, but ideals of any designer or analyst or executive who wants to know answers.

Analysing data retrieved can often be tricky.  Sometimes, one will be required to process through events (like bugs or marketing events) to see if the data is part of a trend or whether something affected the data.  The analyst will also need to identify whether the data they now have is useful or whether they need to dig deeper.  Taking my previous question of “why is my revenue dropping”, we  found that core spenders were often within a range of levels in the game, keeping those spenders happy will result in higher sales.  The users need to have more buying options, room to build those items, space in their inventory to buy the items, incentives to buy them (cool looking) and of course money to buy them items (must be priced properly).  The data would show whether those are addressed, by ensuring there are no sharp drop-offs in the break down of daily active users by level, seeing something like that would often mean the users are missing something at that level that makes them want to continue paying or even spending.

Data driven design, I think, is relatively new and underused.  Larger companies use them and the results are clear.  But looking at smaller or medium sized companies, it is clear this simply isn’t done.  In many cases, likely due to the lack of staff / time or expertise.  This is something I hope everyone could do or use, without someone with a PHD in stats or without needing a team of developers a year to build.