Building a data warehouse

The databases of today are in many cases built for specific purposes.  Some of the more common ones we see every day are relational databases, document-oriented databases, Operational databases, Triplestore, and Column-oriented databases / c-store. Typically relational, document-oriented, operational, and triplestore databases are used to solve frontend database problems.  Then you have databases that are column-oriented or similar that focus on solving warehousing and backend database problems.  These products don’t need to solve those problems, though they are often best suited for them.

Continue reading…

The challenges of scaling your data vertically

There are many reasons for which databases must be scaled.  The majority of the time it must be scaled to accommodate for performance issues as the product grows.  Though NoSQL is making a lot of noise these days, it is to no one’s surprise that SQL is still extremely popular.  In general the same principles are followed while scaling out any SQL product, be it MySQL, MsSQL, Oracle or even DB2. Scaling is often done to overcome performance issues as the product grows. However, dealing with big data scaling is often done to balance the data across multiple hardware nodes or clusters.

Continue reading…

Scaling – boy I’ve matured

Sometimes I like to think of the past as a good example of how much I have learned.  Scaling is a particular domain that I have learned a lot over the past year.  2 years ago, I had always assumed that scaling merely consisted of upgrading servers.  The concept of putting servers together to work side by side was baffling to me.

I do like to poke some fun at my past, only because today I am proud of my accomplishments, proud to say I have scaled a product that can easily handle hundreds of thousands / millions of users with a very small pool of servers.

However, the point of this post is not so much to poke fun at myself, but to teach some of the places I have made mistakes and how to get around them.  Of course, without giving some of my secrets or the key to making extremely responsive APIs, like igapi.

Continue reading…