Looking at the Data

I am fascinated by the rise of “Big Data”. It has become one of those “hot topics” and looks like continuing the trend in 2012. The ability to combine large amounts of data, both structured and unstructured, from private and public sources has the potential to have a profound impact on a range of businesses (and is already in some cases).

So I am learning about Hadoop and Cassandra and Amazon Compute Clusters and k-means clustering and…..

All of this crunching gives me more data. Some of it lovely scatter plots, or charts that I can put trendlines on, and but in the end I am just ‘looking at the data’.

Now I have spent many years “looking at the data” in many different contexts, and it has always amazed me what a ‘black art’ it is. It seems that good data insight comes from an experienced person who understands both the data and the business staring at these charts like some medieval soothsayer staring at lizards gizards before incanting their insights. (For another metaphor, read ‘Data Alchemists’.)

I think one of the reasons that it is such a black art is that the most interesting ‘Insights’ are those that are not part of the collective psyche at that particular point in time. And knowing what is not known seems to be particularly challenging.