I am fascinated by the rise of “Big Data”. It has become one of those “hot topics” and looks like continuing the trend in 2012. The ability to combine large amounts of data, both structured and unstructured, from private and public sources has the potential to have a profound impact on a range of businesses (and is already in some cases).

So I am learning about Hadoop and Cassandra and Amazon Compute Clusters and k-means clustering and…..

All of this crunching gives me more data. Some of it lovely scatter plots, or charts that I can put trendlines on, and but in the end I am just ‘looking at the data’.

Now I have spent many years “looking at the data” in many different contexts, and it has always amazed me what a ‘black art’ it is. It seems that good data insight comes from an experienced person who understands both the data and the business staring at these charts like some medieval soothsayer staring at lizards gizards before incanting their insights. (For another metaphor, read ‘Data Alchemists’.)

I think one of the reasons that it is such a black art is that the most interesting ‘Insights’ are those that are not part of the collective psyche at that particular point in time. And knowing what is not known seems to be particularly challenging.

I am forever being frustrated by my own ability to spend waste huge amounts of time ‘reading’ the internet believing that this is productive. And so it seems are many others, for example, here and here

From time to time I get “motivated” and re-organise my reading habits. Sometimes I cull my regular feed lists, sometimes I limit reading to certain hours of the day, sometimes I attempt to disconnect completely for a day or more, sometimes I make a conscious effort to substitute ‘doing’ for ‘reading’. All of these are worthwhile, and work for a while…. but inevitably, like some sort of drug, it creeps its way back into more and more of my online time.

I think this experience is pretty common, and I know many have written strategies to help address this (I have read them after all!!). But what I haven’t seen discussed much is why it is such a powerful force. What is the neurological/psychological basis for this ‘addiction’. And it does seem to feel like it is some form of addiction. I am reminded of some research into ‘random rewards’ creating addictive behaviours. Does that help explain what is going on? If so, what is it about ‘reading online’ that triggers the shot of serotonin in your brain. Food, or physical drugs I can understand…. but why reading?

I read an article today (that I lost the reference to) that talked a bit about ‘the network’ – and it got me thinking about my earlier post about ‘Humans and the network’. I almost called it ‘Humans and computers’, but somehow ‘the network’ seemed more appropriate. Reading this other post today made me think that this distinction is in fact a big one.

It is only in the last couple of years really – that we have started to interact with ‘the network’ independent of ‘the computer’. For this to happen two things need to be true. Firstly, you must be able to interact with this thing via multiple devices in a seamless fashion. This was starting to become true with laptops and desktops c.2005 – but really started properly with the release of the iPhone and subsequent devices that could browse the internet – significantly better than anything before it. (Remember, that was only 4 years ago!!) The second thing needed for us to interact with ‘the network’ – is true two way interaction. The ability to be able to create as well as consume content – on the fly – and for that content to appear on sites like Facebook or Twitter and then directly impact your network – which can then respond in kind – that is real interaction.

So today – in 2011 – it seems to me that our interactions are no longer with ‘the computer’ but are directly with ‘the network’. To some this just means being able to post and view Facebook on their phone and their laptop. But we have only scratched the surface. In the next few years it will mean much more than this. For example, in the days of the relatively static internet, the internet is just one thing – your internet and my internet are more or less the same. But when we have a fully dynamic and direct interaction with the network – then your network and my network begin to look very different (just like our real world networks are very different). Our social networks, our search results, the ads being displayed to us, the products being displayed to us by Amazon – even colour schemes and news items – all start to become unique. (This gives rise to the much talked about ‘Filter Bubble’.)

To put it another way, we have come to understand that computers are good at many things… they give the same result to calculations all the time, they never get bored, they never forget, they essentially do what we tell them to do – nothing more and nothing less. But these things are not true of ‘the network’. Have you ever posted something to Facebook only to see it disappear a short while later? Have you ever had some email from a trusted source suddenly get routed to your Spam folder? The ‘network’ does forget… and it does make mistakes. ‘The network’ and ‘the computer’ are very different things indeed – and over the next couple of years – we will interact more directly and transparently with the network itself – and we will be amazed.