The “Year of the Data”…

Mar 05, 2012 by Rich Murnane in Big Data, Data Integration, Data Management

According to Chinese Astrology, the “Year of the Dragon” began on January 23rd of 2012, but from my window of the world, I’m thinking 2012 is really going to be the “Year of the Data.”

Signs that the “Year of the Data” was on its way began when The Economist published their “Data Deluge” articles in February 2011. In May of 2011, McKinsey published a white paper titled “Big Data: The Next frontier for Innovation, Competition, and Productivity”. The articles and white paper really made some executives out there start thinking about waking up and smelling the data. 2011 introduced us to the term “Big Data” and the silly little Hadoop elephant which appeared out of nowhere.

Side note: I have to admit, the silly little elephant is much cuter than the silly little penguins which represent the Linux operating system.

So here we are in what is now should be known as the “Year of the Data” and we find ourselves reading about Target’s Data Mining and Analytics programs in the New York Times. Note here that the DataGeeks among us aren’t the only ones reading these articles, Stephen Colbert even decided to poke some fun at the situation which can be viewed here.

Google comes in and starts the “Year of the Data” off with a couple of black eyes: The first of which is the fact that the Wall Street Journal caught them messing around with the Safari browser on iPhones, allowing them to let them monitor user behavior. The second is the fact that they are now explicitly coming out and telling users of Google products that “if you use any Google product, Google will be watching you” which from my perspective kind of feels a bit like using your work email to send someone a personal message, it’s doubtful my Microsoft Exchange Administrator will read the message, but I better not send anything crazy because they can read it if they feel compelled to do so. My friend Phil Simon wrote a little post on the topic of Google and their Data Privacy changes which can be read here.

When Gartner recently asked CIOs about 2012 and what was on their minds and to-do lists, Analytics and Business Intelligence topped the list which again illustrates that we’re now in the “Year of the Data”, don’t you think?

Here in the United States we’ve got a presidential election in the “Year of the Data” and sure enough the candidates are going to recognize the value data mining and analytics bring to their campaigns. I really enjoyed this article here by CNN which outlines what the Obama campaign is going to do in the “Year of the Data.”

The “Year of the Data” brings to London the summer Olympic games and no doubt this is big business. The folks across the pond have already been using analytics to determine the impact of the games on their infrastructure as well as in determining the best strategies for security.

Speaking of data security, some of the folks in the data security circle called 2011 the “Year of the Data Breach” and since I’m no security expert I’m standing clear of that one (while I hope these guys leave me alone).

On a personal note, I’ve been working on a project building out a custom master database for a couple years now and sure enough we’ve got some really big plans for this project during the “Year of the Data.” The database has really begun to add significant value to the client’s operations and I’m thankful that the sponsors of the project have been patient, especially since most of the delays have been because of revolving priorities (wink!). Slow and steady has brought this project to the “Year of the Data” and I’m very excited to experience it. In fact, I have to give credit to my friend Donna who is a program manager on this project for coining the term “Year of the Data.” It was Donna who inspired this post and I’m very grateful for the inspiration!

As for the rest of the world, slow and steady is not however going to be the story in the “Year of the Data” and beyond. Organizations are accumulating data at a much faster rate than any other time in history. The perfect storm of incredible bandwidth, cheaper disks, and faster CPUs coupled with tools that can manage larger data sets distributed across cheaper hardware (like Hadoop) are enabling people to gather and analyze data in a manner never thought possible.

I recently listened in on a presentation where a someone was talking about doing a “Big Data” project where they were loading up tons and tons of static data on a mainframe (into DB2) and I couldn’t help but think OMG while I was trying not to LOL and send a tweet with some serious sentiment. Although this person’s client was happy with the results (the project was on time, on budget, and it worked), I’m doubtful that the DataGeeks among us are going to be able to continue to get away with this type of architecture and thinking for much longer, especially here in the “Year of the Data”. In my opinion, relational is perfect for transactional systems but if you find yourself building a solution which does analytics on static data sets (such as log data), you’d be well served at least to recognize that it might be time to do something different

With all this said, 2012 is no longer the “Year of the Dragon”, it shall now forever be known as the “Year of the Data”.

Until next time…Rich

Tags: big data, Data Integration, Data Management