Big Data: Dangerous to Sit this one Out

May 16, 2012 by Thomas Redman in Big Data

Midway through Big Data Week at the Data Roundtable, Thomas Redman drops by to argue a better definition, while detailing the dangers of a “wait and see” attitude as it pertains to Big Data…

It seems to me that organizations that adopt a “wait and see” attitude with respect to big data play a dangerous game.  I make this claim fully aware that:

1. Like most “shiny new things,” the “hype-to-substance ratio” (see below) is low – and getting lower everyday.
2. Gaining any real, sustained value and advantage will take a lot more dedication than almost anyone imagines.
3. “This time will be different” is almost never true.

Still, I make my claim.  I do so for three reasons. As background, I think the popular definition of “big data,” based on exceeding current storage and processing capabilities, completely misses the point.  For, except for a limited few, technical horsepower was never the limiting factor.  A better definition would reflect overall intellectual, managerial, and organizational capabilities to understand what the data mean and leverage those insights.  To “put the data to work” in other words.

For an organization struggling to interpret spreadsheets properly, a mid-sized transaction system may represent big data.  For others, a warehouse may be the most meaningful “big data.”  In a slightly different vein, a time-series analysis may represent a step up for a company that is currently doing year-over-year comparisons.

Reason one: Viewed in this light, “the big data opportunity” is building a smarter organization.  Never a bad strategy.

Reason two: There are pent-up customer demands in almost every market.  Customers need better financial products, cheaper, better health care, and on and on.  The essential ideas to meet these needs could well lie hidden in the data.  Finding them could yield intoxicating rewards.

Reason three: Your competitors are almost certainly reading the same stuff you are.  The true danger is dithering while your competitor finds new opportunity in data and gains an advantage you can’t match.

Danger indeed!

Note to readers:  As it matures, big data will almost surely introduce new metrics into our lexicon.  I may have done so with “the hype-to-substance ratio.”  The basic notions are surely age-old, but creating a mathematical construct may be new.  I did a quick Google search and don’t find the term.  But maybe I missed something.  Please let me know if you’ve seen it.  If no one has, I’ll get to work on a proper definition and method of measurement!

Tags: big data, Big Data Week