Much of the talk about Big Data these days revolves around the 3 dimensions cited by Gartner: Volume, Variety, Velocity. Companies getting started with Big Data invariably have to come to terms with the challenges of dealing with new storage approaches, unstructured data sources and demands for hyper-speed analytics and processing.

I sometimes feel what we’re missing in the Big Data narrative is the end-game, what are companies going to do with all this data? If we mull on this for a moment then we start to see the importance of enterprise data quality management, not just for our new Big Data sources but for our existing data sets and all along the Information Chain.

For example, imagine your Big Data objective as a logistics company is to store, process and make sense of millions of RFID readings across your entire shipping inventory. The amount of data generated by these sensor networks can be incredibly useful but this information can only be utilised for operational and competitive advantage if an entire Information Chain is in place, connecting the sensory data to conventional corporate data.

The container identifier needs to be connected to the shipment manifest identifier. The manifest needs to be connected to the shipping route. The route needs to be connected to perhaps meteorological “Big Data” or forecasted transit times from the GPS readings relayed continuously off the ship. All of this helps the logistics company predict when the forward delivery trucks need to be in place at the port so that their assets are fully optimised.

It quickly becomes apparent that the customer and shipping records need the highest levels of accuracy, completeness and consistency (amongst many other dimensions) so that they can leverage all this Big Data intelligence to deliver a higher value of service, reduced operating costs and increased competitive advantage.

As we start to map out a typical business use case for Big Data we can see how value can only be delivered when data quality is managed holistically across the entire Information Chain. This may be a wake-up call for companies who see Big Data as somehow separate to their existing data landscape.

To paraphrase the immortal comedian George Carlin, words should simply convey information to people. Far too often, however they are used to confuse people. (In a famous bit on euphemisms and soft language, he talks about the unfortunate evolution of the term shell shock to post-traumatic stress disorder – two words for essentially the same thing.)

I thought of Carlin’s words recently while listening to a recent IBM event entitled Smarter Analytics Leadership Summit Simulcast. After a while, the discussion turned to Big Data as a Platform – and that just rubbed me wrong.

Do you know Big Data as a Platform even means? I sure don’t – and I like to think that I know a thing or two about platforms. The very fact that everything seems to be a platform these days dilutes the meaning of the term.

Disclaimer: I have no bone to pick with IBM in this post. I have nothing against the company or any of its employees.

There are many problems with introducing new business jargon, as I pointed out a few months ago on this site. In this post, I’ll keep my rant to Big Data as a Platform.

If “Big Data as a Platform” catches on (and I sure hope that it doesn’t), I can just see BDAP conferences and self-anointed BDAP “experts” who drop buzzwords in every sentences and confuse the very people they are brought in to help.

Think about this way:

• How many people agree on the “proper” definition of Big Data?
• How many people agree on the “proper” definition of a platform?
• How many people will even know what Big Data as a Platform means?

Simon Says

Now, within any given organization, no one is saying that everyone has to agree on everything for anything to get done. But consider the following questions:

• But, all else being equal, doesn’t a common understanding of terms engender better results?
• Isn’t there less room for confusion, miscommunication, and misinterpretation when everyone is on the same page?
• And how can everyone be on the same page when lofty buzzwords replace common sense and concrete terms?

Feedback

What say you?

It seems to me that organizations that adopt a “wait and see” attitude with respect to big data play a dangerous game.  I make this claim fully aware that:

1. Like most “shiny new things,” the “hype-to-substance ratio” (see below) is low – and getting lower everyday.
2. Gaining any real, sustained value and advantage will take a lot more dedication than almost anyone imagines.
3. “This time will be different” is almost never true.

Still, I make my claim.  I do so for three reasons. As background, I think the popular definition of “big data,” based on exceeding current storage and processing capabilities, completely misses the point.  For, except for a limited few, technical horsepower was never the limiting factor.  A better definition would reflect overall intellectual, managerial, and organizational capabilities to understand what the data mean and leverage those insights.  To “put the data to work” in other words.

For an organization struggling to interpret spreadsheets properly, a mid-sized transaction system may represent big data.  For others, a warehouse may be the most meaningful “big data.”  In a slightly different vein, a time-series analysis may represent a step up for a company that is currently doing year-over-year comparisons.

Reason one: Viewed in this light, “the big data opportunity” is building a smarter organization.  Never a bad strategy.

Reason two: There are pent-up customer demands in almost every market.  Customers need better financial products, cheaper, better health care, and on and on.  The essential ideas to meet these needs could well lie hidden in the data.  Finding them could yield intoxicating rewards.

Reason three: Your competitors are almost certainly reading the same stuff you are.  The true danger is dithering while your competitor finds new opportunity in data and gains an advantage you can’t match.

Danger indeed!

Note to readers:  As it matures, big data will almost surely introduce new metrics into our lexicon.  I may have done so with “the hype-to-substance ratio.”  The basic notions are surely age-old, but creating a mathematical construct may be new.  I did a quick Google search and don’t find the term.  But maybe I missed something.  Please let me know if you’ve seen it.  If no one has, I’ll get to work on a proper definition and method of measurement!

In a previous post, I noted, as many others also have, that Big Data is about more than just data volume. Its other two most commonly cited characteristics – variety and velocity – further complicate the big data challenge. But before I continue, please permit me to greatly oversimplify traditional data management as a two-step process:

1. Structure
2. Qualify

The easiest example of step one is the relational model, which has dominated the data management industry since the 1980s, fostering the long-held belief that data has to be structured before it can be used. The second step is the long-held belief, at least among data quality professionals, that data also has to qualified before it can be used (verifying completeness, validity, accuracy, etc.).

These two steps require a methodical approach that is slower than the velocity of big data, which refers to not only how fast data is being produced, but also how fast data must be processed to meet demand. And the biggest increase in volume comes from the variety of big data, which consists mostly of unstructured or semi-structured data. So, from my perspective, most of the big data angst is about the fear that traditional data management techniques can not effectively and efficiently structure and qualify big data before it can be used.

Different Uses, Different Approaches

We must acknowledge that some big data use cases differ considerably from traditional use cases, requiring us to reevaluate how we structure big data and how we assess the quality of big data.

An excellent example is sentiment analysis, which analyzes large amounts of largely unstructured data in an attempt to understand how customers think and feel about products and services. By its very nature, determining the sentiments your customers have requires a different data management approach than, for example, determining the number of duplicate customer records you have.

In his book The Secret Life of Pronouns: What Our Words Say About Us, social psychologist and language expert James Pennebaker shared insights from his groundbreaking research in computational linguistics – in essence, counting the frequency of words we use – to show that our language carries secrets about, among other things, our thoughts and feelings.

“Sociolinguists,” Pennebaker explained, “focus on broad social dimensions such as gender, race, social class, and power. Their approach is qualitative, involving recording and analyzing conversations on a case-by-case basis. It is slow, painstaking work. Over the course of a year, a good sociolinguist may analyze only a few interactions. Whereas the qualitative approach is powerful at getting an in-depth understanding of a small group of interactions, the methods are not designed to get an accurate picture of an entire society or culture. This is where computer-based text analysis methods can help. By analyzing the blogs of hundreds of thousands of people, for example, the computer-based methods can quickly determine the nature of gender differences as a function of age, class, native language, region, and other domains. In other words, a relatively slow but careful qualitative approach can give us an in-depth view of a small group of people; a computer-based quantitative approach provides a broader social and cultural perspective. The two methods, then, complement each other in ways that the two research camps often fail to appreciate.”

So, the loosely-structured, quantitative approach of counting and categorizing the individual words in a very large data set to assess a general, but broad, aggregated sentiment (e.g., positive, negative, or neutral) is a very different approach than the highly-structured, qualitative approach of evaluating the complete sentences and paragraphs in a very small data set to assess a more specific, but narrow, detailed sentiment (i.e., providing more comprehensive and contextual feedback).

Another excellent example of a data management solution that relies on a loosely-structured, quantitative approach is Internet search engines, which rank their results primarily according to the frequency with which the key words in your search term appear on websites. Of course, as we all know, this doesn’t always guarantee the highest quality search results, but it does enable us to very quickly search a very large number of websites from a wide variety of sources.

Discussing Structure and Quality

Big data discussions often turn into debates due to the misperception that big data always requires sacrificing structured data quality in favor of un-or-semi-structured data quantity. But the reality is that sometimes one of these approaches will be more applicable for certain use cases, and other times, these approaches will complement each other in ways that data management professionals may fail to appreciate, or perhaps just reflexively refuse to accept.

In my opinion, in order to move the big data discussion forward, and, more importantly, enable our organizations to develop strategies for using big data to solve business problems, we have to stop fiercely defending our traditional data management perspectives about structure and quality.

I will cut to the chase here: the current hysteria regarding the all-encompassing benefits of big data analytics, along with its catchy 3 (or 4) “V” formula of “Volume, Variety, Velocity” (and “Value”) is essentially predicated on the expectation that given an operational environment, parallel file system, and parallelizing programming environment, one can accomplish analyses in a much shorter time frame. This compressed “time to information” would enable real-time decision-making (or at least “faster” decision-making), and so each data management tool vendor is working as hard as it can to demonstrate that its tools can be aligned with Hadoop and thereby support “big data.”

There are benefits to providing commodity-based high-performance computing platform for algorithmic implementation. And as long as the massive volumes of data are available, these high-performance computing engines (such as those developed using Hadoop) should perform reasonably well. The bottleneck, though, is the data.

Using a big data analysis platform such as one built using Hadoop, you benefit from the inherent parallelization of execution, but the hidden cost is the latency associated with data access. Accessing data from disk is slow enough, but imagine trying to access and then pump petabytes through the limited network bandwidth to stream the data into the analytical platform. The latency associated with data motion is often glossed over when reporting application execution times, but I wonder whether the fantastic execution times would be as good if you added in the latency associated with the data access.

The challenge is that the latency problem gets worse as the data volumes grow, so if you are developing a big data application, testing it using a reasonably-sized data set, you might not even notice the tax that the data loading process is assessing. However, you must consider the scalability issues, and if your parallel environment hasn’t been fitted with equally scalable networking and I/O channels, you will eventually feel the pinch. In fact, MapReduce is not insensitive to the latency issue, since each transition between Map and Reduce phases will, by necessity, require broadcasting data across the system’s network.

Since data access and exchange latency is the limiting factor for big data performance, it has the potential to be the cloud that rains on the big data parade. That being said, alleviating the latency bottleneck is likely to become a key issue for anyone who wants to tackle really big data. That means high performance data integration.

Yes, even though the term “data integration” is not as sexy or mesmerizing as “big data,” it might be the name of the technology that enables high performance analytics. And to accommodate the massive volumes, it means a number of key considerations for companies providing data integration technology. Here are some things to think about:

• Optimizing communication channels to provide pipelined streaming of data;
• Embedding computation within the communication network (remember “active networks” research?)
• Data federation and virtualization
• High-speed virtual data caching
• Query optimization prior to “pushing-down” to the source
• Integrated event stream processing within the integration layers
• Compression
• Dynamic data realignment (to take advantage of alternate record layouts and orientations)
• Bulk data loading
• Data replication

This is just a short laundry list. I am pretty convinced that solutions for high performance computation without a strategy for high performance data movement is bound to be bound by the latent latency inherent in data access.

According to the all-knowing WikiPedia, when singer/songwriter Phil Collins wrote the 1981 hit In the Air Tonight the lyrics were not about a drowning incident he witnessed like most people thought.  The lyrics were written spontaneously while Collins was reminiscing about the “anger he felt after divorcing his first wife Andrea in 1979″.

So, a misunderstanding and an old problem (love), sounds a bit like “Big Data” to me…

“Big Data” a misunderstanding?

“Big Data” means different things to different people, hence the misunderstanding.  To many techies “Big Data” means technologies such as those in the Hadoop family, along their associated distributed physical data architecture.

To the business executive, “Big Data” is all about making their business better by “data mining” through these piles of crap data to find insight about business which would never have been available had the executive not cut the check for managing all this data.

To DataGeeks among us, it’s all about V³ (Volume, Velocity, Variety).  Organizations which never had to manage terabytes and petabytes of data (Volume) now have to figure out what we need to do to manage this in an effective and efficient manner.  Our data is now growing so much faster than we’ve experienced in our careers, are there things we should be doing differently if our databases are growing at a rate of 500% per year?  What about 2000% a year?  Talk about Velocity.  And talk about Variety, we’re now concerning ourselves with machine generated data such as sensor and log data.  If you add unstructured documents such as video, binary documents, and XML documents and we’re no longer thinking in rows & columns, a brave new world.

“Big Data” an old problem?

Large government agencies along with the big players in the web have been managing “Big Data” for much longer than the rest of us.  There are lessons to be learned from all these folks, particularly about parallel processing and distributed physical data architectures.  Large data processing shops such as credit card companies typically managed large datasets by purchasing big old mainframes and storing everything on file based data stores.  In the late 1990′s when I was a DBA on a Very Large Database (VLDB), I remember opening a trouble ticket with my RDBMS vendor asking them “ahh, is there anything I need to do if my database is growing 200% per month?”.  Folks adding value to their organizations by using data isn’t anything new either, what’s new is the attention this facet of a business is getting these days.

The best part about “Big Data” is that people are really starting to understand that data is an asset to an organization.  The part that keeps me up at night is making sure everyone understands that the same best practices and data management principles apply to data sets of any size, big or small.

“Big Data” is here In the Air Tonight, don’t you think?

Until next time…Rich

What is “Big Data” anyway?  For some companies big data is based on the volume of huge data tables or files.  Is 6 billion rows big data? To some companies it is considered big data.  Some companies consider graphics inclusion in the data warehouse as big data because it brings complexity to the drawing board of BI.  An example would be storing and showing a picture of a product in the reporting and analytics environment alongside product historical sales information.  Other companies believe that big data consists of different types of complex data types, that are brought together on one platform, where sophisticated software crawls thru the data searching for something we didn’t already know.

There is hardware and software available just for crawling thru the complex data, but it is expensive. I get calls almost daily now, wanting to know if I know anything about Hadoop.  That means people are starting to think in the direction of big and complex data.  So, all the more important for us to make we are using it for the right reasons or we could lose the opportunity to learn something new about our data and business.

My biggest fear about implementation of a processor and software to analyze big data is, What if we go to the trouble and expense of processing big data, and no one uses it?  What if we don’t take the results of the analysis and apply to our day-to-day business.

It is crucial to make sure the requirements we gather are RIGHT for our company when we talk about big data.  Not every company has a need for big data – it should all be based on scope and requirements.

Now, for your data-overwhelmed folks, I understand the desire for a more simplistic view of your data universe. I sympathize. While I’m a small business owner, I too am awash in a sea of data. Along these lines, I recently spoke with media expert Catherine Davis on the challenges of knowing where to spend my limited marketing budget.

Uncertainty + Data = Less Uncertainty

You see, it’s not easy for me to know where I should spend my limited time and financial resources. Among the many questions that I face on daily basis are:

• Should I write “exposure-only” guest posts for Inc or Huffington Post? What percentage of links to the book’s site will result in conversions?
• Should I spend $1,000 on an ad that theoretically reaches 200,000 people? Will this ad pay for itself?
• Should I attend a conference at my own expense in the hope that I can meet some people who can help me promote The Age of the Platform?
• Do tweets or other social media mentions matter?

The answer to each of these questions: I honestly don’t know.

And there’s the rub for the independent author: It’s extremely difficult to determine the source of book sales. There are just too many data points for me to know for certain how people found out about my book, much less bought it. Sure, there are major events that drive sales – e.g., a review on Slashdot, an Oprah appearance (back in the day, at least). For the most part, though, I’m clutching at straws.

At least for me, data management is more art than science. If I had written a book in my teens, at least it would have been easier for me to track my sales and assess the potential effectiveness of any marketing efforts. These days, it just ain’t easy to figure out what’s going on. While it might not have been easy for authors 20 years ago, I’d argue that it was probably easier.

Ah, the days of relative data minimalism.

Simon Says

And if it’s this difficult for me, then imagine the CMO of a multinational organization running twelve concurrent international campaigns at different times. Can organizations apply some scientific and mathematical rigor to their data management practices? Of course. But I don’t see a day anytime soon in which the vast majority of organizations will be completely able to eradicate doubt and uncertainty from their decision making processes.

Brass tacks: Embrace the uncertainly – and use your data to minimize it.

Feedback

What say you?

For The Goal is a novel, told in first person by Alex Rogo, a beleaguered plant manager, as he struggles to implement the lessons imparted his old teacher Jonah.  Jonah steadfastly refuses to tell Alex anything, instead forcing Alex to work through the questions he poses by himself.

It is The Goal’s homespun examples that most impacted me.   In one, Alex is taking his son’s scout troop on a hike.  One of the boys is slower than the others and, as Alex works out how to maintain order, he reflects on a related problem at the plant.  The powerful lesson is that quality management is not just for work.  It provides powerful tools for everyday life, raising kids, managing one’s weight, and so forth.  Sometime later I read Coveys’ The Seven Habits of Highly Effective People, which further brought the lesson home.

But wait.  There’s more.  Too many of us have a tendency to think that what we do is inherently complex.  But after The Goal I knew:  If I couldn’t explain it (whatever it is) to a “regular person,” I don’t understand it well enough.

This is a really important lesson.  Many of my clients complain that “their management doesn’t get the data quality joke.”  My response is always the same, “Then you’re not telling it right.”

If you haven’t done so already, read The Goal. It is a fast read.  And even better as a slow, careful, re-read.

Next week:  Drucker

After successfully passing both aspects of this test, I analyzed my smartphone’s log file and was surprised to discover how much data I used – nearly 5 GB in less than 24 hours.

Of course, the reason that mobile providers offer unlimited (as well as a variety of limited) data plans is because, in our increasingly data-constructed world, more and more of our daily activities, both personal and professional, involve using data – and sometimes using a lot more data than we realize.

When most people discuss the unrelenting data management trend of Big Data, many express concerns about Information Overload. But when that term was originally coined over 40 years ago, the primary concern was not about the increasing amount of information, but instead the increasing access to information. I think that the unlimited data plans from mobile providers are an excellent example of this because most smartphone data usage is about access to, not accumulation of, data.

For example, a large percentage of my smartphone data usage was allocated to data streaming services (Pandora for music, Netflix for television shows). With data streaming, I am accessing data without accumulating data, i.e., I am not downloading audio and video files to save on my smartphone’s hard drive (which has a very limited storage capacity anyway).

Of course, one reason for this is financial, e.g., if I wanted to download the MP3 file of a favorite song, I would have to purchase it, whereas Pandora offers a free music streaming service (if you ignore the cost of buying a smartphone and paying for a monthly mobile service plan).

However, to me the more interesting aspect of my smartphone data usage is that I am not creating a smartphone data silo. Unlimited data access has not required unlimited data storage since the vast majority of the data I use via my smartphone is not retained.

Compare this with the data silos that proliferate within the enterprise data management landscape of most organizations. Many, if not most, data silos consist of data that was retained for a very specific and short-term use. However, after this data is used, it remains retained, eventually becoming data that’s managed just because it’s there.

Sometimes, I think that we create data silos just because we can. So, I can’t help but wonder – could unlimited data limit data silos?

In other words, could the enterprise data management equivalent of an unlimited data plan provide users access to unlimited data streaming services without allowing them to create local copies of enterprise data, therefore limiting the proliferation and retention of data silos?