Defending the Data

Nov 18, 2010 by Phil Simon in Data Enrichment, Data Integration, Data Management

For better or (mostly) worse, in my professional career, I have consistently found myself on projects suffering from a bevy of issues, many of which were related to data. By 2008, I had reached a tipping point: I was either going to write a book about IT project failures or see a shrink. I chose the former.

In other words, it’s rare that, as a consultant, I have the ability to influence the direction of an organization’s data management. I find myself these days in such a place. The details of my project aren’t particularly interesting to the average reader. For now, however, suffice it to say that I am building a little ETL tool that takes a bunch of data from a bunch of places, transforms it, and spits it out to a bunch of people. I’d give this about a 4 on my 1-10 scale for complexity. (Yes, I have had to build tools that scored a 14 on that same 1-10 scale before. Take me out for a beer sometime and I’ll tell you a story or two.)

On my current assignment, I am working with a large financial institution in the midst of some M&A activity. I am working closely with a director named Dave. (This is not his real name; I just happen to be listening to the Foo Fighters right now and their front man is ridiculously talented Dave Grohl). Dave is getting pulled in a bunch of different directions and, using more tact than I have, is trying to successfully navigate his organization’s political waters. As soon as we make the changes to file formats for his internal clients, he almost just as quickly comes back with changes and “the new final” formats.

While I’m a big fan of Agile development, constantly changing things is hardly efficient. The other day, Dave told me that we just had to make three changes to “descriptions” that should be quite easy. I furrowed my brow, telling Dave that things weren’t quite as simple as that. His “descriptions” were my fields on multiple tables and queries. Without giving TMI, the changes actually involved a decent amount of work.

But then something strange happened: Dave watched me for an hour make these changes in every place. I’d run the Access-based ETL tool and show him how these ostensibly minor changes caused the whole thing to break. We had developed this on the fly because (spoiler coming) the whole tool was due well before I started.

Armed with a new-found appreciation of the major impact of minor changes, Dave has vowed to do two things now:

Run any changes by me before agreeing to make them, even if they seem insignificant.
Immediately say no to changes that appear to be major.

Dave was minimizing the chance that excessive changes would break the ETL tool and compromise what he sent out. In short, Dave was defending the data.

Simon Says

My life would be a great deal easier if more clients stuck up for their data. In my admittedly jaundiced experience, most people just want what they want when they want it. They don’t understand concepts such as data integrity, referential integrity, master data, and the like. One could argue that not everyone needs to know what we data management professionals know. Perhaps they are right. But an appreciation for the data goes a long way towards ensuring that things don’t completely spiral out of control.

Feedback

What say you?

Tags: consultants, Customer Data Integration, Data Enrichment

8 Responses to “Defending the Data”

Jill Wanless

Nov 18, 2010

Love it. Great new idea (for me anyway). I’m going to start using this term for those few data hero’s I know. Might even use if for a new communication strategy – hope you don’t mind.

Sincerely,
Data Defender Extraordinaire

Reply to this comment
Charles Proctor

Nov 18, 2010

Finance seems to be real culprits in “most people just want what they want when they want it” without thinking through the implications on systems.

Reply to this comment
Paige Roberts

Nov 18, 2010

It certainly makes good sense to minimize changes that aren’t necessary, but it’s pretty much a fact of business life that changes happen, a lot. Having a fast way to measure the impact of changes would make a lot of sense here so Dave would know the difference between a change that would break a lot of stuff, and make a lot of work for you, and one that really is minor.

My other thought is that a more flexible ETL solution that rolls with changes more easily might help a lot as well. I’m sure you built a solid solution, but if you, as the consultant who built the solution, need to be called in for every change, that’s going to be an ongoing problem. Even with a defender of the status quo in house, eventually changes will be needed. You might want to look at shifting some of the things most likely to change into easily accessible variables, and show Dave where he can change them himself. Give the business folks the power to change things for themselves, and you’ve made both his life and yours a lot better.

Paige

Reply to this comment
Phil Simon

Nov 18, 2010

Bring your cape, Jill. We superheroes need you.

Reply to this comment
Marty Moseley

Nov 18, 2010

One upside is that now Dave understands the implications of understanding his data, and it wasn’t involving a $100million program!
It’s a cheap education for him, and some revenue for you!
It’s a win-win situation!!!
M )

Reply to this comment
Phil Simon

Nov 19, 2010

Great comments…

Paige – I totally hear you. Trust me. I don’t want to be called in every time that a change needs to happen. When the client doesn’t have the bandwidth and knowledge, though, something has to give.

Reply to this comment
Debbie

Dec 12, 2011

And yet I still hear managers/directors telling me that data is ‘busy work’. I wish I could find more information regarding the importance of data integrity for decision support. The ripple effect from one little change is part of a project I have been on for over a year.

Most folks believe since they see our end product (report deliverables from an etl) that we can make changes at the drop of a hat.

Thank goodness for folks like you helping us to educate both our user community and upper management.

Reply to this comment
Phil Simon

Dec 12, 2011

Thanks, Debbie. You’ll get no argument from me!

Reply to this comment