The Zen of Solving Data Problems

Jun 14, 2012 by Phil Simon in Data Management

On a recent episode of Charlie Rose, Jonah Lehrer discussed his new book, Imagine: How Creativity Works. During the show, Lehrer mentioned the nature of creativity, of solving many complex problems. When pressed, most people drink a Red Bull or a double expresso. They increase the amount of pressure that they put on themselves and others.

And this is exactly the wrong thing to do.

In his book, Lehrer claims that taking a step back (or five) is often the best course of action during circumstances like these. Hardly a revolutionary concept, but how often do most of us follow this? We press ourselves and others when confronted with a vexing problem. In a word, this is counterproductive. To paraphrase Lehrer, we need to plan to waste time.

Amen.

Data Challenges

I can think of many times on very stressful projects in which people have made unreasonable demands of me–and others. Some of my clients or managers would tell me to “just fix the problem.” They didn’t want to hear about little details like inconsistent data (or even missing data!), poorly documented business practices, and end users who didn’t know what they did and why.

In these cases, I was supposed to work “harder” and stay later yet, much to my clients’ and managers’ chagrins, I knew that I needed to take one of these steps back. Particularly beyond 6 pm, my brain would begin to shut down. I would need to hit the gym, go for a walk, eat a proper meal, or just zone out for a while.

Refreshed after the gym or after a night of what I call sleep, I’d approach the problem with a fresh perspective in the morning. Without fail, I would get in early and, without the distraction of others, test a few new ideas about how to solve the organization’s problem. Removing myself from the confines of a contentious project (even for a few hours) usually helps me see things previously unseen. Maybe there is a simpler way to transform a dataset. Perhaps I need to break five “big” queries into ten small ones.

Simon Says

Emergencies need to be addressed ASAP. I get that. But in’t it easier to connect the dots while not under extreme duress? How often are you able to perform at a peak level while on a burning plank?

Feedback

What say you?

Tags: Data Management, Jonah Lehrer