What Magic Tricks Teach us About Data Science

Jul 19, 2012 by Jim Harris in Data Profiling

Teller, of the magician duo Penn & Teller (Teller is the one who doesn’t speak during their performances), recently revealed in an article for Smithsonian Magazine how magicians manipulate the human mind.

Since predictive analytics is often misunderstood as a clairvoyant magic trick capable of predicting what is going to happen with certainty, when instead, what it actually does is predict some of the possible things that could happen with a certain probability, I think that understanding how magic tricks work can teach us some important things about data science.

“Magicians,” Teller explained, “have done controlled testing in human perception for thousands of years. Magic is not really about the mechanics of your senses. Magic is about understanding — and then manipulating — how viewers digest the sensory information.”

In his article, Teller explains seven principles that magicians employ to alter our perceptions. The first principle is pattern recognition. I have previously blogged about pattern recognition in data-driven decision making. We search for any pattern in the data relevant to our decision, which allows us to discover a potential source of business insight. And once our brain finds a decision pattern, we start making predictions, imagining what data will come next, projecting imaginary order into the data stream, turning bits and bytes into the ebb and flow of the data-decision symphony.

But sometimes the music of the data is the sound of pattern recognition changing our brain in such a way that our search for decision consonance among data dissonance biases us toward comforting, but false, conclusions.

The sixth principle that Teller explained was that “nothing fools you better than the lie you tell yourself. When a magician lets you notice something on your own, his lie becomes impenetrable.” I have previously blogged about the lies we tell data and the lies we tell ourselves about data (e.g., that there’s a single version of the truth). But the most impenetrable lie is when the only thing we notice in data is what we were looking to find.

“Magic is an art,” Teller concluded, “as capable of beauty as music, painting or poetry. But the core of every trick is a cold, cognitive experiment in perception: Does the trick fool the audience? A magician’s data sample spans centuries, and his experiments have been replicated often enough to constitute near-certainty.”

Data science is not a magic trick. However, data science, just like magic, is an art with a cognitive experiment in perception at its core. Using data science effectively requires that we see through our own illusions about data-driven decision making.

Tags: data-driven decision making, predictive analytics