When numbers cast spells

The dangerous art of conjuring certainty from data

The rabbit laughs last / Illustration: Ted Cuzzillo, in collaboration with Thomasin Grim

No doubt about it, data can be a useful, even revelatory. But like any tool, it can be used badly. Instead of creating a parallel reality for observation and analysis, it may seduce decision makers with “data driven” hallucinations — in a profound reversal of control, drawing the decision makers into delusions of their own mastery.

Imagine the old-time magician’s trick of pulling a rabbit out of a hat. With grand gestures and florid speech, the magician draws attention to the rabbit — which has a trick of its own. It takes control of the act and pulls the magician into its own spell.

In plain terms, the “decision maker” cedes whatever knowledge and wisdom has accrued over a career to simple, stupid data. The “magician” become entangled in his own assumptions.

A case that several data experts I’ve talked to all cite a common, infamous story: the disastrous decision in 1986 to launch the Space Shuttle Challenger explosion and the subsequent analysis of the decision to launch.

The Rogers Commission interviewed more than 160 people on their decisions relevant to the final decision to launch that morning. Then in a congressional hearing, the physicist Richard Feynman demonstrated the critical factor on his hearing-room table: He dropped rubber O-rings into a glass of freezing water for a few seconds and then broke them with a tap of a hammer. Fuel leaked, ignited, and it all blew up.

A friend whose identity I can’t disclose but whose familiarity with governmental decision making goes deep after more than 40 years of involvement with it says, “Feynman spent his life with hypothesis, with scientific method,” he said. Feynman demonstrated what the engineers should have known.

Yet they had been caught in a web of such complex data that as a group they had lost their way.

Even simple data can end up disorienting decision makers. The data they think is important reflects the story they tell themselves — so they tell themselves the same story, just “data driven.”

What to do? I suspect the answer isn’t in the data. It’s probably in hiring contrarians or in assigning existing staff to form red teams. They may have to organize their own holiday parties, but the organization might do better because of them.