Data-driven spooks are losing their edge

Parallels for the data industry in Amy Zegart’s “Spies, Lies, and Algorithms”

In Amy Zegart’s new book, Spies, Lies, and Algorithms: The History and Future of American Intelligence, she makes observations that show similarities with the data industry.

Here’s a quick take based on a Wired magazine excerpt.

First, technological breakthroughs are transforming the threat landscape by generating new uncertainties and empowering new adversaries. … The Cold War was a dangerous time, but it was simpler. America’s top intelligence priority was clear. Every foreign policy decision was viewed through the lens of “What would Moscow think?”

The good old days in the data industry were also simple. Just get the right tools and you’re data driven!

In the old days, spy agencies in a handful of powerful countries dominated the collection and analysis of information. … Few countries or companies could come close.

When “big data” came along, you got big, big technology. And you made the best decisions in town.

Much as the data industry talks about being “data driven,” effective and widespread use of data is probably not so common even now. But I like to think that’s changing, especially with a younger generation that seems to have been nursed on data.

In short, data volume and accessibility are revolutionizing sense-making. The intelligence playing field is leveling—and not in a good way. Intelligence collectors are everywhere, and government spy agencies [and everyone else] are drowning in data. This is a radical new world and intelligence agencies are struggling to adapt to it. While secrets once conferred a huge advantage, today open source information increasingly does. Intelligence used to be a race for insight where great powers were the only ones with the capabilities to access secrets. Now everyone is racing for insight and the internet gives them tools to do it.

The competitive edge that data technology provides for organizations is eroding. The new edge comes from good sense-making.

What are the implications for the vendors of information technology? So far, I see two possibilities.

• Organizations could decide to accelerate the spread of data literacy. Training is one path. So is easing the data-illiterate or semi-literate into data literacy with apps made for this. A few tools have shown up lately.  Toucan Toco by Toucan is designed to quickly build elementary data presentations. And Juicebox by Juice Analytics eases people into actual data analysis.

• Experimentation with storytelling support based on deep understanding of narratives. Leaders in the use of data in decision-making might, for example, borrow insights from fields that have already made good use of storytelling, such as law and architecture.