The sorcerer's-apprentice problem in AI-built analytics
Why building a platform is the easy part — and why selling it to an enterprise is something else
Not long ago, a young Harvard graduate approached the now-retired Tableau reseller Dan Murray. The man needed experienced eyes on the analytics platform he had built with AI and a path to market.
Dan was for 13 years the director of BI services at InterWorks Inc and an instrumental driver of growth for the pioneering data-visualization tool Tableau. He retired last year.
“What he did himself in less than three months,” Dan said, “would have taken 25 coders two years.”
At first, Dan declined to sign on. There was no Mac version of this new platform. But then the young man promised to have one within a week -- and he did. Dan accepted, signing on to lend his years of experience to one of many new micro developers building big platforms. And they need it.
Dan was impressed with the young developer’s platform. It was a complete analytics stack, from data ingestion to visualization. It also had forty percent of Tableau’s interconnects, which are the ones Tableau customers use most.
But as miraculous as building with AI may seem at first, it soon proves far from it. What sounds like magic usually turns out to be an illusion -- the truth about which gets around. As Dan looked more closely at the young man’s AI edifice, the more he saw the work of a sorcerer’s apprentice.
“As far as get rich quick schemes with AI go, there might be a few,” says Dan, “But in the real world, it’s usually not that simple.”
There’s such a black box in AI coding, he says, that even the people who’ve built it don’t necessarily know what they’ve built. Sometimes the AI will do great work, and other times it’ll leave you with two years of repair. Some parts work, but many don’t. Or else they do it the way AI often does, in the most inefficient way available.
“If you’re building a thing that’s built by a black box and you don’t have 50 coders who’ve been through every line of code troubleshooting it for a year and a half, you’re gonna have to learn with the client how it actually works.”
A chasm between builder and buyer lies ahead for developers of such machines. Potential clients know this. “A lot of developers are great at building things,” says Dan, “but they’re really crappy at selling them.” So it is with Dan’s young client -- which spawned Dan’s thought that he might be better off licensing instead of selling his product.
On the other side of that builder-buyer chasm, Dan believes he sees a solution: an enterprise-level company willing to run a long-term test with real data.
Data curation is critical. “It’s got to be perfect. Like, perfect,” he says. A human coder knows the code’s logic and can grasp at a glance how it gets from here to there. An AI coder flies through perceptual fog by instrument.
Dan thinks that CIOs will balk at running data on a machine that’s even more automated than what they’ve got already. And often the data that they have already on existing platforms are hard to understand.
He suggests a solution: Enlist an organization to run a year- or two-year test on one use case. Let them ask questions. “I can think of 10 clients off the top of my head that like to experiment like that. I’d say, ‘Here, I got an experiment for you. Here’s why I think you, you should look at it.’ And I’ll give them five reasons. That’s a safe way to deploy these things, a big enough company with the budget and the people and the motivation to run tests. It’ll give them an advantage if they do it first.”
“You don’t know jack until you get a big company to really test it with a full onslaught. Even one use case would do. Maybe in a year or two you’ve got a satisfied client that’s going to say, ‘This is awesome.’”
Dan described how he would pitch that idea to a potential client: “I know your staff. I know what you like to do. I know what your problems are. This has the potential to solve the problem. But it’s new. Something will break. You’re smart enough to know when it breaks and how it broke. And if you spent a year on this, and we worked with you, and we had the vendor work with you, at the end you’d probably have something useful.” On the client’s response: “They’re gonna look at that and say, ‘That’s a believable story.’”
The approach is comparable to InterWorks’ history with Tableau. InterWorks first sold the proof of concept, and only then made money. “We probably sold $50 million worth of Tableau licenses and didn’t get a penny... Then it reached the tipping point where they realized, ‘Oh, these guys are really important.’ And then we made money hand over fist for seven years because we were the first gold partner... we just were the default call.”
Consulting companies that excel will be those that understand AI top to bottom, he said. They’ll grasp the frontier models, the low-cost models, how to optimize compute costs, and how to integrate it all.
“The devil is always in the details,” he said. “But in the world of BI in particular, if your data isn’t right, and you don’t feed it correct, timely, complete information, it’s gonna be a bad outcome.”
To his young client, Dan asked, “Why are you trying to sell this?” It would be smarter, he said, to release it as open source and sell consulting services. It’s easier and more profitable to let clients bill their customers and pay you, the developer, a generous percentage. “Work hard for a few years. Then work sort of, but not very hard, for the next five. Then you’d collect a continuous annuity if you build it right.”
He expects to see AI drive an ever larger need for consulting. “AI is going to be used to develop software,” he said, “and the software won’t be fully sussed out when it first goes out in the field.” Consulting companies that get to be good at the various frontier models and low-cost models, how to optimize compute costs, and how to integrate it into their systems are going to be very valuable.”
“If I were 45, Ted,” said Dan, “I would’ve already built this company”