The uncomfortable truth about quality industry analysis
The methodology that gives Gartner credibility
The dirty little secret about the industry analysis firm Gartner, a mainstay of opinion within the data industry, is that the analysts “hate their jobs.” That’s according to a longtime customer and friend of several Gartner analysts, Dan Graham. He says they’re required to spend long, long days phoning customers — about half of whom are “whiners who dug a hole can can’t figure out how to get out of it.”
This post is the first of several in which I’ll post brief observations from widely respected members of the data analytics community.
Dan calls himself “a Gartner fan boy.” He worked in technical marketing for 19 years, retiring from Teradata as director of technical marketing in 2018. Throughout, he was a regular consumer of analysis by Gartner and others.
Those loathsome days Dan describes add up to research that lays the foundation for solid analysis. Sadly, such methodology appears to be unusual among industry analysts.
The few others whose methodology I know about rely more on their own, let’s say, intuition. We can quibble about each analyst firm’s legitimacy. But in general, I don’t see how opinions on vendors’ products and market strategy based on intuition, or whatever goes on there among the spirits, can rival those based on direct observation.
The outside perspective that analysts provide, well-grounded or not, makes them some of the most welcome of all guests at vendor conferences. Vendors often offer expense-paid travel, one-to-one meetings with technical experts, special dinners, and “fireside chats”with the brass.
Dan preferred Gartner for two reasons. One was research with direct customer contact. The other was its analytical hierarchical process developed by Thomas Saaty that transforms complex decisions into a structured hierarchy. Teams categorize priorities, break them into weighted subcategories, then collectively assign percentage values that must sum to 100 percent. This approach serves two purposes: diverse stakeholders debate the analytical structure while forcing meaningful dialogue during evaluation. The system tends to dilute extreme positions; passionate arguments for specific factors get appropriately contained within their weighted category.
Gartner’s example contrasts starkly with an outfit I observed closely for about one year. No analyst, as far as I could tell, ever talked even once to any vendor’s customers. Nor did any analyst confer with any other one on an evaluation except for a cursory peer review. Yet the firm stays in business.
What is the future of industry analysis? Why doesn’t Gartner’s rigorous methodology roll over those on the other side of justified credibility? Will the market eventually wise up?
Those are my questions. All anyone can do is speculate, but at least I can gather the best speculation I can find.
Next in this series on industry analysis, I have the observations of a second respected member of the analytics community.