What Your Analyst Knows That ChatGPT Doesn’t

Summary

A seasoned analyst's experience and intuition about what's coming are worth more than ever. Two studies, three ways to get more from your analysts and nine questions to ask them below. As LLMs commoditize the published research analyst firms sell, the real value shifts to judgment, real-time signal and intuition about what's next, with three ways to get more from your analyst relationships and nine questions to bring to your next inquiry call.

A seasoned analyst is still worth their weight in gold.

That’s not exactly a popular take these days. I’ve heard many marketing leaders ask their peers whether (or when) AI is replacing the analysts. Many, for example, don’t ask the analysts for help with martech decisions anymore. They ask an LLM.

It’s a fair question, and Wall Street is asking it too.

The market has an opinion

Gartner’s stock has lost about two-thirds of its value from its peak. Some of that came from federal contract cuts. But contract value (the number that tells you whether clients are renewing and expanding) grew less than two percent year over year in its most recent quarter. Forrester’s revenue fell 9 percent in its most recent quarter, and the company recently cut 8 percent of its workforce.

Buyers, of course, are shifting behavior too. G2 recently surveyed more than a thousand software buyers and found 51 percent start their research with an AI chatbot more often than with Google, up from 29 percent less than a year earlier.

So the lead question is whether the analysts themselves still have value, or whether the LLM of your choice does the job well enough.

What your LLM is good at

An LLM is a terrific summarizer of what’s already been written. Feature comparisons, vendor shortlists, review sentiment, the consensus view on a category. It’ll get you up to speed on all of that in minutes, and for a lot of martech (and broader business) decisions that’s plenty.

But look at what it’s summarizing. OtterlyAI recently analyzed more than a million URLs cited by ChatGPT, Perplexity, Google’s AI Overviews and three other AI search tools. Gartner accounted for 81.7 percent of the citations that went to analyst firms, and 96 percent of those pointed to its peer review pages.

In other words, when your LLM tells you what “the analysts” think, it’s mostly telling you what customers posted in reviews. That’s useful, but it’s also a look backward.

What a seasoned analyst does that an LLM can’t (yet)

A good analyst brings three things to a conversation that no model has access to.

Experience.

Years of watching categories form, consolidate and collapse. They’ve seen the same vendor promise three times and know how it played out the first two.

What’s materializing in real time.

Analysts spend their weeks on inquiry calls with buyers and briefings with vendors. Much of what they hear never gets published, which means it never gets into a training set or a citation.

Intuition about what’s ABOUT to happen.

This matters most. Pattern recognition plus fresh signals plus judgement add up to advice and guidance about where a category is heading before the evidence is public. An LLM simply can’t do that (at least yet), because the evidence it needs hasn’t been written down.

Phil Fersht, founder of HFS Research, wrote recently: “The firms that survive won’t be the ones publishing more, they’ll be the ones clients can’t afford to lose because the insight, relationships, and judgment they provide are irreplaceable.”

That sounds about right. The published reports are the part getting commoditized, and the firms seem to know that. AskGartner answers client questions drawing on Gartner research, and Forrester credited “accelerated adoption of Forrester AI” in its most recent earnings release. Both are putting their published research into an AI layer themselves, which leaves the analyst’s judgement as the thing you can’t get anywhere else.

The quadrant still matters, too

None of this means the Magic Quadrant is irrelevant. In IT and cybersecurity especially, it’s still heavily leveraged. Enterprise buying committees use it to shortlist, procurement uses it to justify the decision and nobody wants the uphill battle of defending a new security vendor the analysts have never heard of and can’t vouch for.

So especially if you sell into those markets, keep investing in analyst relationships. Just don’t confuse the report with the value of the person who wrote it. Different objectives, different strategies.

Rethink the analyst relationship

If the value lives with the analyst, it’s worth asking whether it has to come through the firm.

Plenty of seasoned analysts (and analyst-minded operators) now work independently or inside vendors, and many are still doing the same job. They’re talking to buyers, tracking vendors and publishing what they see.

Kyle Poyar, Kerry Cunningham and Cheri Hulse are three who are either independent (Kyle) or smartly hired inside vendors (Kerry and Cheri) to continue developing unique research/insights and serving as an analyst to their firm’s customers and prospects.

Bringing one in as an advisor (fractionally or on retainer) gets you the experience and intuition this piece is about, from someone who works for you. A firm analyst has to stay neutral across every vendor they cover. An independent you hire can help you figure out how to win.

There are tradeoffs. An independent no longer sits on the firm’s inquiry calendar, so their real-time signal depends on the network they’ve kept up since leaving. And they can’t put you in a quadrant.

For a lot of companies the right answer is both: the firm where the quadrant is still a gating item, and an independent where you want someone in your corner.

How to get more from your analysts

Here are three changes I’d make to how most teams use their analyst relationships:

1. Do your homework with the LLM first.

Let the model handle the feature comparisons, the competitive summaries and the review sentiment before you ever get on an inquiry call. Showing up already caught up means the analyst’s time goes to the questions only they can answer. You both benefit.

2. Ask about what’s next.

The answers that matter most live in an analyst’s head and their calendar (nine questions to start with below). When you take the time to extract that gold, you may be the only one in your market who has it.

3. Treat analyst time like advisory time.

Too many AR programs still measure success by placement in a report. Placement matters (see above), but the bigger return often comes from pressure-testing your positioning, roadmap and messaging with someone who spends all day talking to your buyers and competitors.

Nine questions worth asking your analyst

The best inquiry calls go past what’s already in the reports. These questions are built to pull out the parts an LLM can’t reach: what the analyst is hearing, where they think things are heading and what they’d tell you if you asked.

What’s changing

• What are buyers asking you about that they weren’t a few months ago?
• Where are deals in this category stalling or dying after the shortlist, and why?
• When buyers fund this, where is the budget coming from?

What’s about to happen

• Which vendors are quietly losing deals, and to whom?
• What do you expect to consolidate, get absorbed or disappear in the next couple years?
• What are vendors in this space all claiming that buyers have stopped believing?

What you might be missing

• What do you think we’re wrong about?
• Who should we be paying attention to that we aren’t?
• What do you believe about this market that most of your peers don’t?

After almost every answer, ask one follow-up: “What’s that based on?” The number of conversations behind a take tells you whether you’re hearing a real signal, one loud client or just the analyst’s opinion. All three have value, but it helps to know which one you’re hearing.

Will LLMs eventually close that gap? Maybe. I’d be willing to bet the best analysts are already using them to see more, faster. Until then, use your LLM to get caught up and save your analyst time (at a firm, independent or in-house) for what’s coming next.

This post originally appeared on Matt Heinz’s Substack.