Where Does Your Customer Insight Really Come From?

Summary
What a few hundred B2B marketing leaders said recently about AI-powered research, the three lines they drew and the questions to ask before you trust any of it. The real test for synthetic research is whether it beats the instinct-and-a-few-sales-calls baseline most teams are actually running on today, not whether it matches a six-week study nobody was going to fund anyway.
Our Friday CMO Coffee Talk sessions are a live conversation on video with a Zoom chat running alongside it, and the chat is often where the real action is.
A few weeks ago we featured a discussion about how research is changing: synthetic respondents, AI-moderated interviews and simulations trained on your own customer data.
Early in both sessions, where chat was both lively and skeptical in the best way, the room got a simple question. Where does the evidence behind your positioning and messaging decisions come from?
The answers came in fast: Gong call recordings, a survey of 300 people, Wynter testing, “all of the above.”
And my favorite: “What my CEO wants 😅”
A lot of people resonated with that one (for better or worse).
When I recently asked a version of the same question as a LinkedIn poll, the answers looked familiar.
The honest (and thin) baseline
Many of us have made big positioning calls on a sample size of about one (I’ve done this too). Our own instinct, plus whatever came up on the last few sales calls.
That’s the real baseline for a lot of teams, and it’s a long way from the depth and confidence of insights we wish we had (which historically takes time and money we don’t have).
So I’d argue the right test for AI-assisted research is whether it beats what you’re doing right now. For many teams, the six-week study with a top-tier firm was never on the table anyway.
Many times the alternative is doing zero research, and plenty of businesses are already making big decisions on limited data (or none at all).
Sound familiar?
Where CMOs drew the line (three of them)
What people say and what they do are rarely the same. A model trained on your happiest customers learns a flattering version of your market. And one CMO called a synthetic focus group participant “next level executive navel gazing.”
All fair. By the end of both sessions the room had landed on a fairly clear set of lines, and the GatherHQ team (who joined both sessions to lend their experience and expertise) agreed with them.
Keep the respondents real for anything that reaches your board or your market.
Use AI freely to design the study, moderate the interviews, analyze the results and write the report. The people answering the questions should still be people when the output is going in front of executives or customers.
Use simulations to narrow the options.
Synthetic research is useful for taking ten options down to three. The final call still gets tested with real buyers before you commit budget to it.
Keep synthetic quotes out of customer-facing content.
Every quote in a published report or social card should come from a real person. Gather’s own team said they don’t recommend synthetic output for customer-facing assets.
Garbage in, confident garbage out
A synthetic persona is only as good as the data behind it, and we all know that’s where a lot of the real risk lives.
If your recorded call library only includes happy customers, the model learns a happy market. Representative means churned accounts, closed-lost deals, freemium users and buyers who have never heard of you. If you’re training on Gong calls, the model will learn what it’s fed. Make sure it has everything, and, perhaps more important, make sure you’re capturing the good and bad (all of it) in the first place.
The other blind spot is behavior. A few members with e-commerce backgrounds made the case that ad tests, email performance, clickstream and product usage tell you far more than survey answers. Something as small as a period versus an exclamation point can shift response in a statistically significant way. A simulation that only models what customers say (and ignores what they do) is missing its best input.
There’s also a loop problem. You need to understand your ICP well enough to simulate it, and the questions you then ask that simulation risk being leading and inaccurate. The best defense is anchoring simulations in behavioral data and fresh real-world input, and assuming the gaps in your understanding will still come from competitive intel and conversations with people you don’t already sell to.
Trust, but benchmark
Before relying on a simulated audience, run the same study with real participants as a holdback and compare the two. Jay from GatherHQ recommended looking for 85 to 90 percent overlap or better before trusting a simulation, and continuing to run real research in parallel as an ongoing check. It’s a discipline borrowed straight from traditional modeling.
The human baseline has its own problems. A few years ago Kantar found researchers discarding up to 38 percent of the data they collected because of quality concerns and panel fraud. Real respondents still need verifying.
Three questions come up again and again for any research vendor, AI-powered or not:
- Do you train on my data, and who owns the resulting model?
- How do you verify that respondents are human and representative?
- What does it really cost, including the tokens and tooling behind the study?
Get the answers in writing before you upload anything proprietary.
Research as infrastructure
Most of us still fund research as a project. An annual study, a persona refresh for a launch, a win/loss sprint when deals start slipping. Then the work sits still. It never breaks and no alert fires. It just gets a little more wrong every quarter while everybody keeps briefing off it.
Gather’s own research recently found 80 percent of marketing budget owners surveyed calling a customer quotes database important and 62 percent calling it painful to maintain. Nobody in that survey doubts these assets matter, but keeping them current rarely gets an owner (or time, bandwidth and budget).
We see the same thing in the positioning and messaging work we run with clients at Heinz Marketing. The first few weeks are almost never about building something new. They’re about rebuilding what already existed and then quietly went out of date.
Some of the best examples from operating leaders start with data their teams already have. One CMO talked about pulling AE and customer success calls on an ongoing basis to keep an ICP file fresh. Another has every customer call summarized into the CRM and a dedicated Slack channel for each account. That’s a continuous customer signal, and the kind of foundation AI-forward research tools can build on and scale.
For leaner teams, that’s the practical starting point. Structure the call recordings you already have, segment them deliberately so you’re hearing from more than your happiest customers and test what the analysis tells you before you act on it.
Mayank closed the early session with this: “Research is no longer constrained. You should be able to run as much as you need to so you can leverage customers to create better businesses.”
Start here
Most teams can start with the customer signal they already have. Keeping it current at the pace your market moves is where AI-forward research tools come in. Five steps, in order:
- Write down where the evidence behind your last three positioning calls came from. If the honest answer is instinct, a few sales calls or the CEO, that’s your baseline.
- Pull together and structure the calls you already record. Make sure churned accounts, closed-lost deals and prospects who never took a meeting are in there too, and turn the transcripts into data before you analyze them.
- Pick one decision to test with real buyers this quarter. If you use simulations, use them to get from ten options to three, then put those three in front of real people before you commit budget.
- Write down your line. What can be synthetic, what has to come from real people and who signs off on anything that reaches the board or the market. Then share it with your team.
- Scale it with AI-forward research tools. Once you know your baseline and your line, look for tools that can run research continuously with real, verified respondents and keep your core assets current. Bring the three vendor questions above to every conversation.
Keep the judgement
What I’m worried about is teams using AI as a thought replacement instead of a thought partner. A simulation can tell you which three messages are worth testing, but not which one your company should stand behind (and it won’t be in the room when the board asks where the number came from). That judgement call, and the accountability for it, stays with us.
This post originally appeared on Matt Heinz’s Substack.





