Judge the method, not the output, because a weak vendor’s output looks as convincing as a strong one’s. Ask first: does the vendor stick to how clients think, or claim to forecast what they’ll do and pay? And where does their method break? Then test how they handle model bias, averaged personas, numeric ratings and cross-contaminated answers.
Synthetic research vendors now offer to simulate your clients. They build AI “personas” or “synthetic panels” of the people who buy your firm’s work — general counsel, in-house teams, finance directors, procurement leads — and answer research questions on their behalf in days rather than weeks. Used well, the technology is useful. Sold badly, it hands you answers that look authoritative and are wrong.
The difficulty is that the output always looks convincing. Clean charts, confident percentages, plausible quotes. You cannot judge the quality from the deliverable — you can only judge it from the method. And for a professional services firm the stakes are higher than for a consumer brand testing a biscuit wrapper: you are making decisions about positioning, pitch strategy and client relationships on the strength of what the tool tells you your clients think.
This guide is written to be used in a vendor pitch or a procurement call, with no technical background required. A vendor who understands their own method answers each question in a sentence or two. A vendor selling a thin product hedges, points to their client logos, or tells you it’s proprietary. The hedge is the answer.
What “synthetic research” actually means
Before the questions, one clarification that trips up most buyers: “synthetic” is used to describe at least five different things — AI-generated personas, aggregated synthetic insights, respondent-level datasets, digital twins of specific real people, and simulated interviews. Vendors use the same word for very different products. If you’re not clear which one a vendor is selling, you can’t judge whether it fits your question. Ask.
The two problems that hide inside a good-looking result
Two things go wrong beneath the clean chart, and vendors don’t volunteer them.
The model has a bias, and nobody’s cross-examining it. Ask two different AI models the same question about how a general counsel chooses a firm and you don’t get the same answer. Each model carries its own built-in assumptions about what a lawyer is and how one decides, and those assumptions colour every response it gives. So the first question isn’t “is this accurate?” — it’s “accurate according to which model’s assumptions?” A vendor running everything through a single model has no way of knowing which findings are real and which are that one model’s prejudice.
One answer, or a map
You’re being handed the average, then extrapolating from it. A persona built from demographics regresses hard to the mean. It returns the internet-average version of “general counsel at a mid-market firm” and presents it as insight — an answer that corresponds to no actual client, a composite of everything ever written about the category, smoothed flat. This isn’t a hunch. In the largest study of its kind — Park et al., LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals (first published as Generative Agent Simulations of 1,000 People; Stanford and Google DeepMind; arXiv, version 3, June 2026) — AI agents built from demographics alone were 9 percentage points less accurate than agents grounded in a real two-hour interview with the person (74% against 83%), and 12 points behind agents given the interview and the person’s survey answers together (86%), running on the same models. Agents built from a persona description did worse still, at 71%. Demographics alone leave most of the person on the table.
The same research draws the second line you need. These tools reproduce how people think far better than what they do: the interview-grounded agents matched real attitudes on the General Social Survey 83% as accurately as the people matched themselves two weeks later — strong — but on economic games played for real money, accuracy fell to a normalised correlation of 0.66, about two-thirds, and no richer agent significantly beat the demographic one. Attitude is not action, and price sensitivity is where synthetic data is weakest of all.
The questions below are built to surface both problems — and to separate a serious provider from an AI role-play tool.
The two questions that matter most
Ask these first. They separate the serious vendors from the rest.
“Will you tell us what proportion of our clients would move work to another firm — or what they’d pay — or only how they think about those decisions?”
Strong answerA strong answer draws the line between what clients say and what they do, and stays on the “how they think” side. That is the honest, useful application of the technology.
Weak answerA weak answer is a confident behavioural or pricing figure — “62% would switch”, “they’d pay 15% more”. The research is consistent that synthetic tools are far weaker at predicting behaviour than attitudes. A vendor who hands you a switching or pricing number with a straight face is selling a forecast the evidence doesn’t support.
“If we made a positioning or pricing decision on your output and it turned out wrong, which part of your method would have let us down?”
Strong answerA strong answer names the method’s own limits without being pushed. Everyone who understands this technology knows where it breaks.
Weak answerA weak answer claims there are no limits. It means they haven’t found them — which means you will.
Testing the method
“If you asked two different AI models the same question about how a general counsel chooses a firm, would you get the same answer? Which model do you report to us, and why that one?”
Strong answerThey run more than one model and treat disagreement between them as something to investigate.
Weak answer“We use the best model.” A single-model vendor cannot tell genuine signal from that model’s quirk.
“When your simulated clients score something out of five or ten, how do you know the number reflects a real judgement and not the model producing a figure that simply looks plausible?”
Strong answerThey don’t ask the AI for numbers directly — they take answers in words and convert them to scores against a consistent benchmark.
Weak answerPride in clean, tidy rating charts. AI models are poor at producing meaningful numeric ratings; tidy output is not the same as trustworthy output.
“What do you check your results against? And if the answer is ‘real client surveys’, how do you know those surveys were measuring the right thing?”
Strong answerThey can explain what their method captures that a survey cannot.
Weak answer“We match human survey responses 90% of the time.” That only shows the tool reproduces a survey — not that the survey asked the right questions.
“When you run a hundred simulated clients, is each one answering independently, or can the later ones be influenced by the earlier ones?”
Strong answerEach simulated client is fully separate — and they can tell you how.
Weak answerVagueness. If the tool runs them together, later answers are shaped by earlier ones — no different from asking one colleague a question while a second listens in. The second answer is tainted.
“If you describe a client in the brief as ‘cost-conscious’ or ‘time-poor’, haven’t you decided part of the answer before you’ve asked the question? How do you control for that?”
Strong answerYes — and here is how we guard against it.
Weak answerA blank look. A “busy” persona ties every answer back to how busy it is. The words in the brief predetermine the result.
Testing whether they understand your clients
“Do you already know which factors drive how a general counsel or head of legal chooses between firms in our practice areas — or do we need to tell you what to build?”
Strong answerThey bring real knowledge of how professional services are bought.
Weak answerThey need you to supply what matters. If you provide the expertise, you’re paying them to operate a tool while you do the thinking. Worse still is a vendor who won’t let you specify a client beyond a job title and a couple of demographics.
“Will your simulated general counsel admit an unflattering reason for a decision — choosing the familiar firm to be safe, or the cheaper option over the stronger one? Or only the respectable answer?”
Strong answerThey know AI defaults to the professionally respectable answer, and they have ways past it.
Weak answerA claim that synthetic respondents are automatically more honest “because there’s no social pressure”. A simulated GC still wants to sound like a competent GC, and hides the same unflattering truths a real one would.
Testing the data you’ll actually receive
“When the written answers are grouped into themes, is a person or the AI deciding what counts as evidence for each theme?”
Strong answerThey know AI over-attributes — it files an answer under a theme it never mentioned because the ideas sound related — and they check for it.
Weak answer“The AI handles the analysis end to end.” That’s the tool marking its own homework.
“Can we interrogate the output — pick one simulated client, ask why it answered the way it did, and push back — or do we only get the summary numbers?”
Strong answerYes, the reasoning is available and you can question it.
Weak answerA dashboard of percentages with nothing underneath. The number alone is the part most exposed to every weakness above.
How to read the answers overall
You are not looking for a vendor with no limitations — no honest one exists, because the methodology has real limits. You are looking for a vendor who knows their own limits, states them before you ask, and never dresses a stated attitude up as a predicted action. The providers worth your budget find these questions easy. The ones who find them uncomfortable are telling you something.
Synthetic research is not AI role-play at scale, and it is not a replacement for knowing your clients. Done properly, it reveals the landscape your clients’ decisions happen within — the considerations that drive who wins the pitch, keeps the panel, or loses the work — at a depth and scale ordinary research can’t reach without putting client relationships at risk. Done badly, it’s a confident mirror. The questions above are how you tell the difference.
For what the research shows on one question firms can’t put to their own clients, see why corporate clients stay with their incumbent law firm.
Taleist builds decision intelligence for professional services firms — research designed around how your clients actually choose, not how an off-the-shelf model assumes they do. If you’d like to talk through any of these questions against a specific vendor, or against your own client-insight needs, get in touch.