Eighteen months ago, almost nobody was selling answer engine optimization. Today every SEO agency has a page about it, and a suspicious number of those pages appear to have been built from the same deck.
Some of these agencies have gone and built something genuinely new. Most have added a slide to an existing report, renamed the retainer, and increased the price. The awkward part, if you are the person signing, is that both versions pitch identically. Same language, same promises, same confident nodding about ChatGPT. You generally find out which one you bought somewhere around month four.
So here are the questions that pull them apart. Each one has an answer only a real operation can give, and a comfortable non-answer that a repackaged SEO retainer will reach for instead.
1. Which Engines Do You Measure, and Do You Report Them Separately?
What a real answer sounds like: a specific list, with separate numbers for each one. ChatGPT, Claude, Gemini, Perplexity and Google AI Overviews behave completely differently, read different sources, and update on different clocks. A brand can be the most-cited source in one and entirely absent from three.
What should worry you: a single blended "AI visibility score." That number is an average of five different realities, and averaging is exactly what destroys the signal. If three engines have never heard of you and two mention you constantly, the average tells you nothing at all, and the average is the thing you would be paying to improve.
Being cited in one engine tells you nothing about the other four. Any report that hands you one number has averaged away the only thing worth knowing.
2. Show Me the Raw Artifact Behind One Claim
Pick a single finding out of their sample report. Then ask to see what it is built on.
What a real answer sounds like: the exact prompt, the full response, a screenshot, and the list of sources the engine cited, with a date on it. Every claim traces back to a stored artifact. No artifact, no claim.
What should worry you: being shown a dashboard. Dashboards sit downstream of measurement, they are not evidence that measurement happened. If nobody in the room can produce the underlying probe for a specific claim, that claim was inferred rather than observed, and you are buying a directional guess with a good design system.
The difference is between "a competitor is the cited authority on this exact buying question, listed first, three runs out of three, here are the receipts" and "AI seems to favour them."
3. How Many Times Do You Run Each Query?
What a real answer sounds like: at least three runs, spaced at least twenty-four hours apart, reported with the spread between them.
This matters far more than it sounds like it should. AI answers are not deterministic. Ask the same question twice and you can get different sources back. A single run is an anecdote, and a programme built on anecdotes will happily report movement that was only ever noise.
What should worry you: "we check monthly," with no detail on runs. Or any answer that treats one query as one data point and moves on.
4. Do You Measure What the Engine Knows, or Only What It Just Looked Up?
This is the question that most reliably separates the operators from the resellers, mainly because most people selling AEO have never thought about it.
Every AI answer arrives by one of two routes. Either the engine searched the web just now and is summarising what it found, or it never searched at all and is answering from what the model already knows. Those are completely different surfaces with different mechanics, and your position on one says close to nothing about your position on the other.
What a real answer sounds like: every query runs on both routes, and the gap gets reported. We call that gap Stream Divergence and treat it as a durability read. If you show up strongly when the engine searches and vanish when it does not, your visibility is rented rather than owned, and it evaporates the moment a product changes how aggressively it retrieves.
What should worry you: a blank look. Most mention-tracking tools only ever see the retrieval side, because it is the only side that leaves footprints a scraper can follow.
If you only exist while the engine is searching, you disappear the moment it stops. Nobody sends you a notification about it.
5. Do You Report Mentions, or Authorship?
There is a real difference between your name turning up in an answer and your page being the thing that answer was built out of.
We regularly find brands cited four or five times inside a single response where the engine never actually names them. Their content is training the answer, the buyer is reading the answer, and the buyer walks away without ever learning whose argument they just absorbed. That is not visibility. That is donating your research to somebody else's sales process.
What a real answer sounds like: authorship share is the headline number. Who wrote the answer, not who got a passing reference in it.
What should worry you: mention volume as the main metric. It is the friendliest number in the entire category and it flatters absolutely everyone, which is precisely why it keeps getting reported.
6. Is My Question Set Frozen, and Who Decides What Goes In It?
What a real answer sounds like: a defined panel of real decision-stage buyer questions, built with you at the start, then frozen. The same questions every month, so that any movement you see is the system moving rather than the measurement being quietly re-aimed at easier ground.
What should worry you: a question set that shifts between reports, or one you have never actually seen. If the yardstick moves, every number after it becomes unfalsifiable, and you have no way to separate genuine progress from a more flattering list.
Ask to see the list. If those questions do not look like the things your buyers would type at the moment they are choosing a vendor, the entire measurement is pointed at the wrong place, however rigorous the method behind it happens to be.
7. What Happens in Month Four?
This is the question that reveals whether you are buying a project or a programme.
The AI answer layer does not sit still. Across our own ongoing monitoring, the set of sources an engine cites for a given question commonly turns over by roughly 40 to 60 percent month to month. Models get updated. Retrieval behaviour changes. Competitors publish things. Nothing you fixed in January is guaranteed to still be true in April.
What a real answer sounds like: a monthly loop. Re-measure the same frozen question set, look at what moved, redirect the work to wherever the ground shifted, and report what got produced and what changed as a result.
What should worry you: a fixed-scope deliverable with an end date. A one-time AI visibility audit is a photograph of something that is still moving. Genuinely useful for about six weeks, then it is decoration.
8. How Much of This Work Happens Off My Website?
Most of what an engine cites when it answers a buying question is not on your domain. It is comparison articles, community threads, industry press, developer documentation and review profiles. Your own pages matter, but they are a minority of the surface area, and often a small one.
What a real answer sounds like: a serious chunk of the work happens off-site. Getting your claims onto the surfaces engines already trust, with every one of them pointing back to a canonical page you own, so the credit lands with you instead of with whoever republished you.
What should worry you: a scope that is entirely on-site content and technical fixes. That work is necessary, and it is roughly the first half of the job. An agency that stops there is selling you technical SEO with a new cover on it.
9. What Won't You Do?
Save this one for the end. It produces the most informative pause of the whole meeting.
There are fast ways to move these numbers. Seeded reviews, planted community threads, mass-generated comparison pages that exist for no reason other than to name your brand favourably. All of it works, for a while.
Then an engine adjusts how it weighs sources and everything built that way disappears in a single update. Sometimes it takes the domain's credibility with it, and the climb back is considerably slower than the original build was.
What a real answer sounds like: a specific list of things they will not do, offered without you having to drag it out of them.
What should worry you: "we use every lever available to us." That is not confidence. That is a description of tactics somebody would rather not name out loud in a first meeting.
Take This Into the Meeting
Nine questions, nine answers worth listening for. Print it, screenshot it, or keep the tab open during the call.
| Ask | Good answer | Walk away |
|---|---|---|
| Which engines? | Five, reported separately | One blended score |
| Show me the artifact | Prompt, response, sources, date | A dashboard |
| How many runs? | Three or more, a day apart | "We check monthly" |
| Memory or retrieval? | Both, with the gap reported | A blank look |
| Mentions or authorship? | Authorship is the headline | Mention volume |
| Is the question set frozen? | Frozen, and you've seen it | It shifts between reports |
| What happens in month four? | A monthly re-measurement loop | A deliverable with an end date |
| How much is off-site? | A serious share of the work | On-site content and fixes only |
| What won't you do? | A specific list, offered freely | "Every lever available" |
The Short Version
Nearly every question above is the same question wearing a different hat: can you show me the evidence, or are you telling me a story?
Plenty of people in this category tell the story extremely well. The measurement discipline sitting underneath it is what you are actually paying for, because it is the only thing that will tell you, in month six, whether any of this worked.
If you want to see what direct per-engine measurement looks like on your own brand, that is what our AEO and GEO work is built on, and the free visibility check runs your real buyer questions across all five engines and shows you where you currently stand.