AI SEARCH / CARBIDE INSIGHTS

How to measure AI search visibility

AI answers are not rankings. There is no position one, the same question can produce different sources on different days, and a screenshot proves nothing. That does not make it unmeasurable. It makes it measurable differently, in the way opinion polling is measurable without being able to ask every voter. A single query answered once is a sample of one from a system that changes its output by phrasing, location, login state and model version, none of which are under your control. What is under your control is running the same fixed set of questions the same way, repeatedly, and reading the pattern instead of any single result. Here is a method that produces something you can track over time.

#Why the usual approach tells you nothing

Someone asks ChatGPT 'best car dealer in [city]', the store appears, and it goes in a slide. Next week it does not appear and nobody mentions it. Both results are noise, because a single query is one draw from a non-deterministic system.

The answer also changes with phrasing, with whether the user is logged in, with location, and with which model version is serving that day. None of that is under your control and all of it moves the result.

So the unit of measurement cannot be a query. It has to be a fixed set of queries, run the same way, repeatedly.

#Build a prompt set and freeze it

Write thirty to forty questions a real buyer might ask, spread across four types: commercial ('who should I use for…'), definitional ('what is…'), problem-shaped ('why is my… not working'), and brand ('what is [your store]').

Freeze them. The moment you edit a prompt you have started a new time series and lost the comparison. Add new prompts at the bottom instead: the numbering is the contract.

Run them across the systems that matter: ChatGPT with search, Google AI Overviews, Perplexity, Copilot, Claude, Gemini. Fresh session, logged out where you can, no memory, and record which location you used because it affects local answers.

#Record four things, not one

Whether your store was named. The obvious one, and the least useful on its own.

Whether your site was cited or linked. Being mentioned without a citation is worth less. It does not send anyone anywhere.

Which sources were cited instead; this is the most actionable column by a wide margin. If three directories own every answer in your market, you now know exactly which corroboration you are missing.

Whether anything said about you was wrong. Log it verbatim. Incorrect hours, wrong departments or an invented claim are a business problem, and finding them is half the value of doing this at all.

#Reading the results

Expect definitional prompts to move first. A clean definition on a crawlable page is the cheapest thing for a system to quote, and it clears no trust threshold to do so.

Brand prompts move next, and they are the entity check. If a system cannot say what your store is, or gets it confused with a similarly named business, the fix is external corroboration (a complete Google Business Profile, consistent listings, a company page that exists) not more content on your site.

Commercial prompts move last and slowest. Being named next to a buying recommendation requires enough confidence in the entity to put it there. It follows the other two; it does not lead.

#What to pair it with

Branded search volume in Search Console. Most AI referrals arrive with no referrer, so a rise in people searching your name is often the only visible trace.

Referral traffic from the assistant domains, where it exists. It undercounts badly, but the direction is informative.

Direct traffic to deep pages. Nobody types a URL for a specific service page from memory: that traffic came from somewhere you cannot see.

And server logs. Filter for the AI crawler user agents. If they are not fetching your pages at all, no amount of content strategy is the problem. Retrieval is.

#What the log looks like in practice

Six columns: date, system, prompt number, named yes or no, cited yes or no, and which sources were cited instead. A spreadsheet is entirely adequate: doing it consistently matters more than the tool.

Add a seventh column for anything inaccurate, recorded word for word. This is the column that produces immediate business value: a system stating wrong hours, or a service you do not offer, is a problem you can act on this week regardless of any visibility goal.

After three months the sources column becomes the most valuable thing you have. It is a ranked list of who currently owns the answers in your market, which tells you precisely which directories, publications and profiles are worth the effort instead of leaving you to guess.

TABLE

The tracking log, column by column

One row per prompt, per system, per month

Column	What it records	Why it's there
Date	When the prompt was run	So the time series is comparable
System	ChatGPT, AI Overviews, Perplexity, Copilot, etc.	Different systems move at different rates
Prompt #	The frozen prompt's fixed number	Never edit prompt text: add new ones at the bottom instead
Named (Y/N)	Whether your store was mentioned at all	The obvious signal, and the least actionable alone
Cited (Y/N)	Whether your site specifically was linked or credited	A mention with no citation sends nobody anywhere
Sources cited instead	Who the answer credited if not you	The single most actionable column after three months of data
Inaccuracies (verbatim)	Anything wrong the system said about you	Produces immediate business value regardless of any visibility goal

Framework described in this article's own method, compiled for reference.

Questions.

How often should we run the prompt set?

Monthly. Weekly is noise at this sample size and creates busywork. Establish a baseline before any major change, then measure direction over quarters instead of weeks.

Can we pay a tool to do this?

Several tools track AI mentions and they save time on collection. Check what they query and how often before trusting the trend line: a tool using different prompts from yours is measuring a different thing, and switching tools resets your history.

What if we never appear for commercial prompts?

That is the normal starting position and it is usually an entity problem instead of a content problem. Check the brand prompts first: if a system cannot describe your business confidently, it will not recommend it. Fix corroboration before writing more pages.

How many prompts is enough to track?

Thirty to forty, spread across commercial, definitional, problem-shaped and brand questions and including a couple of comparison questions against a named local competitor, run the same way every time. Fewer than that is too noisy to see a trend; far more rarely adds signal for the extra time it costs.

Does being cited by an AI system bring in customers?

Indirectly, mostly. A citation functions closer to a trust signal than a click source, since most AI answers are read without a follow-up visit. The measurable effect usually shows up in branded search and phone volume over time rather than in a referral line in analytics.

What is a realistic timeline to see movement in the prompt set?

A quarter, at minimum, for anything beyond a fast entity-corroboration fix. AI systems update their sense of a business slower than a search index updates its ranking, so a single month's flat result is not yet a signal either way.

Why is one screenshot poor visibility evidence?

Because one query is one draw. A single answer comes from a system that changes its output by phrasing, location, login state and model version. Your store appearing one week and missing the next is noise either way. The unit of measurement has to be a fixed set of prompts run the same way repeatedly.

What belongs in an AI search prompt set?

Four types. Commercial questions such as who to use for a service, definitional questions such as what something is, problem-shaped questions such as why something is not working, and brand questions about your store by name. Spreading across all four shows which type of question your store is ready for.

Which AI systems should we check?

The ones your buyers use. That means ChatGPT with search, Google AI Overviews, Perplexity, Copilot, Claude and Gemini. Use a fresh session, logged out where you can, with no memory, and record which location you used because it affects local answers.

In what order do AI search results improve?

Definitional prompts move first, because a clean definition on a crawlable page is the cheapest thing for a system to quote. Brand prompts move next and act as the entity check. Commercial prompts move last and slowest, since being named beside a buying recommendation needs enough confidence in the store to put it there.

What else tracks AI search visibility?

Branded search volume in Search Console, referral traffic from the assistant domains where it exists, direct traffic to deep pages, and server logs filtered for AI crawler user agents. Most AI referrals arrive with no referrer, so a rise in people searching your name is often the only visible trace.

What if an AI system misstates our dealership?

Log it word for word. Put it in the inaccuracies column and act on it that week. Wrong hours, a department you do not have or an invented claim is a business problem whatever your visibility goal, and finding these is half the value of running the prompt set.

PRIMARY SOURCES