Identical prompts produce different brand lists across runs because generation is probabilistic. Any visibility number from a single run is a sample of one, which is why single-check audits are close to worthless.
The variance is not a defect and it is not going away. It is how the systems are designed, and the vendors document it.
People assume a wrong answer means the engine is broken and a right answer means the work is done. Both assume the answer is a fixed property of the question, and it is not.
What exactly varies between runs?
Three things, and they matter to you in ascending order.
The same facts phrased differently. Cosmetic, and irrelevant to visibility measurement unless you are string-matching too strictly.
IgnoreThe same brands listed in a different sequence. Matters if you count position, which is why a position-weighted metric needs more runs than a presence metric.
DependsA different set of brands entirely. This is the one that moves your number, and the reason a single run cannot support a rate.
The whole problemMembership variation is what makes brand tracking statistical rather than deterministic. A brand sitting near the model’s threshold for inclusion appears in some runs and not others, and that borderline position is where most B2B companies live.
Why is generation probabilistic in the first place?
Because the model samples from a distribution over possible next tokens rather than always taking the most likely one. That is a design choice, and it is what makes the output readable rather than repetitive.
The vendors state the consequence plainly. OpenAI’s advanced usage guide says “Chat Completions are non-deterministic by default (which means model outputs may differ from request to request).”
So two identical requests are two draws from the same distribution, in the same way two coin flips are. Getting heads once tells you the coin can produce heads. It does not tell you the coin is biased.
Does pinning a seed or a model fix it?
It reduces the variance and does not remove it, and you usually cannot do it anyway.
OpenAI’s documentation offers a seed parameter for “(mostly) deterministic outputs” and exposes a system_fingerprint so callers can detect backend changes, noting that if that value differs “you may see different outputs due to changes we’ve made on our systems.” Mostly is the operative word, and the documentation is explicit that determinism is not guaranteed.
Version pinning is firmer. Anthropic’s models overview states that “Every Claude model ID is a pinned snapshot,” so at least you know which model answered.
None of this is available to you when the surface is a consumer product. Nobody asking ChatGPT a question in the app is setting a seed, and the answers your buyers see are the unpinned ones. Measuring the pinned version would be measuring something your market does not experience.
What else varies besides the model?
The retrieval layer, which moves independently and often faster. An engine that searches before answering is subject to whatever the index held at that moment.
Google is explicit that its two AI surfaces differ from each other, noting in its documentation on AI features that “AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary.” Two surfaces, same company, same query, different answers by design.
Underneath that, the index itself changes on a published schedule. Google’s Search status dashboard records ranking updates with their rollout durations, and the May 2026 core update ran for 11 days and 21 hours. A measurement window overlapping a rollout is measuring the rollout.
Two sources of variance stack: the model sampling differently, and the retrieval layer returning different material to sample from.
Can you tell which source produced a difference?
Often, from the answer itself. A run naming different brands with the same citations is model sampling. A run citing different pages is retrieval. A run naming a product you discontinued is training-data recall rather than either, which is the distinction in why training data and live retrieval differ.
This is worth doing once on a prompt that matters, because it tells you which lever to pull. Model-sampling variance is something you accommodate with more runs. Retrieval variance is something you can act on, because the pages being returned are visible and changeable.
Why do borderline brands vary most?
Because inclusion is a threshold effect, and variance is largest exactly at the threshold. A brand the model considers an obvious answer appears in nearly every run. A brand it has never associated with the category appears in almost none. Both are stable.
The unstable case is the brand sitting near the boundary, where small differences in sampling tip it in or out. That produces the frustrating pattern where a company appears in roughly half its runs and its reported rate swings widely week to week without anything changing.
This has a useful corollary. High variance in your own results is itself a finding: it says the engines consider you a plausible but not automatic answer in that category. Brands that are genuinely invisible do not have noisy numbers, they have flat ones at zero.
It also means the amount of sampling you need is not a fixed number. A brand near the boundary needs more runs to pin down than a dominant incumbent does, because the underlying quantity being estimated is the one with the most spread.
What does this mean for a single-check audit?
That it can prove existence and nothing else. One run showing your brand proves the engine can name you. One run not showing it proves nothing at all, because the same prompt might name you on the next attempt.
Free single-shot checkers are built on exactly this misunderstanding. They return a number with no interval and no trend, and the number reads as a measurement because it has a percent sign after it.
The practical consequence for buying decisions is that requery frequency belongs in the evaluation alongside price, because it decides whether you are buying a measurement or an anecdote. That is the arithmetic in how much do AI visibility tools cost.
How do you work with variance instead of against it?
Sample it deliberately and report it honestly. Three habits cover most of it.
Run every prompt more than once per engine per day, and publish the count alongside the result. A rate derived from a stated number of runs is a measurement; the same rate with no stated count is a claim. Vendors that publish their requery frequency are making themselves checkable, and the ones that do not are worth asking directly before you buy.
Widen the prompt set before deepening the runs. Prompts vary more than repeated runs of one prompt do, so breadth buys precision faster, and it also protects you from a set that quietly measures the wrong market.
Report movement against a threshold rather than against last week. Compute the threshold once from your own observation count, then ignore anything beneath it, which is the decision rule in is my AI visibility change significant.
Variance is also the reason to keep the raw answers. When a brand appears in three runs of five, the interesting question is what the other two said instead, and only stored answer text can tell you. That habit is the foundation of how to baseline AI visibility, and it is what turns an unstable number into a diagnosable one.