The short version

Run-to-run variation on an unchanged brand is large enough that small movements are usually noise. Treat a change as real only when it holds across repeated runs and more than one engine.

At five observations a 40% result carries a 95% interval running from about 12% to 77%. Almost any movement you could observe fits inside that.

Nothing here requires trusting a vendor’s number. The uncertainty on a proportion is arithmetic, and once you compute it for your own sample size, most of the arguments about whether a change is real answer themselves.

Why is there variation at all?

Because generation is not deterministic. OpenAI’s advanced usage guide states that “Chat Completions are non-deterministic by default (which means model outputs may differ from request to request),” and even with a fixed seed the documentation promises only mostly deterministic output.

Google says the same about its own surfaces, noting in its documentation on AI features that “AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary.”

So every visibility figure is an estimate from a sample, and every estimate has an interval around it. The only question is how wide.

How wide is the interval, actually?

Computable. The NIST/SEMATECH engineering statistics handbook publishes the interval for a proportion, and applying it to a 40% observed mention rate at different sample sizes gives this.

Observations	Observed rate	95% interval	Interval width
5	40%	11.8% to 76.9%	65 points
15	40%	19.8% to 64.3%	44 points
30	40%	24.6% to 57.7%	33 points
50	40%	27.6% to 53.8%	26 points
150	40%	32.5% to 48.0%	16 points
450	40%	35.6% to 44.6%	9 points

These are computed from the published formula rather than measured from any brand, and an observation means one prompt on one engine once. A 30-prompt set run 5 times on 3 engines is 450 observations, which is the bottom row.

Read the width column rather than the interval itself. It falls fast at the start and slowly afterwards, which is the practical argument for widening a small prompt set before doing anything else: going from 5 observations to 30 buys 32 points of precision, while going from 150 to 450 buys 7.

What the interval width means in practice

A five-observation check Cannot detect anything

The interval spans 65 points. A move from 40% to 55% is entirely inside it, so the check cannot distinguish the two.

A 450-observation programme Detects roughly ten-point moves

The interval spans 9 points. A move from 40% to 55% is now outside it and worth investigating.

Sample size does not make your number more accurate, it makes it more decidable. The point estimate is 40% in both columns; only the claim you can defend changes.

So when is a change real?

When the two intervals do not overlap, and when the movement shows up in more than one place. Two tests, and a change should pass both before you act on it.

The interval test is arithmetic. Compute the interval for last period and this period at your actual observation count. If they overlap substantially, you have not measured a change, whatever the point estimates say.

The corroboration test is structural. A real change in your visibility usually appears on more than one engine, or across more than one prompt, or in both directions consistently. A movement isolated to one engine and one prompt is the shape noise takes.

Two overlapping intervals with different midpoints is not a trend. It is two samples from the same underlying rate.

How do you apply this without doing statistics every week?

Compute one threshold, once, and reuse it. That is the noise floor from your baseline, and it turns every future comparison into a lookup rather than a calculation.

A weekly decision rule

Count your observations honestlyPrompts times engines times runs. Not prompts. This is the number that goes into the interval, and it is usually far smaller than people assume.

Compute the interval width onceAt your observation count and your typical rate. Write it down as a single number of points.

Ignore anything smaller than itNot “investigate cautiously”. Ignore. Reporting movement inside your own margin is how a programme loses credibility with anyone who can read a chart.

For anything larger, check the shapeOne engine or several? One prompt or many? The pattern tells you whether it is yours before you investigate the cause.

Step one is where most of the error enters. A “50 prompt” plan checked once a day on two engines is 100 observations a day, not 50, and a plan checked weekly is a seventh of what you assumed.

If the movement clears the threshold, the shape of it identifies the cause faster than any investigation will, which is the triage in AI visibility dropped suddenly. And if you have not established a floor yet, that is the exercise in how to baseline AI visibility.

What if you cannot increase the sample?

Then widen the reporting period rather than the claim. Four weeks of a small sample is a larger sample, provided nothing about the protocol changed in between.

And report the interval alongside the number. “38%, plus or minus 26 points, from 50 observations” is a defensible sentence. “38%, down from 41%” from the same data is not, and the difference matters most in the room where budget is decided.

The habit of reporting the uncertainty with the estimate is the same discipline that keeps mention, citation and recommendation rates from being blended into one figure, covered in mention rate vs citation rate. A number with no stated uncertainty is being presented as more solid than it is.