A single run tells you what one sample looked like. Stable brand-presence estimates need repeated runs per prompt per engine, and the count needed rises as your mention rate approaches fifty percent.
Reaching plus or minus ten points takes about 25 observations at a 5% rate and about 93 at a 50% rate. The closer you are to a coin flip, the more evidence you need.
This is a planning question rather than a judgement one. Before you build a prompt set, you can work out how many observations it has to generate for the resulting number to be worth reporting.
Why does one run tell you so little?
Because generation is probabilistic and one run is one draw. OpenAI’s advanced usage guide states that “Chat Completions are non-deterministic by default (which means model outputs may differ from request to request).”
So a single run yields 0% or 100% for that prompt. Neither is a rate, and averaging a set of ones and zeros from single runs does not rescue it, because the sample size is still the number of observations rather than the number of prompts.
How many runs do you actually need?
It depends on the precision you want and the rate you are estimating. Applying the interval for a proportion published in the NIST/SEMATECH engineering statistics handbook gives the following.
| Your mention rate | For ±15 points | For ±10 points | For ±5 points |
|---|---|---|---|
| 5% | 13 observations | 25 | 83 |
| 10% | 17 | 37 | 141 |
| 20% | 26 | 60 | 245 |
| 30% | 33 | 78 | 320 |
| 40% | 38 | 89 | 366 |
| 50% | 39 | 93 | 381 |
These are computed from the published formula, not measured from any brand. An observation is one prompt, on one engine, once. So 30 prompts run 5 times on 3 engines produces 450 observations, which clears every cell in the table.
Precision is expensive at the margin. Going from plus or minus ten points to plus or minus five costs four times the observations, not twice.
Why does the number rise toward fifty percent?
Because the uncertainty on a proportion depends on the proportion itself, and it peaks in the middle. A rate near zero or near one is easy to pin down, because almost every observation agrees with the last one.
At fifty percent the observations disagree as much as they possibly can, so each new one carries less information about where the true value sits. That is why the last column runs from 83 observations at a 5% rate to 381 at a 50% rate for identical precision.
This has an awkward implication for anyone whose visibility is actually improving. The measurement gets harder as you move from invisible toward contested, which is precisely the period when you most want to prove the work is landing. Budget for it before you start rather than discovering it mid-quarter.
Should you add runs or add prompts?
Prompts, in almost every case, and the reason is not statistical.
Both raise the observation count identically, so on the arithmetic alone they are interchangeable. But running the same prompt fifty times measures one question very precisely, and one question is not your market. Fifty prompts run five times measures your category, and it protects you from a set that is quietly asking the wrong things, which is the failure in realistic prompts for AI visibility tracking.
Add runs only where you need per-prompt precision, which is usually a small number of high-value prompts you are actively working on. Everything else is better served by breadth.
How do you get there in practice?
Once the set is sized and running, the count converts into a single threshold you can reuse weekly rather than recomputing, which is the decision rule in is my AI visibility change significant. And if you have not yet established what your numbers do while nothing is changing, that measurement comes first, in how to baseline AI visibility.
What if the numbers here are out of reach?
Then report the interval rather than pretending to a precision you do not have. “31%, plus or minus 15 points, from 38 observations” is honest and still useful for direction.
What is not acceptable is reporting the point estimate alone and letting the reader assume it is solid. The underlying variation does not go away because the sample was small, which is the mechanism in why ChatGPT gives different answers to the same question.
There is also a cheaper route to precision than more runs, and it is worth trying first: narrow what you are estimating. A rate computed across your whole prompt set pools questions you win with questions you never could. Splitting the set by intent gives you several smaller estimates that each sit nearer the extremes, where intervals are naturally tighter, instead of one pooled figure parked near the middle where they are widest.