How we handle
variance in AI answers

What we measured

In June 2026 we ran the same ten prompts for a counseling client twice in one afternoon. The first run turned up eight questions where a competitor was named and the client was not. The second run, an hour later, turned up zero. Same prompt set, same engine, same website, and nothing about the client changed in that hour.

Our tooling did not break. Web search results move between requests, the engine samples its own wording, and what comes back differs every time you ask. So if you ask once and write the answer down, you have recorded one draw and called it a rate.

What each run records

Every question runs five times per engine as standard. In each run we record one state for you and one for every competitor in the set:

Then we count. Cited or mentioned counts as named. If you were named in three of five runs, the report says named in 3 of 5 runs. Every raw run stays in the snapshot file we hand you, so you can recount it yourself.

Why the report shows a range, not a number

Three of five is 60%. The question is what a 60% built from five runs is worth. The margin-of-error formula most of us learned in school assumes a large sample, and it breaks in three ways on samples this small.

The grey line is 100%, the edge of what a share can be. The first bar runs past it. The other two have no width at all, which is the formula claiming five runs settled the question.

Being named in every run or in none of them is the most common result on a small prompt set, so those last two turn up in almost every snapshot we build. We use the Wilson score interval instead. Edwin Wilson published it in 1927 for this exact problem. Rather than building a range outward from the share we observed, it solves for the range of underlying rates that could plausibly have produced what we saw. The result never leaves 0% to 100%, and with few runs it pulls toward the middle, with the pull fading as runs accumulate.

Same three results, same scale. Every bar has width, and none of them cross the line.

That last one is the one worth saying out loud. Five runs where an assistant never named you are a reason to look into it. They are not on their own enough to justify rebuilding your website.

What the range covers, and what it does not

The interval describes one specific thing: how much the run count alone pins down the rate at which you were named, for that question, on that engine, through that API, during that sampling window. It is sampling uncertainty and nothing else. Four limits sit outside it, and none of them get smaller when we add runs.

How many runs, and why that many

We picked the run budget before the run starts, from how wide the interval is expected to be. We do not watch results come in and stop when the range looks good enough, because stopping on the answer breaks the arithmetic that produced it. Holding the observed share at 60%, here is what more runs buy:

The white tick sits at 60% in all four. Only the band around it changes.

Five runs is the screening budget. It is enough to separate a business the engines name most of the time from one they almost never name, and that is the finding most Snapshots turn on. It is not enough to resolve a close call. When a decision depends on telling two numbers apart, or on watching one number move, we raise the count to ten or more for those questions and say in the report which count each number came from. The extra runs are cheap. Ten questions on three engines at five runs each is 150 calls, and roughly 300 runs costs $8 to $15 in API spend, so the run budget is a reporting decision rather than a cost one.

How we compare one week to the next

We flag a change only when the two intervals separate completely. That rule is deliberately conservative, and it cuts in one direction on purpose: we would rather miss a real improvement for a week than tell you that you gained ground and take it back a month later.

Two things that rule does not mean, which we would rather say than have you assume. Overlapping ranges are not proof that nothing changed, only that this many runs cannot tell the two weeks apart. And separated ranges are evidence that the rate moved, not evidence of what moved it. Non-overlap is also a blunt test: two ranges can overlap while a direct comparison of the difference still finds a real gap. The sharper method computes an interval on the difference between the two rates rather than eyeballing two separate intervals, and adding it is on our list. Until it is in the report, our rule stays the conservative one, and now you know which way it errs.

Why there is no rank in the report

We never report that you came third in an answer, and we will not build that number on request. The reason is not that the position moves between runs. This whole page is an argument that something which moves between runs can still be measured by repeating it.

The reason is that a position in a generated paragraph is not a ranking. Search results have a defined order with a fixed number of slots. A written answer has an order that follows sentence construction, and the order changes when the question is reworded even where the recommendation does not. There is nothing stable underneath it to estimate, so repeating it more times would give you a precise measurement of something that does not mean anything. Being named is a real thing that either happened or did not in each run, which is why that is the thing we count.

The interval, in code

This is the whole calculation, exactly as it runs in our snapshot tool. The n == 0 branch matters: with no runs there is nothing to estimate, so it returns the whole range. An earlier version returned a zero-width interval at zero, which a report prints as "0% to 0%", which is the same false certainty the interval exists to avoid.

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """Wilson 95% interval for k hits in n runs."""
    if n < 0 or k < 0 or k > n:
        raise ValueError(f"wilson: need 0 <= k <= n, got k={k}, n={n}")
    if n == 0:
        return (0.0, 1.0)          # no runs, so no estimate
    p = k / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (max(0.0, centre - half), min(1.0, centre + half))