The Ten-Run Rule - Telling a Real Regression From Ordinary Noise

The problem

I had a refund-eligibility prompt running at production temperature, 0.7, deciding approve, deny, or escalate on fifteen seeded test cases, with a reason attached to each decision. I ran it through five scripted edits, some of them real behavioral changes, some of them neutral rewordings I wrote specifically to not change anything the model should do differently. The test harness I had in place ran each of the fifteen cases exactly once per edit and reported pass or fail against a ground-truth answer.

Here's what I found once I checked the harness's verdicts against a separately computed ground truth: at least two of the five edits produced a harness verdict that disagreed with the actual ground truth, and neither disagreement had anything to do with what the edit actually changed. A single run at nonzero temperature is one draw from a distribution the model is sampling from. It's a sample, never the full shape of that distribution. Run a case once and you get one sample. Sometimes that sample happens to land on the wrong side of a decision the model would get right nine times out of ten. Sometimes it happens to land on the right side of a decision the model has actually started getting wrong more often than not. Either way, a single-run harness can't tell the difference between an edit that changed behavior and an edit that just happened to draw an unlucky sample.

That's a genuinely uncomfortable finding once you sit with it, because it means the harness itself was manufacturing false alarms on edits that hadn't broken anything, and simultaneously missing real regressions that happened not to surface on that particular draw. A test suite that can't distinguish signal from noise actively erodes trust in the testing process, because every "regression" it flags now needs a human to manually re-check whether it's real, which is exactly the manual verification step you built the harness to eliminate in the first place.

The pattern

The fix keeps the temperature exactly where it was. Dropping temperature to zero to make the harness deterministic would solve the noise problem by changing the thing you're actually testing, since production runs at 0.7, and a harness that tests some other temperature is measuring a different system than the one that actually ships. This harness runs each test case ten times per edit, holding every input constant across those ten runs, and reduces the ten outputs to a single verdict two ways: the modal output, whatever the majority of the ten runs actually produced, and the intra-case variance, how much the ten outputs disagreed with each other.

A genuine behavioral change shows up as a shift in the modal output itself: what nine or ten runs agree on changes from one edit to the next. Ordinary sampling variance shows up differently: the modal output stays the same edit over edit, but the spread among the ten runs might widen or narrow without the center of mass actually moving. Those are two distinguishable signatures once you're looking at ten runs instead of one, and they were never distinguishable from a single draw, because one sample can't show you a distribution's shape.

Running the same five edits through this version brings the false-positive and false-negative rate down to near zero against the same ground-truth check. Nothing about the underlying prompt or the model changed between the two versions of the harness. What changed is that the harness stopped mistaking one sample for the whole distribution, and started reporting a verdict that reflects what the prompt actually does across ten draws instead of what it happened to do on one.

Design considerations

Ten is a practical number sized to this specific harness, and here's where it actually came from. It's large enough to reveal a clear modal answer and a meaningful variance estimate on a binary or small-enum decision like approve, deny, or escalate, and small enough to keep the harness affordable to run on every edit. A decision space with many more possible outputs, or a task where the acceptable answers are more continuous than categorical, might need more runs to get a stable modal read, and a cheaper, lower-stakes decision might get away with fewer. The number should track how much the specific decision space actually needs to stabilize a majority. Copying whatever number worked on a different prompt entirely skips that step.

There's a real cost multiplier here that has to be priced in honestly. Running every case ten times instead of once means ten times the inference cost and ten times the latency for every single harness run, on every edit. That's a fair trade for a prompt gating something as consequential as a refund decision, where a false "no regression" verdict could let a real behavioral problem ship silently. It's a much harder trade to justify for a low-stakes internal tool where a single-run harness's occasional false alarm costs a few minutes of a developer's attention and nothing more. This pattern earns its cost in proportion to how expensive it is to ship an undetected regression. Applied as a default setting for every eval harness, regardless of what's actually riding on it, that proportionality gets lost entirely.

There's a mild circularity here, and I'd rather name it than pretend it isn't there. The ground truth I checked the harness against was itself defined as a ten-run aggregate, the same mechanism the harness under test now uses. That doesn't undermine the finding. Ten runs produces a stable enough estimate of the model's actual behavior that comparing modal outputs across edits becomes a meaningful comparison grounded in what the prompt actually does, and that's a real, useful property regardless of where the comparison baseline came from. But I'd rather flag that circularity plainly than let a reader assume the ground truth here came from some independent, run-count-free source of certainty. It didn't. It came from the same statistical logic, applied once, carefully, to establish a baseline before applying it again, repeatedly, to check every edit against that baseline.