The Tie Check - Refusing to Guess When the Evidence Splits Evenly

The problem

I built a diagnostic agent for a manufacturing line, meant to look at sensor logs off a stamping press and recommend a root cause when the line started kicking out defective parts. Feed it two hundred seeded incidents, and in forty-five of them I built the vibration-and-temperature signature on purpose to fit two genuinely distinct causes equally well: bearing wear and voltage sag. Nothing else in the snapshot could tell the two apart. The evidence, as far as the sensor data went, was a real tie.

The agent didn't know how to sit with a tie. It always returned exactly one root cause, with a confidence score attached, defaulting to whichever candidate its scoring function happened to rank marginally first. On all forty-five of those genuinely ambiguous incidents, the output looked exactly like the output on an unambiguous one: one clean answer, one number, no hint that a maintenance engineer was about to order a specific and specifically expensive repair based on a coin flip the system never disclosed.

This is a different failure than a model that doesn't know its own training was thin. This is a model whose evidence, at the moment of the decision, genuinely doesn't point to one answer, built inside a system that's structured to always answer one thing anyway, because "I don't know which of these two" isn't a shape most output formats know how to hold. The tied alternative was sitting right there in the log the whole time. It just never got surfaced, because the interface only ever asked for a single answer and the system dutifully gave it one.

The pattern

I added one check before any diagnosis ships: a comparison between the top two candidate scores. When they land within a configured margin of each other, the system stops pretending it can pick a winner. Instead of resolving the tie by whatever tiebreak logic happened to be sitting in the scoring function, it surfaces both live hypotheses, their scores, and what additional evidence would actually distinguish them, and routes the case to a person for resolution.

The silent-commitment rate on the ambiguous subset drops to zero. Every genuinely tied case now gets surfaced first, before anything gets resolved by default, and once an engineer weighs in with the plant-floor context the sensor log never captured, accuracy on the previously ambiguous cases lands well above whatever the old silent tiebreak was producing. The gate itself doesn't get smarter about which cause is correct, and it doesn't have access to anything the ungated version didn't have. Its contribution is the discipline to say, out loud, that the evidence doesn't determine an answer, before anyone quietly picks one and calls it a diagnosis.

That distinction matters more than it sounds like it should. Detecting an ambiguous case and resolving it are two completely different jobs, and this pattern only does the first one. It tells you, reliably, when the sensor data alone can't tell bearing wear from voltage sag. Resolving which one is actually correct is a separate job, one this gate was never built to do. The decision about which repair to order then gets made by someone with more evidence than the log actually contains.

Design considerations

The margin you set for "close enough to count as a tie" is the whole ballgame, and it's a judgment call every time. Set it too tight and you catch only the most extreme ties, missing plenty of cases where the evidence was genuinely thin even though the scores weren't mathematically identical. Set it too wide and so much traffic gets routed to human review that the gate stops adding value, because now everything looks tied and the engineer stops trusting the flag. The right margin comes from looking at the distribution of score gaps across historical cases and finding where the gap starts correlating with real downstream disagreement. A round number chosen because it felt reasonable carries no such grounding.

This pattern doesn't generalize cleanly to more than two candidates without more thought than it looks like it needs. A tie between exactly two hypotheses is a clean comparison. A near-tie among three or four candidates is a genuinely harder problem, because the question stops being "are these two close" and becomes something closer to a full ranking-uncertainty problem. I wouldn't bolt a two-candidate margin check onto that case without rebuilding the comparison logic from scratch.

There's a broader problem sitting behind this specific pattern, and I want to name it plainly. Two models can be trained on the exact same data and disagree with each other on individual cases, because more than one internally consistent account can fit the same evidence equally well, with neither model actually wrong. Researchers studying this call it the Rashomon effect: many models fit the same data equally well and still land on different answers (Breiman, 2001; Fisher, Rudin & Dominici, JMLR, 2019). A related line of work, semantic entropy, gives a way to measure this kind of ambiguity per query by clustering how diverse a model's own outputs are when you sample it repeatedly on the same input (Farquhar et al., Nature, 2024). Both point at the same structural fact this tie check is a narrow, practical instance of: sometimes the evidence just doesn't pick a winner, and no amount of additional reasoning from that same evidence was ever going to change that.

One caution I'd flag from watching this space evolve. It's tempting, once you understand the tie-check idea, to reach for an ensemble of models and treat their disagreement as your ambiguity signal, skipping the margin check that would otherwise sit directly inside one scoring function. I'd be careful with that move. A study running large ensembles found that which model reads as the "most divergent voice" depends heavily on which models happen to be sitting in that ensemble (arXiv:2608.00285). It has nothing to do with anything stable about the case itself. Ensemble disagreement is real information, but treating raw disagreement as a clean, portable signal of underdetermination is a mistake I've seen teams make, and it doesn't hold up once you swap out which models are in the room.

The gate is cheap to build and easy to justify to a skeptical engineering team, because the failure mode it catches is so visible once you've seen it: a system confidently recommending a specific, costly repair off evidence that, on inspection, was a coin flip. What it costs is a small amount of throughput, since some fraction of cases that used to resolve instantly now wait on a person. That cost pays for itself in domains where the two candidates imply genuinely different actions. It's much harder to justify in domains where the tied candidates would have led to the same next step anyway. Knowing which of those two situations you're actually in is the calibration question that decides whether this pattern earns its keep.