Twenty Clean Runs Before the First Wire

The problem

I built three agents for a claims-automation project: an intake classifier that routes each incoming claim to the right desk, a policy-lookup agent that pulls coverage terms out of a policy database, and a damage-estimate drafter that writes an initial estimate narrative from adjuster notes and photos. None of the three had ever been run to independent stability. Nobody had measured what any of them did on their own, at volume, without the others watching.

Under a roadmap deadline, I wired all three directly into an orchestration the same week each one was first built. That's the honest version of how this usually happens. Nobody sits down and decides to skip the stabilization step. The deadline just makes stabilization feel like a nice-to-have to circle back to once the demo ships.

Here's what wiring three unstabilized boxes together actually does. A bad classification from the intake agent silently sends the wrong policy terms downstream to the lookup agent, which produces a confidently wrong coverage figure, which the estimate drafter then writes a clean, well-formatted, entirely wrong narrative around. Every individual output looks fine in isolation. I ran forty synthetic claims through this setup, with faults injected at specific boxes so I'd know the ground truth, and measured what I call bad-packet attribution rate: of the packets QC flagged as wrong, what share could anyone actually trace back to the box that caused it. It came in under 40 percent. More than six times out of ten, a bad packet arrived at the end of the chain and nobody could say which agent was responsible, because none of the three had ever been baselined on its own. The chain had a defect. The chain had no memory of where that defect started.

The pattern

The fix is to build a readiness tracker before any orchestration exists at all, and to gate wiring behind it. Each box runs standalone against its own labeled test slice until it clears a stability bar: twenty consecutive clean runs and a correction rate under 10 percent. The tracker logs every run and every correction. Only a box that clears both numbers becomes eligible to be wired into the orchestration, and the endpoint that assembles the chain is built to provably refuse an ungated box. The refusal is mechanical, built into the endpoint itself, so no team can wire around it under deadline pressure.

Running the same forty claims through this version, with the same injected faults, changed the attribution number from under 40 percent to near 100 percent. Nothing about the intake classifier, the lookup agent, or the drafter got smarter. What changed is that every fault now traces to a box with a known, individually measured baseline, because the wiring step itself won't accept a box that hasn't earned one.

The sequencing is the whole mechanism. Orchestration design gets deliberately pushed to after the evidence exists, and the evidence has a specific shape: twenty runs, one number for corrections. That's a low bar on purpose. It's cheap enough that skipping it can never be excused as too slow, and specific enough that "it seemed fine" stops being an acceptable substitute for a logged streak.

I read Anthropic's "Building Effective Agents" after I'd already built this pattern, which is a nice feeling, arriving at the same rule from a different angle. Their own escalation ladder, from a single augmented call up through prompt chaining and routing, and only then into orchestrator-workers or autonomous agents, explicitly recommends adding complexity only once a simpler solution has demonstrably fallen short. That's the same instinct sitting at the Stage 3-to-4 boundary here. Don't wire the arrows until the boxes have already proven they don't need fixing.

Design considerations

A box proven stable standalone hasn't earned any guarantee about how it behaves once it's wired to something else. Twenty clean runs against a solo test slice tells you nothing about interaction effects: timing, shared state, one box's output quietly reshaping the input distribution the next box was tested against. The readiness gate buys a clean starting point for attribution. It doesn't certify the orchestration itself, and that's a separate problem that shows up the moment the boxes start talking to each other.

The twenty-run threshold is a floor. Here's what a floor that low actually buys you: twenty runs catches an agent that's obviously broken, and nothing more. It's a much thinner sample than you'd want before trusting a box with anything consequential, and the correction-rate ceiling matters more than the run count once volume climbs, because a 9 percent correction rate sustained across ten thousand real claims is a very different animal than the same rate sustained across twenty synthetic ones. I use twenty as the number that makes a stabilization gap impossible to hide. It says nothing about readiness at scale.

There's also a real cost to insisting on this discipline. A team with a hard deadline and three boxes that all look fine in a demo is going to feel real friction from a tracker that says none of them are wired-eligible yet. That friction is the entire point. Skip the gate and the deadline still gets met on paper, and the unattributable defect just shows up three weeks later, at a much worse time to be debugging it.

The place this pattern doesn't help is a box that's individually stable but was never tested against the distribution it'll actually see once it's live. A classifier that's clean on synthetic training claims can still be unstable against a real claim type nobody thought to seed into the test slice, and twenty clean runs against the wrong distribution will pass the gate just as confidently as twenty clean runs against the right one. Stabilization is a claim about what was tested. What the system will actually see once it's live is a separate question, and answering it honestly is what keeps this pattern from turning into a false sense of safety.