The Delta That Traveled Through Two Different Weeks

The problem

I built an order-fulfillment agent, FulfillBot, that calls a flaky inventory-check tool, and shipped it with a retry-classification gate that decides whether a tool failure is a transient blip worth retrying or a systemic bug worth escalating, instead of blindly retrying its way into a duplicate order. I wanted to know whether that classification still earned its keep now that the underlying tool-calling framework had matured, so I did the obvious thing first. I ran one week with the gate off, the following week with it back on, and compared task success between the two weeks.

That comparison was broken before it produced a single useful number, and I built the scenario specifically to see why. A sequential before-and-after comparison only ever sees two different slices of traffic, and traffic drifts whether anyone wants it to or not. In this case, the injected failure mix genuinely differed between the two weeks. More systemic-bug-type failures landed in the gate-off week purely by chance, which meant the ungated week looked worse for reasons that had nothing to do with the gate. The sequential delta the dashboard computed came out under 3 points. Read at face value, that number said the gate barely mattered anymore. What it actually said was that the two weeks were never comparable, and I'd spent a week measuring noise and calling it a finding.

The pattern

The fix that actually holds up doesn't try to control for traffic drift after the fact. It fixes the comparison at the root, by forking every incoming request into two parallel executions against the identical injected failure at the identical moment: one path runs with the gate on, one runs ungated. The customer only ever sees the gated path's result. The ungated fork exists purely to be measured, never to affect what actually ships.

Because both paths see exactly the same traffic and exactly the same failure, the delta this version computes is a clean, per-request marginal effect with nothing left to argue about. I measured a simultaneous delta of 15 points or more here, against the sequential version's under-3-point illusion, running against the same underlying failure distribution. That gap between 3 points and 15 points is the difference between a comparison that traveled through two different windows of reality and one that never had to.

This is the aerospace test-flight version of an A/B test, and it's the analogy I reach for every time I catch myself about to run a sequential comparison. Nobody validates a new airframe modification by flying the old design on Tuesday and the new one on Wednesday and comparing outcomes, because Tuesday's wind, Tuesday's air traffic, and Tuesday's fuel load are not Wednesday's. Flight test compares configurations against the same conditions, the same flight profile, as close to simultaneously as physics allows. A shadow-run comparison insists on the same thing for a software gate: never trust a delta that had to travel through two different windows of reality to get computed.

Design considerations

Forking every request into two executions costs real compute, doubled tool calls, doubled downstream load on whatever the ungated path touches, for as long as the measurement runs. That cost is the whole reason a sequential comparison is tempting in the first place. It's cheaper to run one path at a time. I only accept the doubled cost when the traffic itself is drifty enough, or the stakes of the wrong answer are high enough, that a confounded delta would actually mislead a real decision. For a low-stakes gate where a rough sequential estimate is good enough to act on, I wouldn't build the full shadow infrastructure.

The ungated fork has to be genuinely inert from the customer's point of view, and that's a harder engineering constraint than it sounds. Any side effect the ungated path can trigger, a real inventory hold, a real downstream API call with consequences, has to be either fully sandboxed or fully suppressed, because the entire value of the pattern depends on the measurement path never leaking into production behavior. A shadow run that accidentally places a real order isn't a shadow run anymore.

There's also a ceiling on what this pattern can tell me. It isolates the gate's marginal effect against the specific failure distribution I injected during the measurement window. A shadow run answers one narrow question: does the gate help against this specific failure distribution. It says nothing about a failure distribution I never seeded. If the real-world failure mix shifts to a systemic-bug type I never modeled, the clean delta I measured doesn't automatically generalize to that new mix. I'd want to refresh the injected mix periodically against whatever failures are actually showing up in production, rather than treating one clean measurement as permanent proof.