The Gate That Only Ever Learned One Question

The problem

I had a fulfillment pipeline running with a quality gate sitting between an order-parsing agent and a warehouse-dispatch agent. I called it the inventory-check gate. It had a test suite, that suite ran on every change, and it had been passing clean for months. Everyone looking at the dashboard had good reason to feel confident about it, because a gate that's been green for months is exactly the kind of signal a team is trained to trust.

Then I shipped a new order type: split shipments, where a single order fulfills out of multiple warehouses instead of one. I built a seeded dataset of forty orders, twenty-five single-item and fifteen split-shipment, and ran the split-shipment orders through the existing gate without touching the gate itself first. It passed all of them. It had always passed everything, because the gate's tests were written against a mocked inventory API that only ever returned single-item, single-warehouse responses. The gate had never once been asked a question about split shipments, so it had no way to fail one. Miscounted orders moved straight through to autonomous dispatch, and the gate's self-reported pass rate stayed above ninety percent the entire time, because a self-reported pass rate only confirms that the gate ran and returned green. It says nothing about whether the questions it asked were still the right ones.

What made this failure mode uncomfortable to sit with is that nothing about it looked like negligence in the moment. The gate wasn't malfunctioning. The tests that existed were genuinely testing something real, the single-item path, and testing it correctly. The gap traced back to a question nobody had asked: does this gate's existing coverage actually include the thing that just changed. A green checkmark answers whether the test ran and passed. It doesn't answer whether the test was still asking about the part of the system that's different now, and those are two completely different claims that happen to look identical on a dashboard.

The pattern

The fix I built is an audit tool that sits between a proposed change and gate promotion. It takes a diff manifest, a record of which order-type paths actually changed in the new feature, and checks that diff against a coverage map of what the gate's existing tests actually exercise. When the two don't line up, when a change touches a path the gate's tests never covered, the audit blocks promotion until new tests exist for the changed path specifically.

For the fulfillment case, that meant the audit caught, before the split-shipment feature ever shipped, that the inventory-check gate's tests only ever exercised the single-item mock path. It blocked promotion until the team built new gate tests against a mocked inventory API that actually modeled multi-warehouse stock correctly, not a single-item response dressed up to look like general coverage. Once those tests existed and passed, the audit released the gate, and ground-truth accuracy on split-shipment orders climbed above ninety percent, up from the roughly fifty percent the ungated version had actually been producing the entire time it was reporting green.

The distinction this pattern enforces is the gap between a gate that ran and passed and a gate that was sufficient for what changed. Those sound like the same claim right up until a system's input space grows past what the gate was ever built to answer. A gate can be perfectly correct about the question it was designed to ask and still be completely blind to the question that matters now, and the only way to catch that gap before it ships is to check the coverage explicitly, against what actually changed, rather than trusting a pass rate that has no way to know its own question went stale.

Design considerations

This audit is only as good as the diff manifest it's checking against, and that's a real limitation, not a footnote. It catches regressions the team declared as changes. A regression introduced sideways, through a shared library update or a dependency bump that nobody logged as touching the inventory-check path, stays completely invisible to an audit built around declared diffs. The audit has no record of what it wasn't told changed, and a change that slips in under a different name is exactly as dangerous as it was before the audit existed.

The coverage map itself carries a similar risk. The audit compares a proposed change against a map of what the gate's tests currently exercise, and that map has to stay accurate as the test suite evolves for the comparison to mean anything. A coverage map that quietly drifts out of sync with the actual tests, because someone modified a test without updating what it's documented to cover, hands the audit the same false confidence the original ungated gate had. It just relocates that false confidence one layer up, from the gate itself to the thing that was supposed to be checking the gate.

What this pattern buys is a narrower claim than catching every blind spot. It catches the specific blind spot created by declared changes outrunning declared coverage, which turns out to be a large share of real incidents, because most regressions do arrive attached to a change someone consciously shipped. It doesn't replace the harder, slower work of periodically re-auditing a gate's coverage against the system's current shape, independent of any specific change, which is the only defense against the sideways regression this pattern can't see. Both matter, and they're catching different failures. Treating the faster one as a substitute for the slower one is how a team ends up surprised by exactly the kind of gap this pattern was built to close somewhere else.