The Retry Gate - Telling a Bug From a Blip Before You Try Again

The problem

I built a small travel-booking agent to run sixty synthetic flight-booking requests through a mocked pricing-and-booking API, because I wanted to watch a specific failure happen under conditions I controlled instead of just believing it happens. The agent wrapped a single tool call in a generic retry decorator: retry on any exception, exponential backoff, five attempts, done. Nobody designed that policy on purpose. It came bundled with the retry library, and I let it ride the way most of us do, right up until it costs something.

The failure table behind the mock had three kinds of requests seeded into it, and only one of them actually wants a retry. True blips: a 429 rate limit, a 503 timeout, the kind of failure where trying again is exactly correct. True bugs: a 400 for an invalid airport code, a 410 for an expired contract version, the kind of failure where trying again just reruns the same doomed request five times at full backoff delay. And a third case I hadn't weighted heavily enough before I built this: requests where the booking actually succeeded on the server, and the client lost only the confirmation on the way back.

A generic retry-on-any-exception wrapper treats all three cases the same, because nobody ever told it to look closer. The 400 gets retried five times, burning real seconds and real mocked dollars on a request that was never going to succeed no matter how politely I asked again. That's the cheap mistake. The expensive one is the third case. A booking that already committed gets retried anyway, because the client has no way to tell "no response yet" from "already done," and the retry succeeds. It books the same seat twice. Nobody notices until someone reconciles the ledger and finds two paying customers where there should be one.

I put a dashboard in front of the run so the failure carried a number instead of a shrug: a live, color-coded feed of retry attempts across the sixty requests, red for a retry burned on a bug, green for a retry that rightly rescues a blip, next to a running count of dollars wasted and duplicate bookings created. Watch it long enough and the failure splits into two distinct colors. The red bars are the cheap mistake, money and seconds spent on a request that was dead on arrival, easy to fix the moment I write the classification down. The duplicate-booking counter is the expensive one, because from the booking system's side it looks like two legitimate customers, not a retry problem at all. Nothing in the ledger entry whispers that a timeout hid a success underneath it. You have to already suspect the retry layer to go looking in the retry layer, which is exactly the kind of failure that sails through a demo and only shows its face once volume is high enough for a duplicate to actually land.

The pattern

Which exceptions get retried has to become an explicit, unit-tested, application-owned decision, expressed as one function with a name and a contract: is_retryable(exception, response) -> bool. Not a blanket policy inherited from whatever retry library happened to be installed. True blips retry: 429s, 503s, connection resets. True bugs fail fast: 400s, 410s, validation errors. And before any retry fires at all, a booking-status check runs against the ledger, so a request that already succeeded server-side can't get retried into a duplicate just because the client never saw the confirmation come back.

I rebuilt the agent with this gate in place, running the identical scenario: same sixty requests, same failure table, same mocked API. The only thing that changed was that the tool call now passes through an application-owned RetryGate module first, a pinned allowlist of retryable exception and status-code combinations backed by its own unit test suite, plus the pre-retry idempotency check against the booking ledger. Nothing about the underlying scenario got easier. The whole difference is whether the system knows, before it retries, what kind of failure it's actually looking at.

I set real bars to clear before I ran anything: wasted dollars and time down by at least 80% against the naive version, duplicate bookings at zero, and blip-recovery rate holding within a few points of what the naive version achieved, because a gate that turns too cautious and stops retrying real blips has just traded one failure mode for another. The mechanism behind that target is straightforward once you see it. Every dollar wasted on a bug-class retry vanishes the moment the classifier fails fast on the same input instead of retrying it five times at exponential backoff. Every duplicate booking vanishes the moment the ledger check runs before the retry instead of never running at all. No new infrastructure, no smarter model required. What this took was writing down, in testable code, what the framework's default retry policy had been silently guessing at the whole time. That's a smaller job than it sounds, because the failure table itself hands you the spec for free: it names every case the classifier has to get right.

The allowlist deserves a moment of attention precisely because it's small enough to read start to finish, and that's what makes it trustworthy. It's a pinned, versioned list: these status codes retry, these don't, backed by a unit test asserting that behavior for every failure type the seed table injects. No machine-learned classifier producing a mystery probability score anywhere in the loop. Anyone reviewing it can read the whole policy in about the time it took you to read this paragraph. Nobody can point to the line in the retry library's default configuration that decided a 410 deserved five attempts, because no such line exists. It just happened by default, which is worse than a bad decision made on purpose.

Design considerations

is_retryable classifies on exceptions and status codes. It has no eyes on the response body at all. A call that returns HTTP 200 with a malformed, truncated, or outright hallucinated payload looks identical to a clean success as far as the gate is concerned, because nothing about that response trips an exception or a bad status code. A documented 2026 incident makes this gap uncomfortably concrete: an agent stuck in a retry loop ran up a $437 bill over eight hours because its failures were semantic, not transport-level, and never tripped anything a status-code classifier could see. A call that fails by lying convincingly is a different problem, and this gate does not solve it.

The calibration question that actually matters day to day is where the allowlist draws its line, and it's easy to get wrong in either direction. Draw it too narrow and you stop retrying real blips, which shows up as a dropped blip-recovery rate on the dashboard and quietly costs you legitimate bookings that a slightly more patient system would have recovered. Draw it too wide, retrying anything that looks even a little transient, and you're back to burning money on bugs, just with a smaller, better-dressed allowlist doing it. I found the right calibration by running the failure table through the classifier before I trusted it with real traffic, which only works because the failure table names every case up front. A system without that kind of seeded ground truth has to earn the same confidence a slower way, watching real production failures accumulate and checking the classification against what actually happened downstream.

There's also a maintenance cost to naming the policy explicitly that a blanket library default never had to pay. Every new downstream dependency, every new status code a partner API starts returning, is a decision someone has to make and encode, not something that gets absorbed silently by a generic retry-on-any-exception rule. I consider that a feature rather than a tax. A policy nobody can see is a policy nobody can review, and the sixty-request failure table taught me that the review is where the actual value sits, not in the retry mechanics themselves, which are the easy part once the classification is written down.

I'll name the ledger check's own limit plainly. It catches an effect the system already recorded on its own side. It has no visibility into the crash window between an external effect landing somewhere and the internal record of it getting written. If the booking succeeds at the airline's system and my own ledger write fails before the confirmation lands, the check has nothing to compare against, and the gap reopens exactly where the internal record stops being trustworthy. Closing that particular window is a different piece of engineering than this one, and I'd rather tell you where this gate stops than let you find the edge the hard way, in production, after the number on the dashboard already told me it was time to look somewhere else.