Two Hundred Tasks and a Rule That Quietly Stopped Holding

The problem

I built an inventory-reconciliation agent and gave it three rules in its system prompt. Never delete a record, only flag it. Always leave an audit note on anything you touch. If a SKU is missing, escalate rather than guess. Three sentences, all reasonable, all the kind of thing you'd write once and assume holds for the life of the system.

I ran it across two hundred sequential tasks, the kind of long unattended run a reconciliation job actually looks like in production: day after day of records to check, flag, or leave alone. Early in the run, the agent did exactly what the three sentences told it to do. I checked the first twenty tasks by hand and found nothing to complain about. Zero violations, clean audit notes, missing SKUs escalated instead of guessed at.

Nothing about that held. I tracked violation rate in twenty-task windows across the full run, and the curve is the whole story. Window one, next to nothing. By window ten, the agent was violating one of its three rules on more than one in five tasks, deleting a record outright instead of flagging it, skipping the audit note, guessing at a SKU it should have kicked upstream. Nobody changed the prompt. Nobody touched the rules. The agent's own adherence to instructions it had followed perfectly on day one simply eroded as the run went on, and nothing in the system was built to notice.

That's the part I keep coming back to. No line of code did the wrong thing here. Context accumulated instead, two hundred tasks' worth of prior actions and outputs piling up in the window the model was reasoning from, and three sentences written at the top of that window lost more and more of their relative weight as everything else crowded in around them. A rule stated once at the start of a long run is competing for attention against the entire history of the run by the time you're two hundred tasks deep, and it loses that fight gradually, not all at once, which is exactly why nobody catches it until the damage is already sitting in the data.

The pattern

The fix doesn't touch the underlying reconciliation logic at all. It changes what the three rules are and what checks them. Instead of three sentences living inside the prompt, competing with everything else for the model's attention, I pull them out into an explicit contract object: preconditions, invariants, and a governance check that runs on a schedule regardless of what the model happens to be attending to at that moment. The rules stop being prose the model has to keep remembering and become a structure something outside the model actively verifies.

The governance check runs every twenty tasks. It looks at the actions taken in that window and computes a violation rate against the same three invariants: was anything deleted instead of flagged, is there an audit note on every touched record, did a missing SKU get escalated or guessed at. When that rate crosses a threshold, a recovery path fires. Two things happen together. The contract gets re-injected into the agent's context, fresh, at full weight, the way it was on task one. And the specific offending action gets rolled back, the deleted record restored to flagged status, the missing audit note added, the guessed SKU reopened for escalation.

Running the identical two hundred tasks through this version doesn't eliminate drift. I want to be precise about that, because it would be easy to claim more than the mechanism actually delivers. The violation rate still rises inside a given twenty-task window, the same underlying pressure toward drift is still there. What changes is what happens next. Instead of climbing past twenty percent by window ten and continuing upward with nothing to arrest it, the rate gets pulled back down at every checkpoint, and it converges to a ceiling under five percent instead of compounding indefinitely. The rules never stop needing reinforcement. What changed is that reinforcement now happens on a schedule instead of never.

Design considerations

The choice to aim for a bounded ceiling instead of zero violations is a real design decision, and it's a different decision than the one I'd make for a single high-stakes action. If I were gating one irreversible transaction, a refund over a dollar limit, a wire transfer, I'd want a mechanical check that blocks the specific bad action outright, every time, with no tolerance for even one slip. That's a different failure shape: one action, one line that must never get crossed. A long-running sequence is a different problem. The realistic goal is a system that drifts, notices, and corrects before the drift compounds into something expensive. Confusing these two problems gets you either an over-engineered gate on something that just needed periodic reinforcement, or a governance check applied to a single action that needed a hard stop instead.

The interval you choose for the governance check is doing more work than it looks like. Twenty tasks was calibrated to this specific scenario, where a full run was two hundred tasks and a violation rate needed room to show a trend before the check fired on it. Check too often and you're re-injecting the contract so frequently that you've basically rebuilt a prompt the model reads in full every single time, which erases most of the reason you split the contract out in the first place. Check too rarely and the violation rate has room to climb well past whatever ceiling you were hoping to hold before the next checkpoint even arrives. There's no universal number here. It has to come from how fast drift actually accumulates in the specific system you're running, which means you need at least one full run of data before you can set the interval with any confidence, not a number picked in advance because it felt reasonable.

The rollback half of the recovery path is the piece I'd defend hardest if someone wanted to simplify this down to just re-injecting the contract. Re-injection alone fixes the agent's future behavior but leaves the damage from the violation sitting in the data, a deleted record that's still deleted, a missing audit note that's still missing. Rolling back the specific offending action is what makes the correction reach the thing that actually went wrong. Skip that half and you've built a system that gets better going forward while quietly leaving a trail of uncorrected violations behind it, which is a strange kind of progress to settle for.

What this doesn't do is catch a violation the moment it happens. There's a real gap between when a rule gets broken and when the next governance check notices, and inside that gap the violation is live and uncaught. If the cost of even one violation slipping through for up to twenty tasks is unacceptable, this is the wrong mechanism, and you want a check sitting at the point of action instead, one that inspects every single action before it executes rather than sampling the aggregate every so often. This pattern is for the failure shape where the real risk is a slow accumulation nobody's watching.