The Write-Back - Turning a Correction Into a Rule Change

I built a customer-support ticket-routing system on a skill registry, a set of rules mapping ticket-type signals to specialized sub-agents, and ran it over a simulated thirty-day window. Human agents corrected twenty-five misrouted tickets during that window. One specific root-cause misroute, a rule matching "sync failed" to the billing skill instead of the integration skill, recurred fourteen more times in the same window after being flagged the first time.

Every correction got appended to a log with no automated consumer. The correction became a data point for a report nobody read in time. The routing system itself never acted on it. The same misroute kept firing until someone eventually noticed the pattern in a dashboard and edited the rule by hand, days after the first correction had already told the system exactly what was wrong.

The problem

A human correction carries a real, specific, immediately actionable fact: this input, this rule, this outcome, was wrong. Most systems I've seen file that fact away as reporting material. Very few feed it back in as an instruction. The correction gets logged, timestamped, maybe surfaced in a weekly review, and in the meantime the exact same failure keeps recurring because nothing in the live system actually changed. Fourteen recurrences of the same misroute after the first correction means the system had already been told the answer and kept acting as if it hadn't heard it.

The data was there. The corrections existed, were logged accurately, and were even visible to a human who eventually acted on them. What never happened was an automatic close of the loop: the distance between someone noticing the mistake and the system no longer making it ran to days or weeks of manual review cadence, when the fix needed only the next request cycle.

The pattern

I treat a correction as a trigger for immediate action. When a correction comes in, an automated write-back step identifies the specific rule or registry entry responsible for the bad outcome and writes an attribution judgment directly into that substrate: downgrading its confidence for that signal pattern, or flagging it for review. That update happens within the same request that carried the correction, before anyone ever opens a dashboard to look for it.

Concretely, a POST /correct call locates the responsible routing-rule entry in the registry the router reads from and updates its weight immediately, so the very next ticket that would have hit the same misroute gets routed differently. On the identical thirty-day window with the identical recurring misroute, this got the repeat-error rate under three total recurrences, against fourteen in the passive version. The loop closes inside the same request cycle instead of waiting for a person to catch the pattern.

The mechanism here reuses whatever attribution judgment the rest of the system has already produced, whether that's a trace reconstruction, a per-node score, a replay result, or a suppression test. The write-back step works from that already-produced judgment: it takes what already exists and makes sure it actually reaches the substrate that needs to change, instead of stopping at a log entry someone might eventually read.

Design considerations

This pattern is only as good as the judgment feeding it. That is the central risk of automating it at all. If an earlier step misidentifies the responsible rule, whether from a bad trace reconstruction, a noisy per-node score, or an unverified attribution, this pattern writes that wrong judgment directly into a live, load-bearing substrate with full automated confidence and no human in the loop to catch it before it takes effect. Automating the write-back automates the mistake as readily as the fix, the moment the upstream attribution is wrong. A system that closes this loop fast is also a system that propagates a wrong diagnosis fast, and speed cuts both ways here.

That's exactly why I don't trust an attribution judgment into this pattern without first checking it against something like an evidence-adequacy gate: does the evidence behind this judgment actually change if the true cause were different, is it tied to the thing it claims to measure or just a proxy for it, and does it reproduce under identical replay. Skipping that check to get the loop closing faster trades a slower, more careful fix for a faster, less trustworthy one, and I don't think that trade is worth it once the substrate being written to is something downstream systems will keep reading from.

There's also a question of blast radius that I think about every time I build one of these. A write-back that targets the specific rule responsible for one recurring misroute is a narrow, contained fix. A write-back mechanism that's tempted to reset or retrain something broader every time a correction comes in is a much riskier proposition, because a single noisy or unusual correction could then ripple into a change with effects well beyond the one misroute it was meant to fix. I keep the scope of every write-back as narrow as the attribution allows, targeting the specific entry the evidence points to and nothing wider, precisely because the automation is what turns a mistake here into something expensive instead of merely inconvenient.

The last thing worth flagging is the boundary of what this handles. Automated write-back takes care of the recurring, well-attributed case efficiently, and stops there. Someone still has to review the registry periodically for drift, contradictory rules, or rules that have quietly accumulated so many small downgrades that the routing logic no longer resembles anything anyone designed on purpose. Automatic correction runs fast and narrow; periodic human review runs slow and wide, and they're answering different questions. I've never found a way to get the second one from more of the first.