The Ledger That Ends the Keep-or-Cut Argument

The problem

I built KBAssist to answer policy and pricing questions out of an internal knowledge base, and shipped it with a fact-level validity gate, because retrieval was measured citing expired facts, superseded pricing tiers, lapsed policy terms, at a meaningful rate. The gate adds a validity-metadata lookup on every retrieved chunk before it can be cited, which costs latency and requires an engineer to keep expiry metadata current, an unglamorous ongoing tax nobody enjoys paying. Over several quarters, the underlying corpus's own auto-expiry tagging quietly got better, shrinking the residual risk the gate still caught. Nobody was tracking cost against risk anywhere, so each quarter's keep-or-cut call got made on whatever anecdote someone happened to remember.

I played that anecdote-driven process out across six simulated quarters with distinct, seeded cost and residual-risk figures that drifted, some quarters net-positive for keeping the gate, some net-negative. The team in this scenario kept the gate in one quarter because of a bad incident, cut it in a later quarter because "it hasn't caught anything lately," and reinstated it after a new hire wandered in and re-raised the original launch story. None of those decisions got computed from anything at all. I measured a decision-correctness rate, against what a properly computed net-value ledger would have recommended in hindsight, under 50%. On that evidence, deciding by vibes was barely better than flipping a coin, and at least the coin is honest about it.

The pattern

The version that actually resolves the argument persists every quarter's engineering and latency cost, and every quarter's measured residual-risk catch rate, to a standing ledger, and computes a documented net-value formula every quarter: the estimated cost of a stale-fact incident times the risk rate the gate is actually catching, minus the gate's own overhead cost. The keep-or-cut call gets read straight off the ledger's sign, not argued from whoever remembers the loudest incident. Because the decision follows the ledger by construction, the decision-correctness rate comes out at 100%. That number comes from arithmetic. Once a decision follows a persisted figure by construction, there's no judgment left in the loop for anyone to get wrong.

An investment portfolio runs the identical discipline, and it's the analogy that clicked hardest for me on this one. Nobody decides whether to keep hedging a position based on which quarter's headline scared them most recently. A hedge earns its cost or it doesn't, measured against a standing ledger of what it cost to carry and what it would have cost to go unhedged when the bad quarter actually arrived. This pattern is that same ledger, aimed at an engineering control instead of a financial one: write down the cost, write down the catch rate, let the sign of the number make the call instead of whoever argued last.

Design considerations

A ledger is only as trustworthy as the two numbers feeding it, and both of those numbers are estimates dressed up as facts. The cost side is usually the easier one to pin down: latency added, engineering hours spent maintaining expiry metadata. The residual-risk side is much softer, because it depends on estimating what a stale-fact incident would actually cost if it happened, a number that stays a guess dressed in a spreadsheet's clothing until an actual incident occurs and hands you a real data point. I'd treat that incident-cost estimate as the single input worth the most scrutiny before trusting the ledger's sign, because a wrong estimate there flips the entire recommendation while looking exactly as authoritative as a right one.

The ledger also needs a real cadence for updating its inputs, not just its outputs. A catch rate measured once and reused every quarter thereafter defeats the purpose as completely as no ledger at all, just with better production values. I'd want the residual-risk figure re-measured each quarter against the corpus as it actually stands then, not carried forward from whenever it was last checked, because the whole reason this pattern exists is that the corpus and the risk it carries move over time.

The place this doesn't fit cleanly is a control where the cost of being wrong isn't well approximated by an expected-value calculation at all: catastrophic tail risk, where the actual cost of one bad incident dwarfs any number a quarterly average would produce. A net-value ledger is built for costs and risks that behave like a rate you can price across many occurrences. For a control protecting against a single unacceptable outcome rather than a recurring measurable rate, I'd keep the mitigation running regardless of what a rolling average says, because a single unaffordable event breaks the entire logic the ledger depends on.