The Time-of-Decision Log - Scoring Judgment Before the Outcome Arrives
The problem
I built an inventory-reorder agent that recommended reorder quantities across eighty retail SKUs, tracked over a twelve-week trace. In week eight, I built in a demand shock: an unforeseeable disruption that hit ten of those SKUs hard. The reorder decisions made on those ten SKUs back in weeks one through seven had been entirely reasonable given everything knowable at the time. By week twelve, they looked terrible, because the SKUs they touched ended up stocked out or badly overstocked.
The only review process running against this system was a postmortem that ranked every decision purely by outcome, worst results at the top of the list, no exceptions. The ten shock-affected SKUs landed right at the top of the "worst decisions" ranking. An event nobody could have predicted happened to land on exactly those ten and nowhere else; nobody had reasoned poorly about any of them. Run a correlation between that outcome-based ranking and an independent measure of how sound each decision actually was given the evidence available at the time, and the two barely relate to each other. The worst-outcomes list and the worst-decisions list turned out to be almost entirely different lists.
This is the failure in its cleanest form. A decision gets judged by how it happened to turn out, when the real question was always whether it was reasonable given what was knowable at the time it was made. It's an old problem in ethics, long before anyone was training a model to make reorder recommendations. Two people run the same red light under identical circumstances. One gets home safely. One hits a pedestrian. The choice was identical. The consequences, and the judgment that follows from them, are not. Philosophers call this moral luck (Nagel, 1979; Williams, 1979), and it shows up in AI liability review with exactly the same shape it has everywhere else: an outcome gets treated as evidence of decision quality, when the two are only loosely connected once anything genuinely unpredictable enters the picture.
The pattern
The fix writes a structured snapshot at the moment of every single recommendation, before any outcome is known: what demand signals were available, what confidence the forecast carried, what reasoning led to the specific quantity recommended. That snapshot gets written and locked in regardless of what happens next. Nobody goes back later and edits it to look better or worse depending on how the SKU actually performed, because it's captured at decision time and never touched again.
The postmortem process gets rebuilt around that log instead of around the eventual outcome. Instead of asking which SKUs stocked out, it asks whether, given what was knowable when this decision was made, it was a sound call. Scored that way, the postmortem correctly classifies the ten shock-affected SKUs as well-reasoned decisions that ran into bad luck. The stockouts and overstock are still real numbers, still tracked, and still matter operationally. They just stop being treated as a verdict on the quality of the thinking that produced them.
I want to draw the boundary here precisely, because it would be easy to claim more than this pattern actually delivers. This anchors liability and performance review to decision quality. It doesn't eliminate moral luck, because outcomes will keep diverging from decision quality by pure chance no matter how good the logging gets. A well-reasoned decision can still turn out badly for reasons no amount of decision-time evidence would ever have surfaced, and that will keep happening after this pattern is in place exactly as much as before. What changes is that the review process stops mistaking the divergence for evidence of bad judgment.
Design considerations
The hardest part of building this well is resisting the temptation to let the decision-time snapshot get contaminated by hindsight. If the people writing the postmortem know the outcome before they evaluate the decision-time log, they'll unconsciously read the log more critically for decisions that turned out badly and more generously for decisions that turned out well, even when the log itself is identical in quality. The cleanest version of this pattern separates the two roles entirely: whoever scores decision quality against the log shouldn't see the outcome data until after that scoring is locked in, the same way a study reviewer works blind to results before assessing methodology.
This pattern presupposes something that isn't always true, and I want to name where it breaks. Scoring a decision against a time-of-decision log only works if you can actually identify whose decision it was and what evidence they had. In a system where a human and a model are genuinely co-producing a recommendation across several turns, adjusting to each other as they go, the clean separation between the decision and the outcome starts to blur, because it's no longer obvious where the decision ends and the next round of human-AI back-and-forth begins. This pattern handles the case where a single recommendation gets made at a single point in time. It doesn't fully handle the messier case where responsibility is smeared across an ongoing, adaptive interaction, and I don't think logging alone solves that harder version.
The other thing to name is what this pattern doesn't protect against, which is a foreseeability boundary that shifts over time. A category of harm that looked genuinely unforeseeable last year can become foreseeable this year, once enough similar incidents accumulate that a reasonable operator should have started watching for them. A static decision-quality log captures what was knowable at one moment. It doesn't automatically update its own standard for what should have been knowable as the world around it changes, and a postmortem process that treats the original log as the permanent word on foreseeability risks freezing a standard that should have moved.
There's real evidence that the bias this pattern exists to counter isn't hypothetical. Research on moral luck has found that people systematically judge identical decisions as less moral when the outcome turns out badly, even when that outcome sat entirely outside the decision-maker's control, and will retroactively relabel the same action, an investment becomes a gamble, the moment it goes wrong (Max & Uhl, PLOS ONE, 2023). This shows up across more than one study, and it's a documented, repeatable pattern in how humans assign blame. A review process that doesn't actively correct for it will drift toward exactly this bias by default.
The stakes here have moved from law-review argument to live litigation. In Mobley v. Workday, litigated in the Northern District of California, rulings through 2026 have allowed discrimination claims against an AI hiring vendor to proceed on an agent-of-the-employer theory, rejecting the argument that liability stays confined to the software vendor alone. That's a court doing, in miniature, exactly what a time-of-decision log is built to support: attaching responsibility to whoever actually held knowledge and control at the moment a decision was made, rather than to whoever sits easiest to blame along a vendor-to-deployer chain.
Bad outcomes will keep happening regardless of how sound the reasoning behind them was, and no amount of logging changes that. The record this pattern produces lets someone later tell a decision that failed because the thinking was bad apart from a decision that failed because the world did something nobody could have seen coming. An outcome-only postmortem never had the information to draw that line at all.