What the Agent Saw When It Decided

I built a screening agent for a recruiting pipeline: fifty candidates, a fixed rubric, and a pass, hold, or reject decision the agent made after reading each candidate's interview notes. The notes weren't static. Interviewers kept submitting feedback for days after the first read, so a given candidate's note record got edited two, three, sometimes four times over the course of a week as more people weighed in.

The problem

The agent made its call the moment it had enough signal to score against the rubric. Pass, hold, or reject, timestamped, logged. What I hadn't built was any record of what the notes actually said at the exact moment the agent read them. I was only storing the current state of the notes table, the same table the interviewers kept editing. The decision lived in one place. The evidence for that decision kept changing underneath it, in the same row, forever.

Three weeks in, a candidate disputed a reject decision. I went back to check the reasoning the way I'd check any logged decision: pull the notes, run them through the same scoring logic, confirm the call. I got a different answer. Cleaner, more confident, and wrong for my purposes, because it wasn't the answer the agent had actually reached three weeks earlier. In the time since, a second interviewer had added detail that resolved an ambiguity the original notes hadn't. The notes had genuinely gotten better. The agent's decision hadn't been made against the better version. It had been made against whatever was there at the time, and whatever was there at the time no longer existed anywhere I could point to.

I went back and replayed every one of the fifty decisions the same way: notes as they stood today, run through the stored decision logic, checked against what the agent had actually called at the time. The replay didn't reproduce the original decision consistently, and the mismatch rate sat well above zero. Every edit, correction, or consolidated round of feedback to a candidate's record was a small chance that a decision made weeks earlier no longer matched the evidence it was supposedly based on. Nobody had tampered with anything. The record was rewriting itself as a byproduct of interviewers doing their jobs, and I only found out because someone pushed back hard enough to make me look.

The pattern

Freezing the notes would break the actual workflow interviewers depend on, so that never made sense as a fix. What I needed instead was to separate two questions that only match at the instant a decision happens: what memory contains right now, and what memory contained when the agent decided. A second later they start to diverge, and nothing in the architecture flags that divergence unless something is built to catch it.

So before the agent scores anything, I write an immutable copy of exactly the notes it retrieved into a separate table, keyed to that specific decision, timestamped, and never touched again after the insert. The live notes table keeps being edited, merged, and corrected exactly as before, because that's normal and desirable for the ongoing hiring process. The snapshot sits outside that process entirely: a record of one moment, filed away the instant the moment happens, before anyone has a chance to improve on it.

Replay now reads from the snapshot instead of the live table. Ask why the agent rejected a candidate and the system pulls up the exact notes the agent actually saw at decision time, independent of whatever the notes have since become. I added an integrity check on top of that, confirming no snapshot row had ever been modified after insert, because the whole value of the record depends on nobody, including me, being able to touch it once it's written.

The mechanism generalizes past hiring. Anywhere a decision gets made against a fact store that keeps evolving after the fact, the same gap opens up: a loan approval against a credit file that gets corrected later, a triage call against a patient chart that gets updated with new lab results, a fraud flag against transaction history that gets reclassified once more information comes in. In every one of these, what the record says now quietly substitutes for what the record said when the call was made, and the substitution stays invisible until somebody with a stake in the outcome asks the second question out loud.

Design considerations

Snapshotting is forensic work, and I want to be precise about what that actually buys, because it's less than it sounds like. The decision quality, the rubric, whether the notes were accurate when they were written: none of that changes. All it guarantees is that the record of what was seen can't be quietly rewritten afterward, a narrower claim than "auditable decisions," and I'd rather undersell it than overstate what it covers.

The real cost is storage, and that cost only grows. I never prune a snapshot in this design, which means the table grows with every decision the agent makes, forever, by construction. An append-only forensic record and a memory store that's supposed to stay bounded pull in opposite directions, and I haven't found a way to split that difference cleanly. For now, I accept the growth for decisions that carry real consequences, the kind a person can dispute, and skip it for decisions that don't carry that kind of stake.

There's a judgment call about granularity I underestimated the first time through. Snapshot too coarsely, the whole conversation history at decision time, and a lot of irrelevant material rides along into permanent storage next to the two sentences that actually mattered. Snapshot too narrowly, just the fields the rubric explicitly scores, and the context that explains why an edge case landed where it did goes missing. I ended up snapshotting exactly what the retrieval step handed to the scoring step, no more and no less, because that's the actual boundary of what the agent saw. Drawing the line anywhere else means guessing at what a future version of me will need to defend a decision that hasn't happened yet.

The last point is a calibration one. This pattern only earns its cost where a decision might need defending later. Not every agent decision does. A suggestion that gets immediately accepted or discarded and never revisited can skip the permanent, append-only trail entirely, and snapshotting everything by default is how the storage problem above gets worse faster than it has to. Snapshot where a decision could plausibly be disputed, reversed, or audited by someone who wasn't in the room when it was made, and let the rest of the system stay lighter.