The Self-Graded Node - Finding a Bad Step Through Arithmetic
I ran a four-step marketing-copy pipeline against twenty synthetic campaign briefs: a brief-intake agent, an outline agent, a draft agent, and a compliance-check-and-polish agent, graded only at the end against a four-part rubric covering accuracy, tone, compliance, and brand fit. I planted a known defect into a controlled subset of those briefs on purpose. The outline agent introduces a wrong claim structure on six of them. The draft agent adds an unapproved claim on another six. The polish step fails to catch a planted issue on another four.
Grading only the final output meant a genuinely low-compliance draft and a genuinely fine draft that the polish step had quietly corrupted looked identical from outside the pipeline. My own accounting on this version was stark: node attribution accuracy, meaning the share of low-scoring runs where the system correctly names the true injected-fault node, came out to zero percent. Not because the mechanism was bad at guessing. Because no per-node scores existed anywhere to compare against. There was nothing to attribute with.
The problem
A single score at the end of a multi-step pipeline collapses four separate decisions into one number. When that number is low, I know something went wrong somewhere in the four steps. I don't know which step, and a low compliance score by itself carries no information about whether the fault originated in the outline's claim structure, the draft's added language, or the polish step's failure to catch either one. Every run that scores badly looks exactly the same from the outside: one number, no breakdown.
This is the ordinary shape of a black-box pipeline, and it's a common one. Teams build a solid eval harness, grade the final artifact against a real rubric, and treat that as sufficient observability. It answers "is the output good," which is a real and useful question. It doesn't answer "which of the four steps that produced this output is the one to fix," and those are different questions with different owners, especially once a team has more than one person responsible for different parts of a pipeline and needs to know whose step to send back for revision.
The pattern
The fix doesn't require a separate judge model or an expensive new evaluation pass. It requires making every node grade itself against the same rubric taxonomy the final layer already uses, before handing off to the next step. The outline agent scores its own outline against "does this claim structure introduce a compliance risk." The draft agent scores its draft the same way. The polish step does too. Every node emits a typed micro-rubric score, on the identical four-dimension scale, alongside its output.
Attribution then becomes arithmetic. Compare each node's self-score against the final score, and flag whichever node carries the largest negative delta as the primary cause. Nothing here requires a model to reason about causation directly. It requires comparing numbers that already exist, because I built the pipeline to produce them at every edge instead of only at the exit.
On the identical twenty runs with the identical injected faults, this got node attribution accuracy to eighty percent or better, against the zero percent baseline where no per-node scores existed at all. That's the number I built the mechanism to hit. It didn't require touching the final eval layer's rubric at all, only sharing its taxonomy with every node upstream of it, so the self-scores and the final score are directly comparable rather than measuring different things under similar-sounding labels.
Design considerations
The mechanism has a real blind spot, and it's not a minor one. Arithmetic over self-scores assumes the causally relevant defect shows up as a score delta at the moment of handoff. It has no purchase on a defect that scores fine at every immediate handoff and only produces a bad outcome much later, once its downstream consequences accumulate. A node can score itself honestly, by its own rubric, at the exact moment it hands off, and still be the true cause of a failure that only becomes visible three steps and several days later. Per-node scoring catches immediate quality problems. It structurally cannot catch delayed ones, because a delayed failure is, by definition, invisible at the moment the score gets computed.
The rubric taxonomy also has to be genuinely shared, not merely similar. If the outline agent's self-score and the final eval layer's score use slightly different definitions of "compliance," even if both are labeled the same way, the delta between them stops meaning anything. I treat the taxonomy as shared code, not independently maintained copies that happen to look alike, because rubric drift between a node's self-assessment and the final judge is exactly the kind of subtle bug that silently degrades attribution accuracy without ever throwing an error.
There's a cost tradeoff too, though a modest one. Every node now does an extra piece of work: score its own output against a rubric before passing it along. For a four-step pipeline that's a small addition. For a much longer chain, the added latency and inference cost per node starts to matter, and I'd want to think about whether every node needs a full rubric pass or whether a cheaper heuristic screen at most nodes, with full scoring reserved for the ones most likely to introduce a compliance-relevant defect, gets most of the benefit at a fraction of the cost. I haven't needed to make that tradeoff yet on anything longer than four steps, but I'd expect the answer to depend heavily on how expensive the rubric evaluation itself is, not on the attribution logic sitting on top of it.
The last thing worth being honest about is what this pattern is not. It's not a replacement for the causal reasoning a person eventually has to do once a node is flagged. It narrows a four-step black box down to one likely culprit. It doesn't explain why that node produced a bad output, only that its self-assessment diverged furthest from the final judgment. That narrowing is genuinely useful. It's also the limit of what arithmetic over existing scores can give me.