The Marker That Has to Survive Generation - Keeping Every Claim Attached to Its Source

The problem

I built a market-research brief assistant for a small analyst team that pulled from three sources into a single answer: a licensed analyst-report database whose license terms required attribution whenever any of it got used, an internal CRM full of client-specific notes, and public news articles. An analyst asked it to summarize the competitive landscape for mid-market ERP vendors and flag any client-specific signals worth knowing. The assistant pulled relevant passages from all three sources, which is exactly what it was supposed to do, and then concatenated them into one undifferentiated block of context with no delimiters and no instruction to preserve where anything came from.

The brief that came back read beautifully. That's what made this failure so easy to miss. The prose blended a licensed analyst's competitive claim, an internal CRM note about a specific client's stated concerns, and a public news item into sentences where no single claim could be traced back to where it actually originated. The licensed content's attribution requirement got violated by default, not through any decision anyone made, just because nothing in the pipeline had ever asked the model to keep the sources apart.

I measured this the same way I measure everything else here, against a ground-truth mapping of which claim came from which source across a 15-brief eval set. Citation attribution accuracy came in under 30 percent. Fewer than a third of the factual claims in a typical brief could be correctly traced back to their true origin, and the brief itself gave no visible sign that anything was missing. A confidently written paragraph that reads as authoritative is the hardest kind of failure to catch, because nothing about the output looks unfinished or wrong.

The pattern

The fix wraps each retrieved passage with a persistent, parseable source marker before concatenation happens, something like a bracketed tag naming the specific report or CRM record a passage came from. The generation prompt then requires the model to carry that marker into every factual sentence of its output, rather than letting the sources blend into one undifferentiated voice the way they had before. A post-generation parser extracts the markers automatically once the brief is written, so I can compute attribution accuracy without reading every brief by hand.

On the identical 15-brief eval set, that single change moved attribution accuracy from under 30 percent to over 90. I also track a second number alongside it, the percentage of factual sentences missing a marker entirely, which tracks toward near zero on the fixed version. Between those two numbers, I can tell whether attribution improved on average and whether any individual sentence slipped through unmarked, and that second check turned out to matter more in practice than the aggregate accuracy number by itself.

What makes this fix cheap relative to what it buys is that it doesn't touch retrieval or ranking at all. The same passages get retrieved. The only difference is that assembly stops erasing where they came from before generation ever sees them, and the model is explicitly told to keep carrying that information forward instead of treating source identity as something that disappears once text enters a prompt.

Design considerations

Here's the limitation that matters most, and I'd rather be direct about it than let the fix's own numbers overstate what it actually proves. This checks whether a syntactic marker survives generation intact. It says nothing about whether the passage behind that marker actually supports the claim attached to it. A model can carry a source marker faithfully into a sentence that misstates what the underlying passage actually says, and my own attribution-accuracy metric would score that as a full success, because the marker made it through, regardless of whether the sentence it's attached to is true to the source. That's a real gap, and it's the one I'd flag first to anyone building this expecting it to catch a misquote.

I've come to think of this as two separate questions that look like one question from the outside. Did the citation survive is a mechanical question about whether text-processing held up through generation. Does the citation actually support the claim is a semantic question about whether the underlying reasoning is faithful to its source, and answering that second question requires checking whether a cited passage genuinely entails the specific claim attributed to it, which is a materially deeper verification step than confirming a bracketed tag made it into the output string. I built this fix to answer the first question well, and I've stopped pretending it answers the second one at all. Anyone who needs the semantic guarantee has to add a separate verification layer on top of this, one that actually reads the source passage and the generated claim together and checks whether the first one supports the second.

The other place this fix needs calibration is deciding how granular the markers themselves should be. I've used per-document markers, tagging a whole analyst report or a whole CRM record, and I've used per-passage markers, tagging the specific chunk a claim drew from within a larger document. Per-document markers are easier to build and easier for a reader to act on, since "check the source report" is actionable advice. Per-passage markers give you a tighter trace when a document is long and only one paragraph in it actually supports a given claim, but they add real overhead to both the prompt and the parsing step, and I've found that overhead isn't always worth it when the underlying documents are short enough that "which document" already answers the practical question a reader has. The right granularity depends on how long your source documents actually are and how much a reader needs to know beyond which source to go check, not on some fixed rule about what counts as sufficiently precise attribution.