The Untagged Aside - Why Source Identity Has to Be Captured at Ingestion

The problem

I built a support copilot once, on purpose, so I could watch it fail. It pulled from three sources indexed into the same Chroma collection: CRM account notes, a ticketing system export, and a compliance handbook, the only one of the three carrying binding policy language. A rep asked it a plain question: can we waive the $150 early termination fee for this account. A note on that account read, in the rep's own words, "customer very upset, I told them we'd probably waive it." That's a rep venting into a text box. Nobody with the authority to approve a waiver ever signed off on anything. The handbook, elsewhere in the same index, said fee waivers above $100 require VP sign-off.

The copilot answered as if the CRM note had settled the matter. It hadn't decided that the rep's comment outranked policy. It had no way to know the two passages came from different systems with different authority at all. Both had been chunked, embedded, and stored as bare text, with nothing recorded anywhere about which system either one came from. By the time a query pulled the top-scoring chunks, the rep's aside and the handbook's rule looked like two interchangeable pieces of text, and the one that happened to phrase itself closer to the question won.

I measured this properly instead of eyeballing it. Against a 30-question labeled set, where I knew in advance which source should have governed each answer, the untagged pipeline misattributed authority on 40 percent of questions or worse. That's not a rare edge case. That's closer to a coin flip on questions where getting it wrong means telling a rep they can waive a fee they can't waive, or worse, confirming to a customer that something got approved when it never did.

What makes this failure mode uncomfortable is that nothing about it looks broken from outside. No exception fires. No error surfaces in a log. The system answers, confidently, from whichever chunk scored highest that day, and the only way anyone finds out something went wrong is when the wrong answer lands on someone's desk and causes a problem downstream.

The pattern

The fix is narrower than it sounds, and I've come to think that's exactly why it works. At the moment a document gets chunked and embedded, before any of it touches the vector store, the ingestion step writes a source_system field into that chunk's metadata. A business_unit field goes in too, where it's relevant, so a handbook chunk can carry "Finance" or "Compliance" alongside "handbook." Nothing about ranking changes. Nothing about permission changes. The tag simply exists now, riding along on the vector store's native metadata support through every retrieval call that touches that chunk from then on.

The reason this has to happen at ingestion, and can't be patched in later, is that the fact itself doesn't survive anywhere else. Once a chunk of text sits in a vector store as a string, there's no property of the string that tells you which system wrote it. A CRM note and a handbook clause can use nearly identical vocabulary. You can't infer source from content after the fact, because the content doesn't encode it. The only moment that fact is knowable is the moment the pipeline is looking directly at the document coming out of the CRM export, or the ticketing system, or the handbook file, and that moment ends the instant the text gets stripped down to a plain string for embedding.

Once the tag exists, the assembly step that builds the final prompt can finally do something it couldn't do before: label each retrieved passage by where it came from before handing anything to the model. On the same 30-question eval set, tagging the source dropped my misattribution rate from over 40 percent to under 10. The tag itself didn't rank anything or block anything. It just made source identity visible again at every step downstream that needed it, instead of losing it the moment ingestion finished.

I ended up thinking of this as the one fact every later fix in a retrieval pipeline depends on. You can't rank by authority tier if nothing tells you which tier a chunk belongs to. You can't enforce a permission scope if nothing tells you which system's permission table applies. You can't require a citation format if nothing tells you which source's licensing terms govern this passage. All of that machinery reads the same field this fix writes. Get the tag wrong, or skip it, and everything built on top of it is reasoning about the wrong thing without knowing it.

Design considerations

I want to be honest about how narrow this fix actually is, because it's tempting to treat "we tag our chunks now" as if the problem is solved. It isn't. Tagging a chunk's source is one thing. Getting anything downstream to actually read that tag and act on it is a completely separate piece of engineering. I can hit 100 percent tag completeness, every single chunk carrying a non-null source_system field, and still misattribute authority if the ranking step, the permission check, or the generation prompt never looks at the field that's sitting right there. The tag is necessary. It was never going to be sufficient on its own, and I put it first in any retrieval build precisely because everything after it depends on it existing, not because it finishes the job by itself.

There's a second calibration question that took me longer to see clearly: not every fact about a chunk behaves the same way once it's tagged. Source identity is genuinely static. A CRM note stays a CRM note forever, and a handbook chunk stays a handbook chunk. Tag it once and you're done, permanently, for that fact. Permission is a different animal entirely, because a permission grant is true until it isn't, and someone's access can be revoked after a chunk was tagged and before a query ever uses that tag. If you try to apply this same "tag once at ingestion, never touch it again" discipline to permission the way you do to source identity, you'll build something that's correct on the day it's tagged and silently wrong the day access changes. I learned to keep those two kinds of facts mentally separate: source identity earns the tag-once treatment because it never changes; permission earns a live recheck against whatever system actually owns the grant, because it does.

The place this pattern genuinely doesn't help is anywhere the source itself is ambiguous or mixed. A single document that blends content from two systems, a handbook appendix that quotes an email verbatim, for instance, doesn't have one clean answer for what source_system should be. I've had to accept that this fix assumes source identity is a property of the record, cleanly resolvable at the moment of ingestion, and it has nothing useful to say about the harder case where that assumption doesn't hold. In practice that's been rare enough not to worry about it as a general rule, but it's the honest edge, and I'd rather name it than pretend the tag resolves every case cleanly.

What surprised me, coming back to this after building it, is how much of the enterprise data-engineering literature arrives at the identical static-versus-dynamic line from a completely different direction. Metadata practitioners describe ingestion-time metadata, document ID, source, authorship, ownership, as static and factual, and retrieval-time metadata, similarity score, rank position, per-user access check, as inherently dynamic. That's the same distinction I built this fix around, reached from a data-engineering angle instead of the governance angle I came at it from. Two disciplines landing on the same seam independently told me I hadn't just found a convenient simplification. I'd found where the actual fault line in the problem sits.