The Obligation Nobody Wired In - Tagging Compliance at the Source Boundary

The problem

I built a legal-research assistant for a small healthcare provider that retrieved from three sources, each carrying a genuinely different compliance obligation. A case-law citation database required a citation whenever any of it got used. An internal incident-report system was subject to a synthetic HIPAA-style rule, any use had to be logged and the content couldn't persist past a defined retention window. A payer contract repository required exact clause citation, no paraphrasing allowed. A paralegal asked whether the practice had seen similar incidents involving delayed lab notifications and what case law applied to them.

Every one of those obligations existed, documented clearly, in a static reference file. Nobody consulted that file at runtime. That's how this failure always starts. A document describes a rule. A pipeline has no mechanism for reading it. No malice required, no carelessness either, just a gap nobody built anything to close. Case law got cited or not depending on what the model happened to do that run. Incident-report usage generated no audit-log entry, ever. I deliberately wired the audit-log endpoint to return empty every time, so the missing trail would be visible instead of something I had to take on faith.

I measured how often the assistant's actual responses satisfied every obligation attached to every source it drew on, cited, logged, quoted exactly, across a 20-question eval set spanning all three sources. The obligation-compliance rate came in under 40 percent. Less than half the time, the response actually did what its own source systems required of it, and nothing in the output looked wrong to a reader who didn't already know the rule existed.

The pattern

The fix moves the obligation mapping out of a document nobody reads and into metadata attached at the moment of ingestion, because ingestion is the only place that mapping was ever going to survive contact with an actual runtime pipeline. Each chunk gets an obligation type tag, something like citation required, audit log required, or exact clause required, sourced once from the documented mapping. Nothing re-derives that tag by guesswork when a query runs. That tag then travels with the chunk through retrieval and into assembly the same way any other piece of metadata does.

The part that actually closes the gap is a post-generation enforcement step that checks the response against every used chunk's tag before it goes out. A citation gets force-included when the tag requires one. An audit-log entry gets written automatically the moment incident-report content is used, with no reliance on anyone remembering to log it. Payer-contract clauses get quoted verbatim instead of paraphrased, because the tag says exact clause required and the enforcement step actually reads that field. On the identical 20-question eval set, the obligation-compliance rate rose from under 40 percent to over 90, and the audit-log endpoint stopped being permanently empty and became a real, growing count of every time incident-report content actually got used.

What I like about this fix is that nobody has to trust a downstream step's judgment about which obligation applies. The obligation was decided once, at the boundary, sourced from a documented mapping a human actually reviewed. Enforcement downstream reads that decision and never re-derives it, so a model having a bad day and forgetting to cite something turns into a simple, testable fact: a tag either got read correctly or it didn't. That's a much easier thing to catch than trying to audit a model's judgment call after the fact.

Design considerations

The limitation that matters most here is granularity. This fix assumes a chunk's obligation maps cleanly onto the source system it came from, and that assumption holds most of the time because most source systems really do have one dominant obligation. It breaks down for a document that mixes obligations at a finer grain than the chunk boundary happens to fall, part of a single document requiring citation while another part of that same document requires something stricter, like the exact-clause rule. My current schema has no answer for that case. I treat it as a real edge case: the source systems I've built this against tend to carry one obligation type consistently enough that the chunk-level tag captures the right answer almost all the time. It's a working assumption I've tested against, and I'd check it carefully against any new source system before staking a compliance program on it.

The other calibration question is how wide the obligation schema itself should be. I started with a narrow field, just the obligation type and a retention window where relevant. Current data-governance guidance, particularly around transparency obligations tied to regulatory deadlines in 2026, specifies a considerably wider field set than that: source system, legal basis, consent status, retention window, and lineage to the specific model that consumed the record. I think the schema should grow to match that wider set, and I've started widening mine accordingly. The mechanism underneath stays the same either way. Tag the obligation once, at the boundary, from a mapping someone actually reviewed, and enforce it downstream by reading the tag instead of guessing at runtime, whether that tag carries one field or five. The number of fields is a scope decision. The discipline of tagging once and never re-deriving is the part that actually does the work, and it's the part I wouldn't compromise on even as the schema itself keeps growing.

There's a version of this that's tempting to skip if a compliance obligation feels obvious enough that surely the model will just handle it. I don't build that version anymore. Leaving an obligation untagged means it depends entirely on a model behaving correctly on every single run, with nothing checking whether it did. That was precisely the setup that produced a 40 percent compliance rate in the first place.