The Schema You Add After the Test Fails

The problem

I built a meeting-notes tool that extracts action items from transcripts, and the first version I tried jumped straight to a fully structured output. A strict JSON schema, owner as a typed enum that had to match an exact employee-ID lookup, due_date required as ISO 8601 even when the transcript only ever said "by next week," priority as an enum of exactly three values even though half the meetings never stated a priority at all. It felt like the responsible way to build it. A structured extraction feeding a downstream system should be structured, right up until you test it against how people actually talk in meetings.

I ran it against twenty seeded transcripts of varying clarity, some crisp, some the usual meandering mess of a real conversation. The rigid schema produced exactly the failure a rigid schema produces on ambiguous input. When the transcript said "by next week" instead of a real date, the model didn't leave the field blank or flag it as uncertain. It fabricated an ISO 8601 date, invented a specific day that satisfied the schema's requirement without any basis in what was actually said. When someone referred to an owner by a nickname instead of their exact employee ID, the model either guessed at a match that might be wrong or produced malformed output trying to force a fit that didn't exist. Both outcomes are worse than a looser, plainer answer would have been, because a fabricated value looks exactly as confident as a correct one, and a downstream system consuming the extraction has no way to tell the difference without checking every single field by hand.

The deeper problem is that all that constraint complexity got front-loaded before anyone had a shred of evidence it was needed. Three enum types, a strict date format, an exact-match lookup, all specified up front because it seemed like the thorough thing to do. Thoroughness aimed at guesses about failure modes that hadn't actually been observed yet is a strange kind of effort. It's engineering time spent hardening against a shape of failure nobody had confirmed would happen, while the schema simultaneously created new failure shapes, fabrication and malformed output, that wouldn't have existed under a looser spec.

The pattern

The fix inverts the order everything got built in. Start at the simplest plausible instruction and only add structure to the specific places testing shows it's actually needed. Version one of the extractor was plain natural language: list action items as short bullets, what, who, roughly when. No schema, no enums, no forced date format. I ran that against the same twenty transcripts first, before writing a single line of structured validation, and looked at exactly where it actually failed.

Two places failed, and only two. Date parsing, when the model had to decide what to do with a vague reference like "by next week," sometimes handled it well and sometimes didn't. Owner disambiguation, when someone was referred to by a nickname or a role instead of a full name, occasionally produced an ambiguous or wrong attribution. Everything else, the basic shape of extracting a task, a rough owner, a rough timeframe, worked fine in plain language and didn't need any additional structure at all.

Version two escalated constraints only for those two fields. Date parsing got a lighter structured hint, not a hard ISO 8601 requirement, just enough guidance to reduce the specific ambiguity that had actually shown up in testing. Owner disambiguation got a similar targeted nudge. Everything else stayed exactly as loose as version one, because nothing about testing had shown a need to tighten it. The result matched or beat the fully rigid baseline's success rate while carrying meaningfully less total constraint complexity, and fabrication dropped to near zero, because the only fields that got hard structure were the ones that had actually earned it through a demonstrated failure.

The order matters as much as the outcome. I didn't guess which two fields would need help and design for them in advance. I let the plain-language version run first and used its actual failures as the spec for where to add anything at all. Constraint complexity ended up proportional to demonstrated need instead of proportional to how thorough I felt like being on a Tuesday morning before I'd seen a single transcript.

Design considerations

The obvious objection is that this approach requires running the unstructured version first, which means shipping something looser than the ideal, at least temporarily, before knowing where to tighten it. That's a real cost, and I won't pretend otherwise. If the task is genuinely high-stakes on every field from the start, a wire transfer amount, a medical dosage, something where an unstructured first pass could cause real damage even briefly, this ordering is the wrong call. Discovering the constraint you needed only after someone's already been hurt by not having it isn't a cost anyone should accept. Progressive escalation assumes the cost of an early, loosely-constrained failure is recoverable, the kind of thing a wrong owner attribution corrects with a follow-up email rather than something that does lasting damage.

The harder calibration question is deciding when a failure in testing is real signal versus noise from a small sample. Twenty transcripts is enough to notice that date parsing struggles with vague references and owner disambiguation struggles with nicknames, because those failures showed up repeatedly and in a recognizable pattern. It's not enough to rule out a rarer failure mode that just didn't happen to appear in this particular set of twenty. I'd treat this as a discipline I keep applying, not a one-time exercise I run once and close out. The right move is never to escalate once and declare the schema finished. It's to keep the plain-language baseline as the default assumption and add structure only when a new failure shows up in production with enough repetition to be a pattern rather than a fluke, the same test that separated real signal from noise in the original twenty transcripts.

There's a version of this that goes too far in the other direction, where every single observed failure gets its own bespoke structural fix and the schema grows one field at a time until it's just as rigid as the version you started by rejecting. "Add structure when something fails" is too blunt a rule on its own; the actual discipline is narrower: add the least structure that closes the specific gap the failure revealed, and stop there. Owner disambiguation didn't need a full employee-ID enum lookup. It needed enough of a nudge to resolve the specific ambiguity nicknames created. Matching the fix's size to the failure's size, not to the fix's theoretical maximum thoroughness, is what keeps this from quietly turning back into the rigid schema it replaced.