The Schema the Model Cannot Violate

The problem

I built a scheduling agent for a clinic that takes a request like "can I move my Tuesday checkup with Dr. Lee to Thursday afternoon" and turns it into a structured booking action: an action type, a date, a time, a provider ID. That structured object feeds directly into a booking API downstream, and the API has zero tolerance for anything that isn't exactly the shape it expects. The instruction I gave the model was the instruction almost everyone gives a model in this position: respond with valid JSON matching this schema.

I ran thirty requests of varying phrasing complexity through it, some plainly worded, some with the kind of hedging and side commentary real patients actually type. At least five of those thirty broke the downstream parser outright. Some came back wrapped in a sentence of explanation before the JSON even started, "Sure, here's the booking request:" sitting in front of a perfectly valid object that a strict parser chokes on anyway because it isn't looking at bare JSON. Some used an action value that wasn't one of the enum options the schema defined, close enough in English to sound right and wrong enough structurally to fail validation. Some were just malformed, a trailing comma, a missing quote, the kind of thing that reads fine to a human and throws to a parser.

Here's the sentence I keep coming back to about this failure: an instruction to emit valid JSON is a request, and the model is allowed to say no to a request. It usually says yes. Five times out of thirty it didn't, and none of those five times were predictable from the phrasing complexity of the request itself. The failure wasn't rare enough to ignore and wasn't concentrated enough in one type of input to patch around with a better prompt.

The pattern

The fix doesn't ask the model to try harder at following the schema. It removes the model's ability to fail at following the schema, by moving the enforcement out of the prompt and into the generation process itself. A schema-constrained decoder, built against the same JSON schema the downstream parser validates against, restricts the token vocabulary the model is allowed to sample from at every single step of generation. If the schema says the next token has to be one of four enum values, the decoder makes every other token unavailable to sample at that position, not unlikely, unavailable. If the schema says a field has to close with a specific character, the decoder won't let generation proceed past a token that violates that structure.

Run the same thirty requests through this version and the parse and validation failure rate drops to zero, by construction. The model didn't get better at reading instructions. The class of failure the prompt-only version suffered from, a token sequence that doesn't match the schema, stopped being a thing the model was physically capable of producing. The guarantee moved from "the model was asked nicely and usually complies" to "the model was structurally incapable of doing otherwise," and those are two different kinds of guarantee even when they produce the same JSON on the cases where the prompt-only version happened to work.

I want to be exact about what that guarantee covers, because it's easy to overstate it. Constrained decoding guarantees syntactic validity. It says nothing at all about semantic correctness. A booking action with a perfectly well-formed structure and the wrong date, or the wrong provider, sails through every check this mechanism runs, because the mechanism was never checking whether the extracted date was the date the patient actually meant. It was checking whether the output has the shape a parser can accept. Those are separate questions, and a system that treats "it parsed" as "it's correct" is trading one blind spot for a different one.

Design considerations

The obvious tradeoff with this approach is that it requires infrastructure the prompt-only version doesn't: a grammar or logit mask wired against the actual schema file, not a copy of the schema pasted into prompt text that can drift out of sync with what the decoder enforces. If the schema changes upstream, both the validator and the decoder need to load the current version, and if either one is reading a stale copy you've reintroduced exactly the kind of silent mismatch this pattern exists to close. The infrastructure cost is real, and I'd rather state it plainly than gloss over it: this is a decoding-time integration decision, made deliberately, before anything ships, rather than a prompt tweak discovered halfway through a production incident.

Because the guarantee is syntactic and not semantic, the design has to keep a separate check for correctness rather than assuming the schema check covers it. In practice that means spot-checking a sample of outputs against ground truth on the actual date, time, and provider extracted, not just confirming the JSON parses. Skipping that step because "the schema validates" is the single easiest way to lose the value of this pattern while still believing you've solved the problem, because the dashboard will show zero parse failures and the booking system will still be scheduling the wrong appointment with total structural confidence.

I'd also flag where this pattern stops mattering, which is anywhere the downstream consumer of the output isn't strict about format to begin with. If a human is reading the model's response directly and would notice a stray sentence of preamble or forgive an oddly worded field, the cost of building and maintaining a constrained decoder against a schema isn't buying you much. This pattern earns its keep specifically at the boundary between a model's output and a system that has zero tolerance for deviation, an API, a database write, a downstream parser with no forgiveness built in. That's a narrower audience than "anywhere you want structured output," and building this everywhere regardless of whether anything downstream actually enforces strictness is effort spent on a guarantee nobody was demanding.

This is also one of the few places in my own practice where I've watched a technique go from clever workaround to standard infrastructure inside about a year. A ten-thousand-schema benchmark now exists specifically to evaluate how well different constrained decoders handle real-world schema complexity, cost analyses of the draft-conditioned variants are being published as a normal part of the literature, and Anthropic shipped structured outputs as a standard feature on its own developer platform at the end of 2025. None of that changes what the mechanism guarantees. It does mean that building this by hand from scratch is increasingly less necessary than it was when I first started working with schema-constrained generation, because the platforms themselves are absorbing more of what used to be custom integration work.