LLM structured outputs: what they fix and what breaks next

Automation Engineering · By Caleb Sakala · March 31, 2026

Structured output mode (what OpenAI calls “strict mode,” what Anthropic implements through tool use schemas) works by constraining the model’s token generation at decode time. At every step, the decoder masks out tokens that would produce invalid JSON relative to your schema. So you get valid JSON on every call. That part is solved, and if you spent time last year writing retry loops to handle models wrapping their response in markdown code fences or forgetting a closing bracket, you already know how much that matters.

But here’s what happened on a contract processing workflow we built at Chase Agents, maybe two months after switching to constrained decoding. Everything looked good: zero parse failures, schema validation passing on every run. And then someone in operations noticed that a termination notice from a vendor had been routed to accounts payable and processed as an invoice. Structurally, the JSON was perfect. Fields populated, types correct, enum values all within the allowed set, no validation errors anywhere. The category field said "invoice" when it should have said "termination_notice", and because the entire downstream automation keys off that field (routing decisions, approval workflows, payment processing, even reporting), the mistake propagated silently through four systems before a human caught it in monthly reconciliation.

That kind of failure is what this post is about. Format compliance is a solved problem now. Semantic accuracy is not, and the gap between those two things is wider than most teams realize when they first adopt structured outputs.

How field names quietly shape model behavior

Jason Liu’s Instructor library (open-source Python, somewhere around 2.5 million monthly downloads last I checked) published benchmarks on the GSM8k math dataset that changed how I think about schema design, and probably should change how you think about it too.

They renamed a single output field from final_choice to answer. Nothing else about the setup changed; the prompt was identical, the model was the same, the schema structure and examples were untouched. Accuracy went from 4.5% to 95%.

I want to dwell on this because it’s genuinely strange if you haven’t encountered it before. The field name isn’t an instruction. It’s not in the prompt. It’s metadata, schema plumbing. And yet it moved accuracy from completely unusable to production-grade, because language models treat field names and their description annotations as implicit instructions that shape what kind of value belongs in that slot. A field called final_choice apparently activates different completion patterns (more hedging, more ambiguity) than one called answer. It’s the same underlying phenomenon that makes “what is 2+2?” and “compute the sum of 2 and 2” produce different token distributions, except here it’s happening inside the schema rather than the prompt.

The Instructor team found two other things worth knowing. First, schemas that included a reasoning field before the answer fields produced 92-94% accuracy on the same dataset, versus 33% without it. The model uses earlier fields as scratch space while generating later ones, so if you force it to jump straight to an answer with no room to work through the problem, you get well-formatted garbage. Second, JSON mode showed 50% more sensitivity to field renaming than tool calling did. So the enforcement mechanism you pick has consequences for how fragile your accuracy is to schema changes, which is the kind of thing that never comes up in the implementation guides because it makes the whole thing sound harder than the pitch wants it to sound.

Practical upshot: treat your field names like they’re part of the prompt. Add description properties. Put a reasoning or analysis field before any classification or extraction output. And actually test this. Rename a field, run your eval set, see if accuracy moves. If it does, you have a schema problem that no amount of prompt engineering will fix.

Optional fields and the bugs they hide

There’s a specific failure mode with optional fields that I keep seeing teams discover the hard way, usually through a bug report from their users rather than from their test suite.

Say you have a ticket metadata extraction schema. Required fields: ticket_id, priority, category, assigned_agent. Optional fields for when the information is available: customer_tier, affected_product, escalation_reason, previous_ticket_ref. Your test data comes from resolved tickets where most of those optional fields had values, so everything looks fine during development.

In OpenAI’s strict mode, every property has to appear in the required array. If a field might be absent, you type it as a nullable union ("type": ["string", "null"]) and the model returns null when it’s not confident. That’s predictable, you can code around it.

Without strict mode, using plain JSON Schema, the model just skips optional fields it’s unsure about. No null, no placeholder, the key simply isn’t in the response object. And now ticket["affected_product"] throws a KeyError in Python or returns undefined in JavaScript, and both of those will pass silently through most downstream code until something visibly breaks. Most teams find out about this in production because their development test data was too complete to surface it.

Forced enums and the ambiguity they swallow

When you constrain a field to an enum (say, APPROVE, REJECT, ESCALATE, HOLD) and the model encounters an input that doesn’t cleanly map to any of those options, the grammar constraint forces it to pick one anyway. There is no way for the model to express uncertainty through a constrained enum unless you’ve explicitly included an uncertainty option.

So an ambiguous case, the kind where a human reviewer would flag it and ask a colleague, gets silently shoved into whichever bucket scored highest and the automation proceeds as if the classification were confident. Nobody finds out the model was uncertain because the schema gave it no mechanism to say so.

Adding NEEDS_REVIEW or UNCERTAIN to your enums is an obvious fix that almost nobody implements on the first pass, because development test cases tend to be unambiguous. You find out you needed the catch-all option after deployment, when someone asks why a borderline contract got auto-approved.

Why two-step reasoning beats single-step constrained calls

For anything more complex than a straightforward binary classification, the most reliable pattern I’ve seen in production runs two LLM calls instead of one.

The first call gets a free-form prompt with no schema constraints at all. It reads the input and works through the problem in plain text — no JSON, no schema, just unstructured analysis of what it sees and what it recommends. Then the second call takes that analysis as context and formats it into your schema. At this point the reasoning is already done; the second call is just translating finished analysis into structured fields.

Why does this help? Because the first call’s reasoning isn’t being shaped by your schema’s field names and enum constraints. Remember the Instructor benchmarks: field names change model behavior. In a single-step constrained call, the model is simultaneously trying to reason about the problem and produce valid structured output, and the schema’s semantics influence the reasoning in ways you probably don’t want. Splitting the two steps apart means the reasoning happens before the schema has a chance to distort it.

On Chase Agents, we use this for contract approval routing. A first call analyzes contract terms against business rules and produces free-text reasoning. A second call converts that into action_type, required_approvers, risk_flags, routing_metadata, confidence_score. Routing accuracy on edge cases improved noticeably compared to the single-step approach, which is consistent with what the Instructor team documented about accuracy degradation in constrained single-step calls on harder tasks.

The trade-off is cost and latency, since you’re making two LLM calls instead of one. For simple, high-confidence decisions, a single well-designed schema call works fine and the extra round trip isn’t worth it. The two-step pattern pays for itself on the cases where the model would otherwise be uncertain, which are exactly the cases where getting it wrong matters most.

Measuring what actually matters after you’ve got valid JSON

Once you have constrained decoding running, schema compliance will sit at 100%. That number tells you the decoder is working. It tells you nothing about whether the values inside the JSON are correct.

The useful next step: add a confidence field to any schema where wrong outputs have downstream consequences. Then build routing around it. Outputs above 0.9 go through the standard automation path, 0.7 to 0.9 get flagged for human review, 0.5 to 0.7 go to a secondary verification step, below 0.5 gets escalated. The exact thresholds depend on your domain’s tolerance for error, but having any threshold-based routing at all puts you ahead of most production deployments I’ve seen, where the only check is “did the JSON parse.”

On our contract extraction workflows at Chase Agents, we go further than confidence scoring. Each step has semantic validation criteria alongside the schema: extracted party names can’t be empty, effective dates have to fall after the document date, obligation types have to match a defined set of contract categories, monetary values have to parse as valid currency. A step that returns valid JSON with empty party names fails its validation criteria and halts the pipeline before bad data gets into downstream systems. That gap between “the JSON was valid” and “the extraction was correct” is where reliability actually lives in these systems, and structured outputs on their own don’t close it. You have to build the validation layer yourself.

Where this is all heading

Anthropic, OpenAI, and Google have all shipped or announced features during 2026 that move past format compliance toward output-quality assessment: confidence scoring baked into the API, self-consistency checks, retrieval-grounded verification. It’s pretty clear where this is going. By late 2026 or early 2027, the baseline expectation for production structured outputs will include some form of semantic validation — not just schema compliance.

Teams that have already started building confidence scoring and semantic validation into their workflows are ahead of that curve. The rest will get there eventually, but right now most of the implementation guides and tutorials stop at “here’s how to define a schema,” and that leaves out the part of the work that actually determines whether your automation produces correct results or just well-formatted ones.