Force a model into strict JSON too early and it reasons worse, then hands you a clean, well-shaped, wrong answer. Structured outputs guarantee valid JSON, not correct JSON. Your JSON parses; that does not mean it is true. Constrained decoding forces the model to emit a valid shape, and forcing that shape too early can make the model reason worse on hard tasks. OpenAI's documentation says Structured Outputs ensure responses adhere to your supplied JSON Schema, and a controlled study found reasoning accuracy can drop when models are locked into a rigid format before they finish thinking.
Valid JSON is not correct JSON. Constrained decoding shapes the output, not the reasoning, so a clean parse tells you nothing about whether the answer is true.
Quick takeaways: (1) Valid JSON is not correct JSON. Schema adherence is a syntax guarantee, not a truth guarantee. (2) Constrained decoding shapes the output, not the reasoning. (3) Forcing a strict schema on reasoning-heavy tasks can cost accuracy. (4) Classification and extraction tolerate strict schemas well. (5) The reliable pattern: let the model reason in free text first, then format the answer in a second step. (6) Keep validating outputs even when JSON is guaranteed.
Valid JSON is not correct JSON
Structured outputs solve a parsing problem, not a correctness problem. When you enable a strict schema, the platform guarantees the model will not omit a required key or emit an invalid enum value, so your parser never crashes. It says nothing about whether the number in that field is the right number, whether the summary is faithful, or whether the classification matches reality.
A response that parses cleanly looks trustworthy. Engineers see well-formed JSON and assume the pipeline is healthy, then ship a system that returns wrong but well-shaped data. Grammar-guided decoding guarantees syntactic validity, not semantic correctness: the values can be hallucinated or low-quality even when the structure is perfect.
What structured outputs actually guarantee (and what they don't)
Structured Outputs guarantee schema adherence; JSON mode only guarantees valid JSON. OpenAI's documentation draws this line: both produce valid JSON, but only Structured Outputs ensure the response matches your supplied schema, with every required key present and every enum drawn from your allowed set. JSON mode gives you syntactically valid JSON with no structural promises beyond that.
Even the schema guarantee has documented edges. OpenAI tells developers to handle cases where the model refuses for safety reasons, where the response hits a token limit and arrives incomplete, and where, in some cases, the model still fails to produce a valid response matching the schema. Check for refusals and truncation before you trust the parse.
JSON mode vs strict schema vs function calling
These three terms describe different levels of constraint, and conflating them is a common mistake. JSON mode is the loosest: valid JSON, no schema. Strict structured outputs add schema enforcement through constrained decoding. Function calling, or tool use, is a related mechanism where the model selects a tool and fills its arguments, and on modern platforms those arguments can be enforced against a strict schema too.
| Mechanism | Valid JSON | Schema enforced | Room to reason | Best for |
|---|---|---|---|---|
| JSON mode | Yes | No | No | Loose, when any JSON is fine |
| Strict structured outputs | Yes | Yes | No | Extraction, classification, fixed shapes |
| Function / tool calling | Yes (args) | Optional strict | Partial | Routing actions and tool arguments |
| Free text + parse later | Not until step 2 | Applied in step 2 | Yes | Reasoning, math, multi-step analysis |
Schema enforcement is a property you opt into, not a default of every JSON-shaped call. OpenAI's comparison table makes the binary clear: JSON mode and Structured Outputs both yield valid JSON, but only Structured Outputs adhere to the schema. Rule of thumb: if removing the schema would change the answer's content, not just its shape, the task is reasoning-heavy, so keep the model free until it finishes thinking.
How constrained decoding works (logit masking explained)
Constrained decoding masks invalid tokens at each generation step. A logit processor sits between the model's raw output and the sampling step. It tracks a grammar or finite-state machine built from your schema, computes which tokens are legal next given the tokens so far, and sets the logits of every illegal token to negative infinity so they can never be sampled. The output is forced to stay inside the grammar.
The model's weights are never modified. Constrained decoding only reshapes the sampling distribution by applying a logit mask. It does not improve the model's reasoning. It narrows what the model is allowed to say, which is exactly why it can interfere when the next best token for thinking is not the next legal token for the schema.
The documented accuracy tax: what Tam et al. found
Lock a model into a strict schema before it finishes thinking, and on hard problems it gets measurably worse at thinking. That is the accuracy tax, and the effect is task-dependent. In Tam et al.'s controlled study, models showed a measurable decline in reasoning ability under format restrictions, and stricter constraints generally produced larger drops on reasoning-heavy tasks. The same paper found a different trend on some classification tasks, where a JSON format performed competitively or even better because constraining the answer set reduced selection errors. Format is not uniformly good or bad; it interacts with the kind of task.
A rebuttal complicates this. The team behind the Outlines library published a response arguing the original gap shrinks or disappears when the structured and unstructured prompts are made truly comparable, because the structured prompts in the study often gave the model less room to reason before answering. Their reading: the cost comes largely from how the format instruction is delivered, not from constrained decoding as a mechanism.
Both findings point to the same engineering rule. Whether the tax comes from the decoding constraint or from a cramped prompt, the safe move is to stop forcing the model to produce its final structured answer in one shot when the task needs real reasoning. Independent practitioner write-ups report the same pattern: structured outputs can hurt on some tasks and help on others, so measure on your own data rather than assuming.
Why the format instruction itself can cost quality
Force a strict JSON answer and the model spends its first tokens opening a brace and a key instead of working through the problem. Part of the cost is the prompt, not the decoder. When you tell a model to answer only in a strict JSON object, you implicitly tell it to skip the intermediate reasoning that chain-of-thought relies on. The Outlines rebuttal makes this case directly: structured prompts in the original study often did not give the model adequate room to solve the task before committing to a shape.
This explains why the same model can ace a math problem in prose and stumble on it when forced to answer with only a single answer field. The reasoning was load-bearing, and the schema deleted the space where it would have happened. The fix is not to abandon structure. It is to give reasoning a place to live inside the structure, or before it.
The fix: reason first, format second (two-step prompting)
Reason in prose. Format in a second step. That is the whole fix. The most reliable pattern is a two-step flow: generate an unconstrained answer with full reasoning, then make a second, cheap call that converts that reasoning into your strict schema. You keep the parsing guarantee without paying the reasoning penalty.
- Add a reasoning field first. Put a free-text reasoning or analysis property before the final answer field in the same schema so the model thinks before it commits.
- Use two calls for hard tasks. Call one solves the problem in prose. Call two extracts the answer into strict JSON. The extractor call rarely needs the big model.
- Order schema fields by dependency. Properties are generated top to bottom, so put intermediate values above the conclusions that depend on them.
- Reserve one-shot strict output for tasks with little reasoning: extraction, tagging, routing, and fixed enums.
Field order silently changes your answers. Constrained decoding generates keys in the order your schema lists them, so a final-answer field placed above a reasoning field forces the model to commit before it thinks. Put reasoning fields first.
Which tasks tolerate strict schemas
Match the constraint to the task. Low-reasoning, high-structure tasks are where strict schemas shine and where the documented accuracy tax is smallest or reversed. High-reasoning tasks are where you should keep the model free until the thinking is done.
- Safe for strict one-shot schemas: entity extraction, classification into fixed labels, form filling, data normalization, tool-argument selection.
- Risky for strict one-shot schemas: multi-step math, symbolic reasoning, planning, ambiguous analysis, anything where chain-of-thought usually helps.
- Hybrid: keep the strict schema but add a reasoning field, so reasoning-heavy tasks still get structured output without losing their scratchpad.
Validation you still need
Guaranteed JSON does not remove the need for validation; it changes what you validate. Schema enforcement covers shape and types. It does not cover business rules, value ranges, referential integrity, or whether the answer is true. You still need checks that a date is in range, an ID exists, a total sums correctly, and a refusal or truncated response gets caught before it reaches a user.
- Check for refusals and incomplete responses before parsing, as the platform docs instruct.
- Validate values against business rules the schema cannot express, such as ranges, cross-field consistency, and lookups against real records.
- Spot-check correctness on a labeled sample. Parse rate near 100 percent tells you nothing about accuracy.
- Log the free-text reasoning when you use a two-step flow, so you can audit why a value came out wrong.
Where MemX fits: a lot of the format pressure on a prompt is context pressure. When you stuff schemas, few-shot examples, and source documents into one giant request, you crowd out the model's room to reason. MemX is an external memory layer at memx.app that captures your documents, screenshots, voice notes, and forwarded messages, indexes them, and lets any model retrieve only the relevant context at query time. Keeping examples and reference material in a retrievable store, instead of pasting them into every structured-output call, leaves the model more room to think before it commits to a shape. MemX does not enforce your JSON schema; it reduces the prompt bloat that makes strict formatting hurt.
Key takeaway: Structured outputs guarantee that your JSON parses, never that it is correct. Constrained decoding shapes tokens, not reasoning, so forcing a strict schema before the model finishes thinking can lower accuracy on hard tasks. Reason in free text first, format second, reserve one-shot strict schemas for extraction and classification, and keep validating values no matter how clean the JSON looks. Valid JSON is not correct JSON, and it never was.
01Does JSON mode reduce accuracy?
Yes, on reasoning-heavy tasks. A controlled study found models reason worse when locked into a strict format before finishing their thinking, while classification tasks often hold up or improve. The cause is partly the format and partly a prompt that leaves no room to reason. Add a reasoning step to avoid the hit.
02Why does my JSON parse fine but the values come out wrong?
Because schema adherence is a syntax guarantee, not a truth guarantee. Constrained decoding forces the output to match your shape by masking invalid tokens, but it never checks whether the values are correct. A perfectly formed JSON object can still contain hallucinated or miscalculated fields.
03Should I use a strict schema for reasoning tasks?
Not as a one-shot final answer. For math, planning, or multi-step analysis, let the model reason in free text first, then convert to your schema in a second step or via a reasoning field placed above the answer. Reserve one-shot strict schemas for extraction and classification.
04What is the difference between JSON mode and structured outputs?
JSON mode guarantees valid JSON only. Structured Outputs also guarantee the response adheres to your supplied schema, so required keys are present and enums are valid. Per OpenAI's docs, both return valid JSON, but only Structured Outputs enforce the schema itself.
05How does constrained decoding work?
A logit processor tracks a grammar built from your schema and, at each step, sets the logits of every token that would break the grammar to negative infinity, so only valid tokens can be sampled. The model's weights are never changed. It constrains output, not reasoning.
