Structured outputs with JSON Schema in LLM calls: why we still validate the response

There are two very different ways to ask a model for JSON, and the gap between them matters more than it looks. The first is the one we all used for years out of necessity: describe the expected shape in the prompt ("reply only with a JSON object with these keys") and hope the model cooperates, then parse whatever came back and discover, often enough, that it prepended a sentence before the JSON, wrapped it in a markdown code block, or renamed a field because it seemed more natural. The second is what providers now call structured output: pass an explicit JSON Schema alongside the request and let the server constrain token sampling so the response can only take shapes that satisfy that schema.

The second option is a real shift, not a cosmetic one. Instead of politely asking for a format and hoping the model complies, constrained decoding drops, at every step, any token that would lead to an output violating the schema. The model literally cannot open a brace without closing it, use the wrong type for a field, or invent a key that isn't in the definition. That removes, at the root, an entire category of errors we used to patch with defensive regexes and retries.

What structured output actually guarantees

It's worth being precise about what this guarantee covers, because it's narrower than the name suggests: it guarantees shape, not content. The schema forces "price" to be a number instead of a string, forces "items" to be an array instead of an object, forces a key marked as required to show up in the response. It does not guarantee that number is the correct price, that the array has the right number of elements, or that the value inside a required field makes any sense for the case we're actually solving.

That becomes obvious the moment the schema requires a field the model has no real way to fill in. If we're extracting data from a document and mark "order_number" as required, but that particular document doesn't contain any order number, the model can't return null unless the schema explicitly allows that as a valid type — so it produces something syntactically correct and semantically made up: an empty string, a value that looks like an identifier but doesn't come from anywhere in the text. The schema is satisfied. The data is false.

Where it still fails, even with the mode turned on

There's a second group of problems that has nothing to do with the model "cheating," and everything to do with the real limits of the mechanism:

• Deeply nested schemas or $ref-based ones. Each provider's support for the full set of JSON Schema keywords isn't uniform or static: a constraint like pattern, format, or a oneOf with a discriminator can be documented but only partially enforced during sampling, or quietly ignored without the API returning any error. That means a validation we thought we'd delegated to the provider was, in practice, never being applied.
• Truncation from the token limit. Constrained decoding operates token by token on whatever the model is generating, but it has no control over how many tokens we let it generate. If the response gets cut off by max_tokens halfway through a nested array, the result is incomplete JSON — an unclosed brace, an array missing its final bracket — even though every individual token that did get emitted was valid against the schema at that point.
• Enums that are syntactically fine but semantically wrong. The schema can restrict "currency" to an enum of three valid values, and the model will always pick one of those three. That says nothing about whether it picks the right one for the document in front of it — only that it won't invent a fourth value.
• Consistency across fields. No standard JSON Schema natively expresses "the sum of quantity * unit_price across every item in items must equal total." The schema validates each field in isolation; the relationship between fields is on us.

A concrete case: extracting line items from a document

In a document-extraction pipeline handling detail lines — purchase orders, delivery notes, any document with a line-item table — we defined a schema with an items array, each entry carrying description, quantity, unit_price, plus a document-level total field. Structured output immediately eliminated the formatting errors that used to force us to retry the call: zero malformed-JSON responses, zero fields with the wrong type.

What it didn't eliminate was that, on documents with poorly scanned tables or a row split by a page break, the model would generate a complete, syntactically perfect row for the part of the table it couldn't read properly, with a quantity and a price that matched nothing visible in the original document. The schema has no way to catch that, because from its point of view there's no error at all: every field has the right type and is present. The only way to catch it was to add, after receiving a response that had already passed schema validation, a check that the sum of the line items matched the total declared in the document, and treat any mismatch as a signal that at least one line was suspect and needed review before the data could be trusted.

What we still validate on our side

from pydantic import BaseModel, ValidationError

class Item(BaseModel):
    description: str
    quantity: float
    unit_price: float

class Document(BaseModel):
    items: list[Item]
    total: float

def validate_response(data: dict, finish_reason: str) -> Document:
    if finish_reason == "length":
        raise ValueError("response truncated by max_tokens, JSON can't be trusted")

    doc = Document.model_validate(data)

    items_sum = sum(i.quantity * i.unit_price for i in doc.items)
    if abs(items_sum - doc.total) > 0.01:
        raise ValueError(
            f"inconsistency: items sum {items_sum} != declared total {doc.total}"
        )

    return doc

This validation runs after the provider has already applied its own constrained decoding, and it isn't redundant with that step — it checks exactly what the schema can't check. First, finish_reason: if the response got truncated, we discard the result instead of trying to patch it, because "almost complete" JSON isn't data worth building on. Second, we parse against our own data model (Pydantic, Zod, or whatever validator fits the language) — not because we distrust that the schema was enforced, we do trust that, but because we want a typed object in our own code, not a loose dictionary. Third, and this is the part that actually matters, we apply the business rules no standard JSON Schema can express: consistency across fields, sane ranges, agreement with other data we already had about the same document.

What we check before trusting a structured output as validated data

Before treating a structured-output response as already-verified data, we confirm which subset of JSON Schema keywords the provider we're using actually enforces, instead of assuming it supports the full spec just because the docs list the keywords without spelling out the coverage; we check the finish_reason on every response before parsing anything, because a truncated response can be invalid JSON despite structured mode being on; we mark as required, in the schema, only the fields the model can fill with real information present in the context, leaving everything else optional or nullable so it isn't forced into inventing a value; and for every schema we put into production, we write at least one post-validation check that verifies a relationship between fields the schema can't express on its own, because that's exactly the class of error structured output was never designed to prevent.