LLM structured outputs explained

Why free-form text breaks production pipelines

Large language models are trained to produce natural language. Even when you ask for JSON, models routinely wrap output in markdown fences, prepend explanatory sentences, omit required keys, invent enum values that do not exist, or nest objects one level deeper than your parser expects. At low volume you can hand-fix; at scale every half-percent parse failure becomes a support ticket, a stuck workflow, or a silent data corruption bug.

Structured outputs solve a different problem than hallucinated facts. The model may still invent a customer ID that does not exist — schema enforcement guarantees the shape of the response, not its truth. You still need retrieval, verification, and human review where stakes are high. But shape errors — invalid JSON, wrong types, missing required fields — are the ones that crash your code before you even get to fact-checking.

JSON mode vs structured outputs vs function calling

These three features overlap in marketing but behave differently in engineering:

JSON mode

JSON mode (offered by several providers) nudges the model toward valid JSON syntax — an object or array, no markdown wrapper. It does not guarantee adherence to your schema. The model might return {"status": "maybe"} when you required status to be one of approved | rejected. JSON mode is a reasonable first step for simple prototypes; it is not a contract.

Schema-constrained structured outputs

Structured outputs bind generation to a JSON Schema (or equivalent) at decode time. The model's token sampler masks illegal continuations — if the schema says quantity is an integer, the decoder will not emit a decimal point mid-number. Major APIs expose this as response_format: { type: "json_schema", ... } or similar. Constrained decoding dramatically raises parse success rates on complex nested schemas compared to JSON mode plus post-hoc validation.

Function (tool) calling

Function calling is structured output aimed at actions: the model selects a named tool and fills its argument object. The schema describes parameters for create_invoice or search_database, not a final answer document. Use function calling when the LLM's job is to decide which API to hit and with what args; use structured outputs when the job is to produce a data artifact (classification label, extracted entities, evaluation scorecard) that your code consumes directly. Many agent frameworks use both in the same turn: structured planning object, then a function call to execute.

Designing schemas that models can fill reliably

A schema is an API contract you hand to a probabilistic partner. Design it like you would for a junior developer — explicit, minimal, and hard to misread:

• Keep nesting shallow. Deeply nested objects with six required fields at each level fail more often than flat records with clear names.
• Use enums for closed sets. Prefer "sentiment": { "enum": ["positive", "neutral", "negative"] } over free-text labels the model can paraphrase.
• Split large extractions. Asking for twenty fields in one shot invites omissions; two sequential calls with smaller schemas often beat one giant object.
• Match field names to your prompt vocabulary. If the prompt says "ticket priority", the schema key should be priority, not urgency_level_code unless you define that mapping explicitly.
• Document formats in the schema description. Many APIs let you attach a description per property ("ISO 8601 date, UTC"). Models read these during constrained generation.
• Avoid additionalProperties: true in production. Open schemas invite spurious keys that pollute logs and break strict deserializers.

Libraries like Pydantic (Python), Zod (TypeScript), and OpenAPI specs can generate JSON Schema automatically — a single source of truth for validation, API docs, and LLM constraints. When the same schema validates inbound user data and outbound model data, you eliminate an entire class of mismatch bugs.

Validation, repair loops, and graceful degradation

Even constrained decoding is not mathematically perfect on every provider and model tier. Production systems wrap the LLM in a validate → repair → fail pipeline:

1. Parse the raw string with a strict JSON parser. Reject trailing content after the closing brace.
2. Validate against JSON Schema (or your typed model). Collect structured error paths: /items/2/price: expected number.
3. Repair on failure: send a short follow-up prompt with the invalid JSON and the validation errors, asking for a corrected object only. Cap retries at one or two — unbounded repair loops burn tokens and latency.
4. Degrade if repair fails: return a safe default, queue for human review, or fall back to a smaller schema (e.g. extract only category and skip optional metadata).

Log schema failure rates by model version and prompt template. A spike after a model upgrade is a signal to tighten schemas, add few-shot examples, or pin the previous model until evals pass — the same regression discipline as LLM benchmarking for quality metrics.

Streaming structured JSON

Streaming improves perceived latency for chat UIs, but partial JSON is not valid JSON until the final token arrives. Strategies:

• Buffer-and-parse: accumulate tokens server-side, parse only on completion. Simplest; no incremental UI for fields.
• Incremental parsers: libraries that emit events when top-level keys complete (on("field", "title", value)). Useful for showing a headline before the body finishes generating.
• NDJSON / JSON Lines: ask the model to emit one complete object per line for list outputs. Each line parses independently — good for streaming search results or log-style agent traces.

Constrained streaming is harder than constrained batch generation; some providers only guarantee schema adherence on non-streaming calls. Read the provider docs before building a real-time dashboard on streamed structured fields.

Security and abuse considerations

Structured output does not immunize you against prompt injection. An attacker who controls part of the input can still try to steer enum values, inject strings into fields that later become SQL or shell commands, or overflow array lengths if your schema allows unbounded maxItems. Treat every model-produced field as untrusted user input: parameterized queries, allow-listed enums at the application layer, maximum string lengths, and never eval() on model JSON.

When structured output feeds tool execution in agents, apply the same authorization checks you would for a human operator. The model choosing {"action": "delete_user", "id": 42} should not bypass RBAC because it arrived as valid JSON.

Provider landscape (2026)

Capabilities differ; verify current docs before shipping:

• OpenAI-compatible APIs: structured outputs via JSON Schema on recent GPT-class models; function calling with parallel tool support on agent paths.
• Anthropic: tool use with JSON Schema input definitions; structured text blocks for extraction patterns.
• Open-source stacks: vLLM, llama.cpp, and guided generation (Outlines, Guidance) apply grammar constraints locally — valuable for on-device inference without round-trips to cloud APIs.
• Frameworks: LangChain, Instructor, Marvin, and similar wrappers unify schema definition, retry, and parsing across providers — worth adopting if you multi-home models.

For integrations wired through Model Context Protocol (MCP) servers, tool parameter schemas are the structured-output surface between host and server. Keep MCP tool schemas as small and testable as your REST API contracts.

Production checklist

• Define one canonical JSON Schema (Pydantic/Zod) shared by prompts, validation, and docs.
• Prefer provider structured-output / constrained decoding over JSON mode alone for any field that triggers business logic.
• Cap schema complexity: shallow objects, enums for closed sets, split huge extractions across turns.
• Implement validate → repair (max 2 retries) → safe degrade; alert on failure-rate SLO breaches.
• Version and regression-test schemas in CI with golden inputs — same suite as prompt changes.
• Treat parsed values as untrusted; enforce authz, length limits, and injection defenses before side effects.
• Log token usage and latency per schema template; oversized schemas cost context window and money.
• Document which calls require non-streaming mode for strict schema guarantees.

Key takeaways

• JSON mode helps syntax; structured outputs enforce schema shape at generation time.
• Function calling is for tool selection and action args; structured outputs are for data artifacts your code consumes.
• Schema design matters as much as model choice — shallow, explicit, enum-heavy schemas parse more reliably.
• Validate and repair in a bounded loop; never assume 100% success without metrics.
• Security does not end at valid JSON — treat every field as hostile until sanitized and authorized.
• Eval regressions when models or schemas change; structured output failure rate is a first-class metric.

Related reading

• Prompt engineering explained — system prompts and few-shot patterns that complement schema constraints
• AI agents and tool use explained — when structured plans hand off to function calls and external APIs
• LLM evaluation and benchmarking explained — regression suites for parse success and field accuracy
• Prompt injection explained — defending structured pipelines against adversarial inputs