A well-worded prompt is not a contract

Asking a model to "respond only in JSON matching this format" works most of the time, which is exactly the problem. The failures are rare enough to pass a demo and common enough to reach production: a string where an integer belongs, a missing required field, an extra key the downstream system was never built to handle.

Treat the model's output the way you would treat a payload from an external system you do not control: verify its shape before anything acts on it. That is the same posture the OWASP Top 10 for LLM Applications takes under Improper Output Handling, which calls for validating and sanitizing model output before it reaches a downstream component, rather than trusting it because it looks well formed.

Where the gate belongs

Put the check between the model call and the first place its output has consequences: a database write, an API call, or a handoff to a human reviewer. A workflow that only checks the shape of the answer by eye during a demo, and never in the code path that fires the write-back, has not actually built the control.

Some providers now let you push part of this check upstream. Anthropic's strict tool use, for example, constrains a model's token sampling so its structured output matches a supplied JSON Schema before your code ever sees it, closing the class of failure where the model's own answer disagrees with your schema. That reduces how often the downstream check has to reject something, but it does not replace it: a strict-mode guarantee still assumes the schema captures the business rule correctly, and rules that live outside the schema, such as cross-field relationships or values tied to another system, still need to be checked after the call returns.

Define valid narrowly, and reject loudly

A useful schema does more than name the fields. It constrains types, marks which fields are required, sets enumerations for closed sets of values, and disallows unexpected extra keys. The JSON Schema specification is the standard way to express those constraints in a form both a validator library and a human reviewer can read.

When output fails the check, do not silently coerce or drop the offending field. Route the case the same way the workflow already routes a failed business rule: record what came back, record what the schema expected, and send the case to the same review or retry path an operator already trusts. A schema failure is workflow-relevant information, and quietly patching over it removes the evidence a team needs to see the failure rate change over time.

  • Constrain types and required fields, not just field names.
  • Reject unexpected extra keys instead of ignoring them.
  • Log the rejected output next to the schema it failed, not just a pass or fail flag.