Valid JSON, Wrong Answer: find where your LLM's structured output goes unchecked

Your schema check passes. Your JSON parses. The value in it is still wrong, and nothing in your pipeline noticed.

This is the most common way an LLM feature fails in production, and it has a name now. In the 2025 Stack Overflow Developer Survey, the single biggest frustration developers report with AI, cited by 66% of them, is “AI solutions that are almost right, but not quite.” A field comes back plausible, well-formatted, and incorrect. It flows straight into a database write, a tool call, or a knowledge graph, and you find out when a customer does.

This post gives you a way to find every place that can happen in your codebase, and a single command to measure each one.

Valid is not correct

Constrained decoding and JSON schema validation have quietly solved one half of the problem. It is now easy to force a model to emit output with the right shape: the right fields, the right types, parseable every time. That is the shape. What no schema can check is the meaning: whether the value inside the correctly-shaped field is actually right.

A record like {"refundable": true, "amount": 4900} passes every validator you own. It is also a lawsuit if the order was not refundable and the amount was 49. The schema guaranteed the form. It said nothing about the content.

Most teams handle this by checking the output far downstream of the model that produced it, then re-running or patching after the fact. That is expensive and it never fully works. The fix belongs where the output is made.

Do not eyeball it. Locate it.

The first problem is just finding every place in your app where an LLM’s output becomes a real value that other code trusts. In a codebase of any size, they hide: an extraction call three functions deep, a tool-argument schema, a text-to-SQL query, a graph populater.

So we wrote an Agent Runbook: a machine-readable file you paste into your coding agent (Claude Code, Cursor, ChatGPT, whatever you use) with the instruction “Run this against my repo.” It reads your code, traces where each model output actually goes, and reports the sites where the shape is enforced but the meaning is unchecked, with a one-line reason drawn from the code.

Get it here: AGENT_RUNBOOK.md.

It classifies every model-output site into one of four findings:

no_die_yet — the output is consumed as a real value, but its shape is not even enforced yet.
die_no_dough — shape enforced, meaning unchecked. The core case.
covered — shape enforced and a real value-correctness check exists.
out_of_scope — human-facing text, or an agent orchestrating tools, not a structured deliverable.

Crucially, it decides by reading the code, not by matching keywords, because a keyword lies: tools= can be an output schema or just an agent’s tool list, and only the code tells you which.

A concrete finding

Here is the kind of site it flags. A common GraphRAG pattern asks a model to extract entities and relationships from text, under a schema:

class RawKnowledgeGraph(BaseModel):
    entities: list[str]
    relationships: list[str]

# ... format=RawKnowledgeGraph.model_json_schema()
# ... json.loads(response) -> build_graph(entities, relationships)

The runbook’s verdict:

{
  "where": "extract_entities_and_relationships()",
  "finding": "die_no_dough",
  "reason": "The schema enforces two lists of strings; nothing checks the entities and edges are correct, and they are built straight into the graph.",
  "valjson": "valjson --compare --schema kg_schema.json --data extracted.jsonl --gold labeled.jsonl"
}

The schema guarantees list[str]. It does not guarantee the entities are real, the relationships are entailed by the source text, or that an edge points at an entity that exists. Every one of those errors is valid JSON and a wrong graph.

Measure it with one command

Once you know where, valjson measures the what. It is MIT-licensed, and most of it needs no model:

pip install valjson

If you have reference answers, --compare gives you per-field accuracy, and it is role-aware: it scores each JSON grammar role (boolean, enum, number, key, free-text) separately, so a noisy free-text field does not drown out the constrained fields you actually care about.

valjson --compare --schema schema.json --data output.jsonl --gold gold.jsonl --ignore-role STRING

Why role-awareness matters is easiest to see after fine-tuning. Aggregate loss improves, everyone celebrates, and one field quietly gets worse:

STRUCTURAL   5.33 -> 0.00   -100%   OK
KEY          0.47 -> 0.00   -100%   OK
BOOLEAN      0.46 -> 1.05   +130%   !! REGRESSION
TOTAL        0.55 -> 0.17    -69%

Aggregate loss dropped 69%. Boolean prediction got 130% worse. The average hid it. valjson surfaces it.

No gold labels yet? You still have options:

valjson --anatomy --schema schema.json --data output.jsonl — profile where values concentrate and which fields look fragile.
valjson --confidence --schema schema.json --data output.jsonl — see which fields the model is guessing at.
valjson --gate --data probs.jsonl --margin-threshold 0.30 — in production, abstain on fields where the top-two probability margin is thin, instead of committing a confident-wrong value.

And to keep a fixed field honest over time, wire it into CI:

valjson --compare --schema schema.json --data output.jsonl --gold gold.jsonl --strict-exit

Exit code 1 on regression. Your pipeline fails loudly instead of shipping a silently-wrong field.

The short version

Schema validation proves your LLM output is well-formed. It does not prove it is correct.
Run the Agent Runbook to find every site in your repo where the values go unchecked.
Measure each one with valjson, role-aware, mostly no model required.

“Almost right, but not quite” is a solvable engineering problem. Start by finding out how wrong you actually are.

valjson is an open-source (MIT) tool from validjson.com. The thinking behind it is in the paper Valid JSON, Wrong Answer, and in an in-progress book on engineering reliable structured output. Follow along at validjson.com.