How to work with validJSON

Before you hire us, do this. Most structured-output problems can be diagnosed and fixed without consulting. Here’s the order to try things, cheapest first.

Step 1 — Diagnose what’s actually wrong (free, ~30 minutes)

pip install valjson

Run the diagnostic ladder against your current outputs:

• valjson --extract — find JSON in messy model output (code fences, prose, the works).
• valjson --validate --schema schema.json — check against your schema. Surfaces missing keys, wrong types, invalid enum values, key coverage.
• valjson --anatomy --schema schema.json — see your output by grammar role (structural vs. key vs. enum vs. boolean vs. free-text). Most tokens are trivial structure; the few that matter are where things break.
• valjson --compare --schema schema.json --gold gold.jsonl — per-field accuracy against gold. Role-aware, so noisy free-text fields do not drown out the constrained-field signal.
• valjson --diff --schema schema.json --data a.jsonl --data2 b.jsonl — compare two systems (baseline vs. fine-tuned, before vs. after a prompt change) without ground truth.

At the end of Step 1 you will know whether your problem is:

• Format — output doesn’t parse, missing keys, wrong types. Step 2 fixes most of these.
• Wrong values — output parses, schema validates, values are wrong. Step 3 fixes most of these.
• Per-field regression at scale — one specific field is silently wrong, often after fine-tuning. Step 4 is when to call us.

Step 2 — Try the cheap fixes (free, ~1–2 days)

In order:

1. Grammar-constrained decoding. llguidance, Outlines, XGrammar, or your provider’s native structured-output mode. Solves format completely.
2. Retry on validation failure. Up to ~5×, with the validation error fed back as context. Catches most remaining format edges. Note: retries cannot fix semantically wrong output — a confidently-wrong value will be confidently-wrong on every retry.
3. Better prompt with schema in-context. Put the full JSON Schema in the prompt, plus one in-context example. Underrated.
4. Try a different model. Some foundation models are markedly better at structured output than others; the cost of swapping is ~1 hour.
5. Inference-time abstention. valjson --gate --margin-threshold 0.30 — commit when the model’s top-two value margin is wide enough, abstain otherwise. Catches confidently-wrong outputs without retraining.

If Step 2 closed the gap, you’re done. Tell us — we like hearing it.

Step 3 — Try the structural fixes (free + your engineering time, ~1–2 weeks)

When the cheap fixes don’t move the field:

• Schema-level ambiguous pattern. Add ambiguous to the allowed values of the field, and instruct the model (prompt or system message) to emit it when the input does not determine the value. Roughly 90% of the win from presupposition labeling, with zero training required. Documented in the valjson docs.
• Synthetic data generation. Bootstrap a labeled set from a single real example + a schema, for evaluation or for downstream training. Documented in the valjson docs as SDG.
• Bootstrap Extraction Pipeline (BEP). One real document to a production extraction pipeline: speculative schema → seed → grammar constraint + one-shot → diagnose → iterate. Documented in the valjson docs.

Step 4 — When to call us

You’ve worked through Steps 1–3 and you still have one of these:

• A per-field regression you can’t shake at scale. The field is silently wrong on a fraction of inputs and stakeholders cannot accept it.
• A fine-tuning project in a regulated or proprietary-data domain (legal, financial, healthcare) where data cannot leave your infrastructure and the cheaper fixes have plateaued.
• A per-field deployment gate or monitoring setup that has to ship with quantified guarantees.
• A quantified before/after for stakeholders — board, customer, regulator — that requires defensible methodology.

We will spend 30 minutes with you, free, if you bring the valjson output from Step 1. From there we scope a paid pilot: typically 4–6 weeks, fixed price; the deliverable is a per-field reliability report plus a fix — prompt/schema, gates, or a fine-tuned model — whichever the diagnosis warrants.

Get the free consult

Send a one-line description of the problem plus the output of valjson --anatomy --schema your_schema.json --data your_outputs.jsonl to breck@validjson.com. We reply within two business days with whether it’s a Step 2/3 fix or a Step 4 conversation.