valjson — JSON-from-LLM diagnostics

valjson is the open-source diagnostic and gating tool from validjson.com.

pip install valjson

GitHub: github.com/validjson/valjson · PyPI: pypi.org/project/valjson

What it does

• Per-grammar-role loss decomposition. Categorize teacher-forced loss tokens by JSON grammar role (boolean, enum, free-text, structural). Surfaces field-level regressions that aggregate metrics hide.
• Per-field confidence analysis. For each constrained field, extract the model’s probability distribution over allowed values. Tells you which fields the model is guessing at.
• Margin gating (valjson --gate). Commit when the top-two probability margin exceeds a threshold; abstain otherwise. Improves precision at a controlled recall cost.

When to use it

• Before training. Profile your base model on the schema. Identify which fields the pretrained model is already confident on, and which it is not.
• After training. Detect counter-current fields where fine-tuning made things worse on a constrained role even though aggregate loss improved.
• In production. Gate low-margin field predictions to abstain rather than commit a confident-wrong answer.

When valjson is not enough

If a critical field regresses and abstention is not acceptable for your application, training-time fixes (presupposition labeling, custom data preparation) become relevant. Talk to us.