The judge is lying to you

LLM-as-judge calibration report · 48 claims · generated 2026-11-15 10:08 UTC · task: grounding verification against a real README

HEADLINE NUMBERS

97.9%
accuracy vs human labels
98%
F1 (supported class)
0.96
Cohen's kappa
2.1%
expected calibration error

VERDICT

The judge agrees with human labels 97.9% of the time (kappa 0.96). Its stated confidence is well-calibrated: expected calibration error is 2.1%. When it is wrong, it is wrong confidently — see the misses below, sorted by confidence.

CALIBRATION CURVE

If the judge were perfectly calibrated, every point would sit on the dashed diagonal (stated confidence = observed accuracy). Points below the diagonal = overconfidence.

judge confidenceempirical accuracy

CONFUSION MATRIX

judge: supported28 gold=supported0 gold=unsupported
judge: unsupported1 gold=supported19 gold=unsupported

THE LIES (1 misses, overconfident first)

idclaimgoldjudge verdictjudge reason
ex034PR_REVIEW_IGNORE_PATHS accepts glob patterns such as *.lock to skip files.…supportedunsupported @ 1.00The context mentions PR_REVIEW_IGNORE_PATHS defaults to lockfiles, minified, vendor/*, dist/* and is described as comma-

LATENCY

1549 ms
p50
20636 ms
p95
0
transport/API errors

METHODOLOGY

Task: grounding verification. A context document (the README of a real open-source project) and 48 claims about it — some faithful paraphrases, some with altered numbers, invented features, or subtle negations. Gold labels are human-authored.

Judge: a single LLM call per claim with a strict system prompt (literal reading, no outside knowledge), temperature 0, JSON output containing verdict + confidence. The confidence — not just the verdict — is what this report calibrates.

Reproduce: clone the repo, set GEMINI_API_KEY, run python3 src/run.py && python3 src/calibrate.py. The CI re-runs the whole evaluation on every push and republishes this report.