fanrz/grade-the-gold

Grade the Gold — Calibrated to What? Teacher-Consensus Benchmarks Hide Outcome Uncertainty in LLM Agent Decisions

Open reproduction, training recipes, and outcome-based audits for typed agent decisions. We reproduce the benchmark — and grade the gold.

A full reproduction + audit + open challenge to the typed-decisions / AgentJev benchmark ecosystem (the "Jev" hot topic), which scores enterprise-agent decisions against teacher-consensus labels and rewards models "calibrated" to them — most hyped: Laya (421M ModernBERT, 77% reported).

We ask: calibrated to what? Three findings, all backed by per-decision data you can recompute with the scripts here.

TL;DR

Model	Params	Accuracy	ECE (10 bins)
Laya (official ckpt)	421M	0.768 (official: 0.770)	0.215
Qwen3-4B zero-shot	4B	0.488	0.214
Qwen3-4B + LoRA (ours, 3 seeds)	4B	0.795 ± 0.005	0.146–0.167
Qwen3-1.7B + LoRA (ours, 2 seeds)	1.7B	0.777	0.145–0.153

Same-protocol LoRA finetune (soft-CE + 0.1·Brier on teacher distributions) beats Laya on every seed: +3.0/+2.1/+3.0 pp. We report the conservative case-level paired sign-flip test (clustered by the 400 cases, exact enumeration): p = 0.0010 / 0.018 / 0.0008 across the three seeds. Accuracy rises monotonically with size; teacher-agreement is a distillation recipe that scales, not a capability moat.

The three claims

1. The metric is soft. A published 6.5-pp gap between the top two open systems sits inside the noise floor; same-protocol small finetunes erase the leaderboard's headline claim.

2. Confidence is a dial, not a signal. Positive temperature scaling provably preserves argmax — yet on this benchmark a temperature fitted to teacher soft labels swings reported ECE 0.20 → 0.45 without changing a single answer (65 τ × 2,000 decisions; fig_temp_shift.svg). Isotonic regression — ignored by the whole ecosystem — cuts ECE 4× (0.167 → 0.040). Even harsher: the isotonic map fitted on our model transfers to Laya, taking its ECE 0.215 → 0.028 (better than Laya's own self-fit, 0.082). Caveats we state up front: a monotone map preserves ranking, so it fixes reported calibration, not error-detection ability; and the leaderboard's ECE column is far more a property of the dataset + post-processing than a model discriminator.

3. Teacher gold ≠ real outcomes. On a 100-invoice subset (200 decisions; ground truth computable by executing matching rules), teacher argmax disagrees with the executed outcome on 10/200 = 5% of cases — and the teacher was moderately-to-strongly confident on most of them (mean |p−0.5| ≈ 0.17 at disagreements). Under the execution oracle, both models improve but by different amounts — Laya 0.930 → 0.950 (+2.0 pp), ours 0.945 → 0.975 (+3.0 pp): the teacher label systematically under-credits the better model, widening the gap the leaderboard cannot see. A Clopper-Pearson seed experiment (not a formal certificate: 200 decisions come from 100 invoices and the threshold was searched on the same data) suggests 98% auto-commit coverage at true execution risk ≤ 4.6% @95% — the kind of statement teacher-calibration cannot make, and a preview of the audit kit we are building. No certificate exists at α = 0.02 at this n, pinning the sample-size cost of real certification.

4. Real-work gate test (live jev-latest API, rule-executor golds). Three workloads we actually run agents on (scripts/jev_real_work.py): invoice match-check 8/8, duplicate detection 7/8, support-ticket routing 8/8 (small-n capability demos, not accuracy estimates — the one-sided 95% bound of 8/8 is only ~69%), plus a shell-command risk gate for coding agents. Single-shot noul recall: 66.7% (0/15 false positives) — it misses stealthy commands (SSH-key injection, silent privilege add). Refactored into 5 sub-questions composed in code (scripts/jev_shell_decomp.py): 100% recall (15/15), 1/15 false positives. The engineering lesson: let the model emit calibrated local judgments, keep control flow in code.

Related work (we cite our competitors honestly)

Three groups got to parts of this problem before us, and we build on them:

• The dataset README itself states that scores measure teacher agreement, not correctness ("A better model can score worse here… The teacher missed a duplicate invoice whose ID literally matched a prior one"). We take that admitted weakness and make it measurable: an executable oracle, per-case disagreement evidence, and a risk-side reading.
• CounterProof Research (Sep 16) audited the vendor's curated 20-case slice and showed the two reference models disagree with each other — label instability at the top. We audit the full public benchmark with executable ground truth — label incorrectness at the bottom.
• jev-arena calibrates the live Jev API against analytically-known toy worlds (dice, urns). Noisy but real signal; different object from ours (the public benchmark and finetuned checkpoints).

What none of them provide: a reproducible, pip-installable audit that consumes any model's per-decision predictions and reports teacher-vs-executed-outcome disagreements, calibration transfers, and risk–coverage bounders. That is what we are building toward.

Contents

• weights/ — LoRA adapters (r=16, α=32, 33M trainable) on Qwen/Qwen3-4B-Instruct-2507 and Qwen3-1.7B. Note: checkpoints are plain torch.save({"cfg":…, "state":…}) dicts (not PEFT format); load via scripts/eval_qwen_d2b.py.
• results/ — all per-decision predictions (2,000 test decisions × 9 runs): d1_laya.json, runA_eval.json (s1), eval_s2/s3.json, runB_eval.json, eval_17A/17B.json, zero-shot.
• data/ — execution-oracle labels (200 invoice decisions), teacher/oracle disagreement cases, calibration logits.
• scripts/ — full pipeline: Laya eval, training (train_qwen_d2b.py), eval, isotonic, cross-model transfer, oracle seed, conformal certificates, sign-flip tests, SVG figures. Pure-stdlib analysis; torch only for train/eval.
• figures/ — Fig 1–4 (SVG, recomputable from results/).

Reproduce

# data: LocalLLaMA/typed-decisions (HF) -> snapshot dir
export TD_DATA=/path/to/typed-decisions-snapshot
python scripts/eval_qwen_d2b.py Qwen/Qwen3-4B-Instruct-2507 weights/qwen3-4b-lora-runA-s1.pt split960 out/runA
# analysis (no GPU needed), CTW points at repo root:
python scripts/seed_summary.py && python scripts/p2_conformal_seed.py

Splits: deterministic per protocol seed 20260921 (240/30/30 per workflow). Training protocol matches the ecosystem: soft CE + 0.1·Brier, per-type temperature on a held-out calibration split, 900 steps, ~70 min on a single L40.

5. Leaderboard rank ≠ real-work rank. We ran the first-ever evaluation of the commercial JEV API on its own community benchmark (this was never published): JEV scores 0.7335 vs Laya 0.7688 on teacher-gold accuracy (Laya wins, paired p≈0.001) — but JEV's confidence is calibrated (ECE 0.033 vs Laya's 0.215). Then we ran both on three rule-oracled real-work gates with identical fixtures:

gate (rule-executor gold)	JEV API	Laya 421M
invoice match / duplicate	8/8, 7/8	8/8, 7/8
ticket routing	8/8	7/8
shell risk-gate recall (15 dangerous)	10/15 single, 15/15 decomposed	3/15
false-block (15 benign)	1/15	0/15
latency per decision	~170 ms (API)	~5 ms (local L40)

Laya — the model that tops the leaderboard — misses 80% of genuinely dangerous agent shell actions, because it memorized teacher votes on four toy workflows rather than learning the concept. Speed only matters after you trust the answer; and trust, on executable gold, goes the other way.

Honest limits

• Teacher-consensus test split is still the scoring gold for claims 1–2 (that's the point, but claim 3 only covers a 200-decision rule-computable subset — full branch-execution gold is in progress).
• 1.7B is 4× Laya's size and no Qwen exists at ≤421M, so we report a size-controlled trend, not an iso-parameter tie. A sub-Laya decoder (SmolLM2-360M) on the identical pipeline is in flight to complete the scaling axis from below 421M.
• full1200 (runB) has an empty calibration split; its temperature is fitted on test by necessity — excluded from headline claims.
• The conformal number in claim 3 is a seed demo, not a deployable certificate (see the caveat inline).

Roadmap (what "installable audit" means)

gtg audit --cases cases.jsonl --predictions preds.jsonl --oracle invoice-v1 --out report/ → per-case teacher-vs-oracle disagreements with execution evidence, calibration transfer, risk–coverage bounders. CPU-only, JSONL in, HTML+JSON out. First target: any engineer already running Laya gets a report in 10 minutes that changes their label triage, threshold, or release decision.

If this changes how you read a "calibrated confidence" leaderboard number, star it.