gliner-datause-v3-probe

Multi-origin probe of rafmacalaba/gliner_datause (proposals: NAMED_DATA, DESCRIPTIVE_DATA, VAGUE_DATA). The Luna v2.4 keep/drop boundary (v2.4-patched-2026-09-05) is read off the frozen representation with a small MLP head on [start; end; mean; ±64-token window] span features: extractor proposes, head disposes.

Trained on gold v2 (2,728 Luna-verdict spans over rafmacalaba/datause-extracted-sample, gliner2 config: FCV PADs, JDC operational, refugee PADs, ReliefWeb, JAD PADDY, PRWP) plus 26,618 campaign-v2 Luna single-judge labels drawn origin x specificity x 0.1-decile across the full extraction corpus — 29,346 spans total, doc-disjoint 70/15/15 split, every origin in train, val, and holdout, no passage crosses splits. The full candidate pool (2,729 sample spans + 27,126 campaign spans, passages included) is versioned alongside the splits in rafmacalaba/datause-probe-v3 (candidates config). BCE targets are label-smoothed by 0.1 (eps/2) to curb overconfident scores; scores are ranking-oriented, not calibrated probabilities.

Results (holdout, gold = Luna distillation verdicts)

• head AUROC: 0.8867 (distillation fidelity: how well the head carries the Luna v2.4/v2.5 keep/drop boundary)
• head best-F1 thr 0.3: precision 0.7481 · recall 0.9312 · f1 0.8297
• val AUROC: 0.8915

Per origin

origin	n	keep	AUROC	head best-F1 @ thr
fcv_pads_east_africa	1123	431	0.9110	0.7961 @ 0.5
general_prwp	1859	1299	0.8611	0.8721 @ 0.3
jad_paddy_docs	169	158	0.9574	0.9783 @ 0.3
jdc_operational	242	115	0.9068	0.8128 @ 0.5
refugee_pads	460	210	0.9231	0.8317 @ 0.6
reliefweb	804	447	0.7665	0.7586 @ 0.3

Human check (never trained on)

set	n	keep	AUROC	head best-F1 @ thr
human473	473	315	0.8508	0.8477 @ 0.3
annotator190	190	110	0.9285	0.8772 @ 0.4
jdc283	283	205	0.7980	0.8620 @ 0.1

Labels: gold-v2 verdicts + campaign-v2 Luna single-judge labels (v2.4-patched-2026-09-05, keyed single-span batch, prompt v2.5-draft-2026-09-06); they are a distillation target, not human gold. 7 spans quarantined (unresolved echo) + 1 bad_json excluded — see quarantine.jsonl in rafmacalaba/datause-probe-v3. Head: head.pt; per-span holdout predictions: holdout_predictions.jsonl (key, origin, label, probe_score, head_score); full sweeps: holdout_metrics.json. Human external check: human473_predictions.jsonl (473 annotator190/jdc283 spans, never trained on) with per-set AUROC + best-F1 in holdout_metrics.json → human473 and the table above.