Empirical discourse in congressional hearings: demo ensemble
Three fine-tuned encoders that judge whether a sentence spoken in a U.S. congressional hearing appeals to empirical evidence. They back the live demo at maximg1/empirical-discourse.
Ilay Abramovich & Maxim German, Workshop on Deep Learning, Tel Aviv University.
What is in here
| directory | encoder | class weighting |
|---|---|---|
checkpoints/roberta_none_seed42 |
RoBERTa | none |
checkpoints/deberta_none_seed42 |
DeBERTa | none |
checkpoints/modernbert_sqrt_inverse_seed42 |
ModernBERT | sqrt-inverse |
Plus the two files that describe the instrument rather than its weights:
phase2_config.json: the encoders, seeds, class-weight strategies andinput_formatthe producing run froze beside its checkpoints. The demo reads the ensemble's composition from this rather than assuming defaults.v2_threshold.json: the tuned operating point, 0.60, with the sweep it came from.
Scoring averages the three probabilities and calls a sentence empirical at 0.60 or above.
This is a subset, not the published instrument
The paper's silver labels come from 30 models: 3 encoders x 5 seeds x 2 class-weight arms. This repo holds one seed of the macro-F1 arm, because 30 base transformers is not a demo.
The 0.60 threshold was tuned on the full 30-model ensemble's out-of-fold predictions, so here it is inherited rather than re-derived. Scores from these three models track the full ensemble closely on the sentences we have checked, but they are not the same instrument and should not be cited as it.
Training
Fine-tuned on gold_combined_broad: the legacy audit set's training half (congresses 105-114,
coded by the original paper's coders) plus the modern gold's training half (congresses 115-118,
our coders), under the broad construct. Held-out evaluation uses the two halves those splits
reserve, so nothing in the demo's five curated sentences was trained on.
Intended use and limits
Built for research on U.S. congressional hearing transcripts, 1997-2025. It scores one sentence at a time, out of context, and was trained on floor-style speech from that corpus. Expect it to degrade on other genres. It measures whether a sentence appeals to evidence, not whether the evidence is real or the claim true.
Source
Live demo: maximg1/empirical-discourse. The pipeline that produced these checkpoints is not public yet; the paper is in preparation.