hamdallah/budgeted-exit-modernbert-base-civil-comments

Budgeted early exit for ModernBERT-base (Civil Comments moderation)

Stop computing when the decision is settled. This is a full fine-tune of the 150M-parameter encoder ModernBERT-base with a trained classification head at seven exit layers and a frozen, risk-calibrated exit policy. It answers four moderation questions about a comment (threat, identity attack, insult, obscene language), composes them with a rule, and leaves the network early when the rule's outcome can no longer change within an explicit error budget.

On a locked, pre-registered test it keeps decision quality within 0.35 points of the full model while running 5.88 of 22 layers on average, and a runtime that really stops is 2.64x faster in measured throughput.

Research release. Trained and evaluated on one dataset (Civil Comments) and one seed. Read the limitations before using it.

Results

Rule: threat or identity_attack or (insult and obscene) (about 1.2% positive). Official Civil Comments test split, 96,862 comments, read once after the policy, weights, code and test file were hashed. Pass rule fixed in advance: the 95% interval of the change in balanced accuracy must exclude a 1-point loss.

System	Balanced accuracy	Change vs full depth [95% CI]	Mean depth	Throughput (A100, 16k dev texts)
Full depth	93.66%	-	22	2,764 texts/s
Static truncation at layer 16	93.86%	+0.20 [-0.18, +0.60]	16	3,674 texts/s (1.33x)
Frozen exit policy (this release)	93.31%	-0.35 [-0.65, -0.06]	5.88	7,294 texts/s (2.64x)

Balanced accuracy of each exit on the validation split, if every comment stopped there:

Exit layer	4	7	10	13	16	19	22
Balanced accuracy	91.45%	92.32%	92.27%	92.79%	93.35%	93.38%	93.48%

The policy changed 1.65% of the full model's positive decisions (budget 2%) and 0.90% of its negative decisions (budget 1%). The loss of 0.35 points is small but statistically detectable; it comes from a higher false-positive rate (7.0% to 7.7%) at unchanged recall.

How it works

1. Each exit head outputs a probability per question; Platt scaling maps it to the natural label distribution.
2. The rule's probability is computed from the four probabilities (weighted model count, independence assumed).
3. Per exit, an isotonic map gives the probability that the full-depth decision would be positive. It is fitted without labels.
4. One dial per direction, scaled by the share of layers still to run, decides who leaves. The dials are set by conformal risk control against the error budget. Everything is stored in frozen_policy.json as plain numbers.

Files

• model.safetensors: the full fine-tuned encoder and the seven exit heads, 153.2M parameters.
• config.json: base model, exit layers, questions
• frozen_policy.json: the exact policy evaluated on the locked test (Platt parameters, isotonic maps, dials, cut-offs)

Usage

The loading and runtime code lives in the project repository: github.com/hamdallah90/decision-aware-inference (folder budgeted_exit/).

git clone https://github.com/hamdallah90/decision-aware-inference.git && cd decision-aware-inference
pip install -r requirements.txt && cd budgeted_exit

from huggingface_hub import snapshot_download
from hub import load                      # budgeted_exit/hub.py
from runtime_exits import Runtime         # budgeted_exit/runtime_exits.py

folder = snapshot_download("hamdallah/budgeted-exit-modernbert-base-civil-comments")
model, encode, _ = load(folder)           # logits = model(ids, mask) -> [batch, exits, 4 questions]
runtime = Runtime(tag=None, folder=folder)
decisions, exit_layers = runtime.dynamic(encode(["your comment here"]))

Training

Full fine-tuning, mean-pooled readout, on 55,271 Civil Comments training rows (an enriched sample with extra rare positives), binary cross-entropy summed over all exits, two epochs, one seed. Calibration used a separate uniform sample of 99,140 training rows.

Limitations

• One dataset, one seed, one hand-written rule. Re-measure on your own data before relying on it.
• Throughput was measured with requests available in bulk. Latency under live traffic was not measured.
• The error budget is controlled in expectation on data like the calibration sample; it is not a per-request guarantee and it does not cover distribution shift.
• English only. The training data contains offensive language; the model is for moderation research, not for generating content or for automated decisions about people without human review.
• The runtime calls Hugging Face layers directly and was tested with transformers 5.17.

License

Apache-2.0, following the base model. Dataset: Civil Comments (CC0-1.0).