hniyorurema/burnettbench-scope-classifiers

BurnettBench Scope Classifiers (OPT4 / OPT4v2)

DistilBERT-base-uncased classifiers that predict whether a user query falls within the scope of an institution's llms.txt V2 guide. Trained as part of BurnettBench Study 2 — Scope Coverage Effects in llms.txt V2 (Niyorurema & Estella, 2026).

• Paper: arXiv link TBA
• Replication repo: https://github.com/happy-tcu/burnettbench
• Institution: Department of Communications and Strategy, Anne Burnett Marion School of Medicine, TCU

What's in this repo

Eight DistilBERT checkpoints, one per institution per experiment:

File	Experiment	Institution	Training data
opt4_gate_bakame.pt	OPT4	Bakame AI	233 human-labeled GT (leave-one-out)
opt4_gate_mayo.pt	OPT4	Mayo Clinic	233 human-labeled GT (leave-one-out)
opt4_gate_rice.pt	OPT4	Rice University	233 human-labeled GT (leave-one-out)
opt4_gate_uscis.pt	OPT4	USCIS	233 human-labeled GT (leave-one-out)
opt4_gate_ve.pt	OPT4	Vinson & Elkins	233 human-labeled GT (leave-one-out)
opt4v2_gate_bakame.pt	OPT4v2	Bakame AI	506 (233 human + 273 AI-generated)
opt4v2_gate_mayo.pt	OPT4v2	Mayo Clinic	506 (233 human + 273 AI-generated)
opt4v2_gate_ve.pt	OPT4v2	Vinson & Elkins	506 (233 human + 273 AI-generated)

OPT4v2 USCIS and Rice checkpoints were not retained from the original training run — see paper §VIII.D.

Headline numbers

Variant	Mean OUT_V2 accuracy	Mean IN_V2 recall
A (no guide, baseline)	29.5%	—
V2 (full-file inject)	23.3%	88.2%
OPT4	29.9%	71.5%
OPT4v2	26.2%	76.8%

Both variants recover close to the no-guide baseline on out-of-scope accuracy, but neither matches V2's in-scope recall. The paper documents this as a data ceiling (~70% CV accuracy on 263 labeled examples) and recommends publisher-declared scope (covers: / excludes: spec fields) as the structural fix.

Intended use

Research only. These checkpoints are shared for replication of BurnettBench Study 2. The paper's own evaluation shows they do not meet a production bar — do not ship them in user-facing systems.

How to use

from huggingface_hub import snapshot_download
import torch
from transformers import DistilBertTokenizerFast, DistilBertForSequenceClassification

local = snapshot_download(repo_id="hniyorurema/burnettbench-scope-classifiers")
tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-uncased")
model = DistilBertForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2
)
model.load_state_dict(torch.load(f"{local}/opt4_gate_bakame.pt"))
model.eval()

Full inference code: scripts/burnettbench_eval_opt4.py in the replication repo.

Training

See scripts/burnettbench_retrain_opt4.py and scripts/burnettbench_opt4_263.py in the replication repo. GPU recommended (originally run on NVIDIA GB10 / DGX Spark).

Citation

@techreport{niyorurema2026burnettbench2,
  title       = {Scope Coverage Effects in llms.txt V2},
  author      = {Niyorurema, Happy Herman and Estella, Maricar Hastings},
  institution = {Anne Burnett Marion School of Medicine, Texas Christian University},
  year        = {2026},
  type        = {Technical Report}
}

License

MIT for code and model weights. Training data (probes) released under CC BY 4.0 in the replication repo.

Contact

• Happy Herman Niyorurema — h.niyorurema@tcu.edu
• Maricar Hastings Estella (Research Supervisor) — m.estella@tcu.edu