ehuang5/olmo3-calibration-models

Olmo 3 7B DPO calibration study: trained checkpoints

DPO checkpoints from a study of why DPO makes Olmo 3 7B Instruct overconfident on multiple-choice questions. Every model here is trained from allenai/Olmo-3-7B-Instruct-SFT on Ai2's Dolci-Instruct-DPO preference pairs with Ai2's open-instruct code.

• Code, results and the write-up ("Why Does DPO Make Olmo 3 Overconfident?"): github.com/edwin-huang/olmo3-dpo-calibration
• Raw evaluation outputs: ehuang5/olmo3-calibration-results

These are research checkpoints for reproducing the study's measurements. They were not tuned or safety-tested for use. "Agreement-weighted" DPO multiplies each pair's loss by sigmoid(-5 x the SFT model's preference margin), rescaled to average 1, so pairs the SFT model already gets right count less.

Folders

Each folder is a full model (model.safetensors, config, tokenizer, chat template). Load one with subfolder=, e.g. AutoModelForCausalLM.from_pretrained("ehuang5/olmo3-calibration-models", subfolder="q25_weighted").

Folder	Training
q25_plain/	Plain DPO on 25% of the Instruct DPO pairs (65,000), Ai2's recipe (LR 1e-6, 508 steps)
q25_weighted/	Agreement-weighted DPO, same data and recipe
full_weighted/final/	Agreement-weighted DPO on all 259,922 pairs, Ai2's recipe (2,031 steps); compare with Ai2's released Olmo-3-7B-Instruct-DPO
full_weighted/snap_step_*/	Snapshots of that run at steps 406, 812, 1218, 1624 and 2030
wave/wy_*_s{1,2,3}/	Small tests from the pre-registered replication: 8,192 pairs, LR 4e-6, 64 steps, seeds 1-3

Small tests in wave/:

Name	Loss	Training set
wy_plain	Plain DPO	Mixed pairs
wy_w5	Agreement-weighted DPO	Mixed pairs
wy_simpo	SimPO (no reference model)	Mixed pairs
wy_simporef	SimPO with DPO's SFT reference added	Mixed pairs
wy_hi / wy_lo	Plain DPO	99% / 18% of pairs already right under SFT, matched on source and length
wy_ipohi / wy_ipolo	IPO (tau = 1)	Same two sets
wy_easy	Plain DPO	"Easy" pairs (high margin under a DPO-trained model)
wy_w5hi	Agreement-weighted DPO	The 99% set (follow-up 1)
wy_hiall / wy_w5all	Plain / agreement-weighted DPO	The 99% set minus its disagreed pairs (follow-up 2; seed 1 only)

License

Apache-2.0, following the base model. The training data is Ai2's Dolci preference data (ODC-BY); please credit Ai2's Olmo 3 release and the Dolci datasets when using these models.