slm-125m-dpo

125M-parameter legal/financial model, preference-optimized with DPO on top of prajwal55/slm-125m-sft.

Abstention on held-out unanswerable questions rose from 0.0% to 60.3%. It also declines on 26.3% of questions the passage does answer, and token F1 fell from 0.590 to 0.456.

This is a research artifact, not a usable model. Refusing a quarter of answerable questions is not an acceptable operating point. It is published because the measured trade-off is the result, not because the checkpoint is good.

What this phase was trying to fix

The supervised model answers questions its passage does not support — it invents a plausible fact instead of declining. That failure has no gradient in supervised fine-tuning, because the training data contains 8,000 examples of answering and none of declining.

Two axes only: abstention (decline when unsupported) and faithfulness (when supported, keep every claim traceable to a span).

Results

All checkpoints below were trained from the same SFT policy on the same 1,210 frozen preference triples, so the only variable is the algorithm and its hyperparameters. Held-out set is 121 unanswerable prompts, carved before any training and disjoint by source passage. Greedy decoding.

Metric	SFT baseline	DPO beta=0.1, 1 epoch	DPO beta=0.1, 2 epochs	DPO beta=0.3	DPO beta=0.5	PPO
Abstention (held-out, 121)	0.0%	60.3%	66.1%	0.0%	0.0%	0.0%
False abstention	0.0%	26.3%	34.7%	0.0%	0.0%	0.0%
Token F1 (1,003 answerable)	0.590	0.456	0.400	0.585	0.592	0.563
Exact match	11.4%	6.6%	4.8%	11.3%	11.8%	11.9%
Numeric fidelity	98.1%	99.2%	99.2%	98.8%	99.0%	93.5%
KL from SFT	0.0000	0.3159	0.6792	0.0227	0.0166	0.0628

Abstention alone is gameable — a model that declines everything scores 100%. Read it beside false abstention or not at all.

Why this is the 1-epoch checkpoint

The configured run was 2 epochs. Its own epoch-1 checkpoint strictly dominates the finished one:

	abstention	false abstention	ratio	token F1	KL from SFT
epoch 1 (this model)	60.3%	26.3%	2.29	0.456	0.316
epoch 2	66.1%	34.7%	1.90	0.400	0.679

The second epoch bought +5.8 points of abstention for +8.4 points of false abstention — a losing trade — while doubling the drift from the SFT policy. (Ratio is abstention ÷ false abstention; 1.0 means declining at random.)

The headline finding

DPO can consume off-policy chosen text; PPO cannot. With a 0.00% base rate for the target behaviour, DPO was handed decline text and moved likelihood mass onto it directly. PPO could only reinforce what it sampled, and it never sampled a decline.

Abstention has a KL threshold, not a smooth trade-off. Nothing below KL ~0.06 from the SFT policy abstains at all; token F1 falls monotonically with KL regardless of which algorithm caused the drift.

Usage

from transformers import AutoTokenizer, LlamaForCausalLM

tok = AutoTokenizer.from_pretrained("prajwal55/slm-125m-dpo")
model = LlamaForCausalLM.from_pretrained("prajwal55/slm-125m-dpo")

prompt = ("<|bos|><|system|>You are a legal and financial assistant. Answer the "
          "question using only the passage provided. Be precise and concise."
          "<|user|>" + passage + "\n\nQuestion: " + question + "<|assistant|>")
ids = tok(prompt, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=160, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))

Resolve stop and pad ids from the tokenizer, not from LlamaConfig defaults — this tokenizer is bos=0, eos=1, pad=2.

Limitations

• Not usable closed-book. It is a grounded-extraction model over a supplied passage, at 125.8M parameters.
• Declines on more than a quarter of answerable questions.
• Trained on ~1,200 preference pairs. Nothing here is a general claim about DPO or PPO — it is a claim about these algorithms meeting a target behaviour with a zero base rate.
• Checkpoint: /data/checkpoints/dpo/beta0.1/epoch-1.