shivamfet/slm-125m-qa

The shivamfet/slm-125m-base base model, supervised fine-tuned for closed-book question answering: a question goes in, an answer comes out, with no passage supplied. It answers from its weights, or declines.

This is a sibling of shivamfet/slm-125m-instruct, not a successor. That model reads a passage you give it; this one has no passage to read. Both start from the same pretrained base.

Read this before using it

Roughly half of its confident answers are fabricated. Measured on 327 held-out questions: 48.6% contain invented specifics — figures, dates, holdings, party names that are plausible, well-formed, and wrong. Asked what percentage of a company's products were bought directly from manufacturers, it answers "approximately 90%" where the truth is 72%: same sentence shape, same confidence, fabricated number.

This is not a fixable bug in the fine-tune, it is what 125M parameters can do. A model this size holds on the order of 2 bits per parameter; the specific holding of a 1978 state case was never reliably in there. Fine-tuning taught it the form of a confident factual answer, which it now produces regardless of whether it has the fact.

Use it to study abstention behaviour and small-model limits. Do not use it as a source of facts. For real question answering at this scale, supply the passage and use the grounded sibling model above.

Evaluation

357 held-out questions, never trained on. Answers graded by gpt-5.4-mini against reference answers.

Answerable questions (n=327)

model	correct	partial	wrong	refused	hallucinated
base (no SFT)	0.3%	0.6%	99.1%	0.0%	1.5%
this model	4.0%	16.5%	79.5%	4.6%	48.6%

Unanswerable questions (n=30) — questions with no determinate answer ("What did the Court hold?" with no case named). Declining is correct.

model	refuses
base (no SFT)	0.0%
this model	80.0%

Read those two tables together. The model declines 80% of unanswerable questions while declining only 4.6% of answerable ones, so it learned the distinction rather than a blanket habit of ducking. The base model never declines anything, so this behaviour comes entirely from the 714 refusal examples.

Note what the refusal training actually covers: the refusal examples are questions that are visibly underspecified. The model learned "decline when the question does not pin down an answer." It did not learn where its own knowledge ends — nothing fires on "what percentage did company X purchase directly," which looks perfectly well-formed and simply needs a fact it does not have. That gap is why the hallucination rate stays at 48.6% despite the refusal training working.

The base model's 1.5% hallucination rate is not a virtue: it barely produces answers at all (99.1% wrong is mostly incoherent continuation). The fine-tune traded incoherence for confident invention.

Training data

7,141 pairs, built in two halves:

Mined	4,946 questions from the grounded QA set that stand alone once the passage is removed, judged self-contained by gpt-5.4-mini
Generated	2,195 fresh pairs written to name their own subject, then passed through a self-containment gate and a faithfulness gate
Refusals	714 (10%) — underspecified questions paired with four short declines
Sources	U.S. case law 3,482 · FineWeb-Edu 2,051 · SEC filings 1,608
Dedup	exact + MiniLM cosine 0.92 across both halves jointly

Training

Method	Supervised fine-tuning from the pretrained base, answer-only loss mask
Schedule	4 epochs, 1×H100, AdamW, cosine LR 2e-5 → 2e-6, 256-token rows
Selection	Epoch 2 by held-out val loss; epochs 3–4 overfit
Val perplexity	7.80 (answer tokens)
Architecture	Llama, 12L / 768d / 12h, 16,384 vocab, 1024 context

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("shivamfet/slm-125m-qa")
model = AutoModelForCausalLM.from_pretrained("shivamfet/slm-125m-qa")

messages = [
    {"role": "system", "content": "Answer the question."},
    {"role": "user", "content": "What health problems can high lead exposure cause in young children?"},
]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
enc = tok(prompt, return_tensors="pt", return_token_type_ids=False)

eos = tok.convert_tokens_to_ids("<|eos|>")
out = model.generate(**enc, max_new_tokens=96, do_sample=False,
                     eos_token_id=eos, pad_token_id=eos)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Greedy decoding, no repetition penalty — the same setup the evaluation above used. Sampling raises the hallucination rate further.