shivamfet/slm-125m-instruct

The shivamfet/slm-125m-base base model, supervised fine-tuned into a grounded question-answering model: give it a passage and a question, and it answers using only what's in the passage.

🌐 Live demo: https://slm-125m-sft-shivamk.vercel.app

What it does

Reads a source passage and answers extractive questions about it — who, what, where, which claims, what a clause requires, what a court held. It pulls the answer from the passage and states it directly, rather than answering from memory.

What it does not do

At ~125M parameters this is a small model with real limits, stated honestly:

• No arithmetic. "6% of $50,000", revenue growth deltas — it cannot compute.
• Little world knowledge. Remove the passage and it has almost nothing to say; it is a reader, not a knowledge base. Pair it with retrieval (RAG) for real use.
• Not RLHF-aligned; do not rely on it for legal or financial advice.

Training

Method	Supervised fine-tuning (answer-only loss mask)
Data	13,774 grounded QA pairs, teacher-generated + LLM-judged for groundedness, semantically deduplicated
Sources	U.S. case law, SEC filings, FineWeb-Edu passages
Schedule	2 epochs, 1×H100, ~4 min, AdamW, cosine LR 2e-5→2e-6
Selection	Epoch 2 chosen by held-out val loss (later epochs overfit)
Val perplexity	2.60 (answer tokens)
Architecture	Llama, 12L / 768d / 12h, 16,384 vocab, 1024 context

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("shivamfet/slm-125m-instruct")
model = AutoModelForCausalLM.from_pretrained("shivamfet/slm-125m-instruct")

passage = ("The court affirmed the conviction but remanded for resentencing "
           "because the trial judge improperly considered the defendant's silence.")
question = "Why did the court remand the case?"

messages = [
    {"role": "system", "content": "Answer the question using only the passage."},
    {"role": "user", "content": passage + "\n\n" + question},
]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
enc = tok(prompt, return_tensors="pt", return_token_type_ids=False)

# Greedy decoding, NO repetition penalty: extractive QA must copy spans from the
# passage, and a penalty >1 pushes the model off the grounded answer.
eos = tok.convert_tokens_to_ids("<|eos|>")
out = model.generate(**enc, max_new_tokens=128, do_sample=False,
                     eos_token_id=eos, pad_token_id=eos)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True).strip())
# -> "The court remanded because the trial judge improperly considered the silence."