slm-125m-instruct

Instruction-tuned from thesreedath/slm-125m-base on 9,071 legal/financial Q&A pairs generated from a case-law + SEC-filing corpus.

Training

• Base: 125.8M params (12L / 768d / 16,384 vocab)
• Data: 9,071 Q&A pairs, LLM-judged (90.7% pass rate) from 19,775 raw
• Method: full SFT, prompt tokens masked from the loss (answer-only training)
• Hyperparameters: lr 2e-5, cosine decay, 3 epochs, bf16, 1xA100
• Best val loss: 1.914

Chat format

<|bos|><|system|>{system}<|user|>{question}<|assistant|>{answer}<|eos|>

Note on token ids

The base repo's config.json declares eos_token_id: 2, but id 2 is <|pad|> in the tokenizer -- <|eos|> is id 1. This model ships the corrected ids (bos=0, eos=1, pad=2), so generation terminates properly.

Limitations

125M parameters trained on ~9k pairs. It answers in the right form and stops cleanly, but factual accuracy is limited -- verify anything load-bearing.