slm125mLIVE-base

A 125.8M-parameter Llama-architecture language model pretrained from scratch on a legal and financial corpus. Base model, no instruction tuning.

Corpus

Source	Content	Tokens	Share
HFforLegal/case-law	US court opinions	719.9M	28.8%
PleIAs/SEC	SEC filings	864.2M	34.5%
HuggingFaceFW/fineweb-edu	Educational web text	918.9M	36.7%

2.50B unique tokens. Cleaned, MinHash-LSH deduplicated, and decontaminated against CaseHOLD / LexGLUE case_hold by 13-gram overlap - 24,002 case-law documents (10.3% of that source) removed.

Architecture

Parameters	125,847,552
Layers / hidden / heads	12 / 768 / 12 (MHA)
Context	2048
Vocab	16,384 (byte-level BPE trained on this corpus)
Norm / activation	RMSNorm (pre-norm) / SwiGLU
Position	RoPE (theta 10,000)
Embeddings	tied

<|user|>, <|assistant|> and <|system|> are reserved in the vocabulary, so post-training needs no embedding resize.

Training

WSD schedule (warmup - stable - decay), AdamW, bf16 on 4xH100.

Tokens seen	6.25B (~50 per parameter)
Steps	11,920
Validation loss	2.2734
Validation perplexity	9.71

The earlier checkpoint, and why it is worth knowing about

An earlier revision of this repo held a checkpoint at perplexity 11.06. Further pretraining brought it to 9.71 - a 12% improvement, for about $11 of GPU time.

Judged answer quality moved from 0.38 to 0.47 out of 10, a change whose confidence interval includes zero. The perplexity gain was real and the quality gain was not measurable. Post-training moved the same metric by more than 2.5 points for under a dollar.

Both checkpoints are reachable: this revision is the 9.71 model, and commit 560c780c55 is the 11.06 one, so the comparison can be reproduced rather than taken on trust.

Leaderboard

stage	qa	instruction	grounded	refusal	mean	95% CI
base-v1 (ppl 11.06)	0.33	0.23	0.66	0.37	0.38	[0.27, 0.52]
base (ppl 9.71)	0.40	0.40	0.88	0.26	0.47	[0.35, 0.61]
+ instruction tuning	1.85	1.20	1.81	1.11	1.49	[1.31, 1.69]
+ SFT v2 (refusals)	1.65	1.32	3.50	5.89	2.97	[2.56, 3.40]
+ SFT v3	1.90	1.40	3.84	5.29	2.99	[2.61, 3.39]
+ DPO	2.00	1.35	3.38	6.23	3.13	[2.71, 3.56]
+ RAFT	1.75	1.30	2.97	6.89	3.12	[2.68, 3.56]

147 held-out questions, decontaminated against the SFT training passages, scored 0-10 by an LLM judge. Every model was scored on the same item by the same judge with model identities hidden and re-shuffled per item, so a gap between two rows is a gap between two models rather than between two judging runs. Intervals are 10,000-sample bootstraps.

Paired step-over-step change on the same items:

step	delta	95% CI	verdict
base-v1 to base	+0.09	[-0.08, +0.25]	not distinguishable from noise
base to instruct	+1.02	[+0.80, +1.24]	real
instruct to SFT v2	+1.48	[+1.06, +1.92]	real
SFT v2 to DPO	+0.16	[+0.02, +0.31]	real
DPO to RAFT	-0.01	[-0.35, +0.32]	not distinguishable from noise

Intended use

Research and further post-training. At 125M parameters this is a base for building pipelines, not an assistant. Expect locally fluent, frequently incoherent generation - it scores 0 on 63% of held-out questions. Not legal or financial advice.