slm-125m-base

A 125.8M-parameter Llama-style causal LM pretrained from random weights on a legal-first corpus. No distillation, no fine-tuning from an existing checkpoint: the tokenizer, the corpus and the weights were all built from scratch.

Architecture

Field	Value
parameters	125,847,552
layers	12
hidden size	768
attention heads	12 (head dim 64), MHA
MLP	SwiGLU, inner 3072
norm	RMSNorm, pre-norm
positions	RoPE (theta 10000)
context length	1024
vocab	16,384 (byte-level BPE, trained on this corpus)
embeddings	tied (input = output)

Because embeddings are tied, model.safetensors contains no lm_head.weight — it is shared with the input embedding. That is expected, not a truncated upload.

Training data

Legal-first mix, cleaned and deduplicated from scratch (~40/40/20):

Source	Role
HFforLegal/case-law (US)	US case law, strict OCR gate
PleIAs/SEC	SEC filings
HuggingFaceFW/fineweb-edu	general web, kept as a fluency floor

Pipeline: quality/language/OCR filtering, MinHash near-dedup, exact dedup, and 13-gram decontamination against the CaseHOLD benchmark via LexGLUE's case_hold config, which is held out. That removed 24,002 case-law and 175 SEC documents. Result: 2.04B training tokens packed into 1024-token windows.

Contamination caveat (measured, please read before benchmarking)

Decontamination used LexGLUE's case_hold config (480,908 13-grams). A later audit found the standalone casehold/casehold dataset contributes 796,649 additional 13-grams that were never screened — LexGLUE covers only 38% of them. 5.1% of the retained corpus matches at least one (case-law 5.69%, SEC 2.35%).

Treat that 5.1% as an upper bound rather than a leakage rate: SEC filings should contain no CaseHOLD case-law text, so their 2.35% is largely innocent collision on legal boilerplate, and a 13-gram overlap in a citing context is not the same as leaking CaseHOLD's multiple-choice answer. Still, do not quote a CaseHOLD score for this model and treat any LexGLUE case_hold number as optimistic. The honest metric for this model is held-out perplexity on its own validation split, which is what is reported above.

Training

Knob	Value
tokens seen	2.04B (1 epoch)
steps	3,890
hardware	8x H100, single node, DDP
wall clock / cost	~13.4 min, $7.05
optimizer	AdamW (0.9, 0.95), wd 0.1
LR	0.0006 -> 6e-05, cosine
batch	~524K tokens/step
precision	bf16

Held-out validation perplexity: 10.42.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("narendraalluri/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("narendraalluri/slm-125m-base")

ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80, min_new_tokens=40, do_sample=True,
                     temperature=0.8, top_p=0.95, top_k=50, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

Pass min_new_tokens — a base model at this scale will happily emit EOS immediately.

Honest limitations

This is a base completer, not an assistant. It has never seen instruction data or a chat template (the tokenizer carries <|user|>/<|assistant|> tokens, but training never used them), so it will not follow instructions or answer questions.

It writes fluent legal and financial prose with correct register and citation form, and it invents the content: case names, reporter numbers and dollar figures are plausible fabrications. Perplexity is the honest metric here; factual grounding needs retrieval. Do not use it for legal advice.