nagbhaskar55/slm125mlive-base

SLM125mLIVE base (125M)

A 125.8M-parameter Llama-architecture base language model pretrained from scratch on US court opinions, SEC filings, and educational web text. It is a raw pretrained model: it is not instruction-tuned or aligned, and it is not a source of legal or financial advice.

Model

Architecture	LlamaForCausalLM, 12 layers, hidden 768, 12 heads (MHA), SwiGLU 3072, RoPE θ=10000, RMSNorm
Parameters	125,847,552 (tied input/output embeddings)
Context length	1024 tokens
Tokenizer	byte-level BPE, vocab 16,384, trained on this corpus
Special tokens	<|bos|>=0, <|eos|>=1, <|pad|>=2, <|unk|>=3 (plus unused chat tokens)

Documents were packed into 1,024-token windows separated by <|eos|>; no BOS token was used in training, so prompts work best when prefixed with <|eos|>.

Training data

2.04B unique training tokens (plus a 20.6M-token held-out validation split: every 100th window).

Source	Documents	Train tokens	Share
HFforLegal/case-law	206,684	715M	35%
PleIAs/SEC	45,035	860M	42%
HuggingFaceFW/fineweb-edu	418,405	464M	23%

Processing: line-level filtering, boilerplate stripping, and length, repetition, language, and OCR-garble filters. Then MinHash near-duplicate removal (case law) and exact-duplicate removal. Documents sharing a 13-gram with the CaseHOLD test set were removed (24,177 documents) so the CaseHOLD result below is not contaminated.

Training

Tokens seen	6.12B (11,667 steps, 3.0 epochs)
Batch	524,288 tokens (512 × 1024)
Optimizer	AdamW (β1=0.9, β2=0.95, wd=0.1), grad clip 1.0
LR schedule	200M-token warmup to 0.0006, cosine to 6e-05
Precision	bf16 autocast, torch.compile, DDP
Hardware	8× NVIDIA H100 on Modal, 2.26M tokens/s

Evaluation

Held-out validation (all 20.6M validation tokens):

Source	Loss	Perplexity
case-law	2.189	8.92
sec	1.599	4.95
fineweb-edu	3.105	22.31
All	2.149	8.57

Validation loss during training (2,048-window subset):

Step	Val loss	Perplexity
1,000	2.784	16.19
2,000	2.530	12.55
3,000	2.424	11.29
4,000	2.361	10.60
5,000	2.317	10.14
6,000	2.282	9.79
7,000	2.251	9.49
8,000	2.227	9.27
9,000	2.208	9.10
10,000	2.193	8.97
11,000	2.183	8.87
11,667	2.179	8.84

CaseHOLD (LexGLUE case_hold test, 3,600 examples, 5-way multiple choice, zero-shot). Each candidate holding is scored by its log-likelihood as the continuation of the citing text that precedes it (lm-eval-harness style):

Metric	Accuracy
acc (log-likelihood)	12.0%
acc_norm (per-byte log-likelihood)	17.7%
Chance	20%

The model does not solve CaseHOLD zero-shot. acc matches the "always pick the shortest holding" baseline (11.4%): correct holdings are longer on average (168 vs 152 characters), and summed log-likelihood favors shorter text. acc_norm removes most of that length effect but stays below chance. A lexical baseline (pick the holding that shares the most words with the context) reaches 35.0%. Treat CaseHOLD as a fine-tuning target for this model, not a zero-shot score.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("nagbhaskar55/slm125mlive-base")
model = AutoModelForCausalLM.from_pretrained("nagbhaskar55/slm125mlive-base")
ids = tok("<|eos|>The court held that", return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

• Small base model: it produces fluent-looking but frequently incorrect or invented text, including invented case names, citations, statutes, and financial figures.
• Domain skew: US court opinions (some OCR noise) and SEC filings dominate; general knowledge is limited.
• No safety tuning, instruction tuning, or RLHF.
• License: other. Check the licenses and terms of the source datasets before any use.

Files

• metrics.jsonl: training log.
• training_summary.json: run summary.
• eval_results.json: full evaluation output, including sampled continuations.