slm-from-scratch-phase3-scaling

Three GPT-style language models, hand-written and trained from scratch in PyTorch on a single RTX 5070 Ti, produced by a controlled scaling-law experiment: same architecture family, varying only model size and tokens seen. No HuggingFace transformers model classes were used — every component (attention, MLP, blocks, embeddings, training loop) was hand-coded.

This is Phase 3 of a larger build documented at github.com/Ishaanred/slm-from-scratch. See docs/phase3/results.md for the full scaling-law writeup and plots. These are portfolio and learning artifacts, not production models.

The three checkpoints

Checkpoint	Params	Tokens	Tokens/param	Best val loss
75m-500m-tokens/	69.4M	500M	~7.2x	4.0773
75m-2b-tokens/	69.4M	2B	~28.8x	3.8496
150m-500m-tokens/	135.0M	500M	~3.7x	3.7378

Each subfolder is independently loadable: model.safetensors + config.json. All three share the same model.py and tokenizer (this repo's root).

Note: these are the best validation-loss checkpoints saved during each run (the lowest point reached), not necessarily the literal last training step — the scaling-law docs report final-iteration loss instead, for a compute-matched comparison across runs. The two numbers differ slightly because validation loss is noisy near the end of a cosine-decay schedule.

What they're for

• 75m-500m-tokens vs 75m-2b-tokens — same model size, 4x the tokens. Tests whether more data helps at fixed size.
• 75m-500m-tokens vs 150m-500m-tokens — same tokens, ~1.9x the parameters. Tests whether more parameters help at fixed, scarce data.

Architecture

Type	Decoder-only GPT (pre-LayerNorm)
Attention	Flash Attention via PyTorch SDPA
Context length	1024
Vocabulary	Custom 32K BPE tokenizer (trained on this project's own filtered OpenWebText corpus, not GPT-2's)
Precision	weights stored fp32, trained in bf16

Training

Data	~9.14B-token OpenWebText corpus, filtered (language-ID, quality heuristics, exact dedup) and re-tokenized with the project's own 32K BPE tokenizer
Optimizer	AdamW, cosine LR with warmup, weight decay on 2D params
Hardware	1x RTX 5070 Ti (16GB)

Honest assessment

These are small models trained on a token budget far below Chinchilla-optimal for their size (compute-optimal would be roughly 20x tokens/param; the largest run here reaches ~29x on the small model, but only ~3.7x on the 150M model). They generate grammatical English with local coherence but no long-range consistency, and score near the random-chance floor on standard benchmarks (see docs/phase5/evaluation.html) — expected at this scale, not a bug.

Files

• <checkpoint-name>/model.safetensors — model weights, fp32, optimizer state stripped
• <checkpoint-name>/config.json — architecture and training metadata for that checkpoint
• model.py — the from-scratch model definition needed to load any of these weights
• tokenizer/ — the project's own 32K BPE tokenizer (tokenizer.json, vocab.json, merges.txt)

Usage

This is not a transformers architecture, so load it with the included model.py.

import json, torch
from safetensors.torch import load_file
from model import GPT, GPTConfig
from tokenizers import Tokenizer

CHECKPOINT = "75m-2b-tokens"  # or "75m-500m-tokens" / "150m-500m-tokens"

cfg = json.load(open(f"{CHECKPOINT}/config.json"))
model = GPT(GPTConfig(
    n_layer=cfg["n_layer"], n_head=cfg["n_head"], n_embd=cfg["n_embd"],
    block_size=cfg["block_size"], vocab_size=cfg["vocab_size"], dropout=0.0,
))
model.load_state_dict(load_file(f"{CHECKPOINT}/model.safetensors"))
model.eval()

tok = Tokenizer.from_file("tokenizer/tokenizer.json")
ids = torch.tensor([tok.encode("The meaning of life is").ids])

with torch.no_grad():
    for _ in range(50):
        logits, _ = model(ids[:, -cfg["block_size"]:])
        nxt = torch.softmax(logits[:, -1, :] / 0.8, dim=-1).multinomial(1)
        ids = torch.cat([ids, nxt], dim=1)
print(tok.decode(ids[0].tolist()))

Limitations

• Small and undertrained relative to Chinchilla-optimal, especially the 150M checkpoint.
• Trained on OpenWebText, so it inherits the biases and noise of unfiltered web text scraped from Reddit-linked URLs.
• No instruction tuning, no safety alignment. Raw next-token predictors.

License

MIT. Built as an educational project.