slm-from-scratch-77m

A 77M parameter GPT-style language model written and trained from scratch in PyTorch on a single RTX 5070 Ti. No HuggingFace transformers model classes were used. Every component (attention, MLP, blocks, embeddings, the training loop) was hand-coded as part of a learning project to understand the full LLM training pipeline.

This is Phase 1 of a larger build documented at github.com/Ishaanred/slm-from-scratch. It is a deliberately small, deliberately undertrained checkpoint. It is a portfolio and learning artifact, not a production model.

What it is

Parameters	77.2M
Architecture	Decoder-only GPT (pre-LayerNorm)
Layers	8
Heads	8
Embedding dim	512
Context length	1024
Vocabulary	GPT-2 BPE (50,257)
Attention	Flash Attention via PyTorch SDPA
Precision	weights stored fp32, trained in bf16

Training

Data	OpenWebText
Steps	5,000
Tokens seen	~80M
Optimizer	AdamW, cosine LR with warmup, weight decay on 2D params
Hardware	1x RTX 5070 Ti (16GB)
Wall time	~20 min
Validation loss	5.24

Honest assessment

This model is undertrained on purpose. At ~80M tokens it has seen roughly 5% of what a 77M model needs to converge (Chinchilla suggests ~1.5B tokens for this size). The output is grammatical English with plausible local structure but no coherence across sentences.

A companion 50M model trained on the same 80M tokens reached a lower validation loss (5.12). At a fixed, small token budget the smaller model wins, because the larger one is more undertrained relative to its capacity. That gap is expected to reverse with longer training. Phase 3 of the project runs the controlled scaling-law experiments that test this.

Sample output

Prompt: The meaning of life is

The meaning of life is the first ever-to-to-mused of self-lab and the first time period of time.
The difference between these systems, and even the potential difference to be an example of these
two seasons. The reason it might be, but the first time that it's very well worth noting that we
can build up a new one as a result of the whole way we're talking about...

Files

• model.safetensors — model weights, fp32, optimizer state stripped
• model.py — the from-scratch model definition needed to load these weights
• config.json — architecture and training metadata

Usage

This is not a transformers architecture, so load it with the included model.py.

import json, torch
from safetensors.torch import load_file
from model import GPT, GPTConfig

cfg = json.load(open("config.json"))
model = GPT(GPTConfig(
    n_layer=cfg["n_layer"], n_head=cfg["n_head"], n_embd=cfg["n_embd"],
    block_size=cfg["block_size"], vocab_size=cfg["vocab_size"], dropout=0.0,
))
model.load_state_dict(load_file("model.safetensors"))
model.eval()

# tokenize with GPT-2 BPE, e.g. via tiktoken:
import tiktoken
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode("The meaning of life is")])

with torch.no_grad():
    for _ in range(50):
        logits, _ = model(ids[:, -cfg["block_size"]:])
        nxt = torch.softmax(logits[:, -1, :] / 0.8, dim=-1).multinomial(1)
        ids = torch.cat([ids, nxt], dim=1)
print(enc.decode(ids[0].tolist()))

Limitations

• Tiny and undertrained. Not suitable for any real task.
• Trained on OpenWebText, so it inherits the biases and noise of unfiltered web text scraped from Reddit-linked URLs.
• No instruction tuning, no safety alignment. It is a raw next-token predictor.

License

MIT. Built as an educational project.