pngwn/memcap-l-below-capacity

memcap: 8.44M-param GPT-2, below capacity regime

Part of a tiny-scale reproduction of How much do language models memorize? (Morris et al., ICML 2026 honorable mention): GPT-style models trained from scratch on uniformly random token sequences, where generalization is impossible and every bit of loss reduction is memorization.

This checkpoint sits below the model's storage capacity: the dataset entropy (11.09 Mbits) is smaller than what the model can store, and the model memorized 99.5% of it. It demonstrates the full-memorization regime of How much do language models memorize?.

Model

• Architecture: GPT-2 (via transformers), 8.44M parameters (d_model 256, 10 layers, 4 heads), vocab 2048, context 64, trained from scratch (init seed 0)
• Precision: fp32 (TF32 disabled) | Optimizer: AdamW lr 1e-3, cosine decay | Batch: 512 sequences
• Trained for 8,000 steps

Data

Uniformly random tokens, vocab 2048, 64 tokens/sequence, N = 16,000 sequences, data seed 300. Dataset entropy = 63 predictable tokens x log2(2048) = 693 bits/sequence = 11,088,000 bits total. Dataset: pngwn/memcap-random-data

Memorization measurement

memorized bits = dataset entropy - model NLL on the data (in bits)

quantity	value
dataset entropy	11,088,000 bits
memorized	11,030,454 bits (99.5% of entropy)
bits per parameter	1.31

Full learning curves and all 18 cells: pngwn/memcap-results. Training script: train_memcap.py in that repo. This same checkpoint also lives on branch l_n16000_s0_fp32 of pngwn/memcap-runs.

Usage

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("pngwn/memcap-l-below-capacity")

Note: no tokenizer ships with this checkpoint — the training data is raw token ids in [0, 2048).