lamshiu/lm-optimization

LM optimization: checkpoints

Checkpoints from https://github.com/LamShiuChing/LM-optimization, a from-scratch study of what text a small model should think in and learn from. Every run is a 5M-parameter GPT (6 layers, d=256, no positional encoding, attention cut at document boundaries) trained on TinyStories prose mixed with generated reasoning documents.

One folder per run: ckpt.pt (state dict, loadable with src/model.py from the repo), config.json (all hyperparameters and the dataset version), log.jsonl (loss and task accuracies during training), summary.json (final accuracies). Findings F1-F21 are in the repo's docs/findings.md.