WIKI300M colinear sweep
197 language-model checkpoints trained on English Wikipedia along fixed tokens-per-parameter
ratios, for fitting scaling laws where N and D are not independent. Each .pt is one completed
run.
| Checkpoints | 197 (211 GiB) |
| N | 22.8 M to 203.4 M parameters |
| D | 22.8 M to 671.1 M tokens |
| Tokens per parameter | 1.0 to 3.3 |
| Learning rate | 3e-4, weight decay 0.1, 1 epoch |
| Batch size | 96 to 384, sequence length 256 |
| Wall time per run | 0.06 h to 5.33 h on NVIDIA GH200 |
D is determined by N and the TPP ratio, so the runs lie along lines in the N-D plane rather than filling a grid. The independent grid is wiki300m-nxd-grid.
Layout
N105.33M/D126.40M/WIKI300M_COLINEAR_N105.33M_TPP1.2_D126.40M_SEED43_...pt
Checkpoints sit under N<params>M/D<tokens>M/, two to four per cell depending on how many seed
replicates a point received. Folder names are zero padded so that listing them alphabetically also
orders them numerically. The TPP ratio stays in the filename, since D already encodes it.
Filename schema
WIKI300M_COLINEAR_N<params>M_TPP<ratio>_D<tokens>M_wikipedia_m<params>_token<tokens>
_lr<lr>_distributed_epoch<n>_bs<batch>_seq<len>_wd<decay>
_time<hours>h_completedAt<timestamp>.pt
Every hyperparameter of a run is recoverable from its filename, including the TPP ratio it was placed on.
Loading
import torch
from huggingface_hub import hf_hub_download
path = hf_hub_download("leibnitz-lab/wiki300m-colinear", filename="<checkpoint>.pt")
state = torch.load(path, map_location="cpu")
These are raw PyTorch state dicts from a custom distributed trainer, not transformers models.
Related
- wiki300m-nxd-grid, N and D varied independently
- wiki300m-holdout, runs held out for validation