WIKI300M NxD grid
201 language-model checkpoints from a grid over parameter count N and training tokens D, trained
on English Wikipedia to fit scaling laws. Each .pt is one completed run.
| Checkpoints | 201 (220 GiB) |
| N | 22.8 M to 203.4 M parameters |
| D | 20.5 M to 771.8 M tokens |
| Learning rate | 3e-4, weight decay 0.1, 1 epoch |
| Batch size | 96 to 384, sequence length 256 |
| Wall time per run | 0.03 h to 6.16 h on NVIDIA GH200 |
N and D vary independently here, which is what separates this repo from wiki300m-colinear, where they are tied by a fixed tokens-per-parameter ratio.
Layout
N022.80M/D020.50M/WIKI300M_NxD_GRID_N22.80M_D20.50M_SEED43_...pt
N022.80M/D020.50M/WIKI300M_NxD_GRID_N22.80M_D20.50M_SEED44_...pt
N022.80M/D020.50M/WIKI300M_NxD_GRID_N22.80M_D20.50M_...pt
Six N values by 11 D values, all 66 cells occupied, three seed replicates per cell (SEED43,
SEED44, and an unseeded base run). Folder names are zero padded so that listing them
alphabetically also orders them numerically.
Filename schema
WIKI300M_N<params>M_D<tokens>M_wikipedia_m<params>_token<tokens>
_lr<lr>_distributed_epoch<n>_bs<batch>_seq<len>_wd<decay>
_time<hours>h_completedAt<timestamp>.pt
Every hyperparameter of a run is recoverable from its filename, so the grid can be reconstructed without a separate index.
Loading
import torch
from huggingface_hub import hf_hub_download
path = hf_hub_download("leibnitz-lab/wiki300m-nxd-grid", filename="<checkpoint>.pt")
state = torch.load(path, map_location="cpu")
These are raw PyTorch state dicts from a custom distributed trainer, not transformers models.
Related
- wiki300m-colinear, the fixed-ratio sweep
- wiki300m-holdout, runs held out for validation