Checkpoints for "Two Measurement Failures in Small-Scale Mixture-of-Experts Sparsity Studies"
All 84 checkpoints reported in the paper, one per run, with optimizer state stripped and the configuration snapshot of each run. Code, training logs and the evaluation scripts that read these files are at https://github.com/taeyoung1/moe-sparsity-scaling. Paper: https://doi.org/10.5281/zenodo.23752975 (Zenodo preprint; an arXiv version will be linked here when available).
These are small research models (51M–628M total parameters, 19M–51M active) trained for measurement, not for use. They generate text at the quality of a 655M-token nanoGPT run and have no instruction tuning.
What is here
runs/<run name>/ckpt.pt model weights + model_args + iter_num + best_val_loss + config
runs/<run name>/config.json the training configuration snapshot written by train.py
full_eval_nodrop.json the results file every figure and table in the paper is drawn from
84 runs: a sparsity sweep (N ∈ {1,2,4,8,16,32} experts × d_model ∈ {256,384,512} × 3 seeds, plus N=64 at d_model=384), a data-repetition arm (N ∈ {1,4,8,16} × 2 reduced corpus fractions × 3 seeds at d_model=512), and a capacity-factor-2.5 control (N=32, d_model=512, 3 seeds). Seeds are 42, 1337 and 2024.
Run names follow
fineweb_edu[_d<width>]_N<experts>_aux<aux coefficient>_cap<capacity factor>[_frac<corpus fraction>]_seed<seed>
_d<width> is absent for the 512-wide runs and _frac for the full corpus. Do not parse
the names: every field is stored explicitly in full_eval_nodrop.json.
Using them
Drop the runs/ folder into a checkout of the repository above and run
python eval_all_nodrop.py # re-evaluates every checkpoint under both protocols
python plot_all.py # redraws every figure and table from full_eval_nodrop.json
Each ckpt.pt loads with torch.load(path, map_location="cpu", weights_only=False); the
model_args entry rebuilds the model with GPT(GPTConfig(**model_args)) from the
repository's model.py. The saved checkpoint is the one with the lowest validation loss
recorded during training (step in iter_num), not the end-of-training state; the paper's
Section 8 explains why that matters for the repetition runs.
Model details
8-layer GPT-2-style decoder, top-1 Mixture-of-Experts feed-forward block in every layer, capacity factor 1.25 during training (2.5 for the control), Switch-style auxiliary loss 0.01, GPT-2 BPE vocabulary, 512-token context. Trained on the leading 198M tokens of FineWeb-Edu (49.5M and 24.7M for the repetition arm) for 20,000 steps of 32,768 tokens in bf16 on one RTX 5080. Optimiser AdamW (0.9, 0.95), peak lr 6e-4 with cosine decay to 6e-5, 500 warmup steps, weight decay 0.1, gradient clipping 1.0, no dropout.
Licence
Weights and code are released under the MIT licence. The training data, FineWeb-Edu, is distributed by its authors under ODC-By 1.0 subject to Common Crawl's terms of use; it is not redistributed here.