temporal-moe-ckpts

Megatron-LM training checkpoints for the Temporal-MoE experiments, 71 runs, 140.1 GiB. Checkpoints are in torch distributed checkpoint format (.distcp), not the older mp_rank_* layout.

---

WARNING, read before loading anything

1. Seventeen runs contain a second checkpoint that is not the trained model

Each of the 17 runs listed below contains two checkpoint directories: the genuine end-of-training checkpoint, and an iter_0000006 that is not it.

iter_0000006 was written by a later job that ran with finetune=True, which loads the trained weights and resets the iteration counter to zero. It therefore holds the fully trained model plus 6 additional optimizer steps, with the optimizer and learning-rate scheduler state discarded. It is not an obviously broken artifact. It will load, it will train, and it will emit plausible-looking evaluation numbers that are close to but not equal to the published ones. A run that loads it can produce results that look correct and are not, which is why this is dangerous rather than merely inconvenient.

The cumulative FLOP counter confirms the ordering in every affected run: the value stored in iter_0000006 is larger than the one in the fully trained checkpoint, because those 6 steps came after training finished.

latest_checkpointed_iteration.txt originally read 6 in all 17 runs and has been corrected in this repository, so --load now resolves to the trained checkpoint. See "Pointer corrections applied after upload" below for the per-run values and how they were verified. Both checkpoint directories remain present and unmodified, so passing --load with an explicit iteration is still the safest habit.

run	load this (fully trained)	present but NOT the trained model
g3_tmoe_s0_1e16_ant0p02	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_ant0p02_g0p9	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_ant0p1	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_bursty	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_head	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_head_b05	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_head_c	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_head_tc	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_mom	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_mom_seed2	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_momg_b05	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_momr	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_momr_b1	iter_0002232	iter_0000006
g3_tmoe_s0_1e16_momr_b2	iter_0002232	iter_0000006
g3_tmoe_s2_1e17	iter_0003861	iter_0000006
g3_tmoe_s2_1e17_auxfree	iter_0003861	iter_0000006
g3_tmoe_s2_1e17_karen	iter_0003861	iter_0000006

Unaffected runs have a single iter_* directory and a correct latest_checkpointed_iteration.txt.

2. g3_restorecheck/ckpt/iter_0000003/ is an incomplete write

This directory contains only common.pt. It is missing __0_0.distcp, __0_1.distcp, .metadata and metadata.json. It is an aborted checkpoint write and is not loadable.

It was uploaded deliberately rather than dropped, so the record of what exists on the training disk is complete. Do not attempt to restore from it. g3_restorecheck is an uncited infrastructure check and backs no published number.

---

Pointer corrections applied after upload

The ckpt/latest_checkpointed_iteration.txt files listed here were corrected in this repository after the checkpoints were uploaded. Each originally contained 6.

Reason: 6 resolved to iter_0000006, the finetune-mode continuation described in warning 1 above, written after training finished with the optimizer and learning-rate scheduler state discarded. A naive --load restored that instead of the trained model. Because it loads cleanly, trains, and produces plausible evaluation numbers, the failure was silent rather than visible.

run	original value	corrected value
g3_tmoe_s0_1e16_ant0p02	6	2232
g3_tmoe_s0_1e16_ant0p02_g0p9	6	2232
g3_tmoe_s0_1e16_ant0p1	6	2232
g3_tmoe_s0_1e16_bursty	6	2232
g3_tmoe_s0_1e16_head	6	2232
g3_tmoe_s0_1e16_head_b05	6	2232
g3_tmoe_s0_1e16_head_c	6	2232
g3_tmoe_s0_1e16_head_tc	6	2232
g3_tmoe_s0_1e16_mom	6	2232
g3_tmoe_s0_1e16_mom_seed2	6	2232
g3_tmoe_s0_1e16_momg_b05	6	2232
g3_tmoe_s0_1e16_momr	6	2232
g3_tmoe_s0_1e16_momr_b1	6	2232
g3_tmoe_s0_1e16_momr_b2	6	2232
g3_tmoe_s2_1e17	6	3861
g3_tmoe_s2_1e17_auxfree	6	3861
g3_tmoe_s2_1e17_karen	6	3861

Each corrected value was verified against four independent sources before publishing, and all four agreed in all 17 runs:

1. iters= in run.meta
2. the final successfully saved checkpoint from iteration N line in train.log
3. the iteration field inside that checkpoint's common.pt
4. the learning-rate schedule reaching min_lr (3.000059E-04 and 3.000005E-04 against a configured min_lr of 0.0003)

The corrected files match the format used by unaffected runs: bare digits, no trailing newline.

No checkpoint data was altered, and nothing was deleted. Every iter_0000006 directory remains uploaded and byte-identical to what was trained. Only the pointer files changed. Their sha256 and bytes entries in MANIFEST.csv describe the corrected content published here.

---

What this repository holds

71 run directories, plus _batch_logs/ and _lmeval_scratch/lmeval_out/. Each run directory holds the Megatron checkpoint plus the metadata needed to interpret it. 58 runs are cited by a results table or the paper, 13 are not. See the cited column described below.

Layout

<run_name>/
  ckpt/
    latest_checkpointed_iteration.txt   # see warning 1 above
    iter_<NNNNNNN>/
      __0_0.distcp                      # torch distributed checkpoint shards
      __0_1.distcp
      .metadata                         # DCP tensor index
      common.pt                         # iteration, arg namespace, RNG state
      metadata.json                     # backend versions
  run.meta                              # full knob set for the run
  train.log                             # Megatron stdout, includes the arg dump
  tb/                                   # TensorBoard event files
  lmeval_*/                             # lm-evaluation-harness outputs
  *.log, *.out                          # probe and evaluation logs
_batch_logs/                            # cross-run batch driver logs
_lmeval_scratch/lmeval_out/             # evaluation outputs from the scratch harness

Checkpoints copied into _lmeval_scratch for evaluation were byte-identical duplicates of the originals and were not uploaded. Only their evaluation outputs are here.

Training configuration

Every run directory contains run.meta and train.log. run.meta records the full knob set for that run in two lines, for example:

[run] temporal_fine_g3_1e19 N=185766400 iters=17112 warmup=856 min_lr=0.0003 eval@1711
[run] shape=s19opt H=800 L=14 ffn=4272 moe_ffn=184 grain=3 num_experts=192 shared_int=1100
      dense=0 temporal=1 shared_mult=2 topk=18 heads=50 gb=256 mb=8 lr=3e-3 flops=1e19

train.log is the full Megatron stdout, including the complete argument namespace dumped at startup. Read these before drawing any conclusion from a checkpoint.

MANIFEST.csv and the cited column

A manifest covering every file in all four repositories lives in the code repository. It has seven columns: local_path, hf_repo, hf_path, bytes, sha256, run_name, cited.

The cited column marks whether a run is referenced by a results table in results/ablations/*.csv or by the paper:

• cited, the run backs a published number. There are 58 of these.
• uncited, the run is infrastructure validation, a smoke test, a throughput probe, or an aborted run. It is kept for completeness and reproducibility, not because a table depends on it. There are 13 of these.
• empty, the file is not scoped to a single run, for example a batch log or an evaluation output.

Every sha256 in the manifest was computed on this disk before upload and each file was verified to exist remotely with a matching byte size.

Links

• Code: https://github.com/ncylich/temporal-moe
• Paper: Temporal-MoE (short paper), see the paper/ directory in the code repository
• Upstream platform this work forks: FLAME-MoE, arXiv:2505.20225

Companion repositories

• ncylich/temporal-moe-ckpts, Megatron training checkpoints
• ncylich/temporal-moe-router-adapt, router adaptation safetensors
• ncylich/temporal-moe-extras, captures, merged model, result tables, figures
• ncylich/temporal-moe-corpus, tokenized training corpus

Provenance

Trained with a personal fork of FLAME-MoE. Not affiliated with or endorsed by the FLAME-MoE authors.