attention-120m-ce-6k

120M self-attention transformer, plain cross-entropy, Pile, 6,000 updates (3.146B tokens), seed 1111. The attention reference for the M=1 ahead-router ablation ladder, at the ladder's token budget.

Pile PPL (1,000 validation sequences)	11.79
Rare-AR PPL	3.56
Natural FDA (all 1,102 examples)	26.9%

Files: final.ckpt (weights, optimizer state stripped, global_step 6000), resolved-config.yaml (the exact training config), metadata.json (run identity).

Loading

In recurrent-recall-circuits, this checkpoint is registered as attention_120m_ce_6k in configs/models/ladder_m1.yaml (architecture configs/experiment/lm/120m/attention/ce_6k.yaml):

python -m recurrent_recall_circuits.cli.evaluate --suite baseline \
    --registry configs/models/ladder_m1.yaml --models attention_120m_ce_6k

The checkpoint is downloaded from this repo on first use.