lecore-303m-rerun — matched 303M pair: softmax vs gated-HRR
Result: gated-HRR attention is 41.5% worse than softmax at 303M parameters / 1B tokens. The gap did not narrow with scale, it widened (at 12M params on enwik8 it was ~14%).
Two arms, identical in everything but the attention operator: same seed, same 7,629 steps, same cosine schedule, 131,072 tokens/step, 5,000,002,551-token FineWeb-Edu corpus (GPT-2 BPE), RoPE in both arms, block 1024. This is the run that answers the "does the from-scratch gap close at scale?" question the earlier postmortem left open.
| arm | val loss | bits/byte | steps | tokens |
|---|---|---|---|---|
| softmax | 3.1902 | 4.6025 | 7,629 | 999,948,288 |
| gated-HRR | 4.5138 | 6.5120 | 7,629 | 999,948,288 |
HRR / softmax = 1.415. Speed, measured on an H100 with the repo's Triton scan wired in: softmax 151,269 tok/s, gated-HRR 50,797 tok/s — 3.0x slower, not 11x (the Triton kernel is a 10.2x speedup over the naive path and cut memory 74.9 → 41.8 GB).
Files
| file | size | what it is |
|---|---|---|
runs6b/hrr/final.pt |
1.42 GB | support file |
runs6b/hrr/latest.pt |
1.42 GB | support file |
runs303/hrr/final.pt |
1.42 GB | support file |
runs303/hrr/latest.pt |
1.42 GB | support file |
runs6b/softmax/final.pt |
1.42 GB | support file |
runs6b/softmax/latest.pt |
1.42 GB | support file |
runs303/softmax/final.pt |
1.42 GB | support file |
runs303/softmax/latest.pt |
1.42 GB | support file |
runs303/ and runs6b/ are the two run sets, each with an hrr/ and a softmax/ arm and both
final.pt and latest.pt (identical when the run finished cleanly). Checkpoints are ~1.4 GB each.
What this does and does not say
- It does not say HRR is useless: it is a fixed-size recurrent state with O(T) cost. The loss gap is the price; whether the price is worth paying depends on the workload.
- It does not measure long-context behaviour — both arms trained at block 1024.
- n=1 per arm. Single run, no seed sweep.
Related
- Method, tiny-twin benchmarks, constant-memory footprint numbers: staccs/lecore-hrr-gpt2
- Per-model gated-HRR gate sets: Qwen3-32B, Llama-3.3-70B, QwQ-32B, Qwen2.5-72B, Apertus-70B
- Training harness and corpus prep: https://github.com/staccDOTsol/supercontext
License
MIT — checkpoints and code produced here, no third-party base weights.