Fly Wordbrain — full readout, 10,000 stories (control)
This is a control arm, not a recommended model. It exists to answer one question: was the advantage of a low-rank readout real, or just an artifact of training on 1,000 stories?
It is the same fruit-fly-connectome architecture as fly-wordbrain-rank64, but keeps the original full 50,578,432-parameter output matrix — 51,308,213 trainable parameters in total — and trains on the same 10,000-story corpus at the same 16,800-update budget. Pairing it with the rank-64 arm completes a 2×2.
What it answered
| audit CE | 1,000 stories | 10,000 stories | data effect |
|---|---|---|---|
| full readout (51.3M) | 4.9125 | 3.2619 | −1.6505 |
| rank 64 (3.96M) | 3.2802 | 2.9493 | −0.3308 |
| readout gap | 1.6323 | 0.3126 |
80.8% of the low-rank advantage was a small-data artifact. Ten times the text improves the full readout 5.0× more than the low-rank one — which is what a head that was memorising rather than generalising looks like once it finally has enough data.
On the same held-out population (200 stories, 45,059 targets) this model scores CE 3.2619, PPL 26.10, 35.76% top-1 — now 0.726 nats better than the 52,756,661-parameter released reference, having been 0.924 nats worse when trained on 1,000 stories.
A residual advantage for the low-rank readout survives — 0.313 nats and 1.67 accuracy points — but see the caveat below before reading much into it.
Why you probably want the other one
The rank-64 model reaches CE 2.9493 / 37.43% with 13× fewer parameters. This arm is here for reproducibility of the comparison, not for use.
The caveat that matters
Neither 10,000-story arm converged. 16,800 updates is 18.3 passes over 1,000 stories but
only 1.83 over 10,000. The residual 0.313-nat gap is therefore consistent with either a
genuine remaining advantage for the constrained readout or a convergence-rate difference —
the smaller model simply reaching its plateau in fewer updates. Nothing measured here
separates those. Both arms are budget-capped, so the trainer records them as
debug_stopped with debug: true; that flags the cap, not a failure.
The training corpus is a strict superset of the 1,000-story set, and validation, test and audit rows are byte-identical across both, so the scores are directly comparable.
Neither arm's advantage comes from the fly. A later randomised-graph control retrained the rank-64 arm on the same connectome with every edge rewired at random — in-degree, out-degree and every synaptic weight preserved — and it lost only 0.0100 nats [+0.0026, +0.0175]. Whatever these arms measure, it is about readouts and data, not anatomy.
Files
min-ce.safetensors and max-accuracy.safetensors (both update 16,800), manifest.json
with selector receipts and frozen-graph digests, and experiment-config.json.
The connectome is not duplicated here — it lives in
fly-connectome-49k,
and manifest.json records the frozen-buffer digests that identify which graph these
weights were trained on. No synaptic weight was modified.
Full write-up: Stage 6.
Licence
Weights CC BY 4.0 — derived from MaleCNS v1.0 (FlyEM / HHMI Janelia, University of
Cambridge, MRC LMB, Google Research) and the
ngxson/fly-llm-hf architecture. Code MIT.
TinyStories is not redistributed.