nanofly-decoder-en

_{43,993 neurons at their measured MaleCNS v1.0 coordinates, coloured by this checkpoint's state at one tick while writing "…there was a girl named Lily." Orange excited, blue inhibited, grey at rest. Frontal view; the optic lobes flank the central brain.}

A language model whose recurrent layer is the measured wiring of a fruit fly. The connectome is a frozen echo state network reservoir — no synapse is trained. Only the input projection, per-neuron gain/bias/leak, one global scale and the readout learn.

No GGUF, and llama.cpp cannot run this. The recurrent layer is a 49,393 × 49,393 sparse matrix with 9M non-zeros; ggml has no operator for it. Use transformers with trust_remote_code=True.

• Code: github.com/igorktech/nanoFLY
• Russian sibling: igorktech/nanofly-decoder-ru

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "igorktech/nanofly-decoder-en"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()
model = model.to("cuda" if torch.cuda.is_available() else "cpu")

ids = tok("Once upon a time, there was a", return_tensors="pt").input_ids
ids = torch.cat([torch.tensor([[model.config.bos_token_id]]), ids], dim=1).to(model.device)
out = model.generate(ids, max_new_tokens=80, do_sample=True, top_k=50, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))

• Prepend BOS: every training example started with it.
• Greedy: do_sample=False, and drop top_k or transformers warns.
• Beam search and assisted generation are unsupported (stateful model). Greedy, sampling, top-k, top-p work.
• ~960 forward passes/s on an RTX 5080, ~10/s on a laptop CPU.

Architecture

connectome	MaleCNS v1.0 central brain — cb_sensory, cb_intrinsic, visual_projection, descending_neuron, ascending_neuron
neurons / edges	49,393 / 9,055,280 signed (623,728 dropped: modulatory or unknown transmitter)
edge weight	sign of the presynaptic transmitter × synapse count, rows normalised to unit absolute weight. ACh +1; GABA, Glu, His −1; others 0
token input	11,434 sensory-facing neurons, 8-slot delay line (slot j gets token t−j). No attention, no positional encoding
held out	the 2,635 ORNs stay out of the token input, so the encoder-decoder variant can start from these weights
dynamics	x ← (1−a)·x + a·tanh(ρ·g·(Wx) + u + b), 2 ticks per token; a learned per neuron (init 0.5), ρ learned global (init 1.0 → 4.26)
readout	all 49,393 states → Linear(49393→256) → LayerNorm → Linear(256→1024)
trainable	16.25M — readout 12.90M, input projection 2.93M, embedding 0.26M, per-neuron scalars 0.15M

Training

data	TinyStories, 98,024 stories / 1,976 held out, 28,417,062 tokens, byte-level BPE vocab 1,024
objective	next-token cross entropy, truncated BPTT over 32-token windows, state carried across windows
optimiser	AdamW — body 2e-3 (no decay), readout 5e-4 (decay 0.01), warmup 200 then cosine to 10%, clip 1.0
schedule	3 epochs, 22,115 updates, batch 128
hardware	1 × RTX 5080, 1.74 h at ~13,700 tok/s

Evaluation

Held-out loss. Identical data, recipe, seed and budget; only the recurrent layer differs:

recurrent layer	val loss	ppl	trainable
degree-matched shuffled wiring, frozen	1.979	7.2	16.25M
this model — real connectome, frozen	1.933	6.9	16.25M
real connectome, synapse strengths trained	1.913	6.8	25.30M

The control keeps every neuron's in- and out-degree, the transmitter signs and Dale's law, and randomises only which neuron pairs with which. Two results, each holding at all ten intermediate evaluations:

• The fly's specific wiring is worth 0.046 nats over a random graph with its degrees — and that margin grows through training (0.031 → 0.046).
• Making all 9,055,280 synapse strengths trainable is worth 0.020 nats on top — and that margin shrinks through training (0.032 → 0.020), at 4× the compute. It buys faster convergence more than a higher ceiling. Trained strengths stay close to the measured ones (Pearson r = 0.94).

One seed per condition, so treat the margins as indicative rather than significant. Validation was still improving at the end in all three runs; none is converged.

Greedy samples, prompt in bold:

Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, red ball that was very pretty. She wanted to play with it, but it was too high for her to reach.

Tom and his dog were best friends. They liked to play in the park. One day, they saw a big dog with a big bag.

Limitations

• 16M trainable parameters over 85M tokens of children's stories. It writes about Lily and Tom and nothing else.
• 8-token delay line plus a short leaky recurrent memory. Loss stops improving with context by roughly position 32; names and objects drift within a paragraph.
• A tanh rate neuron is not a spiking model: no spikes, no synaptic delays, no neuromodulation — modulatory edges are removed outright.
• Central brain only; the optic lobes and ventral nerve cord of the 166,700-neuron CNS are absent.
• Synapse count is a proxy for strength, and rows are normalised. Neither is measured physiology — see the third row of the evaluation table for what happens when the strengths are fit to the task instead.

Credits

• Connectome: MaleCNS v1.0 — FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, Google Research. CC BY 4.0. The published buffers derive from that release; keep the attribution when redistributing.
• Transmitter signs: Shiu et al., Nature 2024.
• Connectome as reservoir: Costi, Hadjiivanov, Dold, Hale, Izzo, 2025.
• Prior art: ngxson/fly-llm-hf, whose graph subset this reproduces.
• Data: Eldan and Li, TinyStories, 2023.

Weights CC BY 4.0, matching the connectome. Modeling code Apache-2.0.