GPT-Fly (fly-small): a GPT-2 wired with the fruit-fly connectome

Browser-ready ONNX weights for the GPT-Fly demo: a GPT-2-small-shaped transformer (12 layers, 768 wide, GPT-2 BPE, 1024 context) whose MLP blocks are sparsified with frozen binary masks derived from the FlyWire whole-brain connectome (~0.69% dense), plus learned dense glue. Trained on TinyStories to val loss 1.82 (dense control: 1.14).

Files

File	Contents
fly-small.onnx	Model graph (weights live in the sibling .onnx.data)
fly-small.onnx.data	633MB fp32 weights (external data)
fly-small.tokenizer.json	Descriptor naming the GPT-2 BPE scheme

Use

Load with onnxruntime-web, mounting the external weights explicitly (InferenceSession.create(graphBytes, { externalData: [...] })), since onnxruntime-web does not auto-resolve sibling .onnx.data. Tokenize with any GPT-2 byte-level BPE (e.g. Xenova/gpt2); EOS id is 50256. Input name idx, output name logits.

Provenance

• Training code: eob/gpt-fly (FLY-010 scale-up, 6000 steps, 8xA100 → 4xA100)
• Fly regions → transformer pieces mapping: see the technical writeup on the demo page
• License: MIT