thekoshkin/physlm-fly-4k

PhysLM fly-4k

PhysLM fly-4k is a 4,000-unit recurrent network trained as a character-level language model on TinyStories. The update rule of its recurrent units is limited to operations that have a direct counterpart in analog circuits: weighted sums with a bounded number of inputs (400 per unit), saturating nonlinearities, a multiplicative gate, and leaky integration with fixed, untrained time constants. The design goal is a language model that could be built as a physical dynamical system instead of being run as a long sequence of digital operations. That goal is not demonstrated here: this checkpoint is a float32 software simulation, and no hardware non-idealities were applied to it.

It is a research artifact of the PhysLM project, which asks whether language generation can be the natural behaviour of a physical system (relaxation, fluctuations, oscillations). It is small, it writes simple and often incoherent stories, and it is not meant for any application. The model was designed, trained, evaluated and documented by AI agents, Claude Code and a Claude chat session (Anthropic); see How this model was made.

The name refers to the project's scaling path toward the neuron count of the adult fruit-fly brain (139,255 neurons in the FlyWire connectome; Dorkenwald et al., 2024). Only the count is the target: the wiring of this model is random and modular, not taken from the connectome. This 4,000-unit model is the first step of that path.

Paper from the same project: V. Koshkin, Saturation Traps and Learned Conventions in Emergent Analogue Communication, preprint, 2026, doi:10.5281/zenodo.23945050. In it, two recurrent networks of 4,000 cells of the same kind as this model's (tanh units with fixed leak rates and 400 inputs each, but without the multiplicative gate and with random rather than modular wiring) learn to communicate through a noisy analog wire.

Why this model is interesting

• Language from analog-style dynamics. The recurrent units use only operations with a direct circuit counterpart: weighted sums, tanh, a sigmoid gate, a product and leaky integration at a fixed rate. There is no attention, no normalization layer and no learned time constant. It is a concrete, reproducible example of a character-level language model whose recurrent core is restricted to such operations (as a clocked, discrete-time circuit). The readout and the sampling are digital, and no hardware version or noise test exists yet.
• About as good as a small transformer at equal data. Trained on the same 30 million characters, it predicts text about as well as a 0.8 M-parameter transformer with its own tokenizer (0.884 against 0.885 bits per character on the first 1,024 validation stories, 0.919 against 0.918 on stories 1,025-3,430; one run each, the transformer not tuned), and well ahead of the 1.9 M-parameter character-level transformer the project trained (1.035 and 1.093 in two runs, first 1,024 stories). About 0.07 of its 0.11 gap to TinyStories-1M is the training data: the TinyStories-1M architecture trained on this model's data reaches 0.840. The cost of the constraints themselves has not been isolated.
• Designed to scale. Units are grouped into modules with a fixed number of inputs, so the cost of one step grows linearly with the number of units. At this size the modular wiring matched random wiring within the pre-registered margin (0.884 with one modular seed against 0.886 ± 0.002 over three random-wiring seeds). A 16,000-unit version of the same cell has been trained and will be released after its control experiments; the fly-brain scale is planned and depends on funding.
• Small and easy to study. 4 million parameters (16 MB), trained on one laptop in 2.9 hours, runs on a CPU. The 800 slow units integrate over roughly 3 to 330 characters. A convenient test case for simulating device noise, quantization and mismatch, and for studying fast and slow recurrent dynamics.

It is meant for researchers in analog, neuromorphic and other unconventional hardware, for people who study recurrent dynamics and small language models, and for teaching.

Quick start

pip install torch safetensors huggingface_hub pyarrow

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("thekoshkin/physlm-fly-4k")
sys.path.insert(0, path)
import physlm_fly as pf

model = pf.PhysLMFly.from_dir(path)
alphabet = pf.Alphabet.from_file(f"{path}/alphabet.json")
print(pf.sample(model, alphabet, "Once upon a time", n_chars=400, seed=0))

From a local copy of this repository:

python physlm_fly.py eval      # bits per character on TinyStories validation stories (pinned dataset revision)
python physlm_fly.py sample --prompt "Once upon a time" --seed 0

eval takes about 6 minutes on an Apple-silicon laptop GPU (PyTorch MPS).

The model

One step of the network reads one character u and updates the state x of all 4,000 units:

h      = tanh(W x + Win[:, u] + b) * sigmoid(W2 x + Win2[:, u] + b2)
x_next = (1 - a) * x + a * h
logits = readout([x_next; one_hot(u); 1])          # logits of the next character

Property	Value
Units	4,000, in 80 modules of 50 units
Recurrent inputs per unit	400: each module receives dense input from 8 distinct source modules (a fixed random choice)
Wiring	W and W2 share one modular wiring; the cost of a step grows linearly with the number of units
Time constants	a is fixed per unit: units 0-3,199 are fast (a = 1); units 3,200-3,999 (modules 64-79) are slow integrators with a log-uniform in [0.003, 0.3]
Nonlinearity	tanh multiplied by a sigmoid gate; every state stays in (-1, 1)
Readout	linear in the raw state, the one-hot current character and a constant (which duplicates the readout bias); no normalization of the state
Parameters	4,016,623, all trained (recurrent, input, bias and readout weights)
Vocabulary	the 66 most frequent characters of the first 20,000 training stories, plus one shared symbol for every other character
Start state	x_rest: the mean state of 128 training-text streams after 5,500 characters; every story starts from it

Text is normalized before encoding: non-ASCII characters are removed, runs of whitespace become one space, and the result is stripped. The shared symbol also separated stories during training, so in sampling it ends a story.

Not every part is analog-style. The 400-input limit applies to the recurrent units only; the readout (a dense map from all 4,068 features to 67 outputs), the softmax and the sampling are ordinary digital operations here. The network steps once per character and the 3,200 fast units have no leak, so a physical version of this exact model would need clocked, discrete-time updates.

Results

Bits per character (lower is better) on the TinyStories validation split, after the same normalization. In every story the first character is given and every later character is predicted. PhysLM fly-4k starts every story from x_rest. For the transformer baselines the first token is given, and the bits of all later tokens are divided by the same number of characters (no BOS token, padding on the right, at most 2,048 tokens).

The validation split has 21,990 stories. The project evaluates on its first 3,430 stories, in the order of the pinned dataset revision. The first 1,024 of them were used repeatedly for decisions during the project; stories 1,025-3,430 were not used to select or tune this model.

Published TinyStories models. These were trained by their authors on the full training split (about 2.1 million stories) with the GPT-2 tokenizer, which was learned on a large web corpus. fly-4k saw 33,572 stories, 1.6 % of the split, four times.

Model	Type	Parameters	First 1,024 stories	Stories 1,025-3,430
PhysLM fly-4k (this model)	physically constrained recurrent network, characters	4.02 M	0.8838	0.9192
TinyStories-1M	GPT-Neo transformer, subword tokens	3.75 M (0.40 M outside token and position embeddings)	0.7696	0.8012
TinyStories-8M	GPT-Neo transformer, subword tokens	19.7 M (6.3 M outside token and position embeddings)	0.5134	0.5382

Equal training budget. Digital baselines trained by the project on exactly the same 30,000,177 symbols for 4 epochs. Each number is a single run.

Model	Parameters	First 1,024 stories	Stories 1,025-3,430
PhysLM fly-4k (this model)	4.02 M	0.8838	0.9192
Transformer with the TinyStories-1M architecture and the GPT-2 tokenizer (best of 5 learning rates)	3.75 M	0.8400	0.8712
Transformer with a 4,096-token BPE vocabulary learned on the same training text (one learning rate, not tuned)	0.79 M	0.8853	0.9179
Character-level transformer, width 128 (best of 3 learning rates; two runs of the same configuration)	1.86 M	1.035 and 1.093	—

On all 3,430 stories together: fly-4k 0.9091, TinyStories-1M 0.7921, TinyStories-8M 0.5311.

How to read these tables:

• About 0.07 of the 0.11-0.12 bits-per-character gap to TinyStories-1M is the training data: the TinyStories-1M architecture trained on fly-4k's data reaches 0.840 instead of 0.770. At equal data, fly-4k is level with a transformer whose tokenizer was learned on the same text, 0.044-0.048 behind the TinyStories-1M architecture with GPT-2's tokenizer, and ahead of the character-level transformer. The cost of the analog-style constraints themselves has not been isolated. All of these are single runs, and the 4,096-token transformer was not tuned, so the comparison shows where the model stands, not that one architecture is better.
• Stories 1,025-3,430 are harder for every model (by 0.02-0.04 bits per character); the gaps between models are about the same on both parts.
• fly-4k is a single training run (seed 0). The same cell with random instead of modular wiring (400 inputs per unit) reached 0.8862 ± 0.0024 bits per character on the first 1,024 stories (mean and 95 % interval over three seeds). The modular wiring, which makes larger versions affordable, stayed within the pre-registered margin of ±0.02 at this size (one modular seed).
• The transformer baselines were evaluated at revisions 77f1b168 (TinyStories-1M) and 8612e3b1 (TinyStories-8M) of their repositories.

Example output

Temperature 1, sampling from the full distribution (no top-k, no top-p), prompt "Once upon a time", seeds 0-2, not selected. Command: python physlm_fly.py sample --device cpu --n-chars 500 --num-samples 3 (other devices can give slightly different samples).

Once upon a time, there was a smooth dog swimming in the garden. He felt so secure, and he didn't understand why
he was wearing an elevator. He didn't like that. It felt like he was meant to see his mom. Lisa wanted to make her
son even happier. So she decided to never go back outside again. This way, she was only three years old. She
lofted Pink easily in a counting and smiled back. But, when she opened the box, she saw an ostrich with a picture
of a mountain in someone. She also sprinkle out a piece of block,

Once upon a time there was a naughty boy. He was three years old and had a blue envelope. Dave found a box full of
toys and loved to share his bookcase. Every day, the family would set out the blue balloon. After the summer day,
the by noticed the answer, but the animals felt sad. They didn't know what to do. Suddenly, they heard a loud
roar. It scared them and ran back to the hive. The boy said to the sign, "Let's by these bird and get you
something you want." The man let go of his body and brought it back hom

Once upon a time, there were two friends named Jack and Tim. They were very things and loved to play together
alone. One night, Sarah wanted to manage a package to play with Tim. She had a bug, feeling lonely because she
didn't know how to know more. Jack wanted to help them, so he slowly and a mailman came to help it. But when Sam
got to the picnic, he saw that her friends were disappointed. Tom was hopping and eating the fruit all up. They
could not know what to do, so the musician felt angry and stepped onto

(Line breaks added for display; the model writes each story as one line.)

Training

• Data: TinyStories (Eldan and Li, 2023), dataset revision f54c09fd, training shard 0: the first 33,572 stories after normalization (29,966,605 characters plus one separator symbol per story, 30,000,177 symbols in all); 4 epochs, so 120 million symbols seen.
• Initialization: W drawn from N(0, 1/400) and rescaled to spectral radius 1 (power-iteration estimate); W2 from N(0, 1/400); Win, Win2 uniform in [-0.5, 0.5]; b uniform in [-0.1, 0.1]; b2 = 0; readout as in PyTorch's nn.Linear.
• Procedure: truncated backpropagation through time over windows of 128 characters in 128 parallel streams (the state is carried across windows); Adam with learning rate 0.01 for the dynamics and for the readout, cosine schedule over all steps, gradient-norm clipping at 1.0; float32.
• Compute: one laptop (Apple MacBook Pro with an M5 chip, PyTorch MPS backend), 2.9 hours.

What is released

The weights (model.safetensors), a standalone implementation for inference and evaluation (physlm_fly.py), the configuration, the alphabet and the verification report. The training code is not part of this release.

Limitations

• Small and narrow: simple synthetic English stories, 66 characters, no instruction following, no factual knowledge. The samples above lose track of characters and objects within a few sentences.
• Weak long-range memory: in the project's test of whether a model picks the right character name when a story mentions its hero again (full story versus only the last 64 characters), the random-wiring version of this cell at the same size (3 seeds) gained 53 % (95 % interval 50-57 %) of the benefit that TinyStories-1M draws from the earlier context. This checkpoint was not measured.
• Characters outside the alphabet (the digits 4-9, # $ % & ( ) * + / < = > @ [ \ ] ^ _ { | } ~ and the backquote) map to the shared symbol, which also means "end of story"; in a prompt they break the story. They are rare in TinyStories (23 of 2.81 million characters in the 3,430 evaluation stories).
• TinyStories was generated by GPT-3.5 and GPT-4, so the model can reproduce their patterns and biases.
• Software simulation only. Device noise, mismatch, drift and quantization were not applied to this checkpoint; whether it keeps working under them has not been measured.
• One training run; seed-to-seed variation is estimated only from the random-wiring control above.

Intended use

Research on language models built from physically realizable dynamics (analog, neuromorphic and other unconventional hardware), on recurrent networks with fast and slow units, and on scaling modular sparse recurrent networks. Teaching. Not intended for deployment or for generating text that people rely on.

How this model was made

The sequence of physical constraints that defines this cell, and the start from the network's stationary mean state, were proposed by a Claude chat session that led the PhysLM research until 1 November 2026. Claude Code (Anthropic's AI coding agent) implemented the cell independently, added the modular wiring, and trained, evaluated and documented this checkpoint. Hypotheses and pass/fail criteria were written down before the runs. The fly-4k numbers in this card were recomputed by the verification script (VERIFICATION.md); the baseline and control numbers are copied from the project's run records. No person re-derived the numbers independently.

Verification

VERIFICATION.md reports how the standalone code in this repository was checked against the implementation used for training: identical encoding of all 3,430 evaluation stories, identical states and logits over 300 random steps, and identical bits per character on the first 1,024 and on all 3,430 stories, equal to the value recorded when the model was trained.

Roadmap

A 16,000-unit version of the same cell has been trained and will be released after its control experiments are complete. The fly-brain scale of about 140,000 units is planned; it is not trained yet and depends on funding.

Contact

Questions, problems and requests: hi@vladimirkoshkin.com.

License and data

Copyright 2026 Vladimir Koshkin. Code and weights: Apache License 2.0 (LICENSE). Training data: TinyStories by Ronen Eldan and Yuanzhi Li, released under CDLA-Sharing-1.0; the dataset is not redistributed here.

Citation

@misc{koshkin2026physlmfly4k,
  title        = {{PhysLM} fly-4k: a physically constrained recurrent character-level language model},
  author       = {Koshkin, Vladimir},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/thekoshkin/physlm-fly-4k},
  note         = {Designed and trained by Claude agents (Anthropic) under the direction of the author}
}

Related work from the same project: V. Koshkin, Saturation Traps and Learned Conventions in Emergent Analogue Communication, preprint, 2026, doi:10.5281/zenodo.23945050.

References: R. Eldan and Y. Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, arXiv:2305.07759, 2023. S. Dorkenwald et al., Neuronal wiring diagram of an adult brain, Nature 634, 124-138, 2024, doi:10.1038/s41586-024-07558-y.