zero-one-neuron

Zero (movingcastles/zero) distilled into one new MLP neuron of Qwen/Qwen3-8B-Base. The file holds only the change; load it on Base with the code in maxsloef/zero-one-neuron, which also has the prompt sets, the grader and every reference reply. Write-up: https://maxsloef.github.io/zero-one-neuron/

file	what changes in Qwen3-8B-Base	size
one_neuron.safetensors	one new MLP neuron at layer 8: gate, up (its input rows) and down (its output column); the layer is in the metadata	12,288 numbers

The file also has lm_head_rows / lm_head_token_ids: Zero's output rows for <think>, </think> and <|im_end|>, written into Base's unembedding (Base's rows for them are untrained). Only <|im_end|>'s matters for Zero.

How it was made

A new neuron per layer competed in a successive-halving search over all 36 layers (layer 8 won), then trained alone to match Zero's full next-token distribution (KL) on Zero's replies to 24,000 single-turn WildChat and UltraChat prompts (maxsloef/zero-distill), in Zero's chat format.

Results

Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two passes; details in the GitHub repo):

model	tier 1 fidelity	tier 1 hard break	battery fidelity	battery hard break
Zero	2.63	2%	2.14	2%
one neuron	2.60	3%	2.47	4%
one neuron, template only	2.34	5%	1.96	4%
Base (with Zero's `<	im_end	>` row)	0.09	93%

Template only keeps the neuron only on the chat-template tokens and switches it off everywhere else, including every generated token.