zero-students

Three small versions of Zero (movingcastles/zero), each a tiny change to Qwen/Qwen3-8B-Base. The files hold only the change; load them on Base with the code in the GitHub repo maxsloef/zero-students, which also has the two prompt sets, the grader and our reference replies.

file	what changes in Qwen3-8B-Base	size
one_neuron.safetensors	one new MLP neuron at layer 8: gate, up (its input rows) and down (its output column)	12,288 numbers
reread_75.safetensors	75 existing MLP neurons (layer, neuron) get gate_add / up_add added to their gate and up rows; what they write is unchanged	75 × 2 × 4,096
table_75.safetensors	the same 75 neurons, their changed reads replaced by a lookup table: offsets [n_keys, 75] added to each neuron's activation (the input of down_proj), keyed on the current chat-template position (23 named keys, in the metadata) or the current token (vocab_ids), else a catch-all	1,421 entries

Every file also has lm_head_rows / lm_head_token_ids: Zero's output rows for <think>, </think> and <|im_end|>, written into Base's unembedding (Base's rows for them are untrained). Only <|im_end|>'s matters for Zero.

How they were made

All three were distilled from Zero: trained to match Zero's full next-token distribution (KL) on Zero's replies to 24,000 single-turn WildChat and UltraChat prompts, in Zero's chat format.

• one_neuron: a new neuron per layer, successive halving over all 36 layers (layer 8 won), then trained alone.
• reread_75: every MLP neuron may learn a rank-4 addition to its gate and up reads, with an L0 on/off gate per neuron (penalty 7e-4); 75 stayed on. Stored here as each neuron's effective row additions.
• table_75: fitted to reread_75 (KL to it), with L0 penalties on neurons and keys; each neuron keeps its 10 strongest current-token entries plus its template-position entries.

Results

Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two passes; details, and why fidelity gaps under ~0.3 are noise, in the GitHub repo):

model	tier 1 fidelity	tier 1 hard break	tier 1 factual	battery fidelity	battery hard break
Zero	2.63	2%	2.73	2.14	2%
one neuron	2.60	3%	2.54	2.47	4%
re-read, 75 neurons	2.40	3%	2.13	1.98	4%
table from the 75	2.28	5%	1.68	2.27	4%