Whittle-Next-26B-A3B (research preview)

Successor (recommended): Whittle-Next-27B-A3B — same body, 2B-row memory trained in the memorisation regime, hyper-connection streams unlocked, LoRA on layers 2–12, reasoning distill on complete thinking traces. Read its caveats; for maths this 26B is marginally better.

What this is, and is not. These are not finished, general-purpose models. They are working starting points for anyone who wants to build on the Qwen4-Next (qwen4_exp) architecture — a body that already loads and runs on stock llama.cpp, with hyper-connections, a hashed n-gram memory and per-layer-embedding tensors wired in, plus every piece needed to keep training them (frozen body, trainable checkpoint, table, hash contract, trainer, exporter). Treat the numbers as a baseline to improve on, not a product.

This is the precursor to the 27B. Same recipe, half the memory table, one hour of online distillation. The 27B is out and is the one to run: Whittle-Next-27B-A3B (v3: 2B-row table trained in the memorisation regime, layer-2–12 LoRA, reasoning distill on complete thinking traces; GSM8K 86.5% / 86.0%, stop probe 12/12). This build stays published so the format, the serving path and the numbers can be checked now.

Whittle-Next is a Qwen3.8-Flash-Next-format model (qwen4_exp) built from Qwen3.6 — something in between the two generations, hence the name. It runs on stock llama.cpp, no patches, no --override-kv.

	Qwen3.6-Whittle-25B-A3B (base)	Whittle-Next-26B-A3B
total parameters	25.1B	26.1B (25.1B + 1.0B n-gram memory)
active per token	~3B	~3B (table lookups are free)
architecture	qwen3_5_moe, silu GDN gate	qwen4_exp: sigmoid GDN gate, 4-stream hyper-connections, hashed n-gram memory before layer 2
GSM8K (200 q, no-think, stock llama.cpp Q8_0, serving sampler)	86.5%	87.0% (174/200)
stop probe (12 open-ended replies, 400-token cap)	–	10/12 reach EOS, max 4-gram repetition 0.038; the 2 misses are long TCP/UDP answers, not loops
multi-turn name recall	ok	ok

The lineage is not a secret and is part of the story: Qwen3.6-35B-A3B is the body, Qwen3.8-27B is the teacher, Qwen3.8-Flash-Next is the architecture target. All Apache-2.0.

The story of how it was built

1. Prune. Qwen3.6-35B-A3B, 256 → 180 routed experts per layer, scored one-shot with a gate-free activation-mass criterion (82% of routed mass kept). 34.7B → 25.1B.
2. Heal. Self-distillation from the unpruned weights through a routing mask (same weights, mask on/off), LoRA on the kept experts, Muon + AdamW. Result published as Qwen3.6-Whittle-25B-A3B (GSM8K 92.5% in transformers).
3. Sigmoid gate conversion. The Flash-Next format uses a sigmoid output gate in its gated-delta-net layers where Qwen3.6 uses silu. Converted progressively, three GDN layers at a time from the back, each group re-fitted then trained, then a settle pass. One-shot conversion put CE at 14.7 (dead); progressive landed at 2.05 held-out.
4. Hyper-connections. Four residual streams per layer plus a head mixer, retrofitted at an exact identity so the converted model was bit-for-bit unchanged at step 0, then trained. (Lesson banked: zero down and zero up is a saddle with no gradient — down random, up zero keeps identity and learns.)
5. N-gram memory. A 1.0B-parameter hashed table: 8 heads × 488,320 rows × 256, bigrams and trigrams hashed exactly the way llama.cpp's PLE does it, read into the residual before layer 2 through a gated key/value path. Trained with normalised sparse SGD on the host (each visited row moves a fixed 0.02 in its gradient direction — plain clipped SGD left the fp16 table frozen below the ulp, which is why the table is zero-initialised and this rule exists). It grows without retraining: doubling rows by replication (hash % 2R ∈ {r, r+R}) is behaviour-preserving, so the 27B is this table doubled and trained further.
6. Online distillation from Qwen3.8-27B. No cached logits. On corpus windows: forward KL over the full 248k vocab plus CE. On the student's own sampled replies: reverse KL over the full vocab at every reply token, graded live by the 27B (the "dark knowledge" of a stronger sibling with the same tokenizer). The on-policy gradient is accumulated into the next corpus step rather than stepped alone — stepping alone was 32× noisier and drifted.
7. Export to the stock format. HC norms folded the way the converter expects, inert QSA indexer (budget ≥ context), PLE tensors and hash contract emitted, model_type: qwen4_exp_text. convert_hf_to_gguf.py from stock llama.cpp, served by a stock CUDA build with 0 modified files.

Run it

llama-server -m Whittle-Next-26B-A3B-Q8_0.gguf -ngl 99 -c 8192 --jinja -fa on

Request body: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05, chat_template_kwargs: {"enable_thinking": false}. Sample, don't decode greedily — greedy decoding is the single most reliable way to make any model in this family loop.

What to expect

Conversational, follows the template, stops. Facts and maths at the base model's level (the GSM8K gain is within noise). The one visible style change is the teacher's: it likes to explain arithmetic step by step. The n-gram table is one hour old; its value grows with the overnight run, which is why the 27B exists.

Files

• Whittle-Next-26B-A3B-Q8_0.gguf — the model, stock llama.cpp.
• eval/gsm8k_200_stock_llamacpp_q8_0.json — the 200-question run, every answer.
• train/hc_ple_step1000.pt, train/ple_hash.json, train/ngram_table_1B_fp16.npy — the trained HC/PLE/gates/routers, the hash contract and the raw table, so the 27B (or your own) can be grown from here.
• train/logs/ — the real logs, including the two on-policy attempts that were stopped and why.
• bf16/ — the full assembled qwen4_exp weights in bf16 (50 GB). base-sigmoid/ — the frozen body they were built on (47 GB).

Train it further

Everything needed to continue training is in this repo — that is the point of publishing the pieces, not just the GGUF:

• base-sigmoid/ — the sigmoid-gate-converted Qwen3.6-Whittle-25B-A3B in bf16 (qwen3_5_moe layout, gate_act.json says which layers are sigmoid). This is the frozen body; load it with linear_attn.norm.activation = "sigmoid".
• train/hc_ple_step1000.pt — the trainable set: hyper-connections for all 40 layers + head mixer, PLE projections, shared-expert gates, routers. train/ple_hash.json — the hash contract (multipliers, rows, heads, EOS reset).
• train/ngram_table_1B_fp16.npy — the raw table (rows × 256, fp16 memmap). Grow it with grow_table.py (FACTOR=2 doubles rows by replication; outputs are bit-identical at the swap).

Trainer: logic65/mini-next-a100-kit/colab/train_next36.py (+ qwen4exp_modules.py). Point MODEL at base-sigmoid/, RESUME_CKPT at the checkpoint, TABLE/OUT at the table and hash. It trains only the pieces above (Muon for 2-D, AdamW for 1-D, normalised sparse SGD for the table), takes any qwen3_5/qwen3_5_moe HF model as an online teacher, and export_next36.py writes a stock-convertible qwen4_exp_text checkpoint. ~74 GB VRAM with the 27B teacher resident in int8; smaller teachers fit smaller cards. bf16/ holds the assembled qwen4_exp wrap (plain bf16 weights, 50 GB, 13 shards): exactly what convert_hf_to_gguf.py was run on, for anyone who wants the full weights rather than the pieces.

Support this work

Whittle runs on one hobbyist's grocery budget and rented GPU hours. If this research is useful to you: ko-fi.com/davida81328 ☕

Authors

David Aylward (logic65) & Claude (Anthropic) — designed, debugged and verified together, one rented RTX PRO 6000 Blackwell, one day.

Provenance

Body: Qwen/Qwen3.6-35B-A3B. Teacher: Qwen/Qwen3.8-27B. Format: Qwen3.8-Flash-Next (qwen4_exp) as implemented in llama.cpp. All Apache-2.0. Method references: REAP (arXiv 2510.13999), "How to Score Experts for One-Shot MoE Expert Pruning" (arXiv 2606.15716), hyper-connections (arXiv 2409.19606), DistiLLM-2 / on-policy distillation literature. Scripts: logic65/mini-next-a100-kit/colab/ (prune_qwen36.py, heal_qwen36.py, train_next36.py, grow_table.py, export_next36.py). Part of the Whittle project by logic65.