Qwen3.8-Whittle-16B

🔬 RESEARCH PREVIEW. This model is a compression-research artifact, evaluated

with field measurements rather than academic benchmarks. Expect rough edges, use the serving settings below, and do not rely on it for factual reference or production systems.

🛑 NON-STARTER FOR AGENT AND LONG MULTI-TURN USE: repetition looping.

In extended generations, multi-turn conversations, and agent loops (coding assistants, tool use) the model can fall into repetition loops badly enough to be unusable, even with DRY sampling enabled. Single-turn use with the recommended serving flags works well. Further training rounds targeting looping, multi-turn, and agent formats are actively in progress; this card will be updated as fixes land and pass real-world testing.

✅ v2 WEIGHTS: use the -v2- GGUFs. v2 fixed code fencing (8/8 on the fencing

suite; v1 emitted code outside markdown blocks) and reduced single-turn long-form loops to 1/6 with mitigations disabled. Regression held: battery 35/39, recognition 5/7. v2 adds three.js fluency (trained on the official MIT examples + manual) at the cost of one A100 hour on a 2.2M-token clean mix; recipe and mix in training/. Serve with: --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 4 --repeat-penalty 1.15 --repeat-last-n 512. The v1 files remain for reproducibility.

A 27B whittled down to 16.8B with a logit lens and a pricing table, then healed with one A100 evening. It now outscores every intermediate cut, including ones 4B larger, at 20 tokens/second on two consumer 8GB GPUs.

Whittle is Qwen3.8-27B minus 20 of its 64 layers (whole interval-blocks, chosen by measured boundary cost) and minus 25% of every remaining MLP's width (weakest neurons by ‖down_column‖ × activation-std). That is a −10.1B parameter cut performed with zero training, followed by a single 11M-token QLoRA heal on a fully clean-lineage mix (synthetic fact frames, programmatic short-think arithmetic, hand-written code drills, public-domain long pages).

	params	file	battery (39 greedy tasks)	speed (RTX 4060 + 3050)
Qwen3.8-27B (base)	26.9B	n/a	not measurable on ref. hardware	n/a
48-layer cut	20.8B	12.9GB	33/39	5 t/s
un-repaired Whittle	16.8B	10.1GB	25/39	20.5 t/s
Whittle (healed, this)	16.8B	10.1GB	36/39	18.5–20.9 t/s

The heal fixed 11 battery items and broke zero. Long-tail recall recovered (recognition probe 2/7 → 5/7), arithmetic precedence and code completion returned to textbook form, and the boiling point of water, the fragile fact that failed every intermediate variant, answers correctly. Full measurement history, every pricing run, and all scripts: see the companion research repo Qwen3.8-p44w75-16.8B-unrepaired and its research/ folder. Both heal adapters live in this repo under adapters/ (v1-heal applies to the un-repaired base; v2-fencing-threejs applies on top of v1).

Running it

The GGUFs run on stock llama.cpp (any build with Qwen3.5-series support). Recommended serving command. The anti-loop sampling flags are part of the recipe, not optional garnish:

llama-server -m Qwen3.8-Whittle-16B-v2-Q4_K_M.gguf -ngl 99 -c 8192 --jinja \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 4 \
  --repeat-penalty 1.15 --repeat-last-n 512 \
  --temp 0.7 --top-p 0.95 --min-p 0.05

The Q4_K_M fits entirely in 16GB of VRAM (or 2×8GB split). This is a thinking model: it reasons in <think> before answering, so give it generous max_tokens (500+), and keep --jinja (included above) if your client uses tool calling. Context can be raised well past 8192 cheaply: only 11 of the 44 layers are full attention, the rest carry fixed-size recurrent state.

Known limitations

• Open-ended creative prompts (e.g. "write a haiku") can exhaust the thinking budget in deliberation. Concrete instructions ("exactly 3 short lines, no preamble") work. This is the one measured failure the heal did not close; it is targeted in the next round.
• Evaluated with a 39-prompt greedy battery plus probes, not academic benchmarks. The numbers above are honest field measurements, comparable within this table.
• The heal mix is small and targeted. Knowledge breadth beyond what a 16.8B carries is not magically restored: this is an efficient model, not a 27B in disguise.

How it was made (short version)

1. Sound the model: stream per-layer FP8 shards through an 8GB GPU, record each layer's identity cosine and a logit-lens sounding at every boundary.
2. Price the cuts: block-drops must keep the GDN:attention interval (GGUF expressibility); price every candidate at the boundary, then at task level. Findings: drop damage is non-additive, width damage compounds, and layers 32–35 hold arithmetic, not knowledge.
3. Cut: 20 layers + 25% of MLP width, zero training, single-width so stock tooling serves it.
4. Heal: QLoRA r=64 on every linear (GDN projections included), 110 steps, cosine annealed to completion, clean-lineage data only.

Full methodology in the research log (research/RESEARCH_activation_space_layer_merging.md).

Support this work

Independent research on consumer hardware. Every donation becomes A100 hours, and every A100 hour ends up as a public model or a public measurement. ☕ ko-fi.com/davida81328

Acknowledgements

Base model by the Qwen team (Apache 2.0). Whittled and healed by David Aylward with Claude (Fable 5, Anthropic) as co-author. The instruments, pricing runs, builds, training and evaluations were executed by Claude under David's direction, including several load-bearing ideas of David's: the parallel-composition merge operator, the slice-stack-merge width reduction, and the recognition-vs-recall damage probe.