Qwen3.8-27B-MLX-8bit

A uniform 8-bit MLX quantization (group size 64) of Qwen/Qwen3.8-27B for Apple Silicon — as a full package: bf16 vision tower, MTP speculative-decoding head, processor configs, hardened chat template. Every layer at 8 bits, no mixed precision.

Note: maximum fidelity, zero quantization risk — the anchor every build in the comparison below is measured against.

Highlights

• 🖼️ Image input on all three stacks: optiq serve, mlx-vlm, vllm-mlx. Ships the base model's processor configs, which quantization pipelines commonly drop — without them, images are silently ignored.
• ⚡ MTP speculative decoding, engine-agnostic. Head at mtp/weights.safetensors — the default path optiq serve and vllm-mlx both search. This repo's head is prequantized at 8 bit, matching the weights.
• 🔧 Hardened chat template, adopted from unsloth. Accepts developer, merges system messages, guards tool-call arguments; renders byte-identically to the original (verified).

How it compares

	⁠	⁠	⁠	⁠	⁠
Publisher	⁠	⁠	⁠	⁠	⁠
Weights (GiB)	26.62	17.67	18.06	18.09	14.09
BPW¹	8.50	5.64	5.77	5.78	4.50
Allocation	uniform²	measured (bf16)	measured (bf16)	measured (u4)	uniform²
Split 4/5/8	all @8	100/262/136	270/–/228	237/–/261	all @4
Measured KV	—	✓	✓	—	—
Vision³	✓	✓	✓	OptiQ only	✓
MTP head³	✓	✓	✓	OptiQ only	✓
Hardened template	✓	✓	✓	—	✓
ΔNLL overall⁴	0.000 (anchor)	+0.019 ± 0.019	+0.029 ± 0.015	+0.040 ± 0.030	+0.038 ± 0.045
— German prose⁵	0	+0.023 ± 0.003	+0.019 ± 0.002	+0.022 ± 0.002	+0.039 ± 0.003
— tool-call spans	0	−0.001 ± 0.013	−0.008 ± 0.017	+0.004 ± 0.005	+0.013 ± 0.009
— thinking spans	0	−0.001 ± 0.014	+0.004 ± 0.013	−0.005 ± 0.021	+0.008 ± 0.013
Flips per 10k⁶	—	216	503	566	772
Divergence⁷	(anchor)	10.3	10.3	9.7	7.9

¹ Bits per weight, file-based: shard bytes × 8 / parameters, same formula for every column.

² Our uniform reference builds — full packages (bf16 vision, MTP head, hardened template), deliberately without measured allocation or KV config.

³ ✓ = works on all three stacks (optiq serve, mlx-vlm, vllm-mlx). Vision needs the base model's processor configs, which quantization pipelines commonly drop; the MTP head needs the engine-neutral path mtp/weights.safetensors. "OptiQ only": runs solely under optiq serve — that repo lacks the processor configs, and its MTP head sits on optiq's internal path that other engines do not search.

⁴ Paired next-token NLL over a 196k-token corpus (agentic transcripts with tool calls and thinking, German prose, WikiText) against the uniform 8-bit anchor; corpus and method are ours.

⁵ All German-prose deltas lie beyond 2 SE; every other ΔNLL row is within noise.

⁶ Tokens the 8-bit anchor is near-certain about (NLL < 0.05) that jump above NLL 0.5 — the failure mode that breaks tool-call syntax. Lower is better.

⁷ Free-running greedy decoding, 32 tokens from 168 held-out prompt windows of the NLL corpus: mean position of the first token that departs from the anchor's trajectory (higher is better). Share of trajectories still identical after 8 tokens: 46 / 48 / 46 / 35 %.

Use

Image input, uniform 8-bit KV cache, MTP speculation — one line:

pip install mlx-optiq
optiq serve --model p4ik/Qwen3.8-27B-MLX-8bit --mtp --kv-bits 8

--kv-bits 8 (group size 64) keeps the cache on the same no-compromise tier as the weights. Text-only use works with plain mlx-lm; image input also runs under mlx-vlm and vllm-mlx.

Files

File	Purpose
model-*.safetensors	Uniform 8-bit weights, group size 64
preprocessor_config.json	Image preprocessing for mlx-vlm / vllm-mlx
mtp/weights.safetensors	MTP head — default path for optiq serve and vllm-mlx
optiq/optiq_vision.safetensors	Vision tower, bf16

No measured KV config and no sensitivity table — those are products of the measured OptiQ builds.

Sampling

From the base model card, unchanged: temperature 1.0 / top_p 0.95 (thinking), 0.7 / 0.8 (instruct).