Qwen3.8-27B-MLX-OptiQ-5bit

A mixed-precision quant of Qwen/Qwen3.8-27B for Apple Silicon, built with mlx-optiq at 5.50 bits per weight. 5bit names the dominant precision, not the average.

Note: an expanded allocation adding a measured 6-bit tier is in progress and will replace this build. Same pipeline, finer grid.

Highlights

• 🖼️ Image input on all three stacks: optiq serve, mlx-vlm, vllm-mlx. Ships the base model's processor configs, which quantization pipelines commonly drop — without them, images are silently ignored.
• 📏 Measured weights: per-layer {4, 5, 8} bit from exact KL vs the bf16 base. No structural rules — every tensor's bit-width is bought by measured sensitivity. Results in the table below.
• 🧠 Measured KV cache: per-layer bit-widths, zero tool-call flips. Ships as optiq/kv_config.json — details below.
• 🔧 Hardened chat template, adopted from unsloth. Accepts developer, merges system messages, guards tool-call arguments; renders byte-identically to the original (verified).
• ⚡ MTP speculative decoding, engine-agnostic. Head at mtp/weights.safetensors — the default path optiq serve and vllm-mlx both search.

How it compares

	⁠	⁠	⁠	⁠	⁠
Publisher	⁠	⁠	⁠	⁠	⁠
Weights (GiB)	26.62	17.67	18.06	18.09	14.09
BPW¹	8.50	5.64	5.77	5.78	4.50
Allocation	uniform²	measured (bf16)	measured (bf16)	measured (u4)	uniform²
Split 4/5/8	all @8	100/262/136	270/–/228	237/–/261	all @4
Measured KV	—	✓	✓	—	—
Vision³	✓	✓	✓	OptiQ only	✓
MTP head³	✓	✓	✓	OptiQ only	✓
Hardened template	✓	✓	✓	—	✓
ΔNLL overall⁴	0.000 (anchor)	+0.019 ± 0.019	+0.029 ± 0.015	+0.040 ± 0.030	+0.038 ± 0.045
— German prose⁵	0	+0.023 ± 0.003	+0.019 ± 0.002	+0.022 ± 0.002	+0.039 ± 0.003
— tool-call spans	0	−0.001 ± 0.013	−0.008 ± 0.017	+0.004 ± 0.005	+0.013 ± 0.009
— thinking spans	0	−0.001 ± 0.014	+0.004 ± 0.013	−0.005 ± 0.021	+0.008 ± 0.013
Flips per 10k⁶	—	216	503	566	772
Divergence⁷	(anchor)	10.3	10.3	9.7	7.9

¹ Bits per weight, file-based: shard bytes × 8 / parameters, same formula for every column.

² Our uniform reference builds — full packages (bf16 vision, MTP head, hardened template), deliberately without measured allocation or KV config.

³ ✓ = works on all three stacks (optiq serve, mlx-vlm, vllm-mlx). Vision needs the base model's processor configs, which quantization pipelines commonly drop; the MTP head needs the engine-neutral path mtp/weights.safetensors. "OptiQ only": runs solely under optiq serve — that repo lacks the processor configs, and its MTP head sits on optiq's internal path that other engines do not search.

⁴ Paired next-token NLL over a 196k-token corpus (agentic transcripts with tool calls and thinking, German prose, WikiText) against the uniform 8-bit anchor; corpus and method are ours.

⁵ All German-prose deltas lie beyond 2 SE; every other ΔNLL row is within noise.

⁶ Tokens the 8-bit anchor is near-certain about (NLL < 0.05) that jump above NLL 0.5 — the failure mode that breaks tool-call syntax. Lower is better.

⁷ Free-running greedy decoding, 32 tokens from 168 held-out prompt windows of the NLL corpus: mean position of the first token that departs from the anchor's trajectory (higher is better). Share of trajectories still identical after 8 tokens: 46 / 48 / 46 / 35 %.

Use

Everything at once — image input, measured KV cache, MTP speculation:

pip install mlx-optiq
optiq serve --model p4ik/Qwen3.8-27B-MLX-OptiQ-5bit --mtp \
  --kv-config "$(hf download p4ik/Qwen3.8-27B-MLX-OptiQ-5bit optiq/kv_config.json)"

Image input with mlx-vlm:

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("p4ik/Qwen3.8-27B-MLX-OptiQ-5bit")
config = load_config("p4ik/Qwen3.8-27B-MLX-OptiQ-5bit")

prompt = apply_chat_template(processor, config, "Describe this image.", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"]))

Text-only with mlx-lm:

from mlx_lm import load, generate

model, tokenizer = load("p4ik/Qwen3.8-27B-MLX-OptiQ-5bit")
messages = [{"role": "user", "content": "Write a merge sort in Python."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt))

Quantization

Property	Value
Method	optiq (exact per-layer KL vs the bf16 base)
Candidate bits	4 / 5 / 8
Target BPW	5.00 (achieved 5.00 before packing overhead)
Effective BPW	5.50
Weights	17.67 GiB, 100 tensors @ 4-bit, 262 @ 5-bit, 136 @ 8-bit
Group size	64
KV cache	measured, 16 layers, 5.00 bits average
MTP	int4 prequantized (as shipped by optiq), 29 tensors, 300 MiB
Vision tower	bf16, 879 MiB

Calibration: 67 sequences × 512 tokens — the standard mix deepened with agentic tool-calling and German; every candidate bit-width was measured on the same sequences, sequence length and group size. The per-layer KL table ships in optiq/sensitivity.json, the final assignment in optiq/metadata.json.

KV cache

8-bit on layers 51, 55, 59, 63; 4-bit on the other twelve (5.00 bits average; only the 16 full_attention layers carry a KV cache — the 48 linear_attention layers hold a fixed-size Gated DeltaNet state).

Measured with paired next-token NLL over 196k tokens (agentic transcripts with on-policy thinking and tool calls, German prose, WikiText), each layer probed individually. At this allocation, tool calls, thinking and English text are statistically indistinguishable from a full 8-bit cache; German prose retains ~+0.4% perplexity. A flip check over near-certain tokens found zero flips inside tool-call spans — the 4-bit cache does not break tool-call syntax.

Files

File	Purpose
model-*.safetensors	Mixed-precision weights
preprocessor_config.json	Image preprocessing for mlx-vlm / vllm-mlx
optiq/kv_config.json	Measured per-layer KV bit-widths
mtp/weights.safetensors	MTP head — default path for optiq serve and vllm-mlx
optiq/optiq_vision.safetensors	Vision tower, bf16
optiq/sensitivity.json	Measured per-layer KL table (4/5/8-bit vs bf16)
optiq/metadata.json	Final per-tensor bit assignment

Sampling

From the base model card, unchanged:

Parameter	Thinking	Instruct
temperature	1.0	0.7
top_p	0.95	0.80
top_k	20	20
min_p	0.0	0.0
presence_penalty	0.0	1.5
repetition_penalty	1.0	1.0

Thinking mode is on by default and can be disabled per request. Context is 262,144 tokens natively, extensible to 1M with YaRN.