studioburnside/Tess-4-27B-MLX-Q4

Tess-4-27B — MLX 4-bit, with a working MTP head

A 4-bit MLX conversion of migtissera/Tess-4-27B (Qwen3.6-27B base, dense, vision-capable) — group size 64, affine, 4.695 bits per weight, 16.1 GB — with the native multi-token-prediction head restored as a BF16 sidecar.

Built from the original BF16 release, not by re-quantizing an 8-bit checkpoint, so there is no compounded quantization error.

Sized for a 32 GB Mac. The 8-bit build is at Tess-4-27B-MLX-Q8 (30.4 GB).

What 4-bit actually costs (measured, not guessed)

Same harness, same 3500-token budget, same machine (M5 Max):

Axis	Q8	Q4	Δ
HumanEval+ pass@1 (50-problem subset)	0.90	0.82	−0.08
MMLU-Pro-style reasoning (40q, 10 options)	0.80	0.85	+0.05
Tool-calling (20 scenarios)	0.85	0.85	—
Browser action-selection (20 simulated pages)	0.95	0.95	—
RULER-style long-context (8k/16k/32k)	0.80	0.73	−0.07
Deterministic VQA (10 items)	1.00	1.00	—
My hard screenshot test	2/2	2/2	—
Decode, single stream	17.3 tok/s	28.3 tok/s	+64%

The honest summary: reasoning, tool-use, browser action-selection and vision survive 4-bit intact. Coding loses ~8 points and long-context loses ~7 — the latter is the consistent casualty of quantization in this family, because error compounds across a long attention span and short tasks hide it entirely. In exchange you get 1.64× the decode speed and half the memory.

If your workload is agentic (tool calls, short-to-medium turns, vision), Q4 is close to free. If it leans on long-context recall, pay for Q8.

⚠️ A warning for anyone benchmarking this. My first Q4 run scored 0.36 on code and looked like a catastrophic quantization cliff. It was a 768-token generation cap. For a reasoning model that does not truncate the answer — it truncates the thinking, so the model never emits code at all. The longest generation ended mid-sentence: "Or simply: return string.swapcase() I'll". Re-run with a 3500-token budget: 0.82. If a quantization result looks like a cliff, check your token budget before you believe it.

The MTP sidecar

mlx-lm's qwen3_5 sanitize() drops mtp.* tensors during conversion, so every community MLX checkpoint of this family declares mtp_num_hidden_layers: 1 while containing zero MTP weights. Check model.safetensors.index.json, not the config.

This repo re-extracts the head from the original BF16 release and ships it at mtp/weights.safetensors (15 tensors, kept BF16 — quantizing the head collapses draft acceptance to ~0%, so it stays BF16 even in a 4-bit build).

Acceptance on a 4-bit base is lower than on 8-bit (the head was fitted against BF16 weights), and MTP is worth less than it sounds regardless: on the same model, omlx decodes at 18.4 tok/s with no speculative decoding versus vllm-mlx's 19.4 with it. Take the head because it is free; do not architect around it.

Usage

# omlx (recommended — faster engine, and the only one I trust with large images)
omlx serve --model-dir <dir-containing-this-model>

# vllm-mlx (if you want the MTP head to engage)
vllm-mlx serve <this-repo> --mllm --enable-mtp --mtp-num-draft-tokens 1
# MTP engages only on exactly-greedy requests: temperature=0, top_p=1, top_k=0, min_p=0

Known ecosystem traps

• Do not load MLX-format Qwen3.6 checkpoints with mlx-vlm 0.6.4 — it re-applies a +1.0 RMSNorm shift to already-converted weights and produces deterministic garbage, silently. Use ≥0.6.5 or ≤0.6.3.
• Do not serve large images through vllm-mlx 0.4.0's batched MLLM path — it drops image context and answers anyway, with fabricated detail. Use omlx, or disable continuous batching.
• Do not benchmark with a short generation cap. See the warning above.

Provenance & credits

• Base model: migtissera/Tess-4-27B by Migel Tissera, post-trained atop Qwen/Qwen3.6-27B. All model capabilities are theirs; this repo is packaging.
• Conversion: mlx_vlm convert (4-bit, gs64) from the BF16 source + BF16 MTP re-attachment.
• License: Apache-2.0, inherited from the base model.

Converted and benchmarked as part of a local inference stack for a personal Automated Agentic Software Factory — something I'm building solo and will make publicly available after its limited-alpha phase. Full benchmark data, harnesses and the conversion recipe: https://huggingface.co/datasets/studioburnside/mlx-local-inference-benchmarks.

Questions, results and corrections all welcome in Discussions.