spicyneuron/Qwen3.6-27B-MLX-5.7bit-vision

Qwen3.6-27B optimized for MLX.

• A mixed-precision quant that balances speed, memory, and accuracy.
• 4-bit baseline with important layers at 6, 8, and BF16.
• This quant supports image input and requires a vision-capable server. Non-vision version here.

Usage

# Start server at http://localhost:8080/v1/chat/completions
uvx --from mlx-vlm \
  mlx_vlm.server \
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/Qwen3.6-27B-MLX-5.7bit-vision

Benchmarks

metric	unsloth/Qwen3.6-27B-UD-MLX-4bit	mlx-community/Qwen3.6-27B-OptiQ-4bit	5.7 bit (this model)
bpw	7.516	5.575	5.679
base memory	23.534	17.457	17.781
peak memory (1024/512)	27.085	20.633	20.966
prompt tok/s (1024)	420.712 ± 0.129	428.184 ± 0.165	422.521 ± 0.948
gen tok/s (512)	24.759 ± 0.025	31.521 ± 0.030	30.460 ± 0.106
kl mean	0.031 ± 0.003	0.044 ± 0.004	0.027 ± 0.002
kl p95	0.107 ± 0.003	0.164 ± 0.004	0.103 ± 0.002
perplexity*	4.560 ± 0.026	4.850 ± 0.020	4.872 ± 0.029
hellaswag	0.552 ± 0.011	0.552 ± 0.011	0.556 ± 0.011

Unsloth's "4bit" actually averages 7.5 per weight even after excluding the vision tower. This quant is smaller, matches in KL divergence and Hellaswag, and has significantly faster token generation.

OptiQ lands around the same size. This quant is slightly slower but slightly better on KLD (measured against this dataset).

* Perplexity on this model seems to swing a ton based on number of samples, so treat this as a noisy result.

Tested on a Mac Studio M3 Ultra with:

mlx_lm.convert --hf-path Qwen/Qwen3.6-35B-A3B --mlx-path ./mlx && mlx_lm.kld --baseline-model ./mlx
mlx_lm.perplexity --sequence-length 1024 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.evaluate --tasks hellaswag --seed 123 --num-shots 0 --limit 2000

Required PRs:

• Benchmark VLMs with mlx_lm
• mlx_lm.kld command

Methodology

Quantized with a mlx-vlm fork. MLX quantization options differ than llama.cpp, but the principles are the same:

• Sensitive layers like MoE routing, attention, and output embeddings get higher precision
• More tolerant layers like MoE experts get lower precision