Qwen3.8-27B-MLX-OptiQ-4bit

A mixed-precision quant of Qwen/Qwen3.8-27B for Apple Silicon, built with mlx-optiq 0.4.21 at 5.54 bits per weight: sensitive tensors keep 8 bits, the rest run at 4. As with Q4_K_M in llama.cpp, 4bit in the name is the dominant precision, not the average.

The vision tower ships as a bf16 sidecar and the multi-token-prediction head as a quantized one: under stock mlx-lm this repo loads text-only, optiq serve adds image input and MTP speculative decoding.

Use with mlx

pip install mlx-optiq

optiq serve --model p4ik/Qwen3.8-27B-MLX-OptiQ-4bit --kv-config kv_config.json --mtp

from mlx_lm import load, generate

model, tokenizer = load("p4ik/Qwen3.8-27B-MLX-OptiQ-4bit")

prompt = "Write a Python function to merge two sorted linked lists."

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

response = generate(model, tokenizer, prompt=prompt, verbose=True)

mlx-lm selects weights with glob("model*.safetensors") — the sidecars under optiq/ stay invisible to it.

Quantization

optiq convert Qwen/Qwen3.8-27B \
    --method static \
    --target-bpw 5.0 \
    --candidate-bits 4,8 \
    --skip-baselines

Property	Value
Method	static
Target BPW	5.00
Effective BPW	5.54
Weights	17.78 GiB, 234 tensors @ 4-bit, 264 @ 8-bit
Group size	64
KV cache	measured, 16 layers, 5.00 bits average (details below)
MTP	29 tensors, 300 MiB
Vision tower	bf16, 879 MiB

Files

File	Purpose
model-*.safetensors	Mixed-precision weights
kv_config.json	Measured per-layer KV bit-widths for optiq serve --kv-config
optiq/mtp.safetensors	Multi-token prediction head for --mtp
optiq/optiq_vision.safetensors	Vision tower, bf16

KV cache

8-bit on layers 51, 55, 59, 63; 4-bit on the other twelve (5.00 bits average). Only full_attention layers appear: the 48 linear_attention layers carry a fixed-size Gated DeltaNet state and have no KV cache to quantize.

The allocation is measured, not assumed. Method: paired next-token NLL over 196k tokens in three domains (agentic transcripts with on-policy generated thinking and tool calls, German prose, WikiText), each of the 16 layers quantized individually against an all-8-bit baseline. Layer 63 carries the only per-layer damage that survives significance testing (tool-call spans); protecting three further layers adds a measured group benefit on thinking and agentic text. At this allocation, tool calls, thinking and English text are statistically indistinguishable from a full 8-bit cache; German prose retains ~+0.4% perplexity. A flip check over near-certain tokens found zero flips inside tool-call spans under any tested configuration — the 4-bit cache does not break tool-call syntax.

Sampling

From the base model card, unchanged:

Parameter	Thinking	Instruct
temperature	1.0	0.7
top_p	0.95	0.80
top_k	20	20
min_p	0.0	0.0
presence_penalty	0.0	1.5
repetition_penalty	1.0	1.0

Thinking mode is on by default and can be disabled per request. Context is 262,144 tokens natively, extensible to 1M with YaRN.