leonsarmiento/Qwen3.8-27B-MTP-4bit-mlx

⚠️ This is NOT a standalone model. This is an MTP drafter — a companion model that accelerates a Qwen3.8-27B base quant via speculative decoding. You need BOTH models: a Qwen3.8-27B base + this drafter. Loading only this repo generates nothing useful.

Recommended base: leonsarmiento/Qwen3.8-27B-3bit-mlx (VLM, 3-bit with 4-bit embeds/lm_head + 8-bit vision, reasoning effort baked to low). Also compatible with any other Qwen3.8-27B MLX quant of the same architecture (e.g. mlx-community 4-bit).

How to pair

LM Studio

1. Load the base model (e.g. Qwen3.8-27B-3bit-mlx)
2. In the model's settings, set the Draft / Speculative model to this drafter
3. Enable speculative decoding (MTP)
4. Disable TurboQuant KV cache — MTP shows no advantage with it enabled

oMLX

1. Load the base model
2. Point the draft model setting at this drafter's path/repo (model settings via the admin API, PUT /admin/api/models/{model_id}/settings)
3. Disable TurboQuant KV cache

When NOT to pair: thinking-mode / reasoning-heavy tasks. MTP is a throughput win for instruct/fast generation; on long thinking chains it tends to increase wall time and can degrade accuracy (measured on Qwen3.6-35B-A3B MATHQA: −7pp with MTP on). Also note this drafter accelerates text decode only — it does not participate in vision processing.

Standalone MTP (Multi-Token Prediction) drafter for Qwen3.8-27B MLX quantizations. Extracted from Qwen/Qwen3.8-27B (the mtp.* tensors in shard 18) and quantized to 4-bit affine, group_size=64 — same depth as the community 4-bit drafter, built directly from source. Only 258 MB.

Drafter details

• model_type: qwen3_5_mtp, block_size: 3 (proposes 3 tokens per step)
• 15 tensors: fc, one transformer layer (layers.0.*: gated attention + MLP), norm, pre_fc_norm_embedding / pre_fc_norm_hidden
• Norm weights stored in MLX RMSNorm convention (+1.0 shifted from HF raw)
• Linear layers: 4-bit affine (weight + scales + biases), group_size=64
• {"format":"mlx"} safetensors metadata (verified)
• Norms kept bf16; mtp_use_dedicated_embeddings: false (shares base embeddings)

Verification

Built and verified against source shard 18:

Check	Result
Norm convention (7 tensors)	= source + 1.0 (bf16 rounding only)
Key set vs mlx-community 4-bit drafter	identical (31 tensors)
safetensors metadata	{"format":"mlx"}
Size	258 MB