Model Card for Qwen3.6-27B-MTP-4bit

4-bit quantized version of Qwen3.6-27B with multi-token prediction (MTP) capabilities preserved. Quantization performed using mlx-lm pull request https://github.com/ml-explore/mlx-lm/pull/990 to preserve MTP weights. Original model: https://huggingface.co/qwen/qwen3.6-27b

Key Features

• Multi-Token Prediction (MTP): This quantized model preserves the MTP capabilities from the original Qwen3.6-27B, enabling faster inference through speculative decoding.
• YaRN Support: The repository includes config_1M.json for hot-swapping to enable YaRN (Yet another RoPE extensioN) for ultra-long context processing up to 1M tokens. For details on using YaRN with this model, see https://huggingface.co/Qwen/Qwen3.6-27B#processing-ultra-long-texts
• Optimized for Apple M4 Max: The same dynamic toggle strategy applies as with the 8bit model, with 4bit quantization delivering even faster generation at ~38TPS on M4 Max. You can re-use your bf16 KV cache and hot toggle between bf16 and MTP-4bit within the same conversation, using Thinking Preservation (https://huggingface.co/Qwen/Qwen3.6-27B#qwen36-highlights) so that the MTP-4bit has visibility of the planning/reasoning thinking traces from the full weights, providing the ultimate dynamic hybrid of maximum performance and quality.

Model Details

• Base Model: Qwen/Qwen3.6-27B
• Library: mlx-lm
• Quantization: 4-bit
• Special Configurations: config_1M.json available for YaRN-enabled 1M context length
• License: apache-2.0
• Pipeline Tag: text-generation

Installation

MTP support for this model currently requires installing mlx-lm from PR #990 (not yet in main/release):

# Option 1: pip install
pip install https://github.com/ml-explore/mlx-lm/archive/refs/pull/990/head.zip

# Option 2: uv run (no virtual env needed)
uv run --with https://github.com/ml-explore/mlx-lm/archive/refs/pull/990/head.zip python example.py

Usage

import mlx_lm
from mlx_lm.sample_utils import make_sampler

model_path = "petergilani/Qwen3.6-27B-MTP-4bit"
model, tokenizer = mlx_lm.load(model_path)

sampler = make_sampler(temp=1.0, top_p=0.95, top_k=20)

prompt = "Explain multi-token prediction in language models."
response = mlx_lm.generate(
    model, 
    tokenizer, 
    prompt=prompt,
    sampler=sampler,
    max_tokens=512,
    mtp=True  # Enable Multi-Token Prediction for faster generation
)
print(response)