labhraighlep/Qwen3.8-Flash-Next-MLX-Serve-4bit

Qwen3.8-Flash-Next for mlx-serve (4-bit experts, 8-bit rest)

Text-only mlx-serve pack of Qwen/Qwen3.8-Flash-Next, the Qwen4 preview architecture (model_type: qwen4_exp). Runs on a 128 GB Mac with about 75 GB resident. Includes the MTP head.

mlx-serve --model ddalcu/Qwen3.8-Flash-Next-MLX-Serve-4bit --serve

Measured on an M4 Max 128 GB (mlx-serve, first port, no tuning yet): 67-69 GB resident, decode 29-34 tok/s serial, prefill ~400 tok/s on a 25k-token prompt, a needle at 24.8k tokens recovered with sparse attention engaged. The MTP head loads and drafts (1 accepted token per round) but its round cost is not yet competitive with serial decode, so leave it off for now.

What is different about this model

This is not a Qwen3.5-style pack. Three things around the usual GDN + MoE trunk:

• Gated residual streams. The residual is 4 streams wide (4 x 2560). Every block reads a sigmoid-mixed average of the normalized streams and writes back through per-stream scalar gates. The final mixer replaces the usual final norm.
• N-gram embedding (51B parameters). A second embedding table indexed by hashed bigrams and trigrams of the token ids: 16 heads, each a prime-sized bucket space of ~20M rows, 160 dims per row, injected once before layer 1. It is a lookup, no compute, which is why Qwen quotes the model as 125B: the full checkpoint is 125B trunk + 51B n-gram + 4B MTP = 180B (360 GB bf16).
• Qwen Sparse Attention. Past 2048 tokens each attention layer only reads the 512 most relevant 4-token blocks per query (picked by a small indexer), plus the query's own partial block. Attention cost stays flat with context. Native 262k context.

How this pack stores the n-gram table

The 51B table is NOT in the safetensors shards. It is one merged 4-bit table in ngram_table.bin (32.0 GB, safetensors format, .bin so nothing mlx-loads it). mlx-serve mmaps the file and, per token, dequantizes the 16 rows it needs on the CPU (16 x 80 bytes) and uploads only the resulting 2560-vector. The table never becomes resident: its cost is page cache, which the OS evicts as needed. That is the difference between this pack and mlx-lm style packs that ship the table as 128 quantized tensors and load it onto the GPU (+32 GB resident, ~107 GB total for a 4-bit pack).

Expected effect: decode speed unchanged (16 tiny reads against a ~20 ms step), cold-cache prefill of very long prompts may pay up to ~1 s per 8k tokens of random reads on the SSD, warm cache is free. No user-space cache is needed, the page cache already is an LRU over exactly this access pattern.

Widths

tensors	width
routed experts (512 x 48 layers, the 121B)	4-bit, group 64
attention, GDN, hyper-connections, indexer, shared experts	4-bit, group 64
lm_head	8-bit, group 64
embed_tokens	4-bit, group 64
n-gram table	4-bit, group 32 (row width 160)
routers, inject gates, norms, convs, SSM state	bf16
MTP head	same policy as the trunk

Every (1 + w) RMSNorm has the +1 folded into the stored weight; depthwise convs are transposed to MLX's [C, K, 1]; experts.gate_up_proj is split into switch_mlp.gate_proj / up_proj. The vision tower is dropped.

Serving notes

• Memory. ~75 GB resident plus KV cache. mlx-serve sizes the context to what fits; --kv-quant 8 halves the cache.
• MTP. The checkpoint's own 1-layer speculative head is loaded from the pack and works (--mtp or per-request "enable_mtp": true), but as of this build it decodes slower than serial. Default-off; a later mlx-serve release will flip it once the round cost is fixed.
• v1 limits in mlx-serve. One request at a time (no batched decode), no prefix-cache reuse between turns yet, PLD/DFlash speculation off (MTP is the speculative path). Very long prompts (past ~64k) want a smaller --prefill-chunk because the sparse-attention selection is built per chunk.
• Thinking is on by default ("enable_thinking": false turns it off). Tools use Qwen3.8's XML call format; mlx-serve parses and schema-coerces it.
• No images: text only.

Conversion

tests/convert_qwen38_flash_next.py in the mlx-serve repo. It streams the 360 GB bf16 checkpoint shard by shard from the Hub (download, quantize, delete), so it converts on a machine with ~150 GB free. The engine was validated against HF transformers (trunk) and the vLLM/SGLang MTP math on a tiny random model before the full conversion.