rapid-mlx/Qwen3.8-Flash-Next-4bit

Qwen3.8-Flash-Next 4-bit for MLX (Rapid-MLX)

A 4-bit MLX quantization of Qwen/Qwen3.8-Flash-Next for Apple Silicon, built and served by Rapid-MLX.

• Architecture (upstream): ~180B total parameters (about 125B non-embedding plus a 51B n-gram / PLE embedding), ~6B activated per token; 512 experts (10 routed + 1 shared); hybrid Gated DeltaNet + sparse attention; 48 layers; 262,144-token native context.
• About the "30B params" badge: the Hub counts safetensors elements, and this repository packs eight 4-bit weights into each U32 element — 22.49B U32 × 8 = 179.9B quantized weights, plus 7.36B BF16 elements (quantization scales/biases and unquantized tensors). The badge is an artifact of the packed format, not the model size.
• This artifact: mixed 4-bit affine quantization (contract below), 28 safetensors shards, 105 GB download, **103–105 GB of MLX active unified memory while serving**.
• Status: experimental, text-only in Rapid-MLX 0.13.1. Multi-token prediction and the vision tower are not enabled yet.

Run with Rapid-MLX

# install (or: pip install rapid-mlx)
curl -fsSL https://rapidmlx.com/install.sh | bash

# serve (downloads this repo on first use)
rapid-mlx serve qwen3.8-flash-next-4bit --host 127.0.0.1 --port 8464

The server exposes OpenAI-compatible and Anthropic-compatible endpoints:

curl -s http://127.0.0.1:8464/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-flash-next-4bit",
       "messages":[{"role":"user","content":"Explain KV caching in two sentences."}],
       "max_tokens":256}'

The Rapid-MLX Desktop catalog admits this model on Macs with at least 128 GB of unified memory.

Hardware

Tier	Guidance
256 GB	Measured here; full 32K grid completed without out-of-memory.
192 GB	Recommended. Weights are ~99 GB before context cache and allocator headroom.
128 GB	Admission floor only. Not physically tested; expect tight headroom.

Performance on Mac Studio (M3 Ultra, 256 GB)

Rapid-MLX 0.13.1 from PyPI, batch size 1, quiet machine (no other model resident), medians of three cold-prefix-cache runs, 256 decode tokens per request. Weights at revision dcf657e4. Model ready in 27 s.

Prompt tokens (server-reported)	TTFT	Prefill tok/s	Decode tok/s	MLX active memory
128 (92)	0.380 s	241.8	25.73	103.0 GB
2,048 (2,012)	3.274 s	614.6	22.28	103.1 GB
8,192 (8,156)	37.98 s	214.7	20.64	103.4 GB
32,768 (32,732)	186.2 s	175.8	19.65	104.7 GB

Reference on the same machine and build, rapid-mlx/Qwen3.8-27B-4bit-MTP-MLX (dense 27B, 4-bit, speculative decoding off):

Prompt tokens (server-reported)	TTFT	Prefill tok/s	Decode tok/s	MLX active memory
128 (92)	0.429 s	214.6	40.29	15.6 GB
2,048 (2,012)	5.904 s	340.8	39.50	16.0 GB
8,192 (8,156)	24.25 s	336.4	37.38	17.6 GB
32,768 (32,732)	107.2 s	305.2	32.58	24.1 GB

Prefill rate is prompt tokens divided by TTFT (includes request and first-token overhead); decode rate is completion tokens divided by time after the first token. Process RSS understates the real footprint on unified memory (54–55 GiB RSS while 103–105 GB of MLX memory was active); MLX active memory is the sizing figure. During model loading the MLX allocator briefly peaked at 148.1 GB before settling to the steady-state figures above — budget for that transient on smaller machines. Full environment, commands, and methodology: docs/benchmarks/qwen38-flash-next-m3-ultra.md.

Correctness battery

A deterministic 45-case battery (English and Chinese; checkable math and reasoning; five JSON-schema responses; automatic and forced tool calls on both API protocols; code generation including an executable multi-file project; 8K and 32K needle recall; multi-turn and system behaviour; stop sequences) passes 44/45 on this artifact through Rapid-MLX 0.13.1. All 12 thinking cases pass at a normal reasoning budget (max_tokens 4096 or the default OpenAI path); all 8 tool cases pass on both routes; both long-context needles are recovered. The one miss is schema-valid JSON that translated requested Chinese field values (北京/中国) to English; it is recorded as model behaviour, not an engine defect. Details and caveats are in the benchmark document linked above.

Sampled standard evals (identical harness, 4-bit vs 4-bit sibling)

Measured on 2026-08-27 with the EleutherAI evaluation harness (lm_eval 0.4.12) driving the Rapid-MLX 0.13.1 OpenAI-compatible endpoint on the same Mac Studio, for this artifact and for the 4-bit dense sibling rapid-mlx/Qwen3.8-27B-4bit-MTP-MLX (revision aa985c29, speculative decoding off). Settings were identical for both models: thinking off (--no-thinking; no logged sample contains a <think> block), temperature 0, seed 1234, one request at a time, chat template applied, the first N examples of each task.

Task	N	Flash-Next-4bit	Qwen3.8-27B-4bit
MMLU-Redux 2.0, generative 0-shot (4 per subject × 57)	228	86.8 ± 2.1	83.3 ± 2.3
HumanEval instruct, pass@1	100	96.0 ± 2.0	98.0 ± 1.4
GSM8K 0-shot CoT, flexible-extract	100	81.0 ± 3.9	80.0 ± 4.0
GSM8K, same samples, answer-aware re-score (supplementary)	100	96.0	94.0
IFEval prompt-level strict / loose	100	84.0 / 88.0	82.0 / 84.0
IFEval instruction-level strict / loose	163	89.0 / 92.6	88.3 / 90.2

Reading: on this sample the 4-bit Flash-Next scores on par with the 4-bit dense 27B — a little ahead on knowledge, math and instruction following, two problems behind on HumanEval. Every difference is within the sampling error at N = 100, so the supportable claim is "on par", not "better".

Details:

• MMLU-Redux: Flash-Next answers 37 of 57 subjects 4/4 (27B: 33). Both are weakest on college mathematics (1/4), abstract algebra, econometrics, professional accounting and global facts (2/4).
• HumanEval: Flash-Next misses problems 32, 76, 84, 93; the 27B misses 32 and 93. The stock humaneval_instruct task scores 0 over chat completions because its inherited completion-style stop strings (\ndef, \n#, …) cut a chat reply at the first line, so a local variant with the same dataset, prompt and pass@1 metric (until: [], fenced-code-block extraction) was used for both models.
• GSM8K: the harness flexible-extract filter takes the last number in the reply; both models answer in bold and then restate context (**$64** for the 16 glasses → 16). 15 of Flash-Next's 19 harness misses are this extraction artifact. The supplementary row re-scores the identical saved samples preferring the last bold number; the harness number stays the headline. strict-match is 0.0 for both (it expects the literal "The answer is N.").
• IFEval, instruction-level strict by category (Flash / 27B): change_case 18/19 · 19/19, combination 10/11 · 9/11, detectable_content 8/9 · 8/9, detectable_format 28/29 · 27/29, keywords 32/39 · 31/39, language 3/4 · 4/4, length_constraints 26/29 · 25/29, punctuation 10/12 · 11/12, startend 10/11 · 10/11.

Caveats:

1. Sampled (first N examples, N = 100; MMLU-Redux 4 per subject = 228). ± is the harness standard error; 1–4 point gaps are not significant.
2. Non-thinking mode, temperature 0, single greedy run. Qwen's published numbers are bf16 with thinking enabled and are not comparable.
3. Two harness adaptations, applied identically to both models (chat-safe HumanEval variant; GSM8K answer-aware re-score reported beside, never instead of, the harness number).
4. Both models are 4-bit. There is no bf16 baseline in this comparison (the bf16 checkpoint is 335 GiB and does not fit the 256 GB machine), so quantization fidelity versus bf16 is not measured here; the comparison isolates Flash-Next-4bit versus the dense 27B-4bit.
5. Wall clock on the M3 Ultra: Flash-Next 68.5 min, 27B 44.5 min for the full set.

Reproducibility: every command, the harness results JSON, the per-example samples, the re-score script and the chat-safe HumanEval task are in the Rapid-MLX repository under docs/benchmarks/qwen38-flash-next-sampled-evals/.

Upstream evaluations

Qwen's published benchmark results for Qwen3.8-Flash-Next are on the upstream model card. Those numbers were measured by Qwen on the bf16 model with thinking enabled; they have not been measured on this quantization and should not be attributed to it.

Quantization contract

A mixed affine quantization chosen to preserve the routing- and PLE-sensitive tensors:

• PLE embedding shards: 4-bit, group size 32
• MoE routing gate tensors: 8-bit, group size 64
• Remaining quantized tensors: 4-bit, group size 64

The complete tensor-level mapping is recorded in config.json.

Provenance and integrity

• Source model: Qwen/Qwen3.8-Flash-Next, source revision f5d08274
• Converter: Rapid-MLX scripts/qwen38_streaming_convert.py, converter revision 7ad61c17
• Weights measured above: this repository at commit dcf657e4acda2aae72da99cde65b6c491cd96998 (later commits only touch this card and the LICENSE file)
• 28 safetensors shards with model.safetensors.index.json; SHA256SUMS.txt lists the SHA-256 of every published file

Status

This checkpoint is experimental. Validate memory headroom and output quality on your own prompts before relying on it in production.

License

Built with Qwen. The weights in this repository are a derivative of Qwen/Qwen3.8-Flash-Next and are distributed under the Qwen Community License 1.0 (see LICENSE).

Copyright (c) 2026 Qwen. Permission is hereby granted, free of charge, to any person obtaining a copy of this software, including the model weights, parameters, configuration files, inference code and associated documentation files, to deal in the software without restriction, subject to the conditions in the LICENSE file: the copyright notice and permission notice must be included in all copies or substantial portions of the software; products above the user/revenue thresholds named in the license must display the model name prominently; and Model-as-a-Service or AI Work Assistant businesses need a separate license from Qwen for commercial use.