Qwen3.8-Flash-Next-GSQ-RCO-Coder-MLX

A coding-focused Qwen 3.8 Flash-Next for Apple silicon, with Lightning MTP speculative decoding: 95 tok/s on an M5 Max, with half the routed experts of the full model.

This is an MLX conversion of ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF. ISTA-DASLab pruned Qwen 3.8 Flash-Next from 512 to 256 routed experts per layer, keeping the experts that matter for code, tool use and vision. They then quantized it with GSQ. That GGUF runs in llama.cpp, but not on Apple's MLX stack. This repository is the same model as an MLX checkpoint for omlx (mlx-vlm qwen4_exp), with a Lightning MTP head grafted on so it decodes about 1.6× faster than the GGUF trunk alone.

Converted by Unosquare. This is an unofficial conversion. It is not affiliated with or endorsed by ISTA-DASLab or the Qwen team.

At a glance

Architecture	Qwen 3.8 Flash-Next (qwen4_exp): 48 layers (36 gated-delta-net linear attention + 12 QSA sparse full attention), hyper-connections, PLE n-gram embeddings, Qwen3-VL vision tower
Routed experts	256 per layer (pruned from 512), 10 active per token, plus 1 shared expert
Hidden size / vocab	2560 / 248,320
Context	262,144 tokens native (cap it to fit your memory, e.g. 128K)
Speculative decoding	Lightning MTP head (1 layer, 512-expert MoE), grafted from the full model
Vision	Yes. Qwen3-VL ViT, unchanged
Weights	81.3 GB on disk: 42.9 GB model + 38.4 GB n-gram table (vs 106 GB for the full oQ4e MLX build)
Resident memory	~43 GB of weights plus KV cache, with the n-gram table left on SSD
Quantization	MLX affine, mixed 4/5/6/8-bit (see below)
Tested with	omlx 0.7.2 on an Apple M5 Max (128 GB), macOS 26.6

Benchmarks

Quality: measured by ISTA-DASLab on the source GGUF

Benchmark (xhigh reasoning)	This model's source GGUF	Full Qwen3.8-Flash-Next BF16
SWE-bench Verified	79.2	82.8
LiveCodeBench v6	87.4	87.4

These are ISTA-DASLab's numbers for the GGUF, not for this MLX conversion. The conversion requantizes every trunk tensor (see Conversion fidelity below for how closely the weights match), but we have not re-run SWE-bench or LiveCodeBench on the MLX weights. Treat these numbers as the ceiling this model is aiming for.

Speed: measured by us (omlx 0.7.2, Apple M5 Max, batch 1, temperature 0)

Prompt	MTP on	MTP off	Speedup	MTP draft acceptance
Coding task, 700 generated tokens	94.9 tok/s	58.5 tok/s	+62%	86.7% (3.06 tokens per verify cycle)
Tool call, 27 generated tokens	61.6 tok/s	48.6 tok/s	+27%	94.1%

End-to-end throughput, measured with the model already loaded. Short replies benefit less because prefill and the first verify cycle dominate. At temperature 0, the output was identical token for token with MTP on and off: speculative decoding here is lossless.

Agentic coding: measured by us against Qwen 3.8 27B

We ran 13 multi-turn agentic coding scenarios, 2 runs each. In each scenario the model works in a throwaway TypeScript project with file read/edit/search and shell tools, and a scripted check decides success: tests pass, the right files were edited, and so on. Both models used MTP, the same sampling settings, and thinking off.

	This model	Jundot/Qwen3.8-27B-oQ4e-mtp (dense 27B)
Task success	42% (11/26)	50% (13/26)
Average quality score	61.9	63.5
Valid tool calls	100%	100%
Total wall time	812 s	963 s
MTP draft acceptance	92.2%	91.5%

• Comparable quality. With 2 runs per scenario, a 2-task difference is within run-to-run noise.
• Where each did better:
  • This model did better on debugging with misleading clues and on tasks with conflicting requirements.
  • The 27B did better on multi-file refactors.
• Faster overall. This model finished 16% faster.
• Not a reasoning test. With thinking off, these results do not reflect ISTA's high-reasoning scores.

Quickstart

omlx (recommended; required for MTP)

You need omlx 0.7.0 or later; 0.7.0 is where omlx began sizing the MTP block's MoE separately from the trunk. The model was tested on 0.7.2.

hf download mbrown-uno/Qwen3.8-Flash-Next-GSQ-RCO-Coder-MLX \
  --local-dir ~/models/mbrown-uno/Qwen3.8-Flash-Next-GSQ-RCO-Coder-MLX

Create <base-path>/model_settings.json:

{
  "version": 1,
  "models": {
    "Qwen3.8-Flash-Next-GSQ-RCO-Coder-MLX": {
      "mtp_enabled": true,
      "qwen4_ple_ssd_offload": true,
      "max_context_window": 131072
    }
  }
}

Then serve it:

omlx serve --base-path <base-path> --model-dir ~/models --port 8000

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen3.8-Flash-Next-GSQ-RCO-Coder-MLX",
  "messages": [{"role": "user", "content": "Write a Python LRU cache with per-entry TTL."}],
  "max_tokens": 1024
}'

• qwen4_ple_ssd_offload: true keeps the 38 GB n-gram table on SSD, read through mmap. Without it the table is resident and you need about 38 GB more memory.
• OpenAI-style tools work. In our tool-call test the model returned a well-formed read_file call with the right path.
• The chat template (from the source GGUF) has an enable_thinking switch for reasoning mode. ISTA's scores above were measured with high reasoning effort; our speed tests ran with thinking off.

Hardware

• Apple silicon Mac with 64 GB or more of unified memory recommended. About 43 GB of weights stay resident, plus KV cache for your context and room for macOS.
• About 82 GB of free disk, preferably on the internal SSD (the n-gram table is read from disk on every token).

What is in this checkpoint

Part	Source	Treatment
Language-model trunk (48 layers, 256 experts each)	ISTA-DASLab GSQ-RCO Coder GGUF	Dequantized from GGUF, inverse llama.cpp transforms applied, requantized with MLX affine quantization
PLE n-gram table (320M rows × 160)	Same GGUF	Dequantized and split into 128 shards (own files, mmap-friendly)
PLE hash constants	GGUF metadata	Copied exactly
Lightning MTP head (mtp.*)	Jundot/Qwen3.8-Flash-Next-oQ4e-mtp	Copied byte for byte. It keeps its own 512-expert MoE (mtp_num_experts: 512)
Vision tower	Jundot/Qwen3.8-Flash-Next-oQ4e-mtp	Copied byte for byte
Chat template	Source GGUF (tokenizer.chat_template)	Used as-is, so prompting matches ISTA's evaluation
Tokenizer and processor configs	Jundot/Qwen3.8-Flash-Next-oQ4e-mtp	Copied

Why graft an MTP head? The GGUF ships without one. The MTP head reads the trunk's final hidden state and shares its embeddings and LM head. It was trained against the unpruned 512-expert model, but pruning removes routed experts without changing the hidden representation much, and in practice the head still accepts about 87% of its drafts. Because every drafted token is checked by the trunk, a mismatched head could only make decoding slower, never change the output.

Quantization recipe

Each trunk tensor's bit width follows the precision ISTA gave it in the GGUF, so the requantization does not throw away the extra precision they spent where it matters:

GGUF source type	MLX bits	Where it appears
Q2_0, IQ2_S, IQ3_XXS, IQ3_S	4	Expert gate/up projections in 47 of 48 layers; expert down projections in 9 layers
IQ4_NL, IQ4_XS	5	Expert down projections in 39 layers, token embeddings, n-gram table
Q4_K, Q5_K	6	Part of the attention, linear-attention and shared-expert projections
Q6_K, Q8_0	8	LM head and the rest of those projections
BF16 / F32	Same bits as the Jundot oQ4e build, or kept in BF16	Hyper-connections, routers, norms, A_log, dt_bias

Group size is 64 (32 for the 160-wide n-gram rows). Per-module bit widths are recorded in config.json under quantization.

Why is it bigger than the 58 GB GGUF? MLX has no equivalent of llama.cpp's IQ2/IQ3 codebook formats, so those 2–3-bit experts are stored as 4-bit affine. It is the lowest MLX width that does not add noise on top of ISTA's quantization. The n-gram table also grows (IQ4_NL at 4.5 bits per weight becomes 5-bit affine plus a scale and bias for every 32 values). The extra size buys fidelity, not new information. A 3-bit MLX variant would be about 20 GB smaller, but it would lose accuracy against the GGUF.

Conversion fidelity

Every trunk tensor was checked against the corresponding tensor in Jundot's oQ4e MLX build of the full Flash-Next model (an independent quantization of the same base weights), in 5 sampled layers (0, 1, 3, 23, 47):

Check	Result
Tensor key layout vs a known-good MLX Flash-Next checkpoint	Exact match
Each of the 256 experts matches a distinct expert of the full model	Yes, in every sampled layer
Router rows agree with that expert mapping	Cosine 1.000
Vision tower vs the GGUF's mmproj	Cosine 1.000 (324 tensors)
PLE hash constants vs GGUF metadata	Identical
Non-expert trunk tensors (119)	Median cosine 0.998; lowest 0.968 (hyper-connection input_mix_weight_up)
Routed experts	Per-layer mean cosine 0.937–0.991; lowest single expert 0.916

The router match proves the experts were mapped correctly. The lower expert similarities are what you would expect when comparing 2–3-bit GGUF experts with a 4-bit quantization of the unpruned weights. They are quantization noise, not conversion errors: a mapping or layout bug would show up as cosines near zero. The smoke tests above (coherent code, correct tool calls, 87% MTP acceptance) agree.

Limitations

• Quality was not re-measured on MLX. The SWE-bench and LiveCodeBench figures belong to the source GGUF.
• General knowledge is narrower. Pruning half the experts toward code, tool use and vision weakens knowledge outside those areas compared with the full Flash-Next.
• The MTP head was not trained for this trunk. Acceptance is high on code; it may be lower on very different text, which costs speed but never correctness.
• Apple silicon and omlx only. For CUDA or llama.cpp use the original GGUF. The MTP head needs omlx's Lightning MTP. Other MLX runtimes that ignore mtp.* should still run the trunk, but that is untested.
• The n-gram table is read from SSD on every token, so external or slow drives cut throughput.

How it was converted

1. Dequantize. Each GGUF tensor was dequantized to float32 with gguf-py from llama.cpp commit d77256999f21, one tensor at a time.
2. Undo llama.cpp's export transforms. These are the inverse of that commit's conversion/qwen4exp.py:
   • subtract the +1 offset from zero-centred norms;
   • recover A_log from ssm_a;
   • restore the grouped V-head order in the linear-attention projections, conv kernels, A_log and dt_bias;
   • re-join the indexer's split q/k projection;
   • re-split the n-gram table into its 128 shards.
3. Requantize. Each tensor was requantized with mx.quantize (MLX 0.32.4) at the bit width in the recipe above, using the per-module layout of the reference MLX build.
4. Graft. The MTP head and vision tower were copied from Jundot/Qwen3.8-Flash-Next-oQ4e-mtp, and config.json was updated for 256 trunk experts and a 512-expert MTP block.
5. Verify. The checks under Conversion fidelity were run against the reference build and the GGUF's mmproj.

The whole conversion took about 30 minutes on an M5 Max.

License and attribution

Released under Apache-2.0, the license of the source GGUF. Use is also subject to the license of the base model, Qwen/Qwen3.8-Flash-Next.

• Base model: Qwen team, Qwen/Qwen3.8-Flash-Next
• Expert pruning (RCO) and quantization (GSQ): ISTA-DASLab, Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF
  • GSQ: arXiv:2604.18556
  • RCO: arXiv:2605.00649
• MTP head, vision tower and MLX layout reference: Jundot/Qwen3.8-Flash-Next-oQ4e-mtp
• Runtime: omlx and mlx-vlm
• GGUF tooling: llama.cpp

If you use this model in research, please cite the GSQ and RCO papers above.