jalenluorion/Qwen3-Coder-Next-moved-store

Qwen3-Coder-Next, experts moved to disk

An 80B coding agent on a 16 GB MacBook: 4 to 6 tokens/s, 8 GB in memory, first message in about 5 s under stock OpenCode. The 72 GB of experts stay on the SSD and are read ahead of time.

Qwen/Qwen3-Coder-Next (Apache-2.0), the agentic-coding model on the Qwen3-Next 80B-A3B architecture, repacked so that 3% of the parameters are in memory and the rest on disk. Unquantized except for the expert store: the routed experts are int8 on disk, everything else is the released weights. Nothing was retrained. The multi-token-prediction head of the release is not included.

part	parameters	where
attention and linear-attention layers, shared experts, norms, embeddings, output head, the released router gates (resident.safetensors)	2.36B	memory, 3.8 GB in fp16 with the embedding table read from disk
lag-4 routers (routers_k4.pt)	466M	memory, 0.9 GB at fp16
24,576 routed experts (store/layer_XX.bin, int8, one scale per neuron, 3.0 MiB records)	77.3B	disk, 72.4 GiB

The released model is 151 GB in bf16. Under OpenCode's stock prompt it uses tools from the first message and chains them; the Instruct model tends to answer in text instead. The same conversion exists for Qwen3-Next-80B-A3B-Instruct and Qwen3-30B-A3B; the three repos share the same runner and layout.

Quick start

Apple Silicon with 16 GB or more and about 85 GB of disk:

pip install mlx mlx-lm tokenizers jinja2 numpy huggingface_hub
hf download jalenluorion/Qwen3-Coder-Next-moved-store --local-dir coder-next-store
cd coder-next-store
python mlx_run.py                        # chat; -v prints the per-turn statistics
python mlx_run.py --bench --max-new 80   # the timing JSON behind the numbers below
cd demo && ./demo.sh                     # the same engine behind an OpenAI-compatible API, opened in OpenCode

demo.sh needs OpenCode on the path. It starts the server if it is not running and opens OpenCode on it with demo/opencode.json: OpenCode's stock prompt and tools, no custom instructions. The first message of a thread finds OpenCode's 7,500-token system prompt already prefilled; the first start of a day re-prefills it once (about 80 s) because the prompt carries the date.

How it works

In the released model each layer's router picks its experts from the hidden state at that layer, so an expert is needed the moment it is chosen and every expert has to sit in memory. Here the routed experts are stored once on disk, and a small router per layer reads the token's hidden state four layers earlier and names the experts that layer will want. Those experts are read from the SSD while the four layers in between compute. The released router then makes its usual choice among the experts that have arrived. The only trained parts are these routers, trained to imitate the released routers with the model frozen.

• Prefetch mode (default). No expert is read during the layer's own computation. If the routers named the wrong expert, the released router chooses among what is in memory; the output changes slightly, nothing stalls. --m sets how many experts are named per layer and --keep how many tokens a fetched expert stays in memory while it keeps being named.
• Exact mode (--mode exact). The routers only decide what to read ahead; anything missing is read when needed. Outputs are the released model's with int8 experts, regardless of what the routers predicted.

Expert records are int8 with one fp16 scale per neuron. Attention, embeddings, output head, shared experts and router gates are the released weights, loaded as fp16.

The routers for this model were trained on 10M tokens of wikitext, FineWeb-Edu and chat text, warm-started from the Instruct model's routers (10.7 hours on four A6000s). Released top-10 recall 82 to 83%, the same as the Instruct routers. The released router picks 10 of 512 experts per layer.

Quality

Loss gap to the released model in nats on wikitext-103 test / FineWeb-Edu / held-out UltraChat (released 2.148 / 2.264 / 1.573; a coding model, higher than the Instruct model's 1.798 / 2.087 / 1.492 on general text), with the bytes read per token and the experts held in memory:

named per layer	kept (tokens)	loss gap	read / token	experts held
10	1	+0.009 / +0.010 / +0.006	774 MB	726 (2.2 GB)
10	3	+0.008 / +0.008 / +0.003	598 MB	1,125 (3.4 GB)
10	3, at most 24 slots per layer (runner default)	+0.008 / +0.009 / +0.003	601 MB	1,044 (3.1 GB)

0.002 to 0.003 nats more than the Instruct conversion at the same settings, with routers trained on half the tokens.

Speed and memory

Apple M4, 16 GB, internal SSD. --bench --max-new 80 gives 5.2 to 5.5 tokens/s at 8.2 GB; decode reads about 530 MB per token and waits 63 to 97 ms per token on the drive. The table is the same three messages through the two front-ends, back to back on the same machine, with the server's per-turn log on the OpenCode side:

turn	chat loop (no system prompt)	OpenCode (stock TUI, 7,500-token system prompt and 10 tools)
"hi"	9 tokens prefilled in 1.9 s, 11 tokens at 6.1 tokens/s	request reaches the server 0.9 s after Enter, cached system prompt restored and 9 tokens prefilled in 2.6 s, 9 tokens at 5.7 tokens/s; OpenCode's own clock 5.1 s
a 100-word explanation	26 tokens prefilled in 5.6 s, 158 tokens at 4.8 tokens/s	0.2 s to the server, 26 tokens prefilled in 5.6 s, 163 tokens at 4.7 tokens/s; OpenCode's clock 40.6 s against 40.2 s of model time
a one-line Python function	19 tokens prefilled in 4.1 s, 151 tokens at 4.2 tokens/s	0.5 s to the server, 19 tokens prefilled in 6.6 s, 15 tokens at 4.0 tokens/s; OpenCode's clock 10.9 s against 10.3 s

Prefill is 0.22 s per token in both; generation is 4.0 to 6.1 tokens/s in both, a little lower on code than on prose (the released top-10 is found in memory 79 to 81% of the time on the code turn, 86 to 88% on prose). OpenCode adds under a second before the request and under a second after the last token. The engine prefills OpenCode's 7,528-token prompt from cold in 61 s (two batches, the whole store streamed once per batch) and decodes at 5.0 tokens/s at that context; a thread does not pay that: the prompt is snapshotted and replayed at start-up. A 2,000-token tool result is prefilled in about 25 s, 16,000 tokens in about 2 minutes.

Demo mode: the same engine behind OpenCode

python mlx_run.py --serve 8080 --prompt-cache demo/cache --warm serves /v1/chat/completions (streaming, tools through the release's own chat template, tool calls returned as OpenAI tool_calls). demo/opencode.json points OpenCode at it with OpenCode's stock prompt and tools; demo/demo.sh starts the server if it is not running and opens OpenCode.

What the server adds, shared with the chat loop:

• Prompts of 48 tokens or more are prefilled as one batch: the released routing for every token, each needed expert read once, attention in 512-token blocks against the cache. A 4,096-token batch is one pass over the store.
• The decoded prefix is reused across turns, also when the client re-renders the model's own turn with different token boundaries. Prefixes are snapshotted at message boundaries in memory and on disk, so an agent's system prompt is prefilled once per machine, and --warm replays the last request at start-up and when the date in it changes.
• On macOS the model is GPU memory, and the pager ages pages by CPU references, which GPU reads never produce. Under pressure it compresses the model while it is being read every token. The runner wires what it can and touches one byte per 16 KB page from the CPU on a schedule, with no measurable effect on speed.

This model emits tool calls in the Qwen3-Coder XML format (<tool_call><function=...><parameter=...>). The server parses that format and the Instruct JSON one, types the arguments from the tool schema, and re-renders the model's own turn through the release's chat template so the decoded prefix is reused across tool rounds; only the tool result is prefilled, as one batch.

What is in the repo

file	what it is
store/layer_XX.bin	the routed experts of layer XX, one page-aligned int8 record per expert, in expert order
store/meta.json	shapes, record size and offsets the runner reads
resident.safetensors	everything that is not a routed expert, at the released precision
routers_k4.pt	the lag-4 routers (converted once to routers_k4.npz on first run)
mlx_run.py	the runner: research chat, OpenAI-compatible server, benchmark and check modes
demo/demo.sh, demo/opencode.json	start the server and open OpenCode on it
convert.py	rebuilds the store and resident.safetensors from the released weights
sha256.txt	checksums of the first store layer and the resident file
config.json, generation_config.json, tokenizer files	copied from the release

Limits

A token uses 10 experts in each of 48 layers at 3 MB each, 1.44 GB before reuse; the keep window reduces it by holding experts in memory. The experts are int8 and the rest fp16; decode speed is the drive's bandwidth divided by the bytes a token needs. Past about 8,000 tokens of context a 16 GB machine is near its GPU memory limit and a long tool result may be prefilled in smaller batches. Timings are single runs at batch 1 on one machine. The routers were trained on encyclopedia, web and chat text, not code.

Paper

Paper in preparation. The preprint will be linked here.