rapid-mlx/DeepSeek-V4.1-Flash-1017-MLX

DeepSeek-V4.1-Flash-1017 — Rapid-MLX GPTQ mixed-precision (256 GB Macs)

A mixed-precision MLX build of deepseek-ai/DeepSeek-V4.1-Flash-1017 sized to run on a single 256 GB Apple Silicon Mac with Rapid-MLX.

This is v2 of this repository. It replaces round-to-nearest rounding with GPTQ (error-compensating quantization driven by calibration data). Against the same source-precision reference, mean next-token KL divergence drops by 35–44% and top-1 agreement rises by 3.5–5.9 points on every ruler we measured. The previous build stays available as the v1 tag.

Higher top-1 agreement and lower KL mean closer to the original model. Details in Comparison with a community 2-bit MLX quant.

• No experts removed. All 40 × 384 routed experts are kept.
• GPTQ for every quantized weight matrix in the 40 layers, calibrated on 36 × 40,960 tokens, with routed-expert Hessians weighted by the router. The Engram lookup tables are the exception: they are the same 4-bit tables as v1.
• ~215.8 GB resident (weights the model keeps in RAM), plus 122.9 GB of 4-bit Engram tables that stay on SSD. The download is 338.8 GB in total.

Quick start

Requires Rapid-MLX 0.16.6 or newer; 0.16.7 or newer is recommended (up to ~1.9x faster decode). Older releases keep loading the v1 revision.

pip install -U rapid-mlx
rapid-mlx serve deepseek-v41-flash-1017

This starts an OpenAI-compatible server on http://127.0.0.1:8000/v1. The first run downloads the pinned revision and verifies its file hashes.

Requirements and current limits

Item	Value
Unified memory	256 GB Mac (the server refuses to start below the guard)
Free disk	~340 GB (Engram tables are read from SSD on demand)
Decoding	greedy only (temperature 0); seed is rejected
Input	text only, ≤ 8,192 prompt tokens
Output	≤ 4,096 tokens per request
Tools / function calling	not yet supported by this serving lane
Speculative decoding	built-in DSpark draft (dspark/, 4-bit experts)

Sampling, tool calling, and longer inputs and outputs are planned follow-ups for the serving lane. They are not limits of the weights.

What is in this build

Component	Precision (group size 64 unless noted)
Routed experts, gate/up	2-bit; 3-bit on layers 15 and 35–39
Routed experts, down	3-bit, all layers
Attention and shared experts	4-bit; 8-bit on layers 6, 14–18, 26, 39
Indexer, compressor, attention output A	8-bit
Embeddings, LM head, router	BF16
Engram (layers 1, 14)	4-bit, group 32, round-to-nearest (same as v1), offloaded to SSD
DSpark draft head	8-bit dense, 4-bit experts

How it was built

1. Reference. The source FP4/FP8 checkpoint is streamed one layer at a time. Every ruler below compares against the source-precision model, never against another quantized build.
2. Calibration. 36 slices × 40,960 tokens covering chat, code, math, tool-call transcripts, agentic terminal sessions, professional domains and many languages, including low-resource ones. Calibration shares no corpus file or token window with the held-out slices, and no 64-token window with the WikiText or long-context rulers.
3. GPTQ, layer by layer. Each layer is quantized on the outputs of the already-quantized layers before it, so later layers correct for earlier error. Routed experts are processed in batches, with each token's contribution to an expert's Hessian weighted by its router weight.
4. Bit widths. Sensitivity was scanned on all 40 layers. The eight most sensitive layers keep 8-bit attention and shared experts, and the late layers get 3-bit gate/up projections, within a 216 GB resident budget.
5. Engram tables. The SSD-resident Engram tables are taken unchanged from v1 (4-bit), which keeps every file under 50 GB. A 6-bit variant measured 3.5–6.5% lower mean KL and 0.3–0.6 points higher top-1 agreement on the rulers below, at 49 GB more disk.

Quality

Teacher-forced next-token KL divergence vs. the source-precision model; lower is better. Top-1 agreement is the share of positions where the build's most likely next token equals the source model's. Same evaluator and same reference for both builds.

Ruler	Tokens	Build	Mean KL (nats)	p99 KL	Top-1 agreement
38 held-out slices × 512	19,456	v1 (round-to-nearest)	0.439	4.17	79.2%
		v2 (this build)	0.260	3.30	85.1%
WikiText-2 test, 24 × 1,100	26,400	v1	0.444	4.44	78.6%
		v2	0.288	3.26	83.2%
Long context, 2 × 17,000	34,000	v1	0.269	4.49	90.5%
		v2	0.150	3.29	94.0%

The held-out slices cover chat, code, math, tool calls, agentic terminal sessions, law, medicine, finance, science and 20 languages; v2 improves on every one of the 38 slices. The long-context ruler scores every position of two full 17,000-token contexts (an English and a Chinese public-domain novel). For the WikiText and long-context rulers, KL is computed over the reference's top-128 tokens plus one tail bucket, which slightly underestimates full-vocabulary KL; top-1 agreement is exact.

Per-domain KL on the held-out slices:

Domain	v1	v2
code	0.128	0.055
math	0.179	0.063
agentic terminal	0.156	0.097
tool-call transcripts	0.187	0.093
general chat (EN)	0.251	0.177
general chat (ZH)	0.246	0.141
hardest slice (Chinese folk-customs text)	0.749	0.546

Non-English and long-tail knowledge text still drifts more than code, math and agentic text.

Evaluation method

Teacher-forced top-1 agreement and KL divergence are measured against the source FP4/FP8 weights dequantized to BF16, at source revision 1bae0e1088993b9b82de08f59de5be5a34e42170. Top-1 agreement measures agreement with the reference, not task accuracy.

The rulers are held-out 38 × 512 tokens, WikiText-2 test 24 × 1,100 tokens, and long-context 2 × 17,000 tokens (English and Chinese). The long-context ruler is an offline weight evaluation. Held-out KL uses the full vocabulary; WikiText-2 and long-context KL use the reference's top-128 tokens plus one tail bucket, a lower bound on full-vocabulary KL. Top-1 agreement is exact.

Comparison with a community 2-bit MLX quant

Ruler	Build	Top-1 %	Mean KL	p99 KL
heldout38	Rapid-MLX v2 (ours)	85.1	0.259901	3.304570
heldout38	Community 2-bit MLX quant	80.4	0.393205	3.953150
heldout38	Rapid-MLX v1	79.2	0.439326	4.173760
wiki1100	Rapid-MLX v2 (ours)	83.2	0.288168	3.261922
wiki1100	Community 2-bit MLX quant	79.3	0.402463	4.096575
wiki1100	Rapid-MLX v1	78.6	0.444330	4.438421
long17k	Rapid-MLX v2 (ours)	94.0	0.150136	3.292282
long17k	Community 2-bit MLX quant	not available	not available	not available
long17k	Rapid-MLX v1	90.5	0.269110	4.486991

Resident weight sizes are approximately 216 GB for this build and 190 GB for the Community 2-bit MLX quant, excluding SSD-resident Engram tables. These are weight payloads, not peak process-memory measurements.

Speed and memory (Mac Studio M3 Ultra, 256 GB)

Decode speed with Rapid-MLX 0.16.7 or newer, single stream, default settings (built-in DSpark draft at depth 2), greedy, 256 output tokens, warm server, ~0.7–1k-token chat prompts:

Prompt class	Decode (tok/s), default	With RAPID_MLX_DSV41_DRAFT_DEPTH=4
Long prose (1,008 tokens)	35.2	31.2
Question answering	34.3	33.1
Summarize	36.2	33.6
Code	42.7	49.0
Structured output (JSON)	42.1	46.5

On the same machine and prompts, 0.16.6 decoded at 17.6–33.9 tok/s and 0.16.7 at 32.6–44.9 tok/s, up to about 1.9x faster (same-machine A/B method and reproduce commands). For code-heavy workloads set RAPID_MLX_DSV41_DRAFT_DEPTH=4 (or auto). The target model still verifies every emitted token, so greedy output quality is unchanged. Method and raw tables: measured defaults.

Memory. The weights are loaded into RAM up front: ~215.8 GB resident, plus working buffers. The measured peak process footprint is 213 GB (8k-token prompt, otherwise idle Mac Studio). On a 256 GB Mac, quit other memory-heavy apps before starting the server. If macOS runs out of memory and swap during loading, it terminates the server.

Files

GPTQ provenance ships with the weights: gptq-provenance.json (recipe, calibration and source identities), gptq-layer-provenance.json, gptq-statistics/ (per-layer calibration and held-out error reports) and gptq-heldout-drift.jsonl (cumulative hidden-state drift after each layer). The Engram table swap is recorded under post_assembly_overrides in gptq-provenance.json, with file hashes.

Previous version

The round-to-nearest build is the v1 tag of this repository (revision="v1"), ~206.5 GB resident.

License

MIT, same as the base model. See LICENSE.