MiMo-V2.5-0703-NVFP4

NVFP4/MXFP8 mixed quantization of the updated MiMo-V2.5 target model that ships inside XiaomiMiMo/MiMo-V2.5-DFlash (released 2026-07-03) — the exact weights the DFlash speculative-decoding drafter was trained against.

Why this exists: the widely used lukealonso/MiMo-V2.5-NVFP4 was quantized from the original April XiaomiMiMo/MiMo-V2.5 release. The 2026-07-03 DFlash drop silently includes a refreshed target: every weight shard differs (MoE router gates, routed experts, attention o_proj; the vision/audio towers and audio tokenizer are unchanged). Serving the DFlash drafter against the old base works, but leaves drafter/target mismatch on the table. This repo is the drafter-matched target.

What it is

• Same recipe, layout and tensor structure as lukealonso/MiMo-V2.5-NVFP4 (verified structurally identical tensor-by-tensor: names, dtypes, shapes) — a drop-in replacement wherever that checkpoint serves.
• Routed MoE experts: NVFP4 (E2M1, group-16 FP8 scales + per-tensor scale, calibrated input scales). Attention fused-QKV and dense MLP: MXFP8 (group-32 E8M0 scales, exact conversion from the upstream FP8 release). o_proj, routers, embeddings, lm_head, norms, vision/audio towers: BF16. MTP draft weights carried (model-mtp.safetensors).
• Calibration: seeded with lukealonso's published full 6-pass amax state (text/image/audio/video coverage), then adapted to the new weights with fresh passes (agentic coding, diverse chat, VQA, long-context). Weight scales computed entirely from the new weights.
• Quantized with the open pipeline in local-inference-lab/quant-toolkit.

Running with vLLM (SM120 / RTX PRO 6000 Blackwell)

Use the companion image (custom kernels for this format + DFlash):

docker pull ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash

The DFlash drafter is the dflash/ subfolder of the upstream repo — download it and pass the subfolder as the speculative model:

hf download XiaomiMiMo/MiMo-V2.5-DFlash --include "dflash/*" --local-dir /models/mimo-dflash

docker run --gpus '"device=0,1,2,3"' --ipc host --network host --init --shm-size 32g \
  -v $HF_CACHE:/root/.cache/huggingface \
  -v /models/mimo-dflash/dflash:/dflash:ro \
  -e VLLM_USE_V2_MODEL_RUNNER=0 \
  -e VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash \
  vllm serve mitomtuna/MiMo-V2.5-0703-NVFP4 \
    --served-model-name mimo-v2.5 \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --kv-cache-dtype auto \
    --block-size 64 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 1048576 \
    --max-num-seqs 32 \
    --max-num-batched-tokens 8192 \
    --max-cudagraph-capture-size 32 \
    --attention-backend TRITON_ATTN \
    --kernel-config.moe_backend flashinfer_cutlass \
    --kernel-config.linear_backend b12x \
    --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice \
    --compilation-config '{"cudagraph_mode":"PIECEWISE","custom_ops":["all"]}' \
    --async-scheduling --no-scheduler-reserve-full-isl \
    --enable-chunked-prefill --enable-prefix-caching \
    --speculative-config '{"model":"/dflash","method":"dflash","num_speculative_tokens":7,"num_speculative_tokens_per_batch_size":[[1,4,7],[5,32,3]]}'

Notes:

• On PCIe-only boxes add --disable-custom-all-reduce.
• The checkpoint is self-contained (auto_map + configuration_mimo_v2.py included): no config shim needed.
• Text, image, audio and video inputs all work; at TP4 no special --mm-encoder-tp-mode is needed.
• Need 3 GPUs? Use the exact tensor-parallel-3 transform of this repo: mitomtuna/MiMo-V2.5-0703-NVFP4-TP3.

Credits

• Original model + DFlash drafter + updated target: XiaomiMiMo
• Quantization recipe, calibration seed and pipeline: lukealonso / local-inference-lab

Validation (this exact checkpoint, RTX PRO 6000 Blackwell)

0703-target requant validation — 2026-07-08T07:40:08+00:00

Acceptance (reason_accept_check.py, same prompt/protocol)

--- baseline: prod TP3, OLD target (captured in prep):
reasoning ctx0: 412.3 t/s, accept_len 5.91/8 (2500 tokens, 423 drafts)
--- mimo-vllm-val (NEW target):
reasoning ctx0: 488.0 t/s, accept_len 5.69/8 (2500 tokens, 439 drafts)
--- mimo-tp3-val (NEW target):
reasoning ctx0: 381.5 t/s, accept_len 5.71/8 (2500 tokens, 438 drafts)

Needle (90k)

--- mimo-vllm-val:
prompt_tokens=89935 completion=99
answer: 'PERSIMMON-42'
NEEDLE PASS
--- mimo-tp3-val:
prompt_tokens=89935 completion=103
answer: 'PERSIMMON-42'
NEEDLE PASS

KV cache pool measured at boot (this checkpoint, this command)

--- TP4 (util 0.85, max-model-len 1048576):
(Worker_TP0 pid=247) INFO 07-08 07:35:33 [gpu_worker.py:517] Available KV cache memory: 29.77 GiB
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2202] GPU KV cache size: 3,877,291 tokens
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2203] Maximum concurrency for 1,048,576 tokens per request: 3.70x
--- TP3 (util 0.965, max-model-len 524288):
(Worker_TP0 pid=241) INFO 07-08 07:39:30 [gpu_worker.py:517] Available KV cache memory: 14.9 GiB
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2202] GPU KV cache size: 668,489 tokens
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2203] Maximum concurrency for 524,288 tokens per request: 1.28x

Smoke responses: see s5-smoke-*.json

Result: ALL PASS