mbehr90/Mistral-Medium-3.5-128B-nvfp4

Mistral-Medium-3.5-128B — NVFP4

NVFP4 on the MLP of layers 4-86, FP8_DYNAMIC on everything else, applied to mistralai/Mistral-Medium-3.5-128B with llm-compressor 0.13.0 / compressed-tensors 0.18.0 and verified end to end on 4x NVIDIA H100 80 GB.

88.7 GiB on disk (upstream: 124.4 GiB). AI2D 77.8 vs 51.4 for the upstream FP8 release — see Vision below.

The source is already FP8

Mistral publishes this model quantized — quant_method: fp8, one weight scale per tensor and a static per-tensor activation scale, with the vision tower, the multimodal projector and lm_head in bf16. There is no bf16 release. This build dequantizes back to bf16 on load (FineGrainedFP8Config(dequantize=True)) and re-quantizes from there, so its weights carry one more rounding step than a build made from true bf16 weights would.

What is quantized

group	modules	scheme
group_1	MLP gate/up/down_proj, layers 4-86 (249 layers)	NVFP4 — FP4 E2M1 weights, an FP8 E4M3 scale per 16 values, one FP32 global scale
group_0	every self_attn.{q,k,v,o}_proj (88 layers) plus the MLP of layers 0-3 and 87 (367 layers)	FP8_DYNAMIC — FP8 E4M3 weights with a per-output-channel scale, activations quantized per token at runtime
—	vision tower, multimodal projector, lm_head, embeddings	bf16, untouched

The 4-bit/8-bit split follows nvidia/Mistral-Medium-3.5-128B-NVFP4: attention and the outermost blocks keep 8 bits because their activations have the widest dynamic range. The counts match NVIDIA's exactly (367 / 249). The difference is the 8-bit half — NVIDIA uses static per-tensor FP8, this build uses per-channel weights with per-token dynamic activations, for the reason in the next section.

Measured on 4x H100 80 GB

build	size	GSM8K chat	AI2D vision	GSM8K lm-eval	ARC-C	tok/s @1	tok/s @32
Mistral's own FP8 release (source)	124.4 GiB	99.2	51.4	87.0	66.7	66	1682
Mistral-Medium-3.5-128B-fp8	124.5 GiB	99.2	78.4	89.1	68.3	61	1567
Mistral-Medium-3.5-128B-nvfp4 ← this repo	88.7 GiB	98.4	77.8	88.8	68.4	73	1760
Mistral-Medium-3.5-128B-nvfp4-full	74.8 GiB	98.8	76.2	90.7	68.3	78	1825

GSM8K chat: 250 questions through /v1/chat/completions, greedy, answer read from the last number in the reply. AI2D: 500 diagram questions from lmms-lab/ai2d with the image attached, scored on the answer letter. lm-eval 0.4.12 local-completions, gsm8k strict-match and arc_challenge acc_norm. Throughput: 1024-token prompts, 256 output tokens with ignore_eos.

Vision

The upstream FP8 release scores 51.4 on AI2D under vLLM; both builds here score ~78, with identical text scores (GSM8K 99.2 for the source and for the FP8 build). The vision tower and the multimodal projector are bit-identical between the source and these builds — same tensor names, same bf16 values, maxdiff = 0 — and the images reach the model in every case (the image expands to the same 786 / 3216 prompt tokens). What differs is only how the language model consumes the image embeddings.

It is not weight precision: the NVFP4 build has far coarser weights than the source and still scores ~26 points higher. It is the activation scaling. The upstream checkpoint uses one static per-tensor activation scale, calibrated on text; visual tokens arrive with a different dynamic range and get clipped. Both builds here quantize activations per token at runtime, so there is nothing to clip against.

On a plain description prompt the difference is blunt:

checkpoint	photo (786 prompt tokens)
upstream FP8	"Please upload the image you'd like me to describe."
this build	"Two tabby cats are sleeping on their backs on a pink blanket with two remote controls nearby."

Serving

vllm serve mbehr90/Mistral-Medium-3.5-128B-nvfp4 \
    --tensor-parallel-size 4 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.85 \
    --limit-mm-per-prompt '{"image": 4}'

Keep --gpu-memory-utilization at or below 0.85 when you send images: the Pixtral tower allocates its activations outside the KV pool and OOMs at 0.92 under concurrent image requests.

tekken.json is deliberately absent. With that file present vLLM selects the mistral_common tokenizer backend, whose encode("[IMG]") returns the literal bracket text ([1091, 125250, 1093]) instead of image token 10; the Pixtral processor then finds zero image tokens and the server refuses to start with "Mismatch in image token count between text and input_ids". This happens under --tokenizer-mode auto, hf, slow and mistral alike. Shipping only the HF tokenizer.json avoids it.

vLLM version. 0.25 and 0.26 crash for this model during CUDA-graph capture (custom_all_reduce.cuh: an illegal memory access). Every number above was produced on a 0.1.dev20073 nightly, which captures graphs normally with default flags.

On H100, NVFP4 is weight-only. vLLM's NVFP4 kernel list is FlashInfer and CUTLASS, both SM100; on Hopper it falls through to MarlinNvFp4LinearKernel, a W4A16 GEMM — the weights stay 4-bit in HBM but the matmul runs in bf16 and the checkpoint's activation quantization is not applied. On Blackwell the same checkpoint runs true W4A4. The accuracy figures above are the Hopper numbers.

Reproducing

recipe.yaml in this repo is the exact llm-compressor recipe. Calibration: 256 samples of HuggingFaceH4/ultrachat_200k at 2048 tokens — NVFP4 needs it for the static per-tensor input_global_scale, FP8_DYNAMIC is data-free.