mbehr90/Mistral-Medium-3.5-128B-fp8

Mistral-Medium-3.5-128B — FP8 (W8A8 dynamic)

FP8_DYNAMIC on every language-model Linear, applied to mistralai/Mistral-Medium-3.5-128B with llm-compressor 0.13.0 / compressed-tensors 0.18.0 and verified end to end on 4x NVIDIA H100 80 GB.

124.5 GiB on disk (upstream: 124.4 GiB). AI2D 78.4 vs 51.4 for the upstream FP8 release — see Vision below.

The source is already FP8

Mistral publishes this model quantized — quant_method: fp8, one weight scale per tensor and a static per-tensor activation scale, with the vision tower, the multimodal projector and lm_head in bf16. There is no bf16 release. This build dequantizes back to bf16 on load (FineGrainedFP8Config(dequantize=True)) and re-quantizes from there, so its weights carry one more rounding step than a build made from true bf16 weights would.

What is quantized

group	modules	scheme
group_0	all 616 language-model Linear layers	FP8_DYNAMIC — FP8 E4M3 weights with a per-output-channel scale, activations quantized per token at runtime
—	vision tower, multimodal projector, lm_head, embeddings	bf16, untouched

Same size as the upstream release, because the upstream release is already FP8. What changes is the scaling — and on vLLM that is not cosmetic; see below.

Measured on 4x H100 80 GB

build	size	GSM8K chat	AI2D vision	GSM8K lm-eval	ARC-C	tok/s @1	tok/s @32
Mistral's own FP8 release (source)	124.4 GiB	99.2	51.4	87.0	66.7	66	1682
Mistral-Medium-3.5-128B-fp8 ← this repo	124.5 GiB	99.2	78.4	89.1	68.3	61	1567
Mistral-Medium-3.5-128B-nvfp4	88.7 GiB	98.4	77.8	88.8	68.4	73	1760
Mistral-Medium-3.5-128B-nvfp4-full	74.8 GiB	98.8	76.2	90.7	68.3	78	1825

GSM8K chat: 250 questions through /v1/chat/completions, greedy, answer read from the last number in the reply. AI2D: 500 diagram questions from lmms-lab/ai2d with the image attached, scored on the answer letter. lm-eval 0.4.12 local-completions, gsm8k strict-match and arc_challenge acc_norm. Throughput: 1024-token prompts, 256 output tokens with ignore_eos.

Vision

The upstream FP8 release scores 51.4 on AI2D under vLLM; both builds here score ~78, with identical text scores (GSM8K 99.2 for the source and for the FP8 build). The vision tower and the multimodal projector are bit-identical between the source and these builds — same tensor names, same bf16 values, maxdiff = 0 — and the images reach the model in every case (the image expands to the same 786 / 3216 prompt tokens). What differs is only how the language model consumes the image embeddings.

It is not weight precision: the NVFP4 build has far coarser weights than the source and still scores ~26 points higher. It is the activation scaling. The upstream checkpoint uses one static per-tensor activation scale, calibrated on text; visual tokens arrive with a different dynamic range and get clipped. Both builds here quantize activations per token at runtime, so there is nothing to clip against.

On a plain description prompt the difference is blunt:

checkpoint	photo (786 prompt tokens)
upstream FP8	"Please upload the image you'd like me to describe."
this build	"Two tabby cats are sleeping on their backs on a pink blanket with two remote controls nearby."

Serving

vllm serve mbehr90/Mistral-Medium-3.5-128B-fp8 \
    --tensor-parallel-size 4 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.85 \
    --limit-mm-per-prompt '{"image": 4}'

Keep --gpu-memory-utilization at or below 0.85 when you send images: the Pixtral tower allocates its activations outside the KV pool and OOMs at 0.92 under concurrent image requests.

tekken.json is deliberately absent. With that file present vLLM selects the mistral_common tokenizer backend, whose encode("[IMG]") returns the literal bracket text ([1091, 125250, 1093]) instead of image token 10; the Pixtral processor then finds zero image tokens and the server refuses to start with "Mismatch in image token count between text and input_ids". This happens under --tokenizer-mode auto, hf, slow and mistral alike. Shipping only the HF tokenizer.json avoids it.

vLLM version. 0.25 and 0.26 crash for this model during CUDA-graph capture (custom_all_reduce.cuh: an illegal memory access). Every number above was produced on a 0.1.dev20073 nightly, which captures graphs normally with default flags.

On H100, NVFP4 is weight-only. vLLM's NVFP4 kernel list is FlashInfer and CUTLASS, both SM100; on Hopper it falls through to MarlinNvFp4LinearKernel, a W4A16 GEMM — the weights stay 4-bit in HBM but the matmul runs in bf16 and the checkpoint's activation quantization is not applied. On Blackwell the same checkpoint runs true W4A4. The accuracy figures above are the Hopper numbers.

Reproducing

recipe.yaml in this repo is the exact llm-compressor recipe. Calibration: 256 samples of HuggingFaceH4/ultrachat_200k at 2048 tokens — NVFP4 needs it for the static per-tensor input_global_scale, FP8_DYNAMIC is data-free.