Qwen3-VL-8B-Instruct-fp8

FP8_DYNAMIC quantization of Qwen/Qwen3-VL-8B-Instruct, produced with llm-compressor 0.13.0 (compressed-tensors 0.18.0) and verified end to end on NVIDIA H100 80 GB with vLLM 0.26.0 — text and image prompts.

What is quantized

Scheme	FP8_DYNAMIC
Quantized	all 252 language-model linears, attention included
Kept in bf16	vision tower, multimodal projector, lm_head, embeddings
Calibration	none needed (fully dynamic activations)
Size on disk	9.87 GiB
Format	HF format (config.json + sharded safetensors).

Measured on H100 (vLLM 0.26.0, lm-eval 0.4.12)

variant	size	GSM8K (5-shot, strict)	ARC-Challenge (acc_norm)
bf16 source	–	84.31	60.92
FP8 (this repo's sibling)	9.87 GiB	84.38 (+0.1 pp)	61.01 (+0.1 pp)
NVFP4	7.04 GiB	78.85 (-5.5 pp)	60.24 (-0.7 pp)

GSM8K n=1319, ARC-Challenge n=1172, greedy, served through the OpenAI completions API.

Kernels

FP8 runs on Hopper's native FP8 tensor cores, so this is the faster of the two quantizations on H100; the NVFP4 sibling is the smaller one.

Serving

vllm serve mbehr90/Qwen3-VL-8B-Instruct-fp8 --max-model-len 32768 --gpu-memory-utilization 0.9

The quantization recipe is in recipe.yaml.