multilingual-e5-large-instruct-nvfp4

NVFP4 quantization of intfloat/multilingual-e5-large-instruct, produced with llm-compressor 0.13.0 and measured end to end on NVIDIA H100 80 GB with vLLM 0.26.0.

Only the Linear layers of the encoder are quantized; the vocabulary, position and token-type embeddings, the pooler and any classification head stay in bf16.

Measured

variant	size	SciFact nDCG@10	STS-B de ρ	STS17 en-de ρ	texts/s (bs=256)
bf16 source	1068 MiB	0.7204	0.8347	0.8525	1488
FP8	802 MiB	0.7164	0.8349	0.8526	1511
NVFP4	675 MiB	0.7036	0.8371	0.8440	–

Retrieval costs 1.7 pp nDCG@10 against bf16; STS is unchanged. Size is −37%.

SciFact: 300 queries over 5183 documents, nDCG@10. STS-B de: mteb/stsb_multi_mt (1379 pairs). STS17 en-de: mteb/sts17-crosslingual-sts (250 pairs). Reranking: 1 positive + 19 sampled negatives per query. Throughput over 4096 multilingual texts through the vLLM OpenAI endpoint.

A bug you should know about if you quantize BERT-family models yourself

The first NVFP4 build of this model scored 0.042 nDCG@10 in vLLM — random-level — while the same weights scored 0.66 under plain transformers. The weights were fine; the checkpoint was not.

For NVFP4, every linear carries its own weight_global_scale. vLLM fuses query/key/value into one qkv_proj and then collapses their scales with weight_global_scale.max() (compressed_tensors_w4a4_nvfp4.py:111), silently rescaling two of the three projections. It warns ("the weight global scale is different for parallel layers") but continues.

llm-compressor is supposed to prevent this: fuse_weight_observers() links observers across fused groups so they share a global scale. But its FUSED_LAYER_NAMES (llmcompressor/observers/helpers.py:162) only lists decoder naming — q_proj/k_proj/v_proj, gate_proj/up_proj, w1/w3. BERT and RoBERTa call them query/key/value, so the group is never matched and each projection keeps its own scale.

This checkpoint was built with ("query", "key", "value") appended to that list. Layer 0 q/k/v global scales went from {query: 5344, key: 6048, value: 7904} to a shared 5344, vLLM's warning disappeared, and nDCG@10 went from 0.042 to 0.704.

Serving

vllm serve mbehr90/multilingual-e5-large-instruct-nvfp4 --runner pooling