Qwen3-Embedding-4B-NVFP4A16

Qwen/Qwen3-Embedding-4B quantized to NVIDIA NVFP4 weights with BF16 activations (W4A16). This checkpoint is packaged for the compressed-tensors loader and was runtime-verified with the vLLM pooling runner on NVIDIA Blackwell.

Property	Value
Base model	Qwen/Qwen3-Embedding-4B
Quantization	NVFP4A16: 4-bit NVFP4 weights, BF16 activations
Checkpoint size	2,820,762,536-byte model.safetensors (2.7 GiB directory)
Architecture	Qwen3ForCausalLM, 36 layers, hidden size 2560
Native embedding width	2560
Verified output width	768, unit-normalized
MRL widths in config	128, 256, 512, 768, 1024, 1536, 2048, 2560
Verified serving stack	vLLM 0.26.0, RTX PRO 6000 Blackwell

Quantization

The emitted recipe uses NVFP4A16 for Linear layers and excludes lm_head:

QuantizationModifier:
  targets: [Linear]
  ignore: [lm_head]
  scheme: NVFP4A16

The serialized compressed-tensors configuration records 4-bit float weights, group_size=16, symmetric tensor-group quantization, and FP8 E4M3 scales. Input and output activation quantizers are null, so activations remain BF16.

The included one-shot log reports a data-free quantization pipeline. Exact historical command-line and input-byte provenance were not pinned, and the artifact predates the later English calibration-file preparation. Accordingly, this checkpoint is not described as English-calibrated.

Official MTEB subset evaluation

Official MTEB 2.18.13 evaluators, five complete English test tasks (ArguAna, ArXivHierarchicalClusteringP2P, Banking77Classification.v2, BIOSSES, CQADupstackGamingRetrieval) — a targeted subset, not the full MTEB English v2 suite. Evaluated against the BF16 original and the W4A4 sibling, at every Matryoshka width used in production, same protocol for all three.

TL;DR: 2.7GB weights (vs ~8GB BF16, ~3x smaller) at **99% of BF16 MTEB score**, consistent from 768 to 2048 dims. The W4A4 sibling trades a bit more quality (~97-98%) for faster FP4 activations.

Average score across the 5 tasks:

Dim	BF16 (baseline)	W4A16 (this model)	W4A4
768	0.7329	0.7241 (98.8%)	0.7131 (97.3%)
1024	0.7335	0.7258 (99.0%)	0.7169 (97.7%)
1536	0.7340	0.7266 (99.0%)	0.7157 (97.5%)
2048	0.7351	0.7274 (99.0%)	0.7203 (98.0%)

Per-task scores at each dimension

Task	Dim	BF16	W4A16	W4A4
ArguAna	768	0.7333	0.7114	0.6904
ArguAna	1024	0.7390	0.7159	0.6974
ArguAna	1536	0.7377	0.7150	0.6960
ArguAna	2048	0.7355	0.7158	0.6962
ArXivHierarchicalClusteringP2P	768	0.5936	0.5904	0.5897
ArXivHierarchicalClusteringP2P	1024	0.5871	0.5846	0.5866
ArXivHierarchicalClusteringP2P	1536	0.5851	0.5895	0.5802
ArXivHierarchicalClusteringP2P	2048	0.5897	0.5871	0.5913
Banking77Classification.v2	768	0.8435	0.8404	0.8323
Banking77Classification.v2	1024	0.8442	0.8408	0.8314
Banking77Classification.v2	1536	0.8457	0.8410	0.8337
Banking77Classification.v2	2048	0.8473	0.8418	0.8348
BIOSSES	768	0.8258	0.8278	0.8169
BIOSSES	1024	0.8252	0.8289	0.8232
BIOSSES	1536	0.8248	0.8248	0.8194
BIOSSES	2048	0.8262	0.8300	0.8282
CQADupstackGamingRetrieval	768	0.6681	0.6503	0.6360
CQADupstackGamingRetrieval	1024	0.6722	0.6588	0.6458
CQADupstackGamingRetrieval	1536	0.6767	0.6626	0.6491
CQADupstackGamingRetrieval	2048	0.6767	0.6621	0.6509

Retrieval tasks (ArguAna, CQADupstackGamingRetrieval) show the largest quantization gap (~2-3% relative); clustering and classification are within noise of BF16 at every width. Raw embedding cosine similarity vs BF16 (separate 100-text check, not a task score): mean ~0.95-0.96 across dims.

Matched protocol

• Official MTEB task evaluators, complete test splits, dataset revisions pinned in the uploaded manifests.
• 64,880 encoded texts per arm; 24 concurrent request workers, batch size 32.
• Query instruction applied only to query batches; passages/documents were bare.
• Inputs longer than 8,192 tokens were right-truncated identically across all three arms.
• MTEB cache disabled, isolated per-arm prediction artifacts, no result exceptions.

Raw scores: benchmarks/mteb_dims_comparison/all_scores.json.

Serving with vLLM

vllm serve ironbcc/Qwen3-Embedding-4B-NVFP4A16 \
  --served-model-name qwen3-embedding-4b \
  --runner pooling --convert embed \
  --pooler-config '{"dimensions": 768}' \
  --max-model-len 8192

The tested server returned finite, unit-normalized 768-dimensional vectors and accepted long inputs with explicit endpoint-side truncation.

Queries should carry the base model's retrieval instruction; documents should remain bare:

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query: {your query}

OpenAI-compatible example:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
query = (
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\n"
    "Query: What is NVFP4?"
)
response = client.embeddings.create(
    model="qwen3-embedding-4b",
    input=[query],
    extra_body={"truncate_prompt_tokens": 8192, "truncation_side": "right"},
)
embedding = response.data[0].embedding

Intended use

• Blackwell-oriented, memory-efficient text embeddings served through vLLM.
• Retrieval, clustering, classification, and semantic-similarity workloads after validating the checkpoint on the target domain.
• Query/document asymmetric retrieval using the Qwen3 instruction format.

Limitations

• Evaluation covers five English MTEB tasks, not the full MTEB English v2 or multilingual suites. The table must not be read as a leaderboard aggregate.
• BF16 baseline is now included (see above): W4A16 retains ~99% of BF16 average score across the tested widths, versus ~97-98% for W4A4. This does not isolate activation precision as the sole cause of the remaining W4A16/W4A4 gap.
• Runtime verification here is for vLLM pooling on Blackwell. Direct SentenceTransformers/Transformers loading and other GPU architectures were not revalidated after quantization.
• FP4 kernels require a recent compatible vLLM/FlashInfer stack. The tested environment needed FLASHINFER_DISABLE_VERSION_CHECK=1 because its FlashInfer Python/cubin package versions did not match.
• Inputs were capped at 8,192 tokens for the reported benchmark.
• GPU-memory observations were host-wide point samples with other GPU0 services present; they are not isolated model-allocation measurements.
• Inherits the base model's license, limitations, and biases.

Provenance and checksums

• config.json SHA-256: 0d66a32f634ab1196ff724f536a6f4af1a6c81108470d82eaf05445fb2c3be84
• model.safetensors SHA-256: 2ea69117761426f841907430ec03c8aa979fd12ca6c2bebfb805436554350ce4
• recipe.yaml SHA-256: 051768d850068590f7ac0c0d10441378f8c759e9db444bd228b0fb1ee8c5dcb0

recipe.yaml and oneshot_2026-08-05_03-26-07.log are included as emitted build artifacts. The exact historical build invocation remains unpinned.