paulalesius/Qwen3-Reranker-0.6B-GGUF

Qwen3-Reranker-0.6B — GGUF (NVFP4 / Q6_K)

GGUF quantizations of Qwen/Qwen3-Reranker-0.6B for llama.cpp, built for the llama-server --reranking / /rerank path. These are hybrid quantizations: the 2-way relevance head (cls.output.weight) is kept in BF16, every normalization tensor is kept in F32, and in the NVFP4 build the 155M-row embedding table is kept in Q8_0 — so the yes/no scoring path and the embedding lookup stay at high precision while the bulk of the 0.6B body is compressed. The result is files 2.4–2.9× smaller than the BF16 reference with essentially unchanged ranking scores.

Files

File	Size	Body quant	Embed / head / norms	eff. bpw	vs BF16
Qwen3-Reranker-0.6B.NVFP4.gguf	399.5 MiB	NVFP4	Q8_0 embed, BF16 head, F32 norms	5.63	2.86× smaller (−742.6 MiB)
Qwen3-Reranker-0.6B.Q6_K.gguf	472.0 MiB	Q6_K	Q6_K embed, BF16 head, F32 norms	6.65	2.42× smaller (−670.2 MiB)

SHA‑256

91bbd34601c7b8ddb8a8413cdea90536e9754c010da0be7112246d7441411c7d  Qwen3-Reranker-0.6B.NVFP4.gguf
4e0cb92dd4e1f1c8eb6d9537f95d14b7f89aa48da27d64dc826e5837be2c6b40  Qwen3-Reranker-0.6B.Q6_K.gguf

Quantization layout

Both files are GGUF v3, architecture qwen3, 311 tensors, 595,778,560 parameters (595.8M). The NVFP4 file is a deliberate per‑tensor mapping, not a vanilla llama-quantize X NVFP4 output:

Tensors	Count	NVFP4	Q6_K	Notes
Attention/FFN matmuls (28 blocks × 7)	196	NVFP4	Q6_K	the compressed body (≈ 4.47 / ≈ 6.53 bpw)
token_embd.weight	1	Q8_0	Q6_K	embed table, 1024 × 151,669 — kept at 8.50 bpw in the NVFP4 build
cls.output.weight	1	BF16	BF16	1024 × 2 — the relevance head that emits the yes/no score
All norms (attn_norm, ffn_norm, attn_q_norm, attn_k_norm × 28 + output_norm)	113	F32	F32	every normalization tensor left at full precision

The design keeps the tensors that most directly shape the final score — the embedding lookup, the 2‑way head, and every norm — at high precision and compresses only the large linear layers. In the NVFP4 build that costs 157.4 MiB for the Q8_0 embed table (vs an all‑NVFP4 build at 325.5 MiB) in exchange for keeping the lookup path out of the NVFP4 mid‑band distortion regime.

Fidelity vs BF16

Q6_K — controlled harness

Measured against a BF16 GGUF reference over 12 queries × 5 passages = 60 reranks (2 relevant, 1 near‑miss, 2 off‑topic each), CPU, 32 threads, flash‑attention on, ctx 1024, no prompt caching. Prompts are built from the identical embedded rerank template, so the comparison isolates quantization error.

Metric (vs BF16)	Q6
Spearman ρ (all 60)	0.9962
Kendall τ (all 60)	0.9616
Kendall τ (avg per query)	0.9500
mean |Δscore|	0.0022
max |Δscore|	0.091
Full top‑5 ordering identical	9/12
Per‑query Spearman ρ (avg / min)	0.9750 / 0.900
Mean absolute rank shift per doc	0.100
Decision match at 0.5 threshold	60/60
Top‑1 doc match	11/12

Q6 is effectively indistinguishable from BF16 on this workload — deviations are at most a single adjacent swap, and never across the relevant/off‑topic boundary.

NVFP4 (V1) — live memory bank

Measured against the same BF16 reference on a live Hindsight memory bank: 18 queries × 269 stored documents, served through the production llama-server --reranking path.

Comparison	Kendall τ	median |Δrank|	what it isolates
Q8_0 (CPU) vs BF16 (CPU)	0.97	3.0	quantization only, same device
all‑NVFP4 (V0, superseded) vs itself	0.59	32.0	device only — same file, GPU vs CPU (noise floor)
V1 (GPU) vs all‑NVFP4 (GPU)	0.50	37.5	the Q8_0 embed swap, same GPU
V1 (GPU) vs BF16 (CPU)	0.52	37.0	quantization + device

Reading: on this bank the GPU↔CPU kernel difference (τ ≈ 0.5–0.6) is the dominant term, and the NVFP4‑body deltas sit inside that band — the Q8_0 embed swap (V1 vs V0, same GPU) is of the same magnitude as the device noise floor, so it is not a measurable ranking regression here. The clean quantization statement is the same‑device row: Q8_0 preserves 98.3% of document‑pair orderings vs BF16 (Kendall τ 0.97, median rank shift 3 of 269).

License

Apache‑2.0, same as the source model. Quantized for convenience; the original weights and architecture are © Qwen Team — see Qwen/Qwen3-Reranker-0.6B.

Citation

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}