retrieval-quant: quantized ONNX retrievers

Four open retrieval models, two embedders and two rerankers, quantized to 4 and 8 bit for ONNX Runtime and measured against their own FP32 versions. Every variant we built is published, failed ones included, so the size/quality trade-off can be read from the tables below. Tested with ONNX Runtime 1.31.

Repos: nynefold/pplx-embed-v1-0.6b-ONNX, nynefold/F2LLM-v2-1.7B-ONNX, nynefold/Qwen3-Reranker-0.6B-ONNX, nynefold/zerank-1-small-ONNX, and the evaluation set nynefold/FreshRet-Small. pipeline_example.py in this repo retrieves with pplx-embed and reranks with zerank, using inference.py (ONNX Runtime, tokenizers and NumPy only). All six repos are in the collection retrieval-quant: quantized ONNX retrievers.

Results in short

• Each model has a pick that stays within 1 nDCG@10 point of its FP32 version at 3.4 to 6x smaller files and 1.8 to 3.3x less RAM.
• pplx-embed + zerank-1-small is the recommended pair: 0.870 nDCG@10 on FreshRet Small in 1.9 GB. The reranker adds about 12 points over the embedder alone.
• The cheapest useful setup is pplx-embed alone: 0.746 in 401 MB.
• Qwen3-Reranker-0.6B adds nothing on FreshRet Small over the embedder alone, at FP32 as well as quantized.
• The two Qwen3-1.7B-based models (F2LLM, zerank) have MLP activation outliers. F2LLM needs SmoothQuant for INT8 compute; zerank only gains a little from it at 4 bit. The 0.6B models don't need it.

Picks

One recommended variant per model. Size is the ONNX file; RAM is peak process memory while embedding one 256-token passage (embedders) or scoring 20 candidates of 170 tokens (rerankers), in a process with only ONNX Runtime, tokenizers and NumPy loaded. Quality deltas are paired against FP32 in nDCG@10 points, with 95% bootstrap intervals (10,000 resamples). Picks were chosen on FreshRet Small; MIRACL (mteb/MIRACLReranking) is the held-out check.

model	repo	pick	size MB (FP32)	RAM MB (FP32)	FreshRet Small nDCG@10 vs FP32	MIRACL nDCG@10 vs FP32
pplx-embed-v1-0.6b	nynefold/pplx-embed-v1-0.6b-ONNX	mix02-gptq-a4-pc	401 (2387)	571 (1892)	-0.08 [-0.91, +0.74]	-0.38 [-1.38, +0.55]
F2LLM-v2-1.7B	nynefold/F2LLM-v2-1.7B-ONNX	mix50-gptq-sq50-a4	1629 (6885)	1788 (5603)	+0.19 [-0.44, +0.85]	-0.38 [-1.22, +0.28]
Qwen3-Reranker-0.6B	nynefold/Qwen3-Reranker-0.6B-ONNX	e8-w8-rtn-a4	692 (2386)	1473 (2589)	-0.02 [-0.48, +0.43]	-0.07 [-1.08, +0.89]
zerank-1-small	nynefold/zerank-1-small-ONNX	mix30-gptq-sq50-a4	1483 (6885)	2505 (6445)	-0.30 [-0.93, +0.33]	-0.36 [-1.42, +0.54]

Pipelines

The embedder retrieves the top 20 from FreshRet Small, the reranker reorders them, and the top 10 are scored. All models are the quantized picks.

embedder	reranker	total MB	nDCG@10	R@10	MRR@10	vs best (pt, 95% CI)
pplx-embed-v1-0.6b mix02-gptq-a4-pc	none	401	0.746	0.898	0.775	-12.57 [-14.96, -10.17]
pplx-embed-v1-0.6b mix02-gptq-a4-pc	Qwen3-Reranker-0.6B e8-w8-rtn-a4	1093	0.735	0.908	0.747	-13.68 [-15.95, -11.39]
F2LLM-v2-1.7B mix50-gptq-sq50-a4	none	1629	0.742	0.891	0.762	-12.98 [-15.47, -10.50]
pplx-embed-v1-0.6b mix02-gptq-a4-pc	zerank-1-small mix30-gptq-sq50-a4	1885	0.870	0.935	0.921	-0.16 [-1.02, +0.65]
F2LLM-v2-1.7B mix50-gptq-sq50-a4	Qwen3-Reranker-0.6B e8-w8-rtn-a4	2321	0.737	0.907	0.750	-13.45 [-15.58, -11.31]
F2LLM-v2-1.7B mix50-gptq-sq50-a4	zerank-1-small mix30-gptq-sq50-a4	3113	0.872	0.937	0.922	best

pplx-embed + zerank: 0.870 in 1885 MB, +12.40 [+10.13, +14.69] points over pplx-embed alone (0.746, 401 MB). F2LLM + zerank scores the same within noise at 1.2 GB more. Nothing sits usefully between the two sizes. The embedders already put about 95% of the relevant documents in their top 20, so the reranker's ordering is what moves nDCG.

Rerank depth (zerank pick, pplx-embed pool, vs reranking 20): 15 candidates cost 0.38 points [-0.80, +0.01], 10 cost 1.45 [-2.30, -0.67], 5 cost 3.33. Reranking cost grows with the number of candidates, so 15 is a reasonable saving and 10 still keeps about 11 points over the embedder alone.

On the MIRACL slice (120 queries, BM25 top-20 pools, sparse labels) neither FP32 reranker beats pplx-embed alone; MIRACL has few judged passages per query, so FreshRet Small, fully judged, is the set that separates the rerankers.

Method

Quantizer. Our own writer for ONNX Runtime's MatMulNBits (linear layers) and GatherBlockQuantized (token table) contrib ops, so bits, block size, algorithm and compute mode can be set per tensor. Blocks of 32 input channels, asymmetric unless noted. Algorithms: round-to-nearest (RTN), clip search (MSE, optionally Hessian-weighted), HQQ, and GPTQ with act-order (Hessians from a calibration split of MIRACL dev queries).

Mixed precision. A sensitivity scan in PyTorch (fake-quant on cached layer inputs) estimates the damage of putting each tensor at 4 bit. mixNN starts from all-4-bit and promotes the most damaging tensors to 8 bit until NN% of the size gap to all-8-bit is spent.

Gates. Stage A, fidelity to FP32 on public MIRACL sets: embedders need cosine mean >= 0.99 and p1 >= 0.97; rerankers need per-query Spearman mean >= 0.98. Stage B (embedders) and C (rerankers), retrieval quality on FreshRet Small (300 queries, 6 languages, a third cross-lingual): nDCG@10 may drop at most 1.0 point vs FP32. "Strict" means the whole 95% interval clears -1.0. Embedder picks are the smallest variant passing both gates strictly. For rerankers, stage C was run on candidates near Spearman 0.99, since 0.987 already lost 1.2 points; their picks are the smallest variant that passed stage C strictly, and smaller variants that only just passed stage A were not measured. FreshRet Small is never used for calibration or recipe choice.

Sets. Fidelity: 60 MIRACL queries + 360 passages (embedders), 360 query-passage pairs (rerankers). Quality: FreshRet Small in full; reranker pools are its top 20 by reciprocal-rank fusion of BM25 and both FP32 embedders. Held-out check: 120 MIRACL dev queries (20 per language: en, es, fr, de, zh, ar) with BM25 top-20 pools. All MIRACL data is the dev split of mteb/MIRACLReranking, MTEB's reranking version of MIRACL; calibration, fidelity and held-out queries are disjoint.

What we learned

• GPTQ is what makes 4 bit work for the 0.6B embedder. pplx-embed at uniform 4 bit: RTN, MSE and HQQ all land near cosine 0.90; GPTQ reaches 0.992 at the same size. The base model's own 4-bit ONNX sits at 0.90 too.
• Fidelity damage is uneven. F2LLM's damage piles up in its last layers (layer 27's up-projection alone drops cosine to 0.74); a few percent of the tensors at 8 bit fixes most of it. Qwen3-Reranker's is spread thin, so mixing barely helps and it stays at 8 bit.
• Per-block INT8 compute breaks F2LLM without SmoothQuant. F2LLM at 8-bit weights falls from cosine 0.995 to 0.977 once the activations are INT8 too; the MLP inputs have 2-3x the outlier ratio of pplx-embed's. SmoothQuant (alpha 0.5, folded exactly into the weights) brings it back to 0.992. zerank, the other Qwen3-1.7B model, has the same outliers but doesn't need it: at 8-bit weights INT8 compute keeps Spearman 0.9963 with or without SmoothQuant, and at 4 bit it raises 0.9808 to 0.9865. On pplx-embed and Qwen3-Reranker it changes nothing.
• Stage A has to be tight for rerankers. Spearman ~0.987 still lost 1.2 nDCG points on FreshRet Small (zerank uniform 4 bit); ~0.990 was safe.
• Pruning pplx-embed's vocabulary to the tokens seen in the MIRACL corpus (six languages) saves 69 MB. It keeps every MIRACL token, so MIRACL results are unchanged by construction; on FreshRet Small, which was not used to build it, it costs -0.29 points [-0.78, +0.22] against the unpruned variant, -1.00 [-1.95, -0.27] on Arabic documents.
• The a4 variants (INT8 compute) are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights.

All variants

Stage A fidelity for every variant; FreshRet Small quality where it was measured (finalists only). "Below quality gate" means it failed stage A, or its nDCG@10 drop is more than 1.0 point. "pass (estimate only)" means the estimate is within 1.0 point but the interval is not; "pass (stage A only)" means it was not measured on FreshRet Small.

pplx-embed-v1-0.6b

Base model perplexity-ai/pplx-embed-v1-0.6b, FP32 2387 MB. Files: nynefold/pplx-embed-v1-0.6b-ONNX.

variant	MB	cos mean (gate 0.99)	FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt)	verdict
e4-w4-rtn-a4	385	0.9014		below quality gate
mix02-gptq-a4-pc	401	0.9932	-0.08 [-0.91, +0.74]	pick, pass
e8-w4-rtn-b128-a4	419	0.8669		below quality gate
e8-w4-rtn-sym-a4	458	0.8714		below quality gate
e8-w4-gptq-a4	465	0.9921	+0.19 [-0.53, +0.93]	pass
e8-w4-gptq-sq50-a4	465	0.9921		pass (stage A only)
e8-w4-mse-a4	465	0.9020		below quality gate
e8-w4-rtn-a4	465	0.9019		below quality gate
mix02-gptq-a4	470	0.9932	+0.20 [-0.49, +0.90]	pass
mix02-gptq-sq50-a4	470	0.9931		pass (stage A only)
mix02-mse-a4	470	0.9837		below quality gate
mix02-msew-a4	470	0.9860		below quality gate
mix02-rtn-a4	470	0.9828		below quality gate
mix10-gptq-a4	488	0.9958	+0.06 [-0.54, +0.63]	pass
mix10-mse-a4	488	0.9904		pass (stage A only)
mix10-msew-a4	488	0.9917		pass (stage A only)
mix10-rtn-a4	488	0.9898		below quality gate
e8-w4-hqq-a0	513	0.9055		below quality gate
mix30-gptq-a4	533	0.9977		pass (stage A only)
mix30-mse-a4	533	0.9952		pass (stage A only)
mix30-msew-a4	533	0.9958		pass (stage A only)
mix30-rtn-a4	533	0.9949		pass (stage A only)
e8-w8-rtn-a0	692	0.9994		pass (stage A only)
e8-w8-rtn-a4	692	0.9993	+0.15 [-0.05, +0.40]	pass
e32-w8-rtn-a4	1135	0.9993		pass (stage A only)

F2LLM-v2-1.7B

Base model codefuse-ai/F2LLM-v2-1.7B, FP32 6885 MB. Files: nynefold/F2LLM-v2-1.7B-ONNX.

variant	MB	cos mean (gate 0.99)	FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt)	verdict
e8-w4-gptq-a0	1266	0.9540		below quality gate
e8-w4-gptq-a4	1266	0.9370		below quality gate
e8-w4-gptq-gu0	1266	0.9508		below quality gate
e8-w4-gptq-sq50-a4	1266	0.9490		below quality gate
mix02-gptq-a4	1280	0.9471		below quality gate
mix10-gptq-gu0	1338	0.9808		below quality gate
mix30-gptq-gu0	1483	0.9872		below quality gate
mix30-gptq-sq50-a4	1483	0.9895		below quality gate
mix50-gptq-gu0	1629	0.9943		pass (stage A only)
mix50-gptq-sq50-a4	1629	0.9922	+0.19 [-0.44, +0.85]	pick, pass
e8-w8-rtn-a0	1993	0.9953	+0.33 [-0.01, +0.74]	pass
e8-w8-rtn-a4	1993	0.9772		below quality gate
w8-qkv0	1993	0.9780		below quality gate
w8-qkvgu0	1993	0.9929		pass (stage A only)
w8-sq50-a4	1993	0.9919		pass (stage A only)
w8-sq75-a4	1993	0.9924		pass (stage A only)
e32-w8-rtn-a4	2877	0.9772		below quality gate

Qwen3-Reranker-0.6B

Base model Qwen/Qwen3-Reranker-0.6B, FP32 2386 MB. Files: nynefold/Qwen3-Reranker-0.6B-ONNX.

variant	MB	Spearman mean (gate 0.98)	FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt)	verdict
e8-w4-gptq-a0	465	0.9694		below quality gate
e8-w4-gptq-a4	465	0.9687		below quality gate
e8-w4-gptq-sq50-a4	465	0.9657		below quality gate
e8-w4-rtn-a4	465	0.9209		below quality gate
mix02-gptq-a4	469	0.9700		below quality gate
mix10-gptq-a4	488	0.9700		below quality gate
mix30-gptq-a4	533	0.9704		below quality gate
mix30-gptq-sq50-a4	533	0.9801		pass (stage A only)
mix50-gptq-a4	578	0.9781	-0.92 [-1.77, -0.11]	below quality gate
mix50-gptq-sq50-a4	578	0.9785		below quality gate
e8-w8-rtn-a0	692	0.9976		pass (stage A only)
e8-w8-rtn-a4	692	0.9899	-0.02 [-0.48, +0.43]	pick, pass

zerank-1-small

Base model zeroentropy/zerank-1-small-reranker, FP32 6885 MB. Files: nynefold/zerank-1-small-ONNX.

variant	MB	Spearman mean (gate 0.98)	FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt)	verdict
e8-w4-gptq-a0	1266	0.9825		pass (stage A only)
e8-w4-gptq-a4	1266	0.9808		pass (stage A only)
e8-w4-gptq-sq50-a4	1266	0.9865	-1.22 [-1.91, -0.58]	below quality gate
mix10-gptq-a4	1338	0.9825		pass (stage A only)
mix10-gptq-sq50-a4	1338	0.9872		pass (stage A only)
mix30-gptq-a4	1483	0.9875		pass (stage A only)
mix30-gptq-sq50-a4	1483	0.9899	-0.30 [-0.93, +0.33]	pick, pass
mix50-gptq-a4	1629	0.9859		pass (stage A only)
mix50-gptq-sq50-a4	1629	0.9899		pass (stage A only)
e8-w8-rtn-a0	1993	0.9973		pass (stage A only)
e8-w8-rtn-a4	1993	0.9963		pass (stage A only)
w8-sq50-a4	1993	0.9963	+0.14 [-0.08, +0.45]	pass

Variant names

part	meaning
e8, e4, e32	token table at 8 bit, 4 bit, or kept FP32
w4, w8	linear layers at 4 or 8 bit
mixNN	4 bit, with the most sensitive tensors at 8 bit (NN% of the 4-to-8-bit size gap)
rtn, mse, msew, hqq, gptq	quantization algorithm (msew = Hessian-weighted MSE)
b128, sym	block size 128 (default 32); symmetric (default asymmetric)
a0, a4	ONNX Runtime accuracy_level: FP32 compute, or INT8 compute
sq50, sq75	SmoothQuant on the MLP inputs, alpha 0.5 / 0.75
gu0, qkv0, qkvgu0	INT8 compute except FP32 for gate/up, q/k/v, or both
pc	vocabulary pruned to the tokens seen in the MIRACL corpus (pplx-embed)

Licences

The quantized models are derivatives of their base models and keep their licences: pplx-embed-v1-0.6b MIT; F2LLM-v2-1.7B, Qwen3-Reranker-0.6B and zerank-1-small Apache-2.0. FreshRet Small is CC-BY-4.0.

Licence of this repo

Text and figures: CC-BY-4.0. The example scripts inference.py and pipeline_example.py: MIT. See LICENSE.