nynefold/Qwen3-Reranker-0.6B-ONNX

Qwen3-Reranker-0.6B ONNX, 4/8-bit quantized

Quantized ONNX versions of Qwen/Qwen3-Reranker-0.6B for ONNX Runtime. This is a quantized derivative: the model, its training and its licence (Apache-2.0) belong to its original authors. All 12 variants we built are here, including those below our quality gate, with their measurements.

Recommended: e8-w8-rtn-a4, 692 MB instead of 2,386 MB, 1,473 MB RAM instead of 2,589 MB; nDCG@10 vs FP32 -0.02 [-0.48, +0.43] points on FreshRet Small, -0.07 [-1.08, +0.89] on MIRACL.

Usage

pip install "onnxruntime>=1.31" tokenizers numpy huggingface_hub
hf download nynefold/Qwen3-Reranker-0.6B-ONNX inference.py --local-dir .

from inference import load

model = load("nynefold/Qwen3-Reranker-0.6B-ONNX")  # recommended variant; load("nynefold/Qwen3-Reranker-0.6B-ONNX", variant="...") for another
scores = model.rerank("How do I reset the router?",
                      ["Hold the reset button for ten seconds.", "The router has four LAN ports."])
print(scores)  # relevance in [0, 1], one per document

inference.py (in this repo) needs only ONNX Runtime, tokenizers and NumPy. Given a repo id, it downloads the variant as plain files into ~/.cache/nynefold/<repo id> (set NYNEFOLD_CACHE to change the folder); ONNX Runtime does not load model.onnx.data through the Hub cache's symlinks. load("path/to/folder") uses a local copy. To run the graphs another way:

• Inputs input_ids and attention_mask, int64 [batch, seq], right-padded.
• Input: the base model's fixed prefix, then <Instruct>: {instruction}\n<Query>: {query}\n<Document>: {document}, then its suffix; only the middle part is truncated, to 1024 tokens in total. Default instruction: "Given a web search query, retrieve relevant passages that answer the query".
• Output logit [batch]: logit(yes) - logit(no) at the last token; score = sigmoid(logit).
• config.json holds all of this in machine-readable form.

Variants

Each folder holds model.onnx + model.onnx.data and the recipe.json it was built from. Quality deltas are paired against FP32 in nDCG@10 points with 95% bootstrap intervals.

variant	size MB	RAM MB	stage A Spearman mean (gate 0.98)	FreshRet Small nDCG@10 vs FP32	MIRACL nDCG@10 vs FP32	status
FP32 (not included)	2,386	2,589	1	reference	reference
e8-w4-gptq-a0	465	1,143	0.9694			below quality gate
e8-w4-gptq-a4	465	1,256	0.9687			below quality gate
e8-w4-gptq-sq50-a4	465	1,256	0.9657			below quality gate
e8-w4-rtn-a4	465	1,257	0.9209			below quality gate
mix02-gptq-a4	469	1,260	0.9700			below quality gate
mix10-gptq-a4	488	1,278	0.9700			below quality gate
mix30-gptq-a4	533	1,322	0.9704			below quality gate
mix30-gptq-sq50-a4	533	1,322	0.9801			passes stage A only
mix50-gptq-a4	578	1,365	0.9781	-0.92 [-1.77, -0.11]		below quality gate
mix50-gptq-sq50-a4	578	1,365	0.9785			below quality gate
e8-w8-rtn-a0	692	1,368	0.9976			passes stage A only
e8-w8-rtn-a4	692	1,473	0.9899	-0.02 [-0.48, +0.43]	-0.07 [-1.08, +0.89]	recommended

• Recommended variant chosen on FreshRet Small quality; MIRACL is the held-out check (120 dev queries in en, es, fr, de, zh, ar from mteb/MIRACLReranking, MTEB's reranking version of MIRACL, with BM25 top-20 pools). Calibration and stage A use other, disjoint MIRACL dev queries.
• "Below quality gate": stage A fidelity under the gate, or more than 1.0 nDCG@10 point lost on FreshRet Small. "Passes stage A only": not measured on FreshRet Small (finalists only), so its retrieval quality is unconfirmed.
• RAM: peak process memory while scoring 20 candidates of 170 tokens, in a process with only ONNX Runtime, tokenizers and NumPy loaded.
• a4 variants use INT8 compute (accuracy_level 4) and are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights.
• Tested with ONNX Runtime 1.31. Variant names are explained in nynefold/retrieval-quant.

How they were made

Linear layers use ONNX Runtime's MatMulNBits and the token table GatherBlockQuantized, with bits, algorithm and compute mode set per tensor (GPTQ, RTN, clip search; mixed 4/8-bit from a sensitivity scan; SmoothQuant where noted). The method, gates, all results and the pipeline comparison with the other models are in nynefold/retrieval-quant. The evaluation set is nynefold/FreshRet-Small, with mteb/MIRACLReranking for calibration, fidelity and the held-out check; all six repos are in the collection retrieval-quant: quantized ONNX retrievers.

Licence

Apache-2.0, as the base model: see LICENSE. Please cite and credit the original model, Qwen/Qwen3-Reranker-0.6B.