zerank-1-small ONNX, 4/8-bit quantized

Quantized ONNX versions of zeroentropy/zerank-1-small-reranker for ONNX Runtime. This is a quantized derivative: the model, its training and its licence (Apache-2.0) belong to its original authors. All 12 variants we built are here, including those below our quality gate, with their measurements.

Recommended: mix30-gptq-sq50-a4, 1,483 MB instead of 6,885 MB, 2,505 MB RAM instead of 6,445 MB; nDCG@10 vs FP32 -0.30 [-0.93, +0.33] points on FreshRet Small, -0.36 [-1.42, +0.54] on MIRACL.

Usage

pip install "onnxruntime>=1.31" tokenizers numpy huggingface_hub
hf download nynefold/zerank-1-small-ONNX inference.py --local-dir .

from inference import load

model = load("nynefold/zerank-1-small-ONNX")  # recommended variant; load("nynefold/zerank-1-small-ONNX", variant="...") for another
scores = model.rerank("How do I reset the router?",
                      ["Hold the reset button for ten seconds.", "The router has four LAN ports."])
print(scores)  # relevance in [0, 1], one per document

inference.py (in this repo) needs only ONNX Runtime, tokenizers and NumPy. Given a repo id, it downloads the variant as plain files into ~/.cache/nynefold/<repo id> (set NYNEFOLD_CACHE to change the folder); ONNX Runtime does not load model.onnx.data through the Hub cache's symlinks. load("path/to/folder") uses a local copy. To run the graphs another way:

Inputs input_ids and attention_mask, int64 [batch, seq], right-padded.
Input text: <|im_start|>system\n{query}<|im_end|>\n<|im_start|>user\n{document}<|im_end|>\n<|im_start|>assistant\n, query cut to 2,000 and document to 10,000 characters. Above 1024 tokens only the document's tokens are shortened, so the prompt always ends with the template's tail.
Output logit [batch]: the logit of token Yes; score = sigmoid(logit / 5).
config.json holds all of this in machine-readable form.

Variants

Each folder holds model.onnx + model.onnx.data and the recipe.json it was built from. Quality deltas are paired against FP32 in nDCG@10 points with 95% bootstrap intervals.

variant	size MB	RAM MB	stage A Spearman mean (gate 0.98)	FreshRet Small nDCG@10 vs FP32	MIRACL nDCG@10 vs FP32	status
FP32 (not included)	6,885	6,445	1	reference	reference
`e8-w4-gptq-a0`	1,266	1,935	0.9825			passes stage A only
`e8-w4-gptq-a4`	1,266	2,298	0.9808			passes stage A only
`e8-w4-gptq-sq50-a4`	1,266	2,298	0.9865	-1.22 [-1.91, -0.58]		below quality gate
`mix10-gptq-a4`	1,338	2,367	0.9825			passes stage A only
`mix10-gptq-sq50-a4`	1,338	2,367	0.9872			passes stage A only
`mix30-gptq-a4`	1,483	2,505	0.9875			passes stage A only
`mix30-gptq-sq50-a4`	1,483	2,505	0.9899	-0.30 [-0.93, +0.33]	-0.36 [-1.42, +0.54]	recommended
`mix50-gptq-a4`	1,629	2,644	0.9859			passes stage A only
`mix50-gptq-sq50-a4`	1,629	2,645	0.9899			passes stage A only
`e8-w8-rtn-a0`	1,993	2,687	0.9973			passes stage A only
`e8-w8-rtn-a4`	1,993	2,991	0.9963			passes stage A only
`w8-sq50-a4`	1,993	2,991	0.9963	+0.14 [-0.08, +0.45]		passes

Recommended variant chosen on FreshRet Small quality; MIRACL is the held-out check (120 dev queries in en, es, fr, de, zh, ar from mteb/MIRACLReranking, MTEB's reranking version of MIRACL, with BM25 top-20 pools). Calibration and stage A use other, disjoint MIRACL dev queries.
"Below quality gate": stage A fidelity under the gate, or more than 1.0 nDCG@10 point lost on FreshRet Small. "Passes stage A only": not measured on FreshRet Small (finalists only), so its retrieval quality is unconfirmed.
RAM: peak process memory while scoring 20 candidates of 170 tokens, in a process with only ONNX Runtime, tokenizers and NumPy loaded.
a4 variants use INT8 compute (accuracy_level 4) and are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights.
Tested with ONNX Runtime 1.31. Variant names are explained in nynefold/retrieval-quant.

How they were made

Linear layers use ONNX Runtime's MatMulNBits and the token table GatherBlockQuantized, with bits, algorithm and compute mode set per tensor (GPTQ, RTN, clip search; mixed 4/8-bit from a sensitivity scan; SmoothQuant where noted). The method, gates, all results and the pipeline comparison with the other models are in nynefold/retrieval-quant. The evaluation set is nynefold/FreshRet-Small, with mteb/MIRACLReranking for calibration, fidelity and the held-out check; all six repos are in the collection retrieval-quant: quantized ONNX retrievers.