zerank-2-reranker — MLX (q8)

zeroentropy/zerank-2-reranker converted to MLX for Apple silicon. Both the transformer body and the tied token embedding/readout matrix use 8-bit quantization.

On-disk size: 4.3 GB. For the highest-fidelity MLX conversion, use fcmeyer/zerank-2-reranker-MLX-bf16.

zerank-2 is a cross-encoder reranker built on Qwen3-4B. It reads a query and one document together and returns a single relevance score, which is more accurate than comparing independent embeddings but requires one forward pass per candidate document.

Usage

Download rerank.py from this repo, then:

pip install mlx-lm transformers

from rerank import Reranker, probability

reranker = Reranker("fcmeyer/zerank-2-reranker-MLX-8bit")

ranked = reranker.rank(
    "What causes the northern lights?",
    [
        "Auroras occur when charged particles from the solar wind are funnelled "
        "along Earth's magnetic field lines into the upper atmosphere.",
        "Iceland is a popular destination for tourists hoping to photograph them.",
        "The Earth's magnetic field reverses polarity every few hundred thousand years.",
    ],
)

for document, score in ranked:
    print(f"{score:+.3f}  p={probability(score):.3f}  {document[:60]}")

rank() returns documents best-first. score() takes explicit (query, document) pairs and preserves your input order. Both batch internally.

How scoring works

The model does not have a classification head. A pair is rendered through the chat template as system = query, user = document, followed by an assistant generation prompt; the score is the logit of token id 9454 ("Yes") at the final position. That token id comes from 1_LogitScore/config.json in this repo.

The raw logit is the score, and it is what you rank by. sigmoid(score / 5) maps it to the 0–1 range shown on the upstream model card. Scores are comparable within a single query, not across queries.

Verification

Every score below is a "Yes" logit measured on the same 486 query/document pairs used to verify the unquantized conversion. PyTorch float32 is the reference; MLX bf16 is the verified lossless conversion, and MLX q8 is this lossy build.

vs. PyTorch float32	MLX bf16	MLX q8
mean abs. score delta	0.0548	0.0723
p99 abs. score delta	0.1848	0.2119
max abs. score delta	0.2172	0.2736
Spearman correlation	0.999654	0.999494
mean per-query Kendall τ	0.978934	0.977836
ranking inversions	24	29
resolvable ranking inversion rate	0 / 4719 (0.0000%)	0 / 4719 (0.0000%)
relevance verdict flips	0	0

Per-query Kendall τ is computed over each query's own candidate documents. An inversion is resolvable only when the two documents' float32 scores differ by more than 0.1930, the measured PyTorch bfloat16 p99 error. The rate therefore uses all 4719 same-query document pairs that float32 resolved as its denominator. A relevance verdict flip crosses the sigmoid(score/5) = 0.5 boundary relative to float32.

Independent BEIR verification

The independent fixture has 1449 pairs from 210 complete query pools sampled from FiQA, SciFact, and NFCorpus.

vs. PyTorch float32 (BEIR sample)	MLX bf16	MLX q8
mean abs. score delta	0.0508	0.0807
p99 abs. score delta	0.1727	0.2861
max abs. score delta	0.2904	0.4682
Spearman correlation	0.999930	0.999875
mean per-query Kendall τ	0.988963	0.986915
ranking inversions	10	13
resolvable ranking inversion rate	0 / 4168 (0.0000%)	1 / 4168 (0.0240%)
resolvable relevance verdict flips	0	0

This build has 1 resolvable ranking inversion among 4168 float32-resolved same-query document pairs on BEIR. It does not meet the zero-resolvable-inversion bar that the MLX bf16 build met. It has 0 resolvable relevance verdict flips.

NDCG@10 sanity check

implementation	NDCG@10 against BEIR qrels
PyTorch float32	0.954794
MLX bf16	0.955648
MLX q8	0.955104

These are complete pools of only 6–8 documents, and most queries have a single positive. NDCG@10 is therefore a sanity check, not the sharp metric for the quantization damage; per-query Kendall τ and resolvable inversions see reorderings among all documents.

Quantization ablation on BEIR

build	on-disk size	body / embedding bits	mean abs. delta	p99 abs. delta	resolvable inversion rate	resolvable verdict flips	mean per-query Kendall τ	NDCG@10	NDCG@10 − float32
q4	2.28 GB	4 / 4	1.5064	3.2907	127 / 4168 (3.0470%)	13	0.902443	0.957950	+0.003155
q4-e8	2.47 GB	4 / 8	1.3038	2.9973	123 / 4168 (2.9511%)	10	0.903406	0.957289	+0.002495
q4-e8-mixed46	2.60 GB	4 / 8	1.2570	2.8377	99 / 4168 (2.3752%)	9	0.917504	0.957501	+0.002706
q6	3.28 GB	6 / 6	0.1451	0.5718	9 / 4168 (0.2159%)	2	0.969801	0.954977	+0.000183
q8	4.29 GB	8 / 8	0.0807	0.2861	1 / 4168 (0.0240%)	0	0.986915	0.955104	+0.000310

The tied embedding is also the readout matrix: the score is dot(hidden_state, embed_tokens[9454]). The 4-bit experiment therefore tested whether holding that matrix at 8 bits would protect the score. It was directionally right and practically useless. Relative to q4, q4-e8 removes 23% of the systematic bias (+1.319 to +1.015 logits) and 13% of mean absolute error, but residual noise is essentially unchanged (0.586 to 0.583) and mean per-query Kendall τ is statistically indistinguishable (0.9024 to 0.9034). Protecting the readout row protects the ruler; 4-bit quantization also corrupts the hidden state being measured, and that is what reorders documents.

The decomposition below is computed directly from the BEIR score vectors. Bias is the mean signed error. Noise is the mean absolute residual after a least-squares linear fit of each build's scores against float32.

build	mean abs. error	mean signed error (bias)	mean abs. fitted residual (noise)	negative→positive flips	positive→negative flips
q4	1.506	+1.319	0.586	10	3
q4-e8	1.304	+1.015	0.583	6	4
q4-e8-mixed46	1.257	+1.007	0.501	7	2
q6	0.145	-0.007	0.145	0	2
q8	0.081	+0.031	0.074	0	0

The verdict directions corroborate the distinction: q4 produces 10 negative-to-positive flips against 3 the other way, the expected signature of its positive logit bias at the 0.5 boundary. q6 has only 0 and 2 respectively, so its two flips run the other way and are noise rather than systematic positive drift.

Differences from the source repo

Only the weight format. This repo carries the full upstream file set, including 1_LogitScore/config.json, modules.json, sentence_bert_config.json, config_sentence_transformers.json, and the BPE vocab.json / merges.txt — mlx_lm.convert copies none of those on its own, and without 1_LogitScore/config.json there is no record in the repo of which token id the score comes from.

Reproducing

pip install mlx-lm
mlx_lm.convert --hf-path zeroentropy/zerank-2-reranker \
    --mlx-path zerank-2-mlx-q8 --quantize --q-bits 8 \
    --q-group-size 64

then copy the sentence-transformers metadata listed above across from the source repo.

License and attribution

Apache 2.0, inherited from the source model. All credit for the model itself goes to ZeroEntropy; see their technical report. This repo is a format conversion and adds no training.