Nemotron-3.5-Rerank-8B-mlx-4bit

NVIDIA Nemotron-3.5-Rerank-8B-BF16 converted to MLX for Apple Silicon, with the language tower quantized to 4-bit. A reranker: it scores a query against a document (text, a page image, or both) and returns a relevance logit.

Built by NetProjects. Not affiliated with NVIDIA. All credit for the model goes to NVIDIA; this repository only changes the format and the precision.

What is in this build

Language tower: 4-bit affine quantization, group size 64
Kept at original precision: vision encoder and projector (bf16), norms, score head (float32), mean pooling (float32)
Size: 5.0 GB
Same inputs as the original: NVIDIA's chat template with query and document roles; one image per document; up to 32,768 tokens
Converted from upstream revision f88f8d9104726747791283ab156d2fcdb29e90a7

Quick start

pip install git+https://github.com/internet-projects/nemotron-retrieval-mlx   # or clone it and pip install -e .

from nemotron_mlx import Reranker

r = Reranker("netprojects/Nemotron-3.5-Rerank-8B-mlx-4bit")
docs = ["The Acropolis Museum in Athens displays the Parthenon sculptures.",
        "A heat pump moves heat.",
        {"image": "page.png"},                       # a page image (path, bytes, PIL image or data URL)
        {"text": "Figure 3", "image": "chart.png"}]
for hit in r.rank("Where can I see the Parthenon sculptures?", docs, top_k=3):
    print(round(hit["score"], 3), hit["index"])

Local server (OpenAI-style /v1/embeddings, Cohere/Jina-style /v1/rerank; images as data URLs or base64):

python -m nemotron_mlx.server --rerank netprojects/Nemotron-3.5-Rerank-8B-mlx-4bit --port 8095

Measured results

All numbers below were measured on this build. The reference is NVIDIA's own transformers code from the original card, run in float32.

Same output as NVIDIA's code

Test set	Pairs	Spearman	Same top document	Max score difference
SciFact queries x abstracts	400	0.9920	37/40	0.096
long documents (4K+ tokens) and page images	120 (14 long, 31 image)	0.9975	9/10	0.487

Tokenisation is identical to the original on every input.

Speed (text)

Pairs per second; measured with other model servers stopped.

Tokens per input	Batch	Pairs/s	Tokens/s	Peak memory
513	1	5.86	3,007	5.05 GB
513	16	5.17	2,652	5.77 GB
513	64	5.23	2,681	8.64 GB
4097	1	0.58	2,378	5.46 GB
4097	16	0.58	2,365	12.8 GB
4097	64	0.57	2,346	37.77 GB

Licence

The model is NVIDIA's, released under the OpenMDW License Agreement, version 1.1; the LICENSE file in this repository is that agreement, kept as it requires. It is built on mistralai/Ministral-3-8B-Instruct-2512-BF16 (Apache-2.0); see NOTICE and THIRD_PARTY_NOTICES.md.