Nemotron-3.5-Rerank-8B-GGUF

NVIDIA Nemotron-3.5-Rerank-8B-BF16 converted to GGUF for llama.cpp, in BF16, Q8_0 and Q4_K_M. A reranker: it scores a query against a passage and returns a relevance logit. This build is text only.

Built by NetProjects. Not affiliated with NVIDIA. All credit for the model goes to NVIDIA; this repository only changes the format and the precision.

What is in this build

• Text model only: the bidirectional Ministral-3 8B backbone, attention-masked mean pooling, and the Linear(4096 -> 1) score head. The Pixtral vision tower is removed.
• Prompt format (no BOS/EOS): query: {query}\n\npassage: {document}
• Output: a raw relevance logit; apply a sigmoid if you want a probability
• Two variants:
  • rank: pooling RANK, with the score head stored as cls.output and a built-in rerank template. It gives native /v1/rerank and needs llama.cpp with native-rerank.patch.
  • embed: mean pooling, with the score head split out into score_weight.npy; score = dot(score_weight, embedding). It works with stock llama.cpp and rerank_wrapper.py.

File	Quant	Size	Runs on
Nemotron-3.5-Rerank-8B-rank-BF16.gguf	BF16	15.9 GB	patched llama.cpp
Nemotron-3.5-Rerank-8B-rank-Q8_0.gguf	Q8_0	8.5 GB	patched llama.cpp
Nemotron-3.5-Rerank-8B-rank-Q4_K_M.gguf	Q4_K_M	4.9 GB	patched llama.cpp
Nemotron-3.5-Rerank-8B-embed-BF16.gguf	BF16	15.9 GB	stock llama.cpp + wrapper
Nemotron-3.5-Rerank-8B-embed-Q8_0.gguf	Q8_0	8.5 GB	stock llama.cpp + wrapper
Nemotron-3.5-Rerank-8B-embed-Q4_K_M.gguf	Q4_K_M	4.9 GB	stock llama.cpp + wrapper

Q8_0 is the recommended choice.

Quick start

Native /v1/rerank (rank files)

Upstream llama.cpp does not register Mistral3ForSequenceClassification yet, so apply native-rerank.patch (made against llama.cpp master 0ec4027):

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 0ec4027
git apply ../native-rerank.patch
cmake -B build -DGGML_VULKAN=ON   # or CUDA / Metal / CPU
cmake --build build --config Release -j

Local server (Cohere/Jina-style /v1/rerank):

llama-server -m Nemotron-3.5-Rerank-8B-rank-Q8_0.gguf --reranking -ngl 99 -c 4096 -ub 4096 -b 4096 --port 8095
curl http://127.0.0.1:8095/v1/rerank -H "Content-Type: application/json" -d '{
  "query": "Where can I see the Parthenon sculptures?",
  "documents": ["The Acropolis Museum in Athens displays the Parthenon sculptures.", "A heat pump moves heat."]
}'

relevance_score is the raw logit.

The patch registers Mistral3/Ministral3ForSequenceClassification in the converter, maps score.weight to cls.output, and writes pooling RANK, labels ["score"] and the rerank template. It also adds CLS_OUT to the MISTRAL3 tensors (as an optional cls_out tensor) and defaults the pooling cls to MEAN in mistral3.cpp.

Stock llama.cpp (embed files)

llama-server -m Nemotron-3.5-Rerank-8B-embed-Q8_0.gguf --embedding --pooling mean -ngl 99 --port 8095

Send the formatted prompt with add_special: false; score = dot(score_weight.npy, embedding). rerank_wrapper.py is a minimal example and expects score_weight.npy (and a pairs.json test file) next to it.

Measured results

All numbers below were measured on these files. The reference is the original model run layer by layer in bf16 PyTorch on CPU.

Same output as the original

20 query/passage pairs (10 relevant, 10 irrelevant):

Quant	Max score difference	Mean difference	Pearson	Spearman	Pairwise order	Relevant > irrelevant
BF16	0.0113	0.0034	0.99999	1.0000	1.0000	10/10
Q8_0	0.0164	0.0083	0.99999	1.0000	1.0000	10/10
Q4_K_M	0.0356	0.0130	0.99990	0.9985	0.9947	10/10

Native /v1/rerank scores from the rank files match the wrapper scores from the embed files exactly (max difference 0.0000). The raw numbers are in parity.csv and parity_native.csv.

Limitations

• This was a small sanity set, not a full benchmark.
• The original tokenizer inserts a zero-width non-joiner after [ and <. Native /v1/rerank does not do this; rerank_wrapper.py does. This only matters for text containing those characters.
• Text only: page images are not supported. For multimodal reranking on Apple Silicon, use the MLX builds below.

Licence

The model is NVIDIA's, released under the OpenMDW License Agreement, version 1.1; the LICENSE file in this repository is that agreement, kept as it requires. Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. It is built on mistralai/Ministral-3-8B-Instruct-2512-BF16 (Apache-2.0); see NOTICE and THIRD_PARTY_NOTICES.md.

Modification notice: these files were converted to GGUF and quantized with llama.cpp. The vision tower was removed, and in the embed files the score head was split out into score_weight.npy. They are not official NVIDIA releases and are provided AS IS, without warranty.

Credits

• Model: NVIDIA, nvidia/Nemotron-3.5-Rerank-8B-BF16
• GGUF conversion, llama.cpp patch and tests: Paris Michalas, Partner at NetProjects
• Built on llama.cpp and ggml

Built and tested on ASUS ProArt (AMD Radeon 8060S, Vulkan) · llama.cpp 0ec4027

Also from NetProjects: MLX builds of this model: Nemotron-3.5-Rerank-8B-mlx-8bit · Nemotron-3.5-Rerank-8B-mlx-4bit · NetProjects MLX builds collection · all models at huggingface.co/netprojects