bge-reranker-v2-m3-GGUF

GGUF quantizations of BAAI/bge-reranker-v2-m3, a multilingual cross-encoder reranker. Validated and published by SmartTasks.

What this is (read first)

This is a reranker, not an embedding model. Unlike an embedder (which turns text into a vector), a reranker takes a [query, document] pair and directly outputs a single relevance score. Use it as the second stage of retrieval: first retrieve candidates with an embedder (e.g. BGE-M3), then rerank those candidates with this model for higher precision. It does not produce embeddings and cannot be used for vector search.

Validation

Check	Result
Pairwise ranking accuracy	1 (relevant vs. irrelevant, held-out triples)
Conversion	clean (XLM-RoBERTa arch, llama.cpp)
Quants	Q4_K_M, Q5_K_M, Q8_0

Relevant passages scored strongly positive; irrelevant passages scored strongly negative (clean separation preserved across quantization). SHA-256 hashes in SHA256SUMS.

Usage (llama.cpp)

Serve with the reranking endpoint enabled:

llama-server -m bge-reranker-v2-m3-Q8_0.gguf --reranking -c 2048 -ngl 99

Then POST to /rerank with a query and candidate documents; each returns a relevance_score. Higher = more relevant. Scores can be mapped to [0,1] via sigmoid.

Files

See SHA256SUMS for integrity hashes. Q8_0 for best fidelity; Q4_K_M for smallest footprint (ranking behavior preserved at all three levels).

Attribution

Derived from BAAI/bge-reranker-v2-m3 (Apache-2.0). See the original repo for authoritative model details.

First-party validation by the SmartTasks pipeline (ranking correctness + quantization fidelity). Not an independent certification.