Docs | Blog | Slack | GitHub

Vela Reranker

Bring relevant context to the top. Vela Reranker scores passages for retrieval, RAG, and context selection in your router.

307M encoder · 32K context · Multilingual

Evaluation

Compared with the original mmBERT Reranker. Scores are ×100; higher is better.

Evaluation	Metric	Original mmBERT	Vela
MIRACL · 320 queries, 4 languages	nDCG@10	80.47	87.11
Natural Questions · 128 queries	nDCG@10	97.86	99.13
SciFact · 48 queries, all 5,183 documents	nDCG@10	85.51	91.21
QASPER · 64 queries	nDCG@10	56.73	62.86
Controlled 4K–32K context · 288 pairs	Pair accuracy	55.21	62.50
32K context subset · 72 pairs	Pair accuracy	56.94	56.94
32K end-position subset · 24 pairs	Pair accuracy	54.17	54.17

Matched development subsets with identical candidate pools and complete inputs, using 22 layers, 768 dimensions, and FP32. MIRACL covers Arabic, Spanish, Japanese, and Chinese. The context set tests 24 queries across four lengths and three positions; its views are not independent examples.

Selected instruction-retrieval tasks from MTEB 2.20.12:

Task	Metric	Original mmBERT	Vela
Core17InstructionRetrieval	p-MRR	1.92	2.36
News21InstructionRetrieval	p-MRR	6.56	3.24
Robust04InstructionRetrieval	p-MRR	-1.06	2.25

These results describe the listed tasks and development subsets, not a full MTEB score or ranking.

Quick start

Install torch, transformers, and safetensors. Use a ROCm-enabled PyTorch build for AMD GPUs.

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "llm-semantic-router/Vela-1.0-Encoder-307M-Reranker"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).to(device).eval()

query = "How do I reset my password?"
passages = [
    "Select Forgot password on the sign-in page to receive a reset link.",
    "Your monthly invoice is available in the billing dashboard.",
]
inputs = tokenizer(
    [query] * len(passages), passages,
    padding=True, truncation=False, return_tensors="pt",
).to(device)
with torch.inference_mode():
    scores = model(**inputs).logits.flatten().tolist()
print(sorted(zip(scores, passages), reverse=True))

Higher scores mean greater relevance. The default uses all 22 layers and 768 dimensions. The 32,768-token limit includes the query, passage, and special tokens.

Explore the Vela collection