Docs | Blog | Slack | GitHub

Vela Halu

Vela Halu finds answer spans unsupported by the supplied evidence, for grounded answers and response verification.

307M parameters · Supported input length: 8,192 tokens, including special tokens.

Labels are supported (0) and hallucinated (1). Spans use Unicode character offsets in the answer. Evidence support is distinct from real-world factual truth; an empty result does not guarantee correctness.

Answers requiring arithmetic or multi-step reasoning beyond explicit context may be incorrectly flagged.

Evaluation

Reported scores on the 10,698-example fixed evaluation set. Each cell is Vela Halu / LettuceDetect, on a 0-100 scale; higher is better. Bold Vela scores exceed the paired baseline.

Source	Samples	Span F1	Span P	Span R	Example F1	IoU
ALL	10,698	64.7 / 64.2	70.0 / 68.4	60.1 / 60.5	87.5 / 86.9	67.9 / 67.1
ACL	440	60.3 / 57.9	72.0 / 70.5	51.9 / 49.2	85.9 / 87.3	65.1 / 63.7
Code agent	2,015	53.0 / 50.8	65.5 / 61.9	44.6 / 43.0	78.3 / 77.0	58.0 / 58.1
README	641	76.7 / 75.1	81.8 / 78.9	72.2 / 71.6	91.5 / 90.0	78.4 / 76.8
Tool output	617	64.3 / 58.8	78.0 / 73.6	54.7 / 49.0	82.7 / 76.3	70.3 / 64.5
Wikipedia	1,388	73.4 / 70.8	77.1 / 74.1	70.1 / 67.8	93.0 / 91.7	79.1 / 76.8
PsiloQA (multilingual)	2,897	70.8 / 71.4	70.5 / 69.6	71.1 / 73.3	94.1 / 94.3	62.5 / 62.7
RAGTruth	2,700	52.5 / 52.8	67.4 / 66.8	43.1 / 43.7	74.9 / 74.3	72.8 / 72.4

LettuceDetect scores are from its published model card, rounded to three decimal places on the 0-1 scale before conversion.

Span metrics measure character overlap; example F1 measures whether an answer contains any hallucinated span. IoU is mean character-set intersection over union. Vela evaluation uses bfloat16, an 8,192-token limit, only_first pair truncation, and token scores strictly above 0.5. Full scores include all source groups and truncation statistics.

Quick start

With PyTorch and Transformers 4.57.6:

import torch
from transformers import AutoConfig, AutoModelForTokenClassification, AutoTokenizer

model_id = "llm-semantic-router/Vela-1.0-Encoder-307M-Halu"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
config = AutoConfig.from_pretrained(model_id)
config.reference_compile = False
model = AutoModelForTokenClassification.from_pretrained(
    model_id, config=config, attn_implementation="sdpa"
).eval()

context = "The museum opens at 10:00 on Tuesday."
question = "When does the museum open on Tuesday?"
answer = "The museum opens at 09:00 on Tuesday."
prompt = f"User request: {question}\n\n{context}"
inputs = tokenizer(prompt, answer, return_offsets_mapping=True,
                   return_tensors="pt", truncation=False)
assert inputs.input_ids.shape[1] <= 8192
sequence_ids = inputs.sequence_ids(0)
offsets = inputs.pop("offset_mapping")[0].tolist()
with torch.inference_mode():
    scores = model(**inputs).logits.float().softmax(-1)[0, :, 1].tolist()

spans, current = [], None
for sequence, (start, end), score in zip(sequence_ids, offsets, scores):
    if sequence != 1 or end <= start:
        continue
    if score > 0.5:
        if current is None:
            current = {"start": start, "end": end}
        else:
            current["end"] = max(current["end"], end)
    elif current is not None:
        spans.append(current)
        current = None
if current is not None:
    spans.append(current)
print([{**span, "text": answer[span["start"]:span["end"]]} for span in spans])