katsubakirill/toolaware-modernbert-base-toolace-hallucination

ToolAware_ModernBERT_finetuned

Fine-tuned ModernBERT token classifier for span-level hallucination detection in tool-calling dialogues.

Base Model

KRLabsOrg/lettucedect-base-modernbert-en-v1

Dataset

katsubakirill/toolace-hallucination-span-detection

Input Format

tool-aware input: question, available tools, executed tool call, tool facts, raw tool output, answer

Only answer tokens are trained/evaluated for hallucination labels. Non-answer tokens are ignored with label -100.

Labels

• 0: non_hallucinated
• 1: hallucinated

Test Metrics

dataset_type	n_examples	span_precision	span_recall	span_f1	example_f1	exact_span_match_rate	threshold
overall	330	0.795651	0.757896	0.776315	0.80678	0.506061	0.99
contradiction_with_tool_output	130	0.259182	0.31385	0.283908	0.571429	0.469231	0.99
missing_tool	62	0.964447	0.849658	0.903421	0.95082	0.629032	0.99
overgeneration	138	0.858434	0.802064	0.829292	0.911765	0.485507	0.99