katsubakirill/toolaware-modernbert-large-toolace-hallucination

ToolAware_ModernBERT_large_finetuned

Fine-tuned ModernBERT token classifier for span-level hallucination detection in tool-calling dialogues.

Base Model

KRLabsOrg/lettucedect-large-modernbert-en-v1

Dataset

katsubakirill/toolace-hallucination-span-detection

Input Format

tool-aware input: question, available tools, executed tool call, tool facts, raw tool output, answer

Only answer tokens are trained/evaluated for hallucination labels. Non-answer tokens are ignored with label -100.

Labels

• 0: non_hallucinated
• 1: hallucinated

Test Metrics

dataset_type	n_examples	span_precision	span_recall	span_f1	example_f1	exact_span_match_rate	threshold
overall	330	0.857143	0.773028	0.812915	0.803987	0.509091	0.99
contradiction_with_tool_output	130	0.386185	0.312579	0.345506	0.607843	0.476923	0.99
missing_tool	62	0.910204	0.888952	0.899453	0.920635	0.645161	0.99
overgeneration	138	0.912332	0.810272	0.858278	0.897059	0.478261	0.99