s-nlp/ToolHACE_qwen3.5_2B_span

ToolHACE Qwen3.5 2B Span

Qwen3.5-2B fine-tuned as a tool-calling hallucination auditor. The model compares a user request, available tools, tool outputs, and the assistant's final answer, then returns a JSON verdict with hallucination classes and verbatim answer spans.

This is a generative JSON-verdict model. It is not a conventional token classification or CRF span head.

Intended use

Use it to audit the final answer of a tool-calling assistant after tool execution. The model checks support against the supplied tool output; it does not independently verify facts that are absent from that context.

Model and training

• Base model: Qwen/Qwen3.5-2B
• Training: full-parameter SFT, one epoch, learning rate 1e-5
• Precision: BF16
• Loss: completion-only
• Warmup: 3%; cosine-free schedule
• Training data: toolace-unified-hallucinations v3_release train split (37,135 examples)
• Training date: 2026-08-07

Input and output contract

The model was trained with the following four context fields:

TOOLS: <tool signatures/descriptions separated by semicolons>

USER: <user request>

TOOL_OUTPUT:
<tool output>

ANSWER:
<assistant final answer>

The prompt limits are 1,200 characters for USER, 6,000 for TOOL_OUTPUT, and 4,000 for ANSWER.

The expected output is JSON only:

{"errors": []}

or:

{"errors": [{"class": "answer_mismatch", "span": "exact text from the answer"}]}

Supported classes:

• answer_mismatch: a value contradicts the tool output;
• overgeneration: an unsupported claim;
• missing_tool: an offered follow-up action that no available tool can perform;
• undergeneration: requested information omitted from the answer; its span is null.

Usage

The repository includes the standalone predict.py wrapper:

pip install vllm
python predict.py --model s-nlp/ToolHACE_qwen3.5_2B_span --demo

For serving on two GPUs:

vllm serve s-nlp/ToolHACE_qwen3.5_2B_span \
  --dtype bfloat16 \
  --trust-remote-code \
  --tensor-parallel-size 2

Deterministic decoding used for evaluation was temperature 0 with max_tokens=600 and thinking disabled.

Test evaluation

Evaluated on the pinned s-nlp/toolHACE test split:

• Rows: 11,637
• Dataset revision: dda429f84bac17c546b4dbaf1b82f9af365aeca4
• Model revision: 717dc03263bf15b0cdcd69bda7945866375cbea0
• The evaluated weights are unchanged by the subsequent card/artifact upload.
• Inference: vLLM chat, BF16, tensor parallelism 2, temperature 0, enable_thinking=False
• Prompt construction: the predict.py SYSTEM/build_prompt contract; all user turns and tool-response turns joined in order
• Response metric: binary hallucination detection against the row label
• Span metric: greedy one-to-one matching of localized answer spans at the stated IoU threshold; false-positive spans on clean rows count as FPs

Overall results

Metric	Precision	Recall	F1
Response	0.9820	0.7705	0.8635
Character-level span	0.8609	0.8944	0.8773
Span, IoU ≥ 0.50	0.8063	0.8294	0.8177
Span, IoU ≥ 0.75	0.7792	0.8015	0.7902

Strict JSON was produced for 11,620/11,637 rows (99.85%). There were 62 predicted span strings that could not be located verbatim in the answer.

Breakdown by gold category

Response scores are binary detection scores. Span scores are F1; they are not defined for clean and undergeneration rows because those categories have no gold hallucination span in this dataset.

Gold category	Rows	Response P / R / F1	Span F1 @ IoU .50	Span F1 @ IoU .75
answer_mismatch	1,206	1.0000 / 0.9511 / 0.9749	0.7574	0.7259
overgeneration	2,325	1.0000 / 0.9600 / 0.9796	0.8635	0.8386
missing_tool	1,529	1.0000 / 0.9706 / 0.9851	0.8481	0.8169
undergeneration	1,324	1.0000 / 0.0423 / 0.0812	—	—
clean	5,253	FPR 0.0171 (90 FPs)	—	—

The main weakness is undergeneration: the response recall on this category is only 0.0423. This substantially lowers the aggregate response F1, while the localization scores on the three span-bearing error classes remain much higher.

Limitations

• Training examples contain English tool-calling dialogues and generally one error class per row; multiple simultaneous errors are out of distribution.
• answer_mismatch and overgeneration can be semantically ambiguous when a value is absent from the tool output.
• Answers beyond 4,000 characters are truncated by the prompt contract.
• Span localization uses the first exact substring occurrence in the answer; repeated identical text is therefore ambiguous.
• The test results above use a generative JSON-verdict protocol and should not be compared directly with the CRF/token-tagging leaderboard.