When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with P silo QA (2025)

Abstract: Hallucination detection remains a fundamental challenge for the safe and reliable deployment of large language models (LLMs), especially in applications requiring factual accuracy. Existing hallucination benchmarks often operate at the sequence level and are limited to English, lacking the fine-grained, multilingual supervision needed for comprehensive evaluation. In this work, we introduce PsiloQA, a large-scale, multilingual dataset annotated with span-level hallucinations across 14 languages. PsiloQA is constructed through an automated three-stage pipeline: generating question–answer pairs from Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse LLMs in a no-context setting, and automatically annotating hallucinated spans using GPT-4o by comparing against golden answers and retrieved context. We evaluate a wide range of hallucination detection methods-including uncertainty quantification, LLM-based tagging, and fine-tuned encoder models-and show that encoder-based models achieve the strongest performance across languages. Furthermore, PsiloQA demonstrates effective cross-lingual generalization and supports robust knowledge transfer to other benchmarks, all while being significantly more cost-efficient than human-annotated datasets. Our dataset and results advance the development of scalable, fine-grained hallucination detection in multilingual settings.

## introduction large language models (llms) became a crucial component in a wide range of text generation applications, including summarization, translation, and question-answering systems in various domains. however, even state-of-the-art models, such as @xcite , or open-weight models, such as llama @xcite and @xcite are inevitably prone to production of hallucinations or unsupported facts in their generated output @xcite @xcite . this phenomenon poses a crucial obstacle for the real-world deployment of llms, particularly in safety-critical domains such as medicine @xcite . a single hallucinated word can substantially alter the overall meaning of the generation, potentially causing harm to end-users. consequently, hallucination detection has become a critical challenge in the development and application of llms @xcite . hallucination detection is typically categorized into three standard tasks: sequence-level, spanlevel, and entity-level. sequence-level detection focuses on identifying entire generations that contain some factual inconsistencies. in contrast, spanlevel and entity-level detection addresses the more challenging task of precisely highlighting factually misaligned spans or individual entities within the generated text. several approaches have been proposed for the detection of hallucination of llms. uncertainty quantification (uq) emerging as one of the most prominent research directions @xcite @xcite @xcite . recently, numerous uq methods have been developed specifically for llms @xcite . however, most of these methods focus on sequencelevel verification @xcite , while only a few methods operate at the token or span level @xcite . @xcite proposed combining uncertainty estimation methods with a fine-tuned llm, qwen2.5-7b-instruct, using a weighted averaging approach, where the weights were optimized separately for each language. although uq is a rapidly growing area of research, even the most advanced methods still face significant limitations. for instance, samplingbased methods @xcite require substantial computational overhead and operate only on the sequence level. information-based methods @xcite demonstrate strong performance in span-level tasks but still fail to detect some hallucinations and remain far from ideal performance. another set of approaches focuses on factchecking techniques based on external knowledge sources @xcite or auxiliary llms @xcite . while these methods achieve high performance, they require substantial computational overhead. these approaches first extract atomic claims from the generated response and compare them to a retrieved context using an auxiliary llm @xcite . this process produces a verification score, indicating the degree to which the extracted claims supported with the retrieved evidence. moreover, the performance of such systems is heavily dependent on the quality of both the retrieved context and the auxiliary llm, which typically requires fine-tuning for better performance @xcite . to evaluate the quality of both systems, including those based on uncertainty quantification and external knowledge, we require a dataset with annotated hallucinations. high-quality, fine-grained annotations, particularly at the span level, are labor-intensive, requiring expert human annotators @xcite or costly automated pipelines @xcite . span-level annotation, though more practical for pinpointing unsupported text, introduces additional complexity com-pared to sequence-level verification. multilingual contexts increase these challenges due to linguistic diversity, limited non-english data availability, and the need for language-specific pipeline adjustments @xcite . to address these limitations, we introduce a novel methodology for automatically generating multilingual data with fine-grained hallucination annotations, which includes (i) synthetically creating question-answering pairs from wikipedia article summaries, (ii) generating hypotheses using llms in a zero-context setting to produce both hallucinated and accurate answers, (iii) automating span-level inconsistency annotation by comparing responses to context and ground truth via an advanced llm, and (iv) automated filtering step through both rule-based and prompt-based methods. further details of this pipeline are outlined in section 3. the contributions of this work could be summarized as follows: ## experimental setup in our experiments, we pursue the following objectives: (i) to evaluate the performance of uncertainty quantification (uq) baselines, large language models (llms), and state-of-the-art methods on the psiloqa dataset; (ii) to demonstrate the transferability of knowledge from models trained on psiloqa and ragtruth to a range of downstream benchmarks; (iii) to assess the impact of multilingual training in psiloqa by comparing two configurations of mmbert @xcite : one trained on the full multilingual psiloqa dataset and the other trained separately on each individual language subset.