S murf C at at S em E val-2025 Task 3: Bridging External Knowledge and Model Uncertainty for Enhanced Hallucination Detection (2025)

Abstract: The Multilingual shared-task on Hallucinations and Related Observable Overgeneration Mistakes in the SemEval-2025 competition aims to detect hallucination spans in the outputs of instruction-tuned LLMs in a multilingual context. In this paper, we address the detection of span hallucinations by applying an ensemble of approaches. In particular, we synthesized a PsiloQA dataset and fine-tuned LLM to detect hallucination spans. In addition, we combined this approach with a white-box method based on uncertainty quantification techniques. Using our combined pipeline, we achieved 3rd place in detecting span hallucinations in Arabic, Catalan, Finnish, Italian, and ranked within the top ten for the rest of the languages.

## introduction in recent years, there have been significant advancements in natural language generation (nlg) models, mainly due to transformer-based architectures such as gpt @xcite . nevertheless, the field faces two related challenges: the first is the propensity for current neural systems to create incorrect, yet coherent outputs, and the second is the inefficiency of current metrics in prioritizing accuracy over fluency. this leads to a phenomenon known as "hallucination", where nlg models generate cohere but inaccurate outputs that are difficult to automatically identify @xcite . the shared-task on multilingual hallucinations and related observable overgeneration mistakes (mu-shroom, @xcite ) has been suggested to address this challenge. in particular, the mu-shroom task aims to detect hallucination spans in the outputs of instruction-tuned llms in a multilingual context models (arabic, basque, catalan, chinese, czech, english, farsi, finnish, french, german, hindi, italian, spanish, and swedish). 1 1 https://helsinki-nlp.github.io/shroom/ mu-shroom is the continuation of the competitions in hallucination detection, the first being shroom. its goal was to detect hallucinations and overgeneration errors within various generation tasks, such as machine translation, paraphrasing, and definition modeling @xcite . to address the mu-shroom challenge, we developed an ensemble of approaches. first, we finetuned llm on synthetic span-level hallucination detection data. then, we combined this approach with white-box methods based on uncertainty quantification (uq) techniques. using our combined pipeline, we achieved 3rd place in detecting span hallucinations in arabic, catalan, finnish, italian, and ranked within the top ten for the rest of the languages. our contribution could be summarized as follows: ## multilingual synthetic dataset for hallucination detection all of our approaches require additional data for fine-tuning and calibration. the exact scheme of synthetic data generation is shown in figure 1 . to collect synthetic data, we first generate multilingual question-context-answer triplets based on contexts from wikipedia. generation was performed using gpt-4o for the exact prompt used to generate question-answer pairs. next, we collect the hypotheses by passing the generated questions without any contexts to various llms. finally, we ask gpt-4o to find all inconsistencies between the llm answer and the golden answer. any inconsistent information in the hypotheses that contradicts the golden answer is considered a hallucination. in total, synthetic dataset contains 52 271 samples after all filtering stages. the generation of synthetic training data through llm has been successfully employed to address detoxification @xcite and pii detection @xcite . ## error analysis we conducted a detailed error analysis on a subset of english test examples to pinpoint the most common failures of our span-detection algorithms. we found that the vast majority of errors remain factual misclassifications. notably, both the white-box and black-box methods tend to identify spans that are slightly longer. although they still correctly cover the spans, they occasionally truncate entities -e.g. predicting "ewald klein in the 1930" instead of the distinct facts "chemist", "ewald", "1930s" -a behavior more pronounced in the white-box method, which tends to select larger, statement-level spans over fine-grained annotations. on the quantitative side, we measured inaccuracy in 14 languages (