uir-cis at S em E val-2025 Task 3: Detection of Hallucinations in Generated Text (2025)

Abstract: The widespread deployment of large language models (LLMs) across diverse domains has underscored the critical need to ensure the credibility and accuracy of their generated content, particularly in the presence of hallucinations. These hallucinations can severely compromise both the practical performance of models and the security of their applications. In response to this issue, SemEval-2025 Task 3 Mu-SHROOM: Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes introduces a more granular task for hallucination detection. This task seeks to identify hallucinations in text, accurately locate hallucinated segments, and assess their credibility. In this paper, we present a three-stage method for fine-grained hallucination detection and localization. First, we transform the text into a triplet representation, facilitating more precise hallucination analysis. Next, we leverage a large language model to generate fact-reference texts that correspond to the triplets. Finally, we employ a fact alignment strategy to identify and localize hallucinated segments by evaluating the semantic consistency between the extracted triplets and the generated reference texts. We evaluate our method on the unlabelled test set across all languages in Task 3, demonstrating strong detection performance and validating its effectiveness in multilingual contexts.

## introduction llms have gained widespread application across various fields due to their exceptional performance, making them a core technology of significant interest. however, their reliance on vast training data and probabilistic inference mechanisms makes them prone to generating factually incorrect, misleading, or unsubstantiated content, leading to the phenomenon of "hallucination" @xcite . ## task description and datasets in mu-shroom task 3, the organizers introduce a task aimed at predicting the locations of hallucinated segments in text generated by llms. this task focuses on identifying hallucinations in llm outputs across multiple languages, including modern standard arabic, basque, catalan, mandarin chinese, czech, english, farsi, finnish, french, german, hindi, italian, spanish, and swedish. the dataset consists of a sample set, a validation set, and an unlabeled training set. the sample set includes model-generated question-answer pairs, soft labels, and hard labels. soft labels indicate potential hallucinated spans along with their probability distributions, while hard labels specify the exact boundaries of hallucinated segments within the generated text. compared to the sample set, the validation set provides additional tokenized outputs and logit values for each generated token. the unlabeled training set contains only model-generated question-answer pairs without hallucination annotations. ## methodology we proposed a three-stage method, as illustrated in figure 1 . in the first stage, we applies the technique from refchecker @xcite to extract multiple triples from each user-provided passage. in the second stage, a high-performing language model generates references for each extracted triple based on stored knowledge. in the third stage, the method compares the generated references with the original triples to assess their semantic similarity, determine whether hallucinations are present, and perform hallucination localization and structured output formatting. ## conclusion this study proposes a three-stage approach for hallucination detection and localization, consisting of triple generation, reference text generation, and hallucination detection and localization. the method enables fine-grained identification of hallucinated content in text generated by large language models. experimental results indicate that the method achieves good hallucination detection performance in languages such as english, but its detection accuracy decreases in some low-resource languages and those with complex syntactic structures and data.