NCL - U o R at S em E val-2025 Task 3: Detecting Multilingual Hallucination and Related Observable Overgeneration Text Spans with Modified R ef C hecker and Modified S efl C heck GPT (2025)

Abstract: SemEval-2025 Task 3 (Mu-SHROOM) focuses on detecting hallucinations in content generated by various large language models (LLMs) across multiple languages. This task involves not only identifying the presence of hallucinations but also pinpointing their specific occurrences. To tackle this challenge, this study introduces two methods: modified RefChecker and modified SelfCheckGPT. The modified RefChecker integrates prompt-based factual verification into References, structuring them as claim-based tests rather than single external knowledge sources. The modified SelfCheckGPT ~incorporates external knowledge to overcome its reliance on internal knowledge. In addition, both methods’ original prompt designs are enhanced to identify hallucinated words within LLM-generated texts. Experimental results demonstrate the effectiveness of the approach, achieving a high ranking on the test dataset in detecting hallucinations across various languages, with an average IoU of 0.5310 and an average COR of 0.5669.

## introduction large language models (llms) have significantly advanced in producing human-like text across various domains @xcite . however, one critical challenge remains: hallucinations-instances where the generated output contains logical inconsistencies, factual inaccuracies, or irrelevant information @xcite . these issues are particularly prominent in multilingual settings, where linguistic differences, cultural context, and the availability of external resources introduce additional complexities @xcite . to address this issue, semeval-2025 task-3: the multilingual shared-task on hallucinations and related observable overgeneration mistakes (mu-shroom) @xcite was introduced. this task involves identifying hallucinated text spans in llm-generated outputs across multiple languages and llms. to tackle this task, this work modifies two stateof-the-art methods: refchecker @xcite and selfcheckgpt @xcite . re-fchecker detects fine-grained hallucinations by extracting claim triplets (subject, predicate, object) from llm outputs and comparing them with prebuilt reference data, using text classification and aggregation rules. however, it cannot precisely locate hallucination positions and relies on fixed and incomplete references. the proposed modified refchecker improves upon this by introducing prompt-based fact verification, structuring references as claim-based tests for greater flexibility, and enhancing hallucination detection by calculating hallucination probabilities and providing soft and hard labels for more precise analysis. selfcheckgpt detects hallucinations by prompting the same llm for multiple responses and identifying inconsistencies. however, reliance on internal knowledge may fail when hallucinations are consistent. to address this, we modify selfcheck-gpt by incorporating external knowledge and enhancing the prompt design to identify specific hallucinated words rather than only their presence. overall, unlike the original refchecker and self-checkgpt, which rely on static references and internal prompt-based self-consistency, respectively, our modified methods incorporate external knowledge retrieval and prompt-driven span-level verification to improve hallucination detection accuracy and granularity. ## related work most recent approaches to detecting hallucinations in llm outputs rely on prompting techniques, where the models evaluate the likelihood of hallucinations in their responses. for instance, @xcite proposed prompting llms to generate an answer and then predict the probability of its correctness. @xcite introduced selfcheckgpt, which compares an llmgenerated sentence against multiple alternative generations, asking the model to assess whether the original sentence is consistently supported. friel and sanyal (2023) presented chainpoll, using detailed prompts to guide models in identifying hallucinations. @xcite proposed refchecker, a retrieval-augmented evaluation method that checks the consistency of model outputs against retrieved external references, aiming to identify factual inconsistencies and hallucinations without relying solely on llm self-judgment. however, most existing methods focus on detecting whether a text contains hallucinations or not. identifying the specific parts of a text that are hallucinations remains an open research challenge. therefore, in this work, we modified refchecker and selfcheckgpt, two state-of-the-art methods to handle this task. ## conclusion semeval-2025 mu-shroom introduced the task of detecting hallucination spans in multilingual llm outputs. to tackle this task, this work proposed two methods: modified-refchecker (mrc) and modified-selfcheckgpt-h (mscgh). these methods incorporated external knowledge integration and an improved prompt design, enabling the detection of text-span hallucinations in llm-generated texts. mrc and variations of mscgh (with different keyword extraction techniques, external knowledge sources, and prompt strategies) were evaluated across datasets in 14 languages. three top-performing methods were chosen for discussion in this paper. among the evaluated methods, mscgh using gpt-3.5 for keyword extraction, full google cse api results, and gpt-4o for hallucination detection achieved the best overall performance. although mscgh demonstrated higher performance, it lacked stability when applied across different languages. meanwhile, mrc was more stable but less optimized. one limitation of the proposed approaches is the assumption that external knowledge is accurate. however, the retrieved information may not always be fully factual due to the ever-growing volume of online content. such inaccuracies could reduce the effectiveness of the proposed approaches. future research could focus on refining the prompt design and enhancing external knowledge integration and faulty correction strategies. additionally, adaptive learning for low-resource languages and broader language task expansion could be considered.