AILS - NTUA at S em E val-2025 Task 3: Leveraging Large Language Models and Translation Strategies for Multilingual Hallucination Detection (2025)

Abstract: Multilingual hallucination detection stands as an underexplored challenge, which the Mu-SHROOM shared task seeks to address. In this work, we propose an efficient, training-free LLM prompting strategy that enhances detection by translating multilingual text spans into English. Our approach achieves competitive rankings across multiple languages, securing two first positions in low-resource languages. The consistency of our results highlights the effectiveness of our translation strategy for hallucination detection, demonstrating its applicability regardless of the source language.

## introduction hallucinations in large language models (llms) pose a significant challenge, as they can generate fluent yet factually incorrect or misleading content @xcite . several detection techniques have been developed, either by accessing the llm's internals @xcite @xcite , probing generation inconsistencies @xcite or exploiting tokenbased classification @xcite . while hallucination detection has been widely studied in monolingual contexts, multilingual settings are severely underexplored and accompanied by additional complexities. variations in linguistic structure, resource availability, and training data distribution can lead to uneven model reliability across languages. low-resource languages, in particular, are more susceptible to hallucinations due to limited high-quality training data, making factual consistency a critical issue. recent works in multilingual hallucinations detection verify faithfulness shortcomings @xcite , shift the focus on data rather than model capacity @xcite while also pinpointing evaluation concerns @xcite . to this end, the mu-shroom shared task is proposed in order to fill this gap by providing high-quality data on 14 languages, including lowresource languages such as farsi, czech, finnish, swedish and basque. this effort is accompanied by annotations on hallucinated data spans within given sentences, which participants have to automatically detect. in our work, we leverage llm prompting and translation strategies to address hallucination detection without designing independent systems per language. particularly, we combine two llms, llama 3 @xcite and claude (ai), prompting them in a few-shot manner to detect hallucinatory spans. our language-adaptive system is proven efficient and successful in both highand low-resource languages without any training or fine-tuning. as a result, we achieve first position in the low-resource farsi and czech languages, second position in the high-resource italian language and among the top 15% positions in english, spanish, german, hindi and basque. ## system overview we focus on llm prompting and translation strategies to tackle hallucination detection challenges in a language-agnostic manner. due to the promptheavy nature of our approach, no further training is required to attain high scores in either high-or low-resource languages. specifically, we combine two llms, llama 3.1 405b 1 and claude 3.5 sonnet 2 , prompting them in a few-shot (fs) way to detect hallucination spans. to improve detection performance in low-resource languages, we also experiment with incorporating a translation tool to translate original input-output data to english. finally, given the inputs of each mushroom instance, we instruct llama and claude to generate the corresponding output and incorporate it as a hypothesis to facilitate hallucination detection. based on the results of our preliminary experiments presented in app.c, our final system consists of three components, i.e. three experiments: component 1 we prompt claude to detect hallucination spans given the input text, the output text in the original language and their translations in english, as well as the outputs produced by llama as hypothesis. component 3 we prompt llama to detect hallucination spans given the input text, the output text in the original language and their translations in english without providing extra generated answers as hypothesis. we adopt this approach since generated hypotheses can themselves contain hallucinations, resulting in misleading outcomes. moreover, the llms sometimes place undue emphasis on the provided generated hypothesis rather than relying on their internal knowledge, causing them to miss hallucinatory spans in the outputs. each component produces a list of hallucination spans and then the three lists are combined as follows: for each produced span, the assigned probability is calculated as the ratio of the experiments that characterize it as hallucination over the total number of experiments (three). ## results the results of our experiments are shown in detail in tables 1, 2 for prompting experiments (iou and correlation metrics respectively) and in however, the addition of translations and hypotheses has a more pronounced impact on low-resource languages compared to high-resource ones. regarding the translation experiments, interesting insights emerge. on one hand, in the simplest approach-where prompts are given in english while pairs remain in their original language-the english language score is not the highest. this suggests that translation aids in hallucination detec the results tables also show that for highresource languages such as spanish, chinese, and german, the fs scenario and the incorporation of the generated hypothesis contribute the most towards performance improvements. in contrast, for low-resource languages, translation is a crucial component in achieving similar results. the prompts are detailed in app. d. as a demonstration of our system, in ## conclusion in this work, we detect multilingual hallucination spans in the outputs from the semeval 2025-task 3 mushroom dataset using llm prompting and translation techniques. we showcase the merits of converting all data in english, especially for low-resource languages, achieving highly-ranked results in a language-agnostic manner overall.