MSA at S em E val-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detection (2025)

Abstract: This paper describes our submission for SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. The task involves detecting hallucinated spans in text generated by instruction-tuned Large Language Models (LLMs) across multiple languages. Our approach combines task-specific prompt engineering with an LLM ensemble verification mechanism, where a primary model extracts hallucination spans and three independent LLMs adjudicate their validity through probability-based voting. This framework simulates the human annotation workflow used in the shared task validation and test data. Additionally, a fuzzy matching algorithm is utilized to improve span alignment. Our system ranked 1st in Arabic and Basque, 2nd in German, Swedish, and Finnish, and 3rd in Czech, Farsi, and French.

## introduction large language models (llms) are highly effective in generating text; however, they sometimes produce hallucinations-misleading content that is not properly grounded in the input data @xcite . identifying these spans is essential for improving the reliability of llm-generated outputs in translation, summarization, and conversational ai @xcite . semeval-2025 task 3: mu-shroom tackles this challenge by presenting a multilingual benchmark for detecting character-level hallucinations across multiple languages. the task involves detecting hallucinated spans in instruction-tuned llm outputs, presenting challenges in language diversity, annotation consistency, and accurate span localization. @xcite to tackle this challenge, our system utilizes a hybrid approach that integrates task-specific prompt 1 https://github.com/baraahekal/mu-shroom engineering for weak label generation with an llm ensemble verification mechanism @xcite . our methodology follows a multi-step adjudication process in which a primary llm identifies hallucination spans, and three independent llms subsequently verify their validity through a probability-based voting mechanism @xcite . additionally, we apply fuzzy matching techniques to improve the alignment of hallucination spans with ground truth annotations, thereby enhancing detection accuracy @xcite . by participating in this task, we gained insights into language-specific hallucination challenges and the strengths and limitations of llm-based verification. certain llms demonstrated closer alignment with human annotations, while hallucination patterns varied significantly, particularly in morphologically rich languages where annotation ambiguity was higher @xcite . our results indicate that ensemble verification and span refinement substantially improve hallucination detection, offering a robust approach for mitigating llm hallucinations in multilingual settings. 1 2 related work hallucination detection in large language models (llms) has been studied in machine translation, text summarization, and conversational ai @xcite . earlier approaches primarily relied on sentence-level classification, whereas recent research has transitioned to span-level detection for greater precision @xcite . selfconsistency verification and knowledge-grounded approaches have improved hallucination identification, but many depend on external data, limiting their applicability in multilingual settings. @xcite multilingual nlp models struggle with hallucinations, especially in low-resource languages where confidence scores are unreliable @xcite . morphologically rich languages introduce additional challenges due to intricate annotation inconsistencies @xcite . prior work on translation-based verification has attempted to address this, but these approaches are ineffective in zero-shot scenarios @xcite . ensemble verification methods enhance detection accuracy by utilizing multiple models. approaches such as multi-agent verification and crossmodel adjudication have proven effective in assessing llm outputs @xcite . our system expands on these approaches by integrating weak label generation with an ensemble verification pipeline, while also utilizing fuzzy matching to improve span alignment. unlike previous methods that rely on single-model hallucination detection, our approach leverages an ensemble of llms for adjudication, reducing model bias and improving hallucination span refinement via fuzzy matching. ## system overview our hallucination detection approach integrates task-specific prompt engineering, an llm ensemble verification mechanism, and post-processing refinements. the system is composed of three key components: fine-tuned prompt construction, hallucination span verification through llm ensembles, and post-processing with fuzzy matching. an overview of the full pipeline is illustrated in figure 1 . ## conclusion we presented our system for semeval-2025 task 3: mu-shroom, focusing on hallucinated span detection in llm-generated text across multiple languages. our approach combines promptengineered weak label generation with an llm ensemble verification mechanism, demonstrating strong performance in multilingual hallucination detection. our results confirm the effectiveness of ensemble-based adjudication, ranking among the top systems in several languages. however, challenges such as annotation variability and morphological complexity highlight areas for further refinement. future work could focus on integrating external knowledge for hallucination verification, finetuning llms to better align with human annotations, and refining span localization techniques to enhance character-level precision. these improvements could further advance hallucination detection in multilingual nlp systems.