F unghi F unghi at S em E val-2025 Task 3: Mu- SHROOM , the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes (2025)

Abstract: Large Language Models (LLMs) often generate hallucinated content, which is factually incorrect or misleading, posing reliability challenges. The Mu-SHROOM shared task addresses hallucination detection in multilingualLLM-generated text. This study employsSpanBERT, a transformer model optimized forspan-based predictions, to identify hallucinatedspans across multiple languages. To addresslimited training data, we apply dataset augmentation through translation and synthetic generation. The model is evaluated using Intersection over Union (IoU) for span detectionand Spearman’s correlation for ranking consistency. While the model detects hallucinatedspans with moderate accuracy, it struggles withranking confidence scores. These findings highlight the need for improved probability calibration and multilingual robustness. Future workshould refine ranking methods and explore ensemble models for better performance.

## introduction large language models (llms) are widely used in natural language processing (nlp) applications, including text generation, summarization, and conversational ai @xcite . however, they often generate hallucinated content, information that appears plausible but is incorrect or misleading. these hallucinations pose challenges in ensuring the reliability and factual consistency of generated text @xcite . detecting hallucinations becomes more difficult in multilingual settings. variations in grammar, vocabulary, and training data across languages affect a model's ability to identify and rank hallucinated spans consistently @xcite . lowresource languages tend to exhibit higher hallucination rates due to limited training data, whereas high-resource languages, such as english, may benefit from more extensive supervision. addressing these challenges requires models that generalize across languages while maintaining effective hallucination detection capabilities. the mu-shroom shared task foot_0 aims to advance research in multilingual hallucination detection. participants are required to identify hallucinated spans in llm-generated text across multiple languages by multiple models. the evaluation relies on two key metrics: intersection over union (iou) to measure span detection accuracy and spearman's correlation to assess the consistency of hallucination confidence scores. the task presents challenges, including variations in hallucination patterns across languages and the need for probability calibration to improve ranking reliability. in this study, we present a spanbert-based approach for hallucination detection. spanbert @xcite , a transformer model optimized for span-based predictions, is well-suited for identifying hallucinated text segments. due to the limited availability of training data, we incorporate dataset augmentation through translation and synthetic data generation. our results indicate that while our model detects hallucinated spans with reasonable accuracy, it struggles with ranking consistency, as reflected in low or negative spearman's correlation scores. these findings highlight the need for improved probability calibration techniques and enhanced model robustness across languages. the remainder of this paper is structured as follows: section 2 reviews related work on hallucination detection and multilingual evaluation. section 3 details the mu-shroom task and dataset, including our data augmentation approach. section 4 describes our methodology, covering model selection, preprocessing, and training setup. section 5 presents the experimental setup. section 5.3 provides a combined discussion and interpretation of the results. finally, section 6 summarizes key findings and outlines directions for future research. ## related work hallucination detection in llms is a well-known problem, where models generate factually incorrect or misleading information. detecting these errors is important for improving the reliability of generated text @xcite . in this section, we discuss prior work on evaluation metrics, hallucination detection, and multilingual challenges. existing approaches range from sentence-level classification to span-level annotation, with multilingual settings introducing additional complexities. hallucination detection in large language models @xcite compare different hallucination detection metrics in multilingual settings. their study finds that natural language inference (nli)-based methods often perform better than lexical overlap measures like rouge @xcite . however, these methods struggle with detecting fine-grained hallucinations at the span level, which is a key focus of mu-shroom. @xcite introduce a dataset for detecting hallucinations in news headlines across multiple languages. their work includes fine-grained annotations, showing that different hallucination types require different detection strategies. this aligns with mu-shroom's goal of identifying hallucination spans, though our task focuses on llm-generated text rather than news headlines. ## task description and datasets this section outlines the mu-shroom task and dataset. we describe the task of detecting hallu-cinated spans in multilingual llm outputs and explain dataset augmentation through translation and synthetic data generation. ## experiments all experiments were conducted using google colab, utilizing the free gpu (nvidia tesla t4 or p100, depending on session availability). the model training process was optimized for this environment to account for hardware limitations, including restricted memory and compute time. this setup facilitated efficient model testing and training without the need for additional infrastructure. ## conclusion this study explores a spanbert-based approach for detecting hallucinated spans in multilingual llm-generated text as part of the mu-shroom shared task. the results indicate that while the model performs reasonably well in identifying hallucinated spans, particularly in high-resource languages, it struggles with ranking hallucination confidence scores accurately. this is reflected in low or negative spearman's correlation values. these findings suggest that while spanbert is effective for span detection, further improvements are needed for confidence ranking. future work should focus on refining probability calibration techniques, improving robustness across multiple languages, and exploring alternative training objectives that incorporate ranking-aware learning. additionally, ensemble approaches or fine-tuning architectures specifically designed for multilingual hallucination detection could further enhance performance.