LCT eam at S em E val-2025 Task 3: Multilingual Detection of Hallucinations and Overgeneration Mistakes Using XLM - R o BERT a (2025)

Abstract: In recent years, the tendency of large language models to produce hallucinations has become an object of academic interest. Hallucinated or overgenerated outputs created by LLMs contain factual inaccuracies which can potentially invalidate textual coherence. The Mu-SHROOM shared task sets the goal of developing strategies for detecting hallucinated parts of LLM outputs in a multilingual context. We present an approach applicable across multiple languages, which incorporates the alignment of tokens and hard labels, as well as training a multi-lingual XLM-RoBERTa model. With this approach we managed to achieve 2nd in Chinese and top-10 positions in 7 other language tracks of the competition.

## introduction in recent years, due to the development of transformer-based architectures @xcite , natural language generation models saw immense advancements. however, the field is currently struggling with the tendency of neural systems to produce fluent, yet factually inaccurate outputs, aggravated by lack of adequate accuracy metrics. all of the above causes the models to "hallucinate". the overgeneration of inaccurate facts puts in jeopardy the practical applications based on nlg @xcite , which in turn prompts more interest in tackling the problem of detecting hallucinations in the outputs of nlg models. the problems mentioned above were the motivation for the mu-shroom shared task, aimed at identifying hallucinations and related overgeneration mistakes @xcite . the task builds upon ints previous iteration, but with focus on more languages and llm outputs. in this paper we present our approach to detecting hallucinated output using the multilingual xlm-roberta model @xcite . we use the model inputs and outputs from the provided dataset, which are concatenated and aligned with hard labels and then passed through the model. apart from that, we also provide the description of the shared task and the datasets, as well as the discussion of results. ## related work in this section, we offer a short overview of methods used in previous work on detecting hallucinations in llms' outputs. we will examine modelagnostic and model-aware approaches, as well as black-box detection methods and prompting-based techniques. model-agnostic methods, such as prompt engineering and few-shot learning, focus on improving hallucination detection without relying on model internals. these techniques leverage strategies like meta-regression frameworks and automatic label generation, which allow them to be applied across different llms and tasks, offering a flexible solution to hallucination detection @xcite @xcite @xcite . on the other hand, model-aware approaches take advantage of internal signals from llms, such as layer activations and attention values, to detect hallucinations more precisely. by directly analyzing the model's internal workings, these methods can provide deeper insights into how llms generate outputs. techniques like retrieval-augmented generation (rag) and chain-of-verification strategies have been explored to validate generated content against external knowledge sources, ensuring higher accuracy in detecting hallucinations @xcite . however, these methods are generally less effective than modelagnostic approaches including @xcite and @xcite at shroom shared task. they are limited by their dependence on open ac-cess to the model's internal architecture, which may not be possible for closed-source models like chatgpt @xcite . prompting-based metrics are also used in hallucination detection, leveraging the instruction following capabilities of llms. these methods involve providing llms with evaluation guidelines and both the generated and source content @xcite . various strategies have been explored, including direct and chain-of-thought prompting, and in-context learning @xcite . ## methodology our system utilizes the validation dataset provided by the task organizers, which comprises ten languages. to ensure balanced representation across languages, we first split each language-specific dataset into training and validation subsets using a 90/10 ratio and then merged them to form combined training and validation sets. to address data scarcity, we implemented a cross-lingual label transfer mechanism based on neural machine translation and semantic span alignment. we employed helsinki-nlp's opus-mt bilingual models (tiedemann and thottingal, 2020) to translate labeled text from a source language into target languages. for accurate projection of labeled spans (hard labels) onto the translated text, we adopted a semantic similarity-based alignment approach. we used the paraphrase-multilingual-minilm-l12-v2 model from sentence transformers @xcite to generate embeddings for each labeled span in the source text, as well as for candidate spans in the translated output. to accommodate natural variations in translation length, we introduced a variational span matching strategy: for a source span of length l, we evaluated candidate spans in the translated text with lengths in the range [lδ, , l + δ], where δ is a tunable parameter (set to 5 in our experiments). cosine similarity between embeddings was used to identify the most semantically aligned span, which was then adjusted to remove any leading or trailing whitespace, ensuring precise label transfer. the resulting multilingual dataset includes model input text, model output text, hard labels, and relevant metadata. we focus on three key components: the model input text, which provides contextual grounding; the model output text, which is analyzed for hallucinations and overgenerations; and the hard labels, which annotate hallucinated or overgenerated spans for use in evaluation and supervision. the overall architecture of our system is depicted in figure 1 . the flowchart outlines the entire process, starting from concatenating model input and output text, followed by tokenization using xlm-roberta @xcite . the next steps involve hard label token alignment, model training, and inference. post-inference, the logits are processed to calculate soft labels, followed by thresholding to obtain hard labels. finally, we evaluate performance using the intersection over union (iou) and spearman correlation metrics as provided by the task organizers. ## conclusion in this paper, we present a system for the detection of hallucinated spans in text. our approach successfully applies the small amount of labeled data provided by the task organizers to fine-tune a multilingual xlm-roberta model to this end. by aligning the tokens in a concatenated sequence including both the prompt and its resulting model output to the provided hard labels representing the start and end of a hallucinated span, we are able to leverage the context that the prompt provides at the same time that we make use of the available labeled data. this system however has some limitations, such as its reliance on larger models for improved performance and its tendency to predict a larger number of spans which are shorter in length than desirable.