GIL - IIMAS UNAM at S em E val-2025 Task 3: M e SSI : A Multilmodule System to detect hallucinated Segments in trivia-like Inquiries. (2025)

Abstract: We present MeSSI, a multi-module system applied to SemEval 2025’s task 3: Mu-SHROOM. Our system tags questions in order to obtain semantic relevant terms that are used as information retrieval characteristics. Said characteristics serve as extraction terms for Wikipedia pages that are in turn processed to generate gold standard texts used in a hallucination evaluation system. A PoST-based entity comparison was implemented to contrast the test dataset sentences with the corresponding generated gold standards, wich in turn was the main criteria to tag hallucinations, partitioned in soft labels and hard labels. This method was tested in Spanish and English, finishing 18th and 19th respectively on the IoU based ranking.

## introduction given the increasing use of llms, the creation of hallucination detection and fact checking systems is of great importance. the multilingual shared-task on hallucinations and related observable overgeneration mistakes (mu-shroom @xcite ) focuses on analyzing and tagging llms' responses in order to determine which spans of text correspond to hallucinations and incorrect information, based on a system generated probability. mu-shroom's evaluation is carried out in 14 different languages, and performance is measured in two character-level metrics: intersection over union (iou) of tagged spans (predictions vs gold-labels), and a comparison between assigned probabilities of each span compared to the gold label probability (ρ), the main evaluation being iou. this article describes a three module pipeline for solving this task: the multilmodule system to detect hallucinated segments in trivia-like inquiries (messi). our pipeline uses wikipedia as an information retrieval database. based on the retrieved texts, a gold standard answer is generated using automatic summarizing systems or llms, finally this processed gold standard is compared with each llm output from the mu-shroom dataset. given the nature of the questions, a post (part-of-speech-tag) based comparison was used in order to generate the final tags of each span. the messi system was tested in english and spanish, ranking in the top half of all models in english, and just below the half in spanish (19th out of 44 participants in english, and 18th out of 35 in spanish). working in this task gave us insight regarding the difficulties of creating automated fact checking systems, particularly given the wide variety of questions that these systems have to verify and the vast amount of information needed to cover all possible subjects. similarly, the span tagging portions of the task also proved to be a challenge since the tagging has to be based on factual differences that don't always translate to syntactic structure or post based differences. our system proved to be language-resource dependent, almost doubling performance in english compared to spanish, yet managed to surpass each language's baseline. the code can be found in github foot_0 . ## background llm hallucination and fact-checking has become a widely studied area of research with various approaches. @xcite introduced hover , a dataset for evidence extraction and fact verification. it requires models to extract relevant facts from multiple wikipedia articles and determine whether the claim is supported or not supported based on this information. document retrieval was made by a query-based approach, using cosine similarity between binned uni-gram and bi-gram tf-idf vectors. for the claim verification model, authors fine-tuned a bert model to recognize entailment between the claim and the retrieved evidence. a different methodology that focuses on pairwise comparison over entailment is carried out in vitamin c @xcite . the authors analyze attention values for tokens in both sentences and signal out contrasting pairs of tokens, this particular work serves as a starting point for our comparison module. @xcite proposed fewl, a framework that measures the hallucination score of different llms designed for scenarios where the benchmark datasets lack gold-standard answers. given a set of questions and the correspondent llms-generated answers, the framework computes an intermediate truthfulness score weighted by an individual expertise score for each model. using a set of similar questions, a laziness penalty is applied to the expertise score based on the level of superficialness exhibited by the llm responses. fewl has demonstrated effectiveness in mitigating hallucinations by guiding in-context learning and supervised fine-tuning, even in the absence of gold-standard references. ## system overview since this task analyzes llm outputs that aim to answer trivia-like questions, our approach consisted of obtaining a correct answer to the each question in order to analyze differences between the test set answers and our gold standard. the precise evaluation consists of generating two sets of tags called hard labels and soft labels. hard labels are character-level intervals that our system considers hallucination, whereas soft labels are probabilities assigned to intervals that correspond to annotator agreement of each interval. our system is composed of three modules: question-based information retrieval, text filtering and gold standard generation, comparison between our gold standard and the task's test dataset. the input data corresponds to questions from the test dataset, the output is a fully formatted dataset compatible with the task's evaluator. ## experimental setup the experiments were carried out in python using various packages for each module of the system. for further reference please review our system's repository in github. ## results and analysis full results can be found in mu-shroom's official site 2 . the main scores ranked participants by iou rather than cor, however an alternative ranking is also available in the official site. ## conclusion in this paper we described messi, our pairwise sentence comparison system based on lexical differences, as well as messi's performance and limitations in the semeval task mu-shroom. in ideal cases our system correctly extracts relevant wikipedia articles, generates an adequate gold standard and identifies differences in words between gold standards and test set questions. however, ideal cases aren't always the norm. wikipedia resources heavily depend on the language, almost 7 million articles are available in english compared with the 2 million in spanish @xcite , and wikipedia unfortunately doesn't contain the whole of humanity's knowledge. on top of this, rag pipelines are still prone to inadequate text generation, leading to incorrect gold standards used for pairwise comparison. regardless our system proved to be competitive in both languages and managed to tag soft and hard labels in a way to keep both values correlated, something that even the best performing spanish model failed to do. for future work and possible deployment in a context outside tasks, this system could be improved throughout the pipeline. initially by enlarging the retrieval database using specialized corpora, textbooks, or general purpose datasets. this way our system is less dependent on one single resource and can cover a wider variety of subjects. the gold standard generation module could be improved by a finer segmentation and embedding of the retrieved documents, this would reduce size of extracted documents and benefit the llm based generation as well as the pairwise comparison module. finer text normalization would also improve the system, considering that various wikipedia based texts contain separators like section and subsection names, or characters like "==". finally, alternatives for the comparison module could include a semantic analysis of answers in order to understand which elements coincide to factual information and over just tagged elements, while keeping interpretability of the model. a transformer based approach, similar to the one presented in vitamin c, could improve results and focus on token attention over post as well as including contextual representation of texts previous to comparison.