Swushroomsia at S em E val-2025 Task 3: Probing LLM s’ Collective Intelligence for Multilingual Hallucination Detection (2025)

Abstract: This paper introduces a system designed for SemEval-2025 Task 3: Mu-SHROOM, which focuses on detecting hallucinations in multilingual outputs generated by large language models (LLMs). Our approach leverages the collective intelligence of multiple LLMs by prompting several models with three distinct prompts to annotate hallucinations. These individual annotations are then merged to create a comprehensive probabilistic annotation. The proposed system demonstrates strong performance, achieving high accuracy in span detection and strong correlation between predicted probabilities and ground truth annotations.

## introduction hallucinations in large language models (llms) are a widely-known problem critical for their trustworthiness @xcite . the detection of hallucinations presents a challenge due to the absence of a standardized definition @xcite . moreover, different llms may identify different parts of the same text as hallucinations and in general, different llms have different hallucination rates 1 @xcite . furthermore, despite some on-going research, hallucinations are still not very well explored in multilingual setups @xcite . in order to contribute to the research on multilingual and multimodel hallucinations, mu-shroom ("multilingual shared-task on hallucinations and related observable overgeneration mistakes") @xcite , a semeval-2025 task-3, was proposed. the task focuses on detection of hallucination spans in the outputs of instruction-tuned publicly available llms, covering 14 different languages: modern standard arabic (ar), basque (ba), catalan (ca), chinese mandarin (zh), czech (cs), english (en), farsi (fa), finnish (fi), french (fr), german (de), hindi (hi), italian (it), spanish (es), and swedish (sv). the dataset is divided into validation (labelled), train and test (both unlabelled) sets (for details see appendix, in the remaining of the paper, due to space limitations, we mainly focus on our best performing approach s2 2 , which in the official iou ranking scored 4 th and 5 th for fr and it, respectively, but we also provide some details on s1 in the appendix. the ensemble strategy performs particularly well for corr metric (it ranked 1 st , 3 rd and 4 th for en, de and fr, respectively), but we have identified, as well, some particularly performative models and prompts for iou. we provide a detailed analysis of considered closed-weight large language models' performance. ## related work this mu-shroom task builds upon semeval 2024 monolingual shroom task 6 (mickus et al., 2 our code and data is available at: https://github.com/ idsia-nlp/mushroom/ 2024), which comprised three nlg tasks divided in two streams (model-agnostic vs. model-aware) but was scoped less ambitiously: participants were supposed to only perform binary classification to identify hallucinations, without indicating hallucination spans. nevertheless, some ideas from 2024 edition's winning approaches were inspirational for us. in particular, we noticed that 4 out of 6 best approaches were reporting excellent results using closed-weight models @xcite @xcite as well as that high performance is not readily achieved with off-the-shelf llms and systems @xcite . in particular, the best performing model @xcite resorted to a meta-regressor framework aggregating uncertainty signals from multiple llms, which was the motivation for models' output merging in our s2. moreover, we draw inspiration from direct prompting strategies which have already been explored to evaluate factual consistency @xcite , assess the self-alignment capabilities of llms with respect to factuality @xcite , and detect confabulations, a specific subclass of hallucinations, by eliciting multiple candidate responses @xcite . ## challenges related to ground truth annotations we observed that the ground truth annotations for en lack consistency: the characters included in the span of the same hallucination type differ from sentence to sentence. for example, in some cases where the names of the places used in the question were misspelled in the model outputs, ground truth annotates complete name as a hallucination (see ex. 2 in figure 1 ) while in others, only the added character ('s' in ex. 3 in figure 1 ) was annotated as such. some other languages (e.g. it) did not have this type of issues (or had very few which, however, have not influenced annotations). ## system overview in order to facilitate the readability of our article, we use abbreviations to designate the llms employed (see appendix b.1 for the list of exploited llms and their respective abbreviations). our systems have been evaluated only on the languages included in the validation set. ## quantitative findings we provide simple statistics in the bl column presents the results achieved by the best baseline provided by the organizers for each language. for iou, the baseline is always mark-all. additionally, we noticed that for all languages mismatches related to no-spans were far more frequent for the direction when spans were present in ground truth and missing in s2 (s2-nosp) than vice versa (gt-nosp). however, percentages of s2-nosp vary greatly across languages, being the best for fi, then sv, and the worst for en and zh. we performed yet another analysis, where together with the ratio of overlapping spans (ol_spans) we also looked at the ratio of overlapping characters (ol_chars) for each test instance, comparing the s2 annotations with those of ground truth. comparing inter-quantile ranges (and medians) of ol_spans and ol_chars distributions (see figure 2 ), we can see not only differences in scores between various languages but also that, as expected, reaching span overlap is much harder to achieve than character overlap (the latter also aligns better with iou). iou the official rankings of the mu-shroom task were provided only with respect to the iou score. as showcased in the results for less performing s1 can be seen in appendix ( corr our merged configuration demonstrated particular effectiveness in measuring correlation, yielding the best results across all languages, with performance on corr surpassing even that of iou. specifically, the m \6 configuration achieved the highest performance for eight languages, compared to m \3 and m, which were the bests for only one language each. furthermore, these configurations proved highly effective relative to other teams, allowing us to achieve the best results for three languages (sv, en, and es) and rank within the top five for eight out of the ten languages evaluated (see also mean corr plots in appendix d). however, chinese remains challenging due to annotation difficulties, resulting in a correlation value below 0.5 and performance significantly lower than that of the top-performing teams (ranking 11 th out of 20). ## qualitative analysis we performed qualitative analysis for it and fr comparing our best s2 models for these languages with ground truth. some of the observed patterns in annotation discrepancies between s2 and ground truth are reported in ( s2 involves prepositions and connectors 2 "jelgava, in lettonia" "jelgava", "lettonia" (2) 3 "ottobre del 1991" "ottobre", "1991" (2) 4 "jamie jones e sean tyas" "jamie jones", "sean tyas" (2) 5 "due medaglie d'oro olimpiche nel 1952" "due", "d'oro", "1952" (3) s2 encompasses larger context 6 "fu campione panamericano di nuoto" "panamericano" and "nuoto" (2) 7 "neuchâtel, nella regione della svizzera occidentale" "neuchâtel", "svizzera occidentale"(2) 8 "5 febbraio 1942" "5" and "1942" s2 is less precise than ground truth ( "febbraio" and "mary" are correct) 9 "dr. mary blakiston" "blakiston" when comparing different model hallucination annotation, we perceived that models performing worse for it were tending to annotate more hallucination spans than best performing models (see figure 4 ). the contrary of this behavior was noticed for fr. we also noted that observations with shorter length tend to have higher iou scores. this tendency is particularly pronounced for it while it is less evident for other languages (see appendix, figure 8 ). additionally, figure 7 in appendix shows two examples (in it and fr) of more drastic annotation problems. ## conclusion this paper outlines our system for mu-shroom, a shared task with key challenges, such as multilingualism, hallucination span detection without annotated training data, inconsistencies in human annotations. despite all these, our collective intelligence approach exploiting annotation potential of diverse close-weight llms and accompanying prompts, demonstrated strong effectiveness, particularly in achieving high correlation with the ground truth annotations. our future work will focus on investigating noted imbalances varying across different languages and inputs, as well as more refined comparison exploiting the open-weight models.