S em E val-2025 Task 3: Mu- SHROOM , the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes (2025)

Abstract: We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs).Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task. We received 2,618 submissions from 43 participating teams employing diverse methodologies. The very high number of submissions highlights the interest of the community in hallucination detection. We present the results of the participating systems and provide an empirical analysis in order to better understand the factors that can lead to strong performance in this task. We also underscore current challenges, notably the varying degree of hallucinations across languages and the high annotator disagreement when labeling hallucination spans.

## lets a-go! introduction as generative ai systems become increasingly integrated into real-world applications we expect them to produce fluent and coherent text (e.g., @xcite . however, a critical issue undermines their reliability: these models frequently generate outputs that are highly fluent but factually incorrect, a phenomenon known as hallucination. hallucinations, as presently observed, are characterized by a disregard of the truth value of statements in favor of persuasive or plausiblesounding language, carrying consequences such as the spread of misinformation, and erosion of user trust @xcite . compounding this issue is the tendency of hallucinations to "snowball": when models are prompted to provide evidence or explanations for a false claim, they often generate coherent but false statements, further entrenching misinformation @xcite . addressing hallucinations is crucial for building systems that the public can trust. despite its significance, detecting hallucinations at scale remains a major challenge, with no clear universally effective solution currently available. the mu-shroom task 1 aims to contribute to advancing research in this direction. mu-shroom builds on the shroom shared task @xcite , expanding its scope and addressing key limitations. unlike shroom which focused solely on english, mu-shroom incorporates multilingual data across 14 languages to account for potential variations in hallucination rates @xcite . it also addresses general-purpose llms, reflecting the dominance of such models in current research, and introduces token-level annotations for more precise hallucination detection. by providing a richly annotated multilingual dataset and evaluation metrics, mu-shroom aims to advance research on hallucination patterns, improve detection methodologies, and foster community collaboration in nlg and factual consistency assessment. the mu-shroom dataset consists of a collection of prompts, model outputs, logits, and identifiers for openly available llms. the dataset encompasses 10 languages with validation and test data (modern standard arabic, german, english, spanish, finnish, french, hindi, italian, swedish and mandarin chinese), 4 test-only ("surprise") languages @xcite , as well as unlabeled training data for english, spanish, french, and chinese. supplementary metadata, including raw annotations before post-processing and the wikipedia urls used as references, as well as the scripts used to generate model outputs for all 14 languages and code for the annotation and submission interfaces are all publicly available. 2 the shared task attracted a total of 43 teams, resulting in over 2,600 submissions during the threeweek evaluation phase. the strong participation and diverse methodologies signal the task's success. notably, many teams relied on a few key models, often using synthetic data for fine-tuning or zeroshot prompting. while 64-71% of the teams outperform our baseline, top-scoring systems perform at random for the most challenging items. we present the results and provide a thorough analysis of the strengths and limitations of current hallucination detection systems. more immediately relevant to our shared task are pre-existing benchmarks and datasets. @xcite introduced halueval which is focused on dialogue systems but relies on closed, non-transparent models, limiting reproducibility. other benchmarks, such as those by @xcite and @xcite , use synthetic data for token-level hallucination detection. the shroom dataset @xcite provides 4k multiannotated datapoints for task-specific nlg systems. recently, @xcite introduced ragtruth, a large-scale corpus with 18,000 annotated responses for analyzing word-level hallucinations in rag frameworks. @xcite proposed factchd to specifically study hallucinations due to fact conflation. additionally, @xcite introduced a comprehensive dataset and a vulnerability index to quantify llms' susceptibility to hallucinations. most of these datasets focus on english (or chinese, @xcite . ## collecting the coins: data we begin with a description of the general process, and then note specific ad-hoc departures from this process for each language. the dataset covers 38 llms over 14 languages, out of which 4 (ca, cs, eu, fa) are test-only with about 100 datapoints. the other 10 languages (ar, de, en, es, fi, fr, hi, it, sv, zh) include both a validation split of 50 datapoints and a test split of 150 datapoints. 3 the construction of the dataset started with an automatic extraction of 400 wikipedia pages, with a focus on pages available in multiple languages of interest. we eventually increased this extraction to 762 links to guarantee a large number of wikipedia pages for all languages. from this point, the process we follow for creating the mu-shroom dataset is divided into two phases: datapoint creation and data annotation. data creation. the datapoint creation for each language was spearheaded by one of our organizers proficient in the language. appendix b.3 (esp. figure 7 ) describes the process in detail. in short, we manually selected and read 200 wikipedia pages (100 for test-only languages), and wrote for each page one question that could be answered with the information it contained. due to variations across wiki projects, the set of selected pages and the constructed questions vary across languages. 4 questions had to be factual (i.e., not a matter of opinion) and closed (i.e., answerable with a closed set of answers, such as numbers, places, names, etc). for each question, we then generated multiple llm answers: we identified existing open-weight instruction-tuned llms capable of handling the languages of interest (cf. ## it's a me, wario: metrics and baselines metrics. we compare the participants' submissions using two metrics: an intersection-over-union metric (iou) and a correlation metric (ρ). in order to apply the iou metric, we first binarize annotations by considering whether a majority of annotators (> 50%) marked a character as hallucinated, and then compare the set of indices marked by the system being rated to this binarized set of annotations. formally, for one datapoint: on the other hand, the ρ metric tries to factor in the lack of thorough consensus we observed in section 3. a drawback of the binarized annotation scheme is that it assumes a single ground truth, which may prove inaccurate or overly simplistic @xcite . to sidestep this issue, we consider whether the empirical probability of a character being marked by our annotators aligns with the probability derived from the participants' models. for a given datapoint of length k, we formally measure: baselines. to lower the barrier to entry to the shared task, we provided participants with an xlm-r-based baseline system neural fine-tuned on the entire test set for token-level classification. 7 this classifier directly maps tokens in an llm's answer to binary probabilities, without any intermediate fact verification step. in addition to this neural baseline, we consider two heuristics: mark-all where all characters are marked as hallucinated with probability 1, and mark-none where no hallucination is found, i.e., all characters get a probability of 0. the neural baseline is meant first and foremost as a tool for participants to build upon and demonstrate how to map characters to tokens. without any means of verification of the facts underpinning an llm output, we have low expectations that this baseline will perform well, especially in zero-shot settings. the two heuristics assign probabilities of 0 or 1 uniformly to all characters, which entails that every llm output is mapped to a constant series of probability. this corresponds to a correlation score of ρ = 0 in most cases. as for iou scores, given our data selection protocol (cf. section 3), we expect our dataset to be biased towards samples that contain hallucinations. therefore, the marknone baseline should yield lower iou scores than the mark-all baseline. 5 it's a me, mario: participants' systems 43 teams submitted their systems during the evaluation phase, and 35 teams wrote a paper describing their system. in total, we received 2,618 submissions across all languages. in average, 27.2 teams participated in each language. 41 teams submitted systems for english (en), followed by 32 for spanish (es) and 30 for french (fr). the languages with the least number of participants were our surprise languages: catalan (ca) with 21 teams; czech (cs), basque (eu) and farsi (fa), with 23 teams each. overall, we remark a wide variety of approaches, ranging from qa-or nerbased finetuning, to time series-based analyses of logits (aryal and akomoize, 2025) and to zero-shot rag-based approaches. we present an overview of the participating systems in the ucsc system @xcite ) is designed as a three-stage pipeline: (i) context retrieval, wherein they retrieve relevant pieces of information to assess the factuality of the llm outputs; (ii) hallucinated fact detection, wherein they identify the incorrect facts based on the retrieved contexts; and (iii) span mapping, wherein the incorrect facts are mapped onto specific segments of the output. the approach furthermore employs prompt optimization to maximize performances. multistage frameworks were also deployed by other teams, for instance, iai_msu @xcite developed a three-step approach, with a retrievalbased first step, a self-refine second step, and an ensembling third step. another noteworthy entry is that of ccnu (liu and chen, 2025) -whose report also incorporates information about unsuccessful attempts and some team & paper languages overview advacheck (voznyuk et al., 2025) en ner-based keyword extraction, wikipedia-based rag, llm edition-based prompting. ailsntua (karkani et al., 2025) all translate-test (to en and zh) prompt-based approaches using synthetic few-shot examples. atlantis (kobus et al., 2025) de, en, es, fr rag + llm prompting; rag-based approaches; token-level classifiers. bluetoad (pronk et al., 2025) ar, cs, de, en, es, eu, fa, fi, fr, hi, it, sv, zh qa-finetuned base plms; fine-tuning on synthetic data ccnu (liu and chen, 2025) all prompting & rag cogumelo (creo et al., 2025) en, es ner-finetuning; perplexity-based assessments cuet_sstm ar ner-finetuning. deloitte (chandler et al., 2025) all binary token-level classifiers, trained using web-search results, task instruction and datapoint as inputs. deeppavlov all white-box approaches dutjbd (yin et al., 2025) en -fenji (alberts et al., 2025) all dense passage retrieval for flan-t5 prompting. firc-nlp (tufa et al., 2025) all prompt-based approaches, incorporating external references. funghifunghi (ballout et al., 2025) en, es, fr, it, sv translate-train (to en) and synthetic datasets. gil-iimas unam (lopez-ponce et al., 2025) en, es wikipedia-based rag. hallurag-rug (abdi et al., 2025) en wikipedia-based rag, followed by a summarization step and a zero-shot prompting to annotate the items. hallusearch (abdallah and el-beltagy, 2025) all factual statement decomposition and verification through real-world context retrieval. halluciseekers ar, de, en, es, fa, fr, it, sv -hallucination detectives (elchafei and abu-elkheir, 2025) ar, en semantic role labeling, dependency parsing, and token-logit confidence scores to construct spans hausanlp (bala et al., 2025) en finetuning approaches. howard university -ai4pc (aryal and akomoize, 2025) all time-series anomaly detection across the sequence of logits. iai_msu (pukemo et al., 2025) en rag keepitsimple (vemula and krishnamurthy, 2025) all multiple llm generated responses are compared with model output text by modeling information entropy for detecting uncertainty. lcteam (maldonado rodríguez et al., 2025) ## rainbow road completed: results we include an overview of the highest scoring systems from each team per language in figure 3 . in the interest of space, we defer tables of ranking to appendix c. most teams outperformed the baselines. the mark-none and neural baselines rank extremely low, both in terms of iou and ρ. the mark-all baseline performs better in terms iou, but remains far below the top teams, highlighting the need for more sophisticated strategies. the most consistent top performers coincidentally made submissions to all 14 languages. 8 ucsc @xcite ) appears in the top 3 teams for 11 languages, securing 5 wins (ca, de, fi, it, sv) and 5 second-place finishes (cs, en, eu, fa, hi). their systems demonstrate a stable iou-to-ρ ratio mean(iou/ρ)= 1.01. msa @xcite ranks in the top 3 for 8 languages, winning in 2 (ar, eu) and securing second place in 3 others (de, fi, sv) and mean(iou/ρ)= 1.03. ails-ntua @xcite performs well across multiple languages, winning in 2 (cs, fa), but showing a less balanced performance between the two metrics: mean(iou/ρ)= 0.93. ccnu @xcite ) ranks first in hi and appears in the top-5 in 9 languages with mean(iou/ρ)= 0.93. deloitte @xcite ranks first in fr and places the top-5 in 3 languages. smurfcat @xcite consistently ranks in the top-5 across 7 languages and never falls out of the top-10. at-lantis @xcite participated in 4 languages and won the 1st place in es. however, their ρ scores are near zero in three of their languages, including es and en, where they placed 1st and 3rd, respectively. due to this imbalance, we report the inverse ratio: mean(ρ/iou)= 0.2. iai_msu @xcite competed only in en, where 8 note that we generally do not find evidence that rank differences are statistically significant, cf. appendix c. and cs rank lowest, with es standing out due to its top-performing system achieving an almost zero ρ. this suggests that certain languages pose greater challenges for models, potentially due to dataset properties, linguistic complexity, or limitations in training data. however, as shown in some teams show high ρ but low iou, suggesting they are good at ranking hallucinations but struggle with binary classification. an example when the gap between iou and ρ is small -for teams like ucsc and ailsntua -shows the reliability of both metrics not just in raw intersection but in their robustness in ranking, implying that high iou does not always correlate with high ρ. a big gap in these two metrics when ρ ≪ iou, as we observe for atlantis, indicates that the models are good at making binary decisions but poor at ranking how hallucinated a character is compared to others. conversely, we observed for teams with iou ≪ ρ that their models can rank characters well in terms of hallucination but fail in making the correct binary selections. for instance, hausanlp shows an extremely low iou despite a decent ρ, meaning its predictions are correlated but far from accurate. other trends we observe from the general ranking are: trustai and swushroomsia @xcite ## 1-up! discussion in order to deepen our understanding of the factors relevant to the success of participating teams within our shared task, we now turn to an analysis of metadata collected during the shared task. participants were asked to fill in a form to describe how their systems worked and what type of resources they used. 9 the trends we discuss below are therefore based on self-reporting. we z-normalize performance per language before analysis so as to factor out the varying intrinsic difficulty of the different language datasets. a first obvious trend in our results is that 36.98% of the submissions are reported as prompt-based. the iou scores of prompt-based submissions are not statistically distinct from the iou scores of other submissions, but we do find a statistical difference for ρ scores, which are usually lower than in other submissions (mann-whitney u test: p-@xmath1.002, common language effect size: @xmath2.96%). this echoes findings in the previous iteration of the shared task @xcite , which pointed out that fine-tuning based approaches were usually more successful on hallucination detection. even more prominent is the use of rag: 52.60% of the submissions report using rag, and are assigned statistically higher iou scores (mann-whitney u, p-@xmath3, @xmath4.80%) and ρ scores (p-@xmath5, @xmath6.16%). on a related note, if we focus on the data used by participants, we find that 34.88% of submissions which primarily used the data we provided tend to have lower iou (mann-whitney u, p-@xmath7, @xmath8.60%) and ρ scores (p-@xmath9, @xmath10.28%). the preponderance of retrievalbased approaches and their noteworthy success another factor of interest is whether specific plms stand out as more or less appropriate for the task of detecting hallucinated spans. in the last factor we explore is related to our earlier observations regarding inter-annotator agreement (reported in section 3, the results are summarized in ## the princess is in another article: conclusions the mu-shroom multilingual shared-task was an overall success. we received 2,618 submissions from 43 teams, including a handful of participants from the first iteration of the shroom shared task. whilst the level of participation varied by language, over 20 teams competed in each of the 14 languages. participating teams deployed a vast array of methodologies, ranging from qa-or ner-based pretraining to synthetic data generation and rag approaches, which will serve as starting points for future research. we also observed a high number of student-lead teams. one of the goals of the shared task is to lower the barrier to entry to current challenges in nlp, hence we take the interest of students as a further indicator of success. beyond these participation numbers, the data collected for mu-shroom also allowed us to highlight a number of often-overlooked points in the literature. the prevalence and severity of halluci-nated outputs varies across languages (see if mu-shroom has allowed us to establish the importance of multilingual data for hallucination detection, much remains to be done in order to fully assess llm technologies' tendency to produce non-factual information. one other aspect we have left outside the scope of this shared task is that of mitigating hallucinations, a step that is however necessary and complementary to our endeavors. we have constructed the present shared task as a means to draw the attention of the community towards some challenges tied to hallucination detection -and attention is indeed needed, given that even top-scoring teams do not detect 20% or more of the hallucination spans.