TUM - M i K a N i at S em E val-2025 Task 3: Towards Multilingual and Knowledge-Aware Non-factual Hallucination Identification (2025)

Abstract: Hallucinations are one of the major problems of LLMs, hindering their trustworthiness and deployment to wider use cases. However, most of the research on hallucinations focuses on English data, neglecting the multilingual nature of LLMs. This paper describes our submission to the "{textit{SemEval-2025 Task-3 — Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes}}”. We propose a two-part pipeline that combines retrieval-based fact verification against Wikipedia with a BERT-based system fine-tuned to identify common hallucination patterns. Our system achieves competitive results across all languages, reaching top-10 results in eight languages, including English. Moreover, it supports multiple languages beyond the fourteen covered by the shared task. This multilingual hallucination identifier can help to improve LLM outputs and their usefulness in the future.

## introduction hallucinations are unwanted parts in the llm outputs that are either not aligned with the source document (intrinsic) or non-factual in terms of world knowledge (extrinsic) @xcite . these over-generations are a severe problem in nlp research, and their detection and mitigation are studied widely @xcite . however, most hallucination research focuses on english data. therefore, the "semeval-2025 task-3 -mu-shroom, the multilingual shared-task on hallucinations and related observable overgeneration mistakes" provides hallucination annotations in fourteen languages and asks the participants to create multilingual hallucination detectors @xcite . their data comes from open-source instruction-tuned llms, generated in a qa setting, and humans annotated 0.9 finds extrinsic hallucinations captures structural and semantic characteristics of hallucinated content support vector regression probability of being a hallucination figure 1 : given a question-answer pair, each token in the answer is evaluated for extrinsic hallucinations using a retrieval-based fact verification model, which compares the token against external knowledge. simultaneously, the bert-based model captures structural and semantic characteristics of hallucinated content. the results from both models are then integrated using support vector regression to estimate the final probability of each token being a hallucination. each token to determine whether it belongs to a hallucinated phrase. each token receives two different labels: a binary hard label, obtained as a majority vote between the annotators, and a soft @xmath0, 1] given by the proportion of annotators who labeled the token as a hallucination. in this paper, we present our contribution to the shared task: a multilingual and knowledge-aware non-factual hallucination identifier (mikani). our system is model-agnostic and supports multiple languages beyond the fourteen covered in the shared task. it annotates hallucinations on a token level, providing fine-grained information about the correctness of atomic facts. to obtain these annotations, we propose a two-part pipeline (visible in figure 1 ): the first part is an atomic factchecker based on the retrieval of information from wikipedia. the second part is a fine-tuned bert model incorporating linguistic features. the predictions from the two parts are then combined using a support-vector regression model. our combined system achieves competitive scores across all languages (e.g., 8/44 in english, 2/33 in french). our code and model weights are publicly available for further usage and development foot_0 . ## background and related work many different approaches exist to detect hallucinations in generated texts @xcite . some of them take information about the generating model into account, e.g., by analyzing attention weights or the model's logits @xcite . while the logits of the last model layer were provided in this shared task, we created a model-agnostic detection mechanism that does not require any information about the underlying llm. intrinsic hallucinations occur when the model outputs deviate from the source document. this is often the case in summarization or rag-based tasks that provide long contexts for the model generations @xcite . in contrast, in the mu-shroom data, the model only receives a question that it has to answer based on its internal knowledge. therefore, the main interest point of our hallucination identifier is to find extrinsic hallucinations. this entails a fact-checking or verification objective. the most popular way to check the facts in generated texts is to compare them against world knowledge covered in wikipedia or knowledge graphs @xcite . thus, the first part of our system builds upon this, comparing the atomic facts against retrieved data from the english wikipedia. previous works solved this comparison by predicting the entailment @xcite or the edit operations that are necessary to transfer the retrieved information into the model generations @xcite . ## system overview our approach integrates the strengths of two complementary submodels, whose outputs are combined in a third model to generate the final hallucination prediction. the first submodel is designed to detect extrinsic hallucinations by retrieving relevant wikipedia facts and using them to assess token-level hallucination probabilities. the second submodel is bert-based and trained on tokenlevel hallucination probability-annotated data. it focuses on identifying common hallucination patterns. the outputs of both submodels, along with additional extracted features, are fed into a support vector regression (svr) model, producing the final combined hallucination score. details about this architecture are presented in figure 1 . ## results the hallucinations were evaluated using intersection-over-union (iou) for hard labels and spearman correlation (cor) for soft labels. iou measures the overlap between hallucination spans, while cor assesses the correlation between predicted and reference probabilities @xcite . the submission results, including iou and cor scores along with the corresponding ranks, are presented in ## conclusion in this paper, we present mikani, a multilingual and knowledge-aware hallucination identification system that achieved competitive performance in the mu-shroom shared task. our system combines fact verification against external sources with a bert-based system fine-tuned to detect common hallucination patterns. the system uses the same architecture for all languages and llm outputs, making it strongly multilingual and modelagnostic, generalizing well on the unseen test set languages. the language capacities for further languages are only limited by the availability of spacy or stanza annotations and their support in multilingual bert. in future work, we will try to make the model even more flexible by testing open-source fact verification models instead of relying on ope-nai's gpt-4o.