keepitsimple at S em E val-2025 Task 3: LLM -Uncertainty based Approach for Multilingual Hallucination Span Detection (2025)

Abstract: Identification of hallucination spans in black-box language model generated text is essential for applications in the real world. A recent attempt at this direction is SemEval-2025 Task 3, Mu-SHROOM—a Multilingual Shared Task onHallucinations and Related Observable Over-generation Errors. In this work, we present our solution to this problem, which capitalizes on the variability of stochastically-sampled responses in order to identify hallucinated spans. Our hypothesis is that if a language model is certain of a fact, its sampled responses will be uniform, while hallucinated facts will yield different and conflicting results. We measure this divergence through entropy-based analysis, allowing for accurate identification of hallucinated segments. Our method is not dependent on additional training and hence is cost-effective and adaptable. In addition, we conduct extensive hyperparameter tuning and perform error analysis, giving us crucial insights into model behavior and where improvement can be made.

## introduction hallucination is a situation where large language models (llms) produce outputs that are inconsistent with real-world facts or unverifiable, posing challenges to the trustworthiness of ai systems @xcite . hallucination detection is the process of identifying such sections of text where a model generates content that is untrue, misleading, or unverifiable by any source. as llms are used to generate massive texts in all applications, it is essential to make sure their output is accurate @xcite . undetected hallucinations can propagate misinformation, lower confidence in ai systems, and have severe implications in applications such as healthcare and law. identification of particular spans of hallucinated text, as opposed to 1 the code is available at https://github.com/ sakethreddyvemula/semeval-2025_mu-shroom figure 1 : architecture diagram describing proposed method for detecting hallucination spans. @xcite merely marking whole outputs, is critical for realworld application, as it enables accurate corrections and improved comprehension of where and why a model hallucinate. in this paper, we describe an llm-uncertainty based method for hallucination span detection. our hypothesis builds upon @xcite that if an llm is certain of a given concept, stochastically-sampled responses are likely to be similar and contain consistent facts. however, for hallucinated facts, these sampled responses are likely to diverge and contradict one another. we utilize entropy information to identify the precise spans of hallucinated text using sampled responses @xcite , allowing us to effectively identify inconsistencies that signal hallucination. our approach works well in zero-resource and black-box environments without any extra training. in addition, since our approach is languageindependent, it works equally well in a variety of languages. our model ranks 18th on average among over 40 submissions, achieving its best rank of 10th in chinese (mandarin). foot_0 ## related work the problem of hallucination detection in large language models (llms) has been a focus of much attention recently. hallucinations are defined as cases when llms produce outputs that sound plausible but are factually false or unsupported, compromising their validity for real-world usage. @xcite proposed a technique employing semantic entropy to identify such confabulations through uncertainty estimation in the semantic space of model outputs. this method calculates uncertainty at the meaning level as opposed to actual word sequences and allows for recognizing arbitrary and poor-quality generations for different datasets and tasks without explicit domain knowledge. following this, @xcite introduced semantic entropy probes (seps), which estimate semantic entropy directly from one generation's hidden states. seps are efficient in computation, avoiding repeated model samplings at inference time. their experiments showed that seps have high performance in hallucination detection and generalize well to out-of-distribution test sets, indicating that model hidden states contain semantic uncertainty relevant to hallucinations. in parallel, @xcite introduced selfcheckgpt, a zero-resource black-box method for fact-checking llm responses independent of external databases. the technique exploits the consistency of stochastically generated responses by assuming that when an llm has knowledge about a concept, its sampled responses will be consistent and similar in content while hallucinated facts result in diverse and contradictory responses. their results show that selfcheckgpt efficiently identifies non-factual sentences and evaluates the factuality of passages, providing an efficient solution for situations where model internals are not available. these studies together highlight the need to create effective and efficient techniques for hallucination detection in llms. methods based on semantic entropy, model hidden states, and response consistency provide promising directions for improving the reliability of llm outputs in different applications. ## task description mu-shroom foot_1 (multilingual shared-task on hallucinations and related observable overgeneration mistakes) focuses on detecting hallucinated spans in text output from instruction-tuned llms. the task includes 14 languages: arabic (modern standard), basque, catalan, chinese (mandarin), czech, english, farsi, finnish, french, german, hindi, italian, spanish, and swedish. @xcite evaluation is conducted separately for each language and is based on the following two characterlevel metrics: data format is described in ## methodology in this section we describe our methodology for detecting hallucination spans. given generated text g and stochastically-sampled responses @xmath0, s ′ 2 , ..., s ′ n from models, our method predicts hallicination spans as follows: given a generated text g, we segment it into overlapping spans using a sliding window approach. each span s i is extracted using a window size w and stride t such that: field description lang language of the text. model_input input query provided to the llm. model_output_text generated text from the llm. hard_labels list of pairs (si, ei) representing hallucination spans (start-inclusive, end-exclusive). soft_labels list of dictionaries, each containing: for all valid indices i with step size t. this ensures each part of the text is analyzed with sufficient context. for each span s i , we retrieve the most similar spans from a set of sampled responses @xmath1, s ′ 2 , ..., s ′ n using a lexical matching function based on sequence similarity. the matching spans m i are defined as: we compute the hallucination score for each span s i using a combination of semantic entropy, lexical entropy, and frequency-based scoring. ## results and analysis the findings reveal that our model is aptly suitable for detecting hallucinations for a wide variety of languages that possess intricate morphological and syntactic features. the high correlation scores across numerous languages confirm that our system makes good predictions which correlate well with ground truth annotation. further, the high iou values verify its capacity for good localization of hallucinated text, which enables it to be a trustworthy model in addressing the problems of hallucinations in multilingual environments. ## conclusion in this paper, we utilized an llm-uncertaintybased method for hallucination span detection which works equally well in multiple languages. by using entropy-based uncertainty measures from sample responses, our approach accurately detects hallucinated spans without the need for further training. our model performed competitively in various languages, ranking highly in basque, italian, and hindi. the experiments emphasize the strength of our method, as they show its effectiveness in coping with varied linguistic forms and in yielding precise hallucination span detection. our error analysis also informs on typical failure instances, presenting potential for additional refinements. although our approach is strong, it has limitations, specifically in exploiting supervised learning to achieve better span prediction. our future re-