Deloitte (Drocks) at S em E val-2025 Task 3: Fine-Grained Multi-lingual Hallucination Detection Using Internal LLM Weights (2025)

Abstract: Large Language Models (LLMs) have greatly advanced the field of Natural Language Generation (NLG). Despite their remarkable capabilities, their tendency to hallucinate—producing inaccurate or misleading information-remains a barrier to wider adoption. Current hallucination detection methods mainly employ coarse-grained binary classification at the sentence or document level, overlooking the need for precise identification of the specific text spans containing hallucinations. In this paper, we proposed a methodology that generates supplementary context and processes text using an LLM to extract internal weights (features) from various layers. These extracted features serve as input for a neural network classifier designed to perform token-level binary detection of hallucinations. Subsequently, we map the resulting token-level predictions to character-level predictions, enabling the identification of spans of hallucinated text, which we refer to as hallucination spans. Our model achieved a top-ten ranking in 13 of the 14 languages and secured first place for the French language in the SemEval: Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes (Mu-SHROOM), utilizing the Mu-SHROOM dataset provided by the task organizers.

## introduction the domain of natural language generation (nlg) is witnessing a remarkable transformation with the emergence of large language models (llms) @xcite @xcite . llms have been shown to outperform traditional natural language processing (nlp) approaches across a wide range of applications @xcite . despite the rapid advancements in llms, a concerning trend has emerged where these models generate hallucinations @xcite , resulting in content that appears plausible but is factually unsupported. hallucinations can be categorized into extrinsic errors, where claims conflict with external facts, and intrinsic errors, where claims are not fully grounded in the source material. this issue is particularly critical in sensitive domains such as healthcare, finance, and legal services, where the accuracy of generated content is paramount. hence, the automatic detection of hallucinated content has become an active area of research, aiming to enhance the reliability and trustworthiness of llm-generated content @xcite . recent studies have explored different methodologies for hallucination detection, including natural language inference (nli) and factual consistency checking @xcite @xcite , as well as textual entailment techniques @xcite . additionally, approaches like reference-free (zero context) hallucination detection have been investigated @xcite @xcite , alongside evidence retrieval methods utilizing retrieval-augmented generation (rag) or web search @xcite @xcite . however, fact-checking models often demonstrate inconsistent performance when evaluating text across different languages. @xcite highlights that even state-of-the-art (sota) llms, when used for fact-checking, struggle with text in low and medium-resource languages. moreover, many popular hallucination detection methods classify hallucinations at the sentence or document level, which limits their ability to precisely identify and correct the specific text responsible for these errors. to address this limitation, @xcite introduced the hades dataset (hallucination detection dataset), enabling fine-grained, referencefree hallucination classification at the token level. uncertainty-based and consistency-based methods were proposed to detect token level hallucinations @xcite . for example, @xcite utilized log-probability curve detection, while @xcite estimated semantic likelihoods by clustering generated sequences. furthermore, @xcite explored token type and frequency for detecting hallucinations based on uncertainty. building on these ideas, @xcite developed metrics assessing token cohesiveness through successive rounds of random token deletion and measuring semantic differences. recent advancements have shown promise in using llm internal states for detecting token-level hallucinations. for instance, @xcite focused on identifying hallucinations by analyzing embeddings and gradients to gauge probability distribution differences, while @xcite applied mechanistic interpretability within rag scenarios. many of these solutions, however, encounter common challenges. they often rely heavily on large amounts of labeled training data and necessitate multiple inference calls for each sentence, which can be resource-intensive. they also frequently fall short in testing across low and mediumresource languages, or in conducting comprehensive multilingual evaluations. to boost this area of research further, the se-meval 1 organizers introduced the mu-shroom task. this task focuses on detecting hallucination spans in the outputs of instruction-tuned llms in a multilingual context. the contribution of this study are: ## mu-shroom dataset the mu-shroom dataset contains the following columns: ## proposed approach in this section, we describe our proposed approach for detecting hallucination spans as depicted in figure 1. the approach encompasses three primary components: 1) context generation, 2) extracting token-level internal weights from llm, and 3) constructing a binary classifier to produce token-level predictions, which are subsequently transformed into character-level predictions. ## experimemts this section details the experimental evaluation of our approach. to assess the effectiveness of our method, we employed two established characterlevel metrics such as intersection-over-union (iou) and spearman correlation (s.corr). the iou is calculated as the ratio of the number of characters identified as hallucinations by our model to the total number of unique characters in both the predicted and actual sets. conversely, spearman correlation is calculated by comparing the probabilities assigned by the model that indicate a character's classification as part of a hallucination against the empirical probabilities observed from human annotators. ## conclusion in this study, we tackles the critical challenge of hallucination phenomenon observed in llms. by employing a neural network classifier that utilizes features extracted from various layers of an llm, we enable precise identification of hallucination spans within generated text. our model achieved a top ten ranking across 13 languages achieving first place specifically in french.