REFIND at S em E val-2025 Task 3: Retrieval-Augmented Factuality Hallucination Detection in Large Language Models (2025)

Abstract: Hallucinations in large language model (LLM) outputs severely limit their reliability in knowledge-intensive tasks such as question answering. To address this challenge, we introduce REFIND (Retrieval-augmented Factuality hallucINation Detection), a novel framework that detects hallucinated spans within LLM outputs by directly leveraging retrieved documents. As part of the REFIND, we propose the Context Sensitivity Ratio (CSR) , a novel metric that quantifies the sensitivity of LLM outputs to retrieved evidence. This innovative approach enables REFIND to efficiently and accurately detect hallucinations, setting it apart from existing methods. In the evaluation, REFIND demonstrated robustness across nine languages, including low-resource settings, and significantly outperformed baseline models, achieving superior IoU scores in identifying hallucinated spans. This work highlights the effectiveness of quantifying context sensitivity for hallucination detection, thereby paving the way for more reliable and trustworthy LLM applications across diverse languages. Our code is available at https://github.com/oneonlee/REFIND .

## introduction detecting hallucinated information in responses generated by large language models (llms) has emerged as a critical challenge in the field of natural language generation @xcite . hallucination, in this context, refers to the generation of content that is factually incorrect or lacks grounding in verifiable sources @xcite . this issue is particularly pronounced in knowledge-intensive tasks that demand high factual accuracy, such as question answering @xcite . the consequences of unmitigated hallucination are significant, ranging from the propagation of misinformation to a decline in trust in ai systems, underscoring the need for effective hallucination detection for the development of safe and trustworthy ai. prior research has explored various approaches for hallucination detection. token-level classifiers, for example, leveraging pre-trained language models like roberta @xcite , have been employed for binary classification, labeling individual tokens as either factual or hallucinated @xcite . however, these models often exhibit limitations when applied to low-resource languages and tend to rely heavily on internal knowledge without effectively utilizing external evidence, which can hinder their performance. extrinsic methods, such as retrieval-augmented models, aim to mitigate hallucinations by integrating external knowledge. nevertheless, existing retrieval-augmented approaches, such as fava @xcite , can potentially lead to inaccuracies in aligning the modified responses with the original llm output, due to their multi-step processes involving retrieval, comparison, and editing. to address these limitations, we introduce refind (retrieval-augmented factuality hal-lucination detection), a novel framework specifically designed to identify hallucinated spans within llm-generated text. refind achieves this by quantifying the context sensitivity of each token at the token level. by leveraging retrieved documents, refind calculates a context sensitivity ratio (csr) for each token in the llm's response, measuring the token's dependence on external contextual information. tokens exhibiting high csr values are identified as likely hallucinations, offering a more direct and efficient approach to factuality verification. our contributions can be summarized as follows: detection of hallucinated responses several studies have proposed methods to detect whether a response contains hallucinated information. farquhar et al. (2024); han et al. (2024); arteaga et al. (2025) leveraged semantic entropy @xcite to estimate uncertainty and identify hallucinations. these approaches utilize entropy-based metrics to assess the reliability of generated responses. selfcheckgpt @xcite introduces a method that employs the language model itself to sample multiple responses and detect inconsistencies among them, thus identifying hallucinated outputs. however, this method relies solely on the internal knowledge of the language model, making it less effective when the model's knowledge is limited or incomplete. detection of hallucinated spans beyond identifying whether a response is hallucinated, other works aim to detect specific spans of hallucinated content within a response of llms. token-level classification approaches @xcite utilized pre-trained language models to classify individual tokens as factual or hallucinated. these methods focus on analyzing attention patterns, demonstrating that query input tokens (defined as constraint tokens) exhibit strong correlations with factual answer tokens @xcite . fava @xcite proposes a retrievalaugmented pipeline that integrates retrieval, comparison, and editing steps to identify and correct hallucinated spans. while effective, the multi-step process introduces complexity and alignment challenges, particularly in ensuring that the corrected responses remain consistent with the semantics of the original output. ## conclusion in this study, we introduced refind, a novel framework for detecting hallucinated spans in llm-generated outputs by leveraging retrieved documents to compute the context sensitivity ratio (csr) at the token level. refind was rigorously evaluated on the multilingual semeval 2025 task 3: mu-shroom dataset, demonstrating superior performance across nine languages, including low-resource settings, compared to baseline approaches. by directly integrating retrieved context into the token probability calculation, refind effectively identifies hallucinated spans with greater precision and efficiency. our experimental results highlight the robustness and scalability of refind in multilingual environments, offering a promising solution for enhancing the factuality of llm outputs. moreover, the streamlined detection process avoids the complexities associated with multi-step frameworks, enabling practical deployment in real-world applications. for future work, we aim to extend refind by exploring adaptive thresholding mechanisms to further optimize the balance between precision and recall in hallucination detection.