H allu RAG - RUG at S em E val-2025 Task 3: Using Retrieval-Augmented Generation for Hallucination Detection in Model Outputs (2025)

Abstract: Large Language Models (LLMs) suffer from a critical limitation: hallucinations, which refers to models generating fluent but factually incorrect text. This paper presents our approach to hallucination detection in English model outputs as part of the SemEval-2025 Task 3 (Mu-SHROOM). Our method, HalluRAG-RUG, integrates Retrieval-Augmented Generation (RAG) using Llama-3 and prediction models using token probabilities and semantic similarity. We retrieved relevant factual information using a named entity recognition (NER)-based Wikipedia search and applied abstractive summarization to refine the knowledge base. The hallucination detection pipeline then used this retrieved knowledge to identify inconsistent spans in model-generated text. This result was combined with the results of two systems which identified hallucinations based on token probabilities and low-similarity sentences. Our system placed 33rd out of 41, performing slightly below the ‘mark all’ baseline but surpassing the ‘mark none’ and ‘neural’ baselines with an IoU of 0.3093 and a correlation of 0.0833.

## introduction the rise of large language models (llms) has brought attention to an important limitation they have, a phenomenon often referred to as llm 'hallucinations'. this phenomenon occurs when an ai-generated text contains or describes facts that are not supported by the provided reference. these facts do not necessarily need to be false to be labeled a hallucination. instead, they are cases where the answer text is more specific than it should be, given the information available in the provided context. to further clarify what a hallucination is, we provide the following example introduced by @xcite : as seen in this example, the generated text is fluent but inaccurate concerning the source text. this is noted by the generation of information that is not found originally in the source text, specifically referring to 'android mobile'. the generation of false information can hinder a model's usefulness in many applications. moreover, it can also be the cause for ethical concerns: when a text is syntactically sound, people quickly assume that it is also semantically sound. a user being presented with false information can cause considerable harm in many different domains. the detection of hallucinations is an important task in improving model trustworthiness, so it is vital to develop and improve methods of hallucination detection. in this context, semeval-2025 task 3: mu-shroom, the multilingual sharedtask on hallucinations and related observable overgeneration mistakes was organized @xcite foot_0 . this year, the task inquires about the exact text spans in which hallucinations occur, as opposed to last year's binary classification task @xcite . the organizers provided validation data in ten different languages, from which we only considered english. our approach to this particular task implements a combined model that implements retrieval-augmented generation (rag) in combination with factual information from wikipedia and llama-3, as well as supplementary prediction models based on token probabilities and low-similarity sentences. our method achieved place 33 out of 41 for the english track in semeval-2025 task 3, performing slightly below the 'mark all' baseline. ## related work first, we will discuss some related work that was done during the previous iteration of this shared task, semeval-2024 task 6 @xcite . one of the most common techniques across the papers written on this task is the use of transformerbased models and semantic similarity measures. @xcite employed sentencetransformers to generate embeddings for hypothesis and target texts, comparing them via cosine similarity. the aim was to detect hallucinations based on low similarity scores. other groups pivoted towards prompt-based methods. @xcite focused on zero-shot and few-shot learning. in zero-shot learning, the model relied on its pre-trained knowledge, using carefully crafted prompts. few-shot learning incorporated a small set of labeled data, improving the model's ability to detect more subtle hallucinations. a large number of groups that participated used similar models: models like vectara, mistral/mixtral, deberta, and gpt (3.5 or 4) were used very commonly. the organizers note that especially the gpt-based models work well: four out of six top-scoring teams incorporated it in their approach (e.g., @xcite ; @xcite ). outside of the context of the shared task, much work has been done regarding the use of rag in hallucination reduction. the main intuition behind this approach is that the inclusion of factual information (the 'retrieval' part) will reduce the generation of factually incorrect content @xcite . the use of rag for hallucination reduction has been proven to be effective for multiple use cases, like conversation @xcite or structured outputs like workflow generation @xcite . considering rag's usefulness in hallucination reduction, it will be interesting to see whether the addition of relevant retrieved data also extrapolates to improved hallucination detection. this approach is not well-represented in the literature yet, so the merits of the method are yet to be seen. ## data the shared task data was provided in 14 languages total, but for our approach, only the english data was considered. the organizers of the shared task released both a validation set as well as a test set. the english validation set comprised 50 data points, while the test set, released at the start of the evaluation phase (initially without labels), comprised 154 data points. each of the data points consists of the following elements: for the english datasets, the data points were annotated by up to 13 annotators. each annotator was provided with the model output text and relevant context. then, they were instructed to highlight each span of model output text that was inconsistent with the given context. annotators were instructed to be as conservative as possible when marking hallucinations. ## method as mentioned previously, our approach leverages a form of rag, an llm, and two prediction models. for efficiency, we split our pipeline into two segments: 1. knowledge retrieval, and 2. hallucination detection. this pipeline is illustrated in figure 1 . ## results in ## conclusion and future work in this work, we presented a retrieval-augmented pipeline for detecting hallucinated spans in llm output, focusing on english data from the mu-shroom task. our system combined token probability, factual checks, and summarized wikipedia context to highlight hallucinated spans. while our approach outperformed two out of three baselines, it often detected an overabundance of spans, reducing precision and diluting overall performance. in addition, the practical constraints of our chosen model regarding input length and model parameters restricted performance. despite these challenges, our results suggest that integrating retrieval methods and careful prompt engineering can help with validating llm output.