Medico: Towards Hallucination Detection and Correction with Multi-source Evidence Fusion (2024)

Abstract: As we all know, hallucinations prevail in Large Language Models (LLMs), where the generated content is coherent but factually incorrect, which inflicts a heavy blow on the widespread application of LLMs. Previous studies have shown that LLMs could confidently state non-existent facts rather than answering “I don’t know”. Therefore, it is necessary to resort to external knowledge to detect and correct the hallucinated content. Since manual detection and correction of factual errors is labor-intensive, developing an automatic end-to-end hallucination-checking approach is indeed a needful thing. To this end, we present Medico, a Multi-source evidence fusion enhanced hallucination detection and correction framework. It fuses diverse evidence from multiple sources, detects whether the generated content contains factual errors, provides the rationale behind the judgment, and iteratively revises the hallucinated content. Experimental results on evidence retrieval (0.964 HR@5, 0.908 MRR@5), hallucination detection (0.927-0.951 F1), and hallucination correction (0.973-0.979 approval rate) manifest the great potential of Medico. A video demo of Medico can be found at https://youtu.be/RtsO6CSesBI.

## introduction large language models (llms) have attracted significant interest from academia and industry. major tech companies have introduced solutions like openai's gpt-4 (openai, 2023), google's gemini @xcite , and alibaba's qwen @xcite . llms have shown impressive performance in understanding and generating language. however, their complex structures, vast parameters, and opaque generation processes make it difficult to ensure the accuracy of the generated content, known as hallucination 1 @xcite corresponding author. 1 hallucination can be broadly categorized into factuality hallucination and faithfulness hallucination, referring to figure 1 : motivation example. the generated content and retrieved evidence are marked in yellow and green, respectively. (a) shows the situation of acquiring evidence in a single way and making an erroneous judgment due to outdated evidence. (b) shows the situation, where users are only provided with a veracity label, confusing users about why and where the content is incorrect. recently, an ever-growing body of studies and systems has been focused on verifying llms' generated content in terms of hallucinations, such as fleek @xcite , factllama @xcite , and safe @xcite . they formulate hallucination-checking as the classification task, where the input consists of the evidence and generated content, and the output typically determines the veracity of the generated content into three categories, i.e., supported, not supported, and irrelevant @xcite . however, they commonly acquire evidence in a single way and may fall into the absence of useful evidence. in fact, the accuracy of the generated content involves many aspects, requiring informative evidence from diverse sources. taking the generated content "queen elizabeth ii is the head of state of 16 countries in the commonwealth realm." for example, it might be classified section 5.1 for more details. this work mainly focuses on factuality hallucination. as correct when only using evidence acquired from a non-real-time knowledge base, as shown in figure 1(a ). on the other hand, they usually show users only the veracity label, while the rationale behind such a decision is missing. so these models lack explainability and still require arduous labor from users to manually check why and where the generated content is incorrect, which creates a poor user experience. we show this issue in figure 1(b) . in this work, we propose medico (multi-source evidence fusion enhanced hallucination detection and correction), a hallucination-checking framework, which satisfies the three properties of being multi-faceted, model-agnostic, and explainable. specifically, our framework acquires diverse evidence from multiple sources, including unstructured text, semi-structured knowledge base, as well as structured knowledge graphs. it reranks the evidence candidates and organically fuses them to obtain the fused evidence, which offers sufficient support evidence for the following detection. our framework then leverages the fused evidence to detect whether the generated content is correct or incorrect and also gives the rationale behind the decision. if the classification result is incorrect, it will iteratively revise the hallucinations within the generated content according to the rationale. our main contributions can be summarized as follows: ## methodology figure 2 presents the overall system framework of medico. it mainly consists of three components: (1) multi-source evidence fusion, which in-corporates diverse evidence from multiple sources to provide sufficient support evidence for detection; ## user interface we build medico using the gradio package @xcite , an easy-to-use webui development framework based on fastapi and svelte, which facilitates the deployment of machine learning apps. we can naturally divide the view of medico's system into two parts: (1) retrieval and fusion, and ## experiments in this section, we conduct extensive experiments on a hallucination evaluation benchmark, halueval, to answer the following research questions (rqs): ## conclusion this work presents medico, an innovative hallucination-checking system, which assists users in detecting and correcting factual errors in llms' generated content with multi-source evidence fusion. to the best of our knowledge, medico is the first hallucination detection and correction framework that leverages multi-source evidence fusion, provides the rationale behind the decision, as well as revises the incorrect generated content. last but not least, medico can not only be used as a tool to help users detect and correct hallucinations in response, but also serve as a security plug-in that automatically checks llms' replies in real-time.