D elucion QA : Detecting Hallucinations in Domain-specific Question Answering (2023)

Abstract: Hallucination is a well-known phenomenon in text generated by large language models (LLMs). The existence of hallucinatory responses is found in almost all application scenarios e.g., summarization, question-answering (QA) etc. For applications requiring high reliability (e.g., customer-facing assistants), the potential existence of hallucination in LLM-generated text is a critical problem. The amount of hallucination can be reduced by leveraging information retrieval to provide relevant background information to the LLM. However, LLMs can still generate hallucinatory content for various reasons (e.g., prioritizing its parametric knowledge over the context, failure to capture the relevant information from the context, etc.). Detecting hallucinations through automated methods is thus paramount. To facilitate research in this direction, we introduce a sophisticated dataset, DelucionQA, that captures hallucinations made by retrieval-augmented LLMs for a domain-specific QA task. Furthermore, we propose a set of hallucination detection methods to serve as baselines for future works from the research community. Analysis and case study are also provided to share valuable insights on hallucination phenomena in the target scenario.

## introduction large language models (llms) have brought breakthrough performances for many natural language processing tasks, especially for generationrelated applications such as chatbots and questionanswering (qa). however, state-of-the-art llms (e.g., chatgpt foot_0 , bard foot_1 ) still have a number of weaknesses. notably, the tendency to generate hallucinatory (non-factual) statements is a critical issue in llms that hinders their usage in applications where reliability is a must-have feature (e.g., vehicle-repair assistant). to elaborate, hallucination occurs because the models e.g., llms follow the principle of maximum likelihood while generating the output text without considering the factuality. as a result, they end up generating outputs that are always highly probable but not necessarily factual. this is a longstanding problem in neural natural language generation and it is pervasive in all types of generation tasks ranging from qa @xcite to abstractive summarization @xcite . with the recent technological advances made by state-of-the-art llms, hallucination has been largely reduced compared to models from prior work. however, hallucinatory statements are still prevalent in text generated by llms. the potential incorrect/hallucinated content in the answers provided by llm-based assistants raises significant liability concerns (e.g., vehicle damage) for the applications requiring high reliability. to date, various approaches have been proposed to reduce the degree of hallucination in qa tasks @xcite . one effective way is to leverage information retrieval (ir). in particular, instead of directly feeding the question alone to the llm for answer generation, an ir system first retrieves relevant domain-specific information for the given question to be used as context. next, the question and the retrieved context are used together in prompt for an llm to generate an answer. the retrieved context aids the llm in generating a more reliable answer. however, even with ir and prompt engineering, hallucinations still occur from time to time, typically due to imperfect information retrieval and/or over-reliance on knowledge acquired through pre-training. detection and mitigation of hallucinatory content is thus an essential topic to widen the application of llms to domain-specific qa tasks. in this work, we focus on addressing hallucination by llms in domain-specific qa where high reliability is often needed. specifically, we present delucionqa, a large dataset for facilitating research on hallucination detection in ir-aided car manual qa systems. we also propose a set of hallucination detection approaches, which may serve as baselines for further improvement from the research community. our best-performing baseline shows a macro f1 of only 71.09% on the test set of delucionqa illustrating the difficulty of the task and room for future improvement. insights on the causes and types of hallucinations for the target scenario are provided in this paper. we hope this work paves the way for more future explorations in this direction, which can expand the scope of llm usage to tasks with high-reliability requirement. ## related work to date, various resources (e.g., datasets/benchmarks) have been made available for studying hallucination detection. most of these resources address the existence of hallucinatory content in abstractive summarization and machine translation. for example, @xcite construct a dataset where each example contains the human annotations for faithfulness and factuality of the abstractive summaries generated by state-of-the-art models for 500 source texts taken from the xsum dataset @xcite . @xcite proposes a benchmark named frank for understanding factuality in abstractive summarization. the benchmark dataset is derived by generating 2,250 abstractive summaries from different models and then collecting human annotations related to their factuality. @xcite and @xcite also construct smaller scale resources for hallucination in abstractive summarization. for machine translation, guerreiro et al. ( 2023 ) recently introduced a dataset containing 3,415 examples containing annotations for different types of hallucination. moreover, there have been significant explorations @xcite @xcite for addressing hallucination in machine translation. recently, several datasets have been proposed to study hallucination in general text generation (not specific to any tasks) by llms. @xcite presents hades, a token-level hallucination detection benchmark for free-form text generation. in contrast to abstractive summarization and machine translation, where the input text can be used as a reference for detecting hallucination, hades is reference-free. @xcite released a dataset containing 1,908 sentences with annotations related to hallucination. the sentences are derived by prompting an llm to generate wikipedia articles about different concepts. @xcite attempts to automatically detect the truthfulness of statements generated by an llm by using its activations to train a classifier. to this end, they build a dataset containing true/false human annotated labels from 6 different topics. @xcite presents attributable to identified sources or ais as a framework for evaluating if the text generated by a model is supported by external sources (i.e., not hallucinated). while all of these datasets are valuable resources for the exploration of hallucination detection by llms, none of them cater to customer-facing scenarios where high reliability is essential. ## delucionqa: a dataset for hallucination detection we aim to develop a dataset that represents a domain-specific qa task requiring high system reliability for customer satisfaction. an example of such a scenario is question answering over car manuals, where incorrect llm output could cause damage to cars and endanger lives. however, such domain-specific scenarios pose challenges such as technical terms and language particularities being infrequent during the pre-training stage of llms resulting in their sub-optimal representation. to ensure answer reliability and prevent hallucination in these settings, ir technology is usually employed to inject related context information into the llm prompt. despite this, hallucinations can still occur. the delucionqa dataset seeks to capture instances of hallucination in domain-specific qa after using ir, facilitating research on hallucination detection in this scenario. for this, we select carmanual qa as a domain-specific representative task and, without loss of generality, create the dataset using the manual of jeep's 2023 gladiator model. to collect the dataset, we create a set of questions for the technical car repair manual domain and use a qa system equipped with multiple retrievers to generate various retrieval results for each question. we then generate various answers based on those retrieval results, resulting in (question, retrieval result, answer) triples which are annotated by human annotators. the resulting dataset is of high quality and can be used to investigate hallucination detection/handling approaches for domainspecific qa tasks. we elaborate on the dataset curation processes in the following sections. ## automatic hallucination detection this section describes two automatic approaches to detect hallucinations in context-augmented qa. the evaluation of these approaches is reported in section 5. note that while we evaluate the proposed hallucination detection approaches using only delucionqa, they can be used for other ir-augmented qa tasks as well. ## results and analyses this section will report the evaluation of the previously introduced hallucination detection methods on our newly proposed delucionqa dataset. hallucination detection performance we provide a comparison of our hallucination detection algorithms for all three splits of the delucionqa dataset in similarity match vs keyword match the results show that the similarity-based methods (showing macro f1 scores from 69.45% to 71.09%) perform considerately better than the keyword-match approach with 52.8% macro f 1 . this indicates that simple keyword matching is not enough for hallucination detection, and more sophisticated methods to capture the similarities between the context and answer are necessary. ## case study and error analysis we observe that hallucination may occur due to various reasons, including insufficient retrieval results, excessive retrieval results, and the llm focusing on its learned knowledge instead. to facilitate the discussion, we list a number of examples covering various types of hallucination in p1: using learned knowledge instead of context. when the retriever in the qa system fails to fetch the correct information, sometimes the llm can honestly admit that it does not know the answer, but often the llm simply makes something up, leading to a hallucinated answer which looks probable. as shown in the first example in p2: unrelated information in retrieval results. even if the retrieved result includes the correct context, the llm may still hallucinate, likely due to the additional information presented in the retrieval result. for example, for the second question in question answer how do i activate the uconnect voice recognition system? . . . alternatively, you can also say "hey uconnect" or "hey chrysler" to activate the system . . . . . . or say "hey uconnect" followed by a voice command . . . what does emission inspection check? . . . the inspection also checks for any issues that may cause the mil to be illuminated during normal vehicle operation . . . . . . the inspection also checks for fluid level, exhaust system, and catalytic converter maintenance. how far can i use key fob to lock/unlock the door? . . . in general, the range can be up to several hundred feet . . . . . . from distances up to 66 ft (20 m) using the key fob . . . p1 and p2 can occur jointly. note that the llm does not always hallucinate when the retrieval result includes additional text. use the third question in ## conclusion this work addresses hallucination in text generated by llms in retrieval-augmented question answering, focusing on a domain-specific scenario with high-reliability requirements. our study demonstrates that even after leveraging ir technology to provide additional context to aid the model in providing a factual answer, hallucinations can still occur due to various reasons. therefore, it is crucial to develop automatic hallucination detection and handling methods to improve the reliability of llm-based question-answering systems. to facilitate research in this topic, we release our newly collected delucionqa dataset consisting of 2,038 high-quality human-labeled hallucination ratings for question answering over car manuals. based on that, we propose a set of initial solutions for hallucination detection as baselines in this paper. furthermore, we perform a qualitative analysis to provide insights into why hallucination occurs even after the llm is provided a retrieved-context relevant to the question. while delucionqa is constructed from a single domain (car manual) and a single llm (chatgpt), we believe that the insights gained from it (both by us and the research community in the future) will be valuable in addressing hallucinations by other llms in the other domains as well. in the future, we will diversify our study of hallucination detection to multiple llms and domains, and present more advanced hallucination detection/handling approaches.