Two-tiered Encoder-based Hallucination Detection for Retrieval-Augmented Generation in the Wild (2024)

Abstract: Detecting hallucinations, where Large Language Models (LLMs) are not factually consistent with a Knowledge Base (KB), is a challenge for Retrieval-Augmented Generation (RAG) systems. Current solutions rely on public datasets to develop prompts or fine-tune a Natural Language Inference (NLI) model. However, these approaches are not focused on developing an enterprise RAG system; they do not consider latency, train or evaluate on production data, nor do they handle non-verifiable statements such as small talk or questions. To address this, we leverage the customer service conversation data of four large brands to evaluate existing solutions and propose a set of small encoder models trained on a new dataset. We find the proposed models to outperform existing methods and highlight the value of combining a small amount of in-domain data with public datasets.

## introduction in the last year, large language models (llms) have exploded in popularity, in part due to their ability to convincingly answer arbitrary questions. retrieval-augmented generation (rag), which injects portions of external knowledge bases into the prompt, is an effective method for introducing specific information for a given brand or use case. however, hallucinations, where the system provides an ungrounded response, threatens the viability of this application in an industry setting. this paper proposes and evaluates a novel encoder-based classifier for hallucination detection tailored for enterprise customers. our model, raghalu, is an encoder-based two-tiered solution that leverages one binary classifier in each tier. raghalu first identifies factually verifiable statements and then determines whether each verifiable statement is supported or unsupported by the kb. whereas other works either do not handle non-verifiable (e.g. small talk or information gathering) statements (honovich et al., 2022; gekhman et al., 2023; muhlgay et al., 2023), or group them with other types of verifiable claims (gupta et al., 2022), we developed a 3-label taxonomy to distinguish between the two. our model is trained on both re-annotated and original public datasets, and internal in-domain data. although there are recent studies such as wang et al. (2023) using chatgpt in a similar two-tiered solution, to the best of our knowledge, this is the first hallucination detection solution developed that explicitly identifies verifiable claims and leverages them as atomic claims (cf. min et al. 2023). we compare raghalu against a number of baselines: prompt-engineering openai's gpt-3.5turbo-0613 1 (openai, 2023), a hallucination detection fine-tuned mistral-7b-instruct llm, and open source hallucination detection models by google @xcite we find our two-tiered solution which further fine-tunes a natural language inference (nli) @xcite deberta (he et al., 2021) cross-encoder model performs best and generalizes both across customer service domains and open source data. figure 1 shows raghalu inte- @xcite we refer to this model as chatgpt throughout. @xcite mistral-7b-instruct-v0.1, t5_xxl_true_nli_ mixture,hallucination_evaluation_model 3 cross-encoder/nli-deberta-v3-large grated into a customer service rag system. the paper is organized as follows. we first present our model architecture and the data used to train it. we then outline the baselines, followed by the results and discussion. there has also been work aimed at judging the factual precision of llms without retrieval. muhlgay et al. (2023) assess lm factualness as related to generated token perplexity. they find that while perplexity is related to factualness, it is not enough to identify hallucinations on its own. tian et al. (2023) fine-tunes llms for factualness using model uncertainty. within the area of question-answering and rag, there has been a variety of work aimed at using llms to self-verify factual consistency with prompting (min et al., 2023; wang et al., 2023; manakul et al., 2023). though these prompts were shown to be effective, using an llm to self-judge remains impractical and expensive in a large scale industry setting. fact verification similar to other works, we judge factual consistency on a sentence level (thorne et al., 2018; honovich et al., 2022), we consider a "checkworthiness"/verifiable statement type (wang et al., 2023; gupta et al., 2022; mishra et al., 2024), and we fine-tune an nli model. however, unlike the aforementioned works, we train and evaluate on real commercial data, we train a model to distinguish between verifiable and nonverifiable claims, we fine-tune our model on a new collection of llm generated texts, and we produce an end-to-end solution that does not rely on prompting of an llm for classification. ## experimental setup in addition to evaluating three open source nli-based models on the sup-ported/unsupported examples, we compare the performance of raghalu with prompting chatgpt and fine-tuning mistral-7b. similar to other works (thorne et al., 2018; honovich et al., 2022; wang et al., 2023) we split the response into sentences using the nltk sentence tokenizer (bird et al., 2009) for classification. ## future work future work could include developing a more finegrained hallucination detection model as done in mishra et al. (2024). examples include distinguishing between unsupported and contradicting claims and identifying statements of action such as "i found your account number", which could indicate a need for an api integration. correcting or mitigating hallucinations by improving kb chunking are also important considerations. ## conclusion we developed a novel encoder-based hallucination classifier optimized for performance on customer service rag bots in enterprise. our models are trained on a new collection of open source and private data that generalizes and outperforms other models tested. we demonstrated the need for domain specific training data for hallucination detection, as well as the importance of kb lengths used in rag.