Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation (2024)

Abstract: Language models (LMs) are known to suffer from hallucinations and misinformation. Retrieval augmented generation (RAG) that retrieves verifiable information from an external knowledge corpus to complement the parametric knowledge in LMs provides a tangible solution to these problems. However, the generation quality of RAG is highly dependent on the relevance between a user’s query and the retrieved documents. Inaccurate responses may be generated when the query is outside of the scope of knowledge represented in the external knowledge corpus or if the information in the corpus is out-of-date. In this work, we establish a statistical framework that assesses how well a query can be answered by an RAG system by capturing the relevance of knowledge. We introduce an online testing procedure that employs goodness-of-fit (GoF) tests to inspect the relevance of each user query to detect out-of-knowledge queries with low knowledge relevance. Additionally, we develop an offline testing framework that examines a collection of user queries, aiming to detect significant shifts in the query distribution which indicates the knowledge corpus is no longer sufficiently capable of supporting the interests of the users. We demonstrate the capabilities of these strategies through a systematic evaluation on eight question-answering (QA) datasets, the results of which indicate that the new testing framework is an efficient solution to enhance the reliability of existing RAG systems.

## introduction recent progress on large-scale pre-trained language models (lms) @xcite has demonstrated great potential in revolutionizing a wide array of applications across fields, ranging from natural language understanding and generation to complex problem-solving in scientific research. despite their remarkable abilities, generative lms suffer from poor interpretability and transparency, as well as the intrinsic risk of hallucination and misinformation, which collectively prohibit them from being deployed in safety-critical domains such as healthcare @xcite . retrieval augmented generation (rag) @xcite ) is a promising approach for enhancing language models (lms) by incorporating verifiable, current information from external knowledge databases. incorporating this external context to complement the inherent knowledge of lms has demonstrated notable benefits in reducing occurrences of hallucination and misinformation, thereby improving the reliability of content produced @xcite . still, numerous studies @xcite @xcite show that the effectiveness of rag is dependent on the relevance between the query and retrieved documents. in cases where documents of weak relevance are provided, they can become distractions for the lm @xcite , leading to the generation of incorrect answers. at the present moment in time, there is no viable solution for safety-critical rag systems to possess mechanisms for (1) evaluating the relevance of queries submitted from users to the knowledge corpus and flagging low-relevance queries in real-time that cannot be adequately addressed using the available knowledge or (2) identifying significant shifts in query distribution that are indicative of a potential misalignment between the knowledge corpus and user interests, which would suggest an outdated knowledge database that requires updating. to address these deficiencies, in this paper we establish a statistical framework for accurate assessment of the query-knowledge relevance in retrievalbased lms through hypothesis testing. as illustrated in figure 1 , we introduce two testing procedures: an online testing procedure (left subfigure) that aims at identifying single out-of-knowledge queries and an offline testing procedure (right subfigure) for detecting query distribution shifts. in the online testing scenario, we cast evaluation of query-knowledge relevance as testing against the null hypothesis that the unknown query arises from the same distribution as the empirical in-knowledge queries (i.e., testing for goodness-of-fit (gof)). we utilize the semantic similarity between the query and the retrieved most relevant documents captured by text embedding models @xcite to derive test statistics and reject query samples that are unlikely to occur given the empirical distribution of in-knowledge queries (queries that can be answered with the knowledge corpus). in scenarios when the in-knowledge query distribution is unknown, we generate synthetic in-knowledge queries, by prompting lms with document chunks, to serve as a proxy to the true in-knowledge query distribution. in the offline testing scenario, we employ a two-sample gof test to determine whether the unknown set of queries matches the empirical distribution of historical in-knowledge queries, where a large p-value is suggestive of a significant query distribution shift. to demonstrate the feasibility of these approaches, we report on a systematic evaluation with two biomedical corpus and eight qa datasets, including three general domain qa datasets and five biomedical qa datasets, to investigate seven test statistics and six retrievers. our experiments highlight several notable findings. first, the testingbased methods can more reliably capture the relevance compared to lm-based relevance scores and outlier-detection-based baselines. second, synthetic queries can provide a good approximate to the in-knowledge distribution with similar empirical performance for detecting out-of-knowledge queries. third, there is a misalignment between embedding models' ability to retrieve relevant documents and their ability to detect out-of-knowledge queries. and, fourth, query distribution shifts can be effectively detected through gof testing with high accuracy using a relatively small sample size. ## related work retrieval augmented generation. retrieval augmented generation (rag) @xcite provides a tangible solution to address hallucinations in knowledge-intensive tasks by retrieving verifiable information from external knowledge corpora @xcite @xcite . however, the truthfulness of rag responses highly depends on the relevance between the query and the corpus @xcite @xcite and may suffer from increased risk of hallucination under distribution shifts @xcite . while recent studies mostly focus on improving various stages of the retrieval and generation pipeline @xcite @xcite , in this work, we take a different perspective and improve the rag systems' awareness of their knowledge boundary to indicate what they know and when are sufficiently certain that they should return knowledge to an end user @xcite @xcite . ## conclusion identifying out-of-knowledge queries is an important step in improving the reliability of rag systems and reducing hallucination and misinformation. this work establishes a statistical framework for quantifying the relevance of query-knowledge relevance through goodness-of-fit hypothesis testing. we introduce two testing procedures with different goals of identifying low-relevance query samples and detecting query distribution shifts. we demonstrate the effectiveness of these approaches through extensive experiments on eight datasets from various domains. we hope our findings can provide insights for future research on reliable retrieval-based generation.