Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework (2024)

Abstract: Retrieval-augmented generation (RAG) has emerged as a popular solution to mitigate the hallucination issues of large language models. However, existing studies on RAG seldom address the issue of predictive uncertainty, i.e., how likely it is that a RAG model’s prediction is incorrect, resulting in uncontrollable risks in real-world applications. In this work, we emphasize the importance of risk control, ensuring that RAG models proactively refuse to answer questions with low confidence. Our research identifies two critical latent factors affecting RAG’s confidence in its predictions: the quality of the retrieved results and the manner in which these results are utilized. To guide RAG models in assessing their own confidence based on these two latent factors, we develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers. We also introduce a benchmarking procedure to collect answers with the option to abstain, facilitating a series of experiments. For evaluation, we introduce several risk-related metrics and the experimental results demonstrate the effectiveness of our approach. Our code and benchmark dataset are available at https://github.com/ict-bigdatalab/RC-RAG.

## introduction large language models (llms) have gained considerable attention across a wide range of language tasks @xcite @xcite . despite the exciting performance, llms may suffer from hallucination issues @xcite , due to limited memorization abilities or outdated pre-training corpora @xcite . recently, retrieval-augmented generation (rag) has emerged as a promising solution to enhance factual accuracy @xcite figure 1 : illustration of risk control for rag. given a question, a risk controlled rag model is expected to provide the correct answer if it has knowledge of the question, or alternatively, refuses to answer the question. 2023; @xcite , by synthesizing text snippets retrieved from external resources into final responses @xcite @xcite @xcite . however, directly applying existing rag techniques, particularly for knowledge-intensive tasks @xcite @xcite such as factoid question answering @xcite , introduces significant risks in practice. when confronted with noisy search results, even the most advanced rag models are prone to pro-ducing unreliable answers, often exhibiting overconfidence in these erroneous responses @xcite . such unreliable answers may severely undermine the user's question answering (qa) experience. therefore, for practical applications, especially in sensitive domains like healthcare and legal assistance, it is crucial that rag systems confidently provide answers when they know and state "i don't know" when they do not, as illustrated in figure 1 . this calls for the investigation on the risk control issue of rag, a core research problem we want to tackle in this work. this approach reflects wisdom, as it involves rag models proactively refusing to answer questions when predictions are uncertain. unfortunately, most previous research on risk control has focused on llms @xcite @xcite . there has been little work addressing the predictive uncertainty issue of rag. compared to the uncertainty assessment of llms, which concentrates on internal knowledge boundaries, the assessment for rag requires additional consideration of external knowledge from retrieved results. in this work, we identify two critical factors during the uncertainty assessment of rag: the quality of the retrieved results and the manner in which they are used. this raises an important research question: how can we assess the predictive uncertainty of rag based on these two retrieval results-related factors to determine when to discard or keep the generated answers? in this work, we propose a new task of risk control for rag (rc-rag) to decide whether to keep or discard the rag outputs based on confidence assessment. we then introduce a novel counterfactual prompting framework for rag under the zero-shot scenario, leveraging the counterfactual thinking for confidence assessment based on two latent factors. counterfactual @xcite describes the human capacity to learn from prior experiences by imagining the outcomes of alternative actions that could have been taken. for a language model, we can inject counterfactual thinking into prompt, like "what if..." or "assume that", to imagine or simulate the consequences of changing a factor. here, we induce the model to imagine scenarios where the quality of the retrieved results and their usage are poor, then measure its confidence based on the effect of these imagined scenarios on the answers. specifically, our framework consists of three major modules, i.e., a prompting generation module, a judgment module, and a fusion mod-ule: (i) the prompting generation module generates answers under two scenarios that challenges the improper use and poor quality of the retrieved results, respectively; (ii) the judgment module determines whether to discard or keep the generated answers for both scenarios; and (iii) the fusion module combines the judgment results from both scenarios to produce the final decision for selective output. it is important to note that our method is a general post-processing technique, making it applicable to almost any existing rag method. for evaluation, traditional metrics like exact match and f1 score typically focus on the effectiveness of rag. in this work, we propose four risk-related metrics -risk, carefulness, alignment, and coverage -for risk-aware rag evaluation. due to the limited availability of datasets directly applicable to rc-rag, we have constructed a novel risk control benchmark based on two publicly available qa datasets. extensive experiments on rag with mistral @xcite and chatgpt (roumeliotis and tselikas, 2023) as backbones demonstrate that the proposed framework can effectively abstain, outperforming baselines in 3 out of the 4 settings in terms of carefulness and risk, with up to a 14.76% improvement in carefulness and a 2.88% reduction in risk on average. ## related work retrieval-augmented generation. the typical retrieval-augmented generation (rag) method follows a retrieve-then-generate pipeline, first retrieving relevant documents from a grounding corpus and then generating the final answer by the frozen generators @xcite . the retrieval augmentation is performed for all the questions through a single round @xcite @xcite or multiple rounds @xcite @xcite @xcite . however, such practice sometimes hurt generation performance, due to the unsatisfactory retrieved results @xcite @xcite . the reason may lie in the inconsistency between the relevance judgments in retrieval stage and the utility judgments in generation stage @xcite . besides jointly optimization of the retriever and generator @xcite @xcite , another solution is adaptive retrieval augmentation @xcite @xcite , which actively determines when to retrieve based on internal knowledge boundaries. ## experiment settings baselines. we compare our proposed cf prompting framework with three prompt-based baselines: (i) if-or-else (ioe) prompting framework @xcite , facilitating self-corrections based on llms' confidence. to adapt to the rc-rag, we classify the case of answer correction as discard. ## experiment results we aim to answer six research questions: (rq1) does our cf prompting framework efficiently control the risk of rag compared with the baseline methods? (rq2) does the ability of llms affect the effectiveness of rc-rag? (rq3) does the difficulty of qa task affect the ability of rc-rag? (rq4) does the quality of retrieval results affect the effectiveness of rc-rag? (rq5) how does two cf-prompts affect the effectiveness of rc-rag respectively? (rq6) are our risk control framework interpretable? ## conclusion in this work, we propose a counterfactual prompting framework for assessing the uncertainty of rag results, based on the quality of the retrieved results and the manner in which they are used. we construct a benchmark and design risk-related evaluation metrics. experimental results with two llms on two datasets show that our method can effectively reject unanswerable samples and has a certain interpretability. in the future, we will explore other factors that may affect predictive uncertainty in rag, such as conflicts between internal and external knowledge. additionally, we will attempt to design objective functions based on riskrelated metrics to guide the joint learning of the risk control framework and the rag model.