Not All Contexts Are Equal: Teaching LLM s Credibility-aware Generation (2024)

Abstract: The rapid development of large language models has led to the widespread adoption of Retrieval-Augmented Generation (RAG), which integrates external knowledge to alleviate knowledge bottlenecks and mitigate hallucinations. However, the existing RAG paradigm inevitably suffers from the impact of flawed information introduced during the retrieval phrase, thereby diminishing the reliability and correctness of the generated outcomes. In this paper, we propose Credibility-aware Generation (CAG), a universally applicable framework designed to mitigate the impact of flawed information in RAG. At its core, CAG aims to equip models with the ability to discern and process information based on its credibility. To this end, we propose an innovative data transformation framework that generates data based on credibility, thereby effectively endowing models with the capability of CAG. Furthermore, to accurately evaluate the models’ capabilities of CAG, we construct a comprehensive benchmark covering three critical real-world scenarios. Experimental results demonstrate that our model can effectively understand and employ credibility for generation, significantly outperform other models with retrieval augmentation, and exhibit robustness despite the increasing noise in the context.

## introduction in recent years, large language models (llms) @xcite @xcite have experienced significant growth and demonstrated excellent performance in multiple domains @xcite @xcite . with the ascendancy of llms, retrieval-augmented generation (rag) has attracted significant interest. rag mitigates the knowledge bottleneck of llms by incorporating externally retrieved documents into their generation process. this inclusion helps diminish the occurrences of hallucinations and misinformation during generation, thereby substantially enhancing the quality of output from llms @xcite @xcite . however, rag for large language models remains significantly impacted by flawed information. this is mainly because the retrieval process often provides noisy, outdated, and incorrect contexts which adversely affects rag, substantially reducing its effectiveness. specifically, previous research @xcite has found that llms are highly sensitive to noise, which impacts llms' capacity to discern and trust accurate information, ultimately affecting the outcomes they generate. furthermore, due to the temporal insensitivity of llms @xcite , these models struggle to discern out-dated information solely based on their internal knowledge. more critically, because llms are trained on extensive collections of historical text, there's an inherent risk that outdated information will align with the models' internal knowledge bases. this alignment can encourage llms to favor and perpetuate outdated information. besides, the prevalence of misinformation on the current web poses a significant challenge for large models, which struggle to identify misinformation using only their inherent knowledge @xcite . this difficulty makes them susceptible to misinformation, leading to the generation of incorrect answers. therefore, flawed information, characterized by noisy, outdated, and incorrect information, has substantial negative effects on rag. from a cognition perspective, a common approach humans adopt to combat flawed information is to assess the credibility of external information @xcite . for humans, credibility refers to the acceptability based on the quality of the information, its source, and subjective evaluation. however, llms relying solely on internal knowledge to assess information credibility are unstable and unreliable @xcite . therefore, we aim to guide llms' acceptance of information by utilizing external indicators of credibility. we introduce credibility-aware generation (cag), a universally applicable framework designed to address flawed information encountered during rag. at its core, cag seeks to equip models with the ability to discern and process information based on credibility. by assigning different credibility to information based on its relevance, timeliness, and the reliability of its source, and explicitly distinguishing them in the input, cag significantly mitigates the issues arising from flawed information. unfortunately, we have discovered that existing llms are not inherently sensitive to directly provided credibility in the prompt. this deficiency restricts their capacity to optimally employ credibility for discerning and processing information. to endow models with the capability of cag, we propose a novel data transformation framework. this framework transforms existing question answering (qa) datasets into data that integrates credibility, which can be employed to guide the model for credibility-based generation. specifically, our process comprises two core steps: 1) multi-granularity credibility annotation, which assigns credibility to text units at both document and sentence levels by dividing retrieved documents into varying granularities. 2) credibility-guided explanation generation, which prompts llms to generate credibilityguided explanations given questions, retrieved documents with credibility annotation and golden answers. finally, we employ instruction fine-tuning to train the model, enabling it to generate responses based on credibility. to rigorously assess the ability of the model's credibility-aware generation in managing flawed information, we construct a comprehensive benchmark encompassing various real-world scenarios, including open-domain qa, time-sensitive qa, and misinformation polluted qa. in this benchmark, retrieval relevance, timeliness, and source authority are regarded as established measures of credibility. experimental results on multiple datasets across multiple scenarios demonstrate the efficacy of our approach in utilizing credibility. our model significantly outperforms various prevalent rag approaches applied to both open and closedsource llms of diverse scales. additionally, it exhibits robust resilience against noisy documents, maintaining high performance even as alternative strategies suffer sharp declines. all these results verify the effectiveness of the proposed cag framework and corresponding training algorithm. the main contributions of this study are summarized as follows foot_0 : question docs documents sentences retrieve units retriever high credibility low credibility medium credibility credibility annotated docs ## credibility-aware generation credibility-aware generation is designed to enable models to discern and process information based on its credibility. subsequently, we will provide formal definitions for both rag and cag, illustrating their divergence. definition in the retrieval-augmented generation process, user input x initiates the retrieval of a set of related documents d x from a large corpus c based on how closely these documents match the input. then, it combines the input x with these documents d x to generate responses y, formalized as @xmath0, d x ]), where [., .] denotes the concatenation operation. compared to rag, the credibility-aware generation offers additional credibility for each document. initially, through credibility assessment based on various scenarios, each retrieved document has been assigned a level of credibility. then, these documents d x with their credibility c are synthesized with the user input x as augmented input. lm generates responses y based on this augmented input, formally represented as ## teaching model to credibility-aware generation in this section, we endow llms with the capability of cag. a potential approach involves directly providing the credibility annotations of each document in the prompt. unfortunately, as indicated in ## credibility-aware generation benchmark to rigorously evaluate the ability of credibilityaware model generation to handle flawed informa-tion, we construct the credibility-aware generation benchmark (cagb). this benchmark encompasses the following three specific scenarios where the integration of credibility is essential: ## experiments to demonstrate the effectiveness of our framework in handling flawed information in real-world qa scenarios, we conduct comprehensive experiments within the cagb. all these results verify the effectiveness of the cag framework and the corresponding training algorithm. additionally, our models maintain robustness even with the increasing noise in the context. in the following sections, we will discuss our experiments and conclusions in detail. ## related work retrieval-augmented generation @xcite integrates a retriever with a generator to improve text generation quality by utilizing external knowledge @xcite @xcite . however, the accuracy of rag is compromised by flawed information, as the inclusion of noisy @xcite , outdated @xcite , or false information @xcite during the retrieval negatively impacts the generator's outputs. previous studies have primarily focused on filtering, ranking, or manually evaluating retrieved documents to mitigate the impact of flawed information. for instance, @xcite ; @xcite deploy various filtering algorithms to remove irrelevant text. zhang and choi (2023) utilizes timestamps to identify and discard outdated information. however, these approaches are limited by the accuracy of filtering algorithms, thereby discarding helpful information and impairing the effectiveness of rag. meanwhile, misinformation is primarily addressed by identifying falsehoods through fact-checking @xcite . however, this approach necessitates either human verification or further training of the discriminator @xcite , both of which can be resourceintensive and introduce bias @xcite . in comparison, our work mitigates the impact of flawed information without discarding documents by introducing multi-feature dimensions of external information to assess the credibility level of each document. researchers fine-tune language models to better leverage the context provided in the input. for instance, @xcite train the model using counterfactuals and irrelevant context to prioritize context. @xcite include irrelevant context in the training samples, making the model robust to irrelevant documents. @xcite train the model on contexts with reflective tokens, enabling it to evaluate the relevance of passages during generation. however, these approaches focus mainly on irrelevant documents. meanwhile, the model predominantly learns implicit rules, resulting in opaqueness of the generation. equip the model with cag capabilities, we introduce a data transformation framework aimed at generating credibility-based dataset, upon which we fine-tune the model. to effectively verify the ability of model credibility-aware generation to handle flawed information, we construct a benchmark from different real-world scenarios. experimental results show that our model can effectively utilize credibility, exhibiting robustness in the face of flawed information and significantly outperforming other models with retrieval augmentation. moreover, through customizing the credibility, our approach can be applied to the real-world scenario including personalized response generation, for which we provide a detailed case study in the appendix a.5.