Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts? (2024)

Abstract: While auxiliary information has become a key to enhancing Large Language Models (LLMs), relatively little is known about how LLMs merge these contexts, specifically contexts generated by LLMs and those retrieved from external sources.To investigate this, we formulate a systematic framework to identify whether LLMs’ responses are attributed to either generated or retrieved contexts.To easily trace the origin of the response, we construct datasets with conflicting contexts, i.e., each question is paired with both generated and retrieved contexts, yet only one of them contains the correct answer.Our experiments reveal a significant bias in several LLMs (GPT-4/3.5 and Llama2) to favor generated contexts, even when they provide incorrect information.We further identify two key factors contributing to this bias: i) contexts generated by LLMs typically show greater similarity to the questions, increasing their likelihood of being selected; ii) the segmentation process used in retrieved contexts disrupts their completeness, thereby hindering their full utilization in LLMs.Our analysis enhances the understanding of how LLMs merge diverse contexts, offers valuable insights for advancing current LLM augmentation methods, and highlights the risk of generated misinformation for retrieval-augmented LLMs.

## introduction recent advancements in augmenting large language models (llms) with auxiliary information have significantly revolutionized their efficacy in knowledge-intensive tasks @xcite . this auxiliary information can originate from contexts generated by llms or retrieved from external sources. for the former, @xcite ; @xcite ; @xcite gpt-4 gpt-3.5 llama2-13b llama2-7b instruct llms to initially generate a background context tailored to the given question, which then serves as the basis for producing the final answer. in contrast, retrieval-augmented approaches @xcite include relevant passages retrieved from external corpora, such as wikipedia, as context, thereby notably enhancing llms' capability to address challenges like knowledge updates @xcite and long-tail knowledge @xcite . recent hybrid methods have attempted to integrate these two types of contexts to further improve performance in tasks like question answering (qa) @xcite @xcite . however, we observe an abnormal phenomenon: in certain cases, models relying solely on retrieval contexts succeeded, whereas hybrid approaches failed, as depicted in figure 1 . this observation implies that llms may struggle to effectively integrate diverse types of contexts, overlooking correct information in retrieved contexts. to uncover the reasoning behind this, this study investigates the underlying mechanisms by which llms process the two types of contexts, especially when they contain conflicting information. to facilitate the investigation, we proposed a systematic framework to dissect the process by which llms merge generated and retrieved contexts. we curated tailored context-conflicting (cc) datasets in which each question is accompanied by a pair of generated and retrieved contexts. these contexts are deliberately designed to be inconsistent, with only one containing the correct answer to its corresponding question. these datasets provide a solid foundation for determining whether llms utilize retrieved or generated context to produce responses in qa tasks. in this paper, we conducted a series of controlled experiments using our uniquely designed datasets to empirically study this question, focusing on several state-of-the-art closed (gpt-3.5/4) and open (llama2-7b/13b) llms. surprisingly, our findings reveal a pronounced bias in llms to favor generated contexts across various llms ( §4.2), even when the generated contexts offer incorrect information while the retrieval contexts hold the correct answers. further analysis shows that the bias is prevalent across various retrieval models ( §4.3). these findings highlight a critical challenge for existing llms in effectively merging contexts from diverse sources, especially in light of the increasing prevalence of llm-generated content on the internet, which may contain potential misinformation @xcite . through extensive empirical analyses, we reveal several factors contributing to this bias: (i) confirmation bias @xcite is not a key factor ( §5.1): llms maintain a significant preference for generated contexts even when they are inconsistent with llms' parametric knowledge. (ii) text similarity is a significant factor ( §5.2): compared to retrieved contexts, generated contexts typically exhibit a higher degree of similarity to the questions, even when they contain incorrect information. the samples with a larger similarity gap between generated and retrieved contexts exhibit a more pronounced bias. (iii) semantic completeness matters ( §5.3): llms tend to favor contexts with semantic integrity. the segmentation process used in retrieved contexts may disrupt their completeness, thereby hindering their full utilization in llms. this finding emphasizes the need for optimizing passage segmentation in current retrieval systems. in summary, we explore the challenges llms face when utilizing conflicting contexts and make the following contributions: • we uncover a critical bias in existing llms, where they heavily rely on generated contexts regardless of correctness, indicating an insufficient use of diverse information sources. ## background & study formulation in this section, we briefly review three categories of llms augmented with auxiliary information for qa tasks: retrieval-augmented, generationaugmented, and hybrid approaches. additionally, we introduce the framework of our investigation. ## experimental setup to facilitate our investigation into how llms merge generated and retrieved contexts, this sec-tion elaborates on the construction of our contextconflicting datasets and the evaluation metric. ## how llms merge contexts? this section conducts experiments on the constructed datasets to investigate the merging mechanism of the llms in hybrid approaches. we first consider a typical setting where the generator and reader share a single llm, to explore how llms merge retrieved and self-generated contexts ( §4.1). then, we extend our experiments to include more flexible combinations of generator and reader ( §4.2), and various retrieval models ( §4.3), to investigate their effects. ## why llms prefer generated contexts in this section, we investigate the causes of the observed bias from several perspectives: the effect of parametric knowledge in §5.1, context similarity to the question in §5.2, and context completeness in §5.3. this section primarily presents the result for llama2-13b and gpt-3.5, while the results for the other llms are provided in appendix b. ## conclusion and future work in this study, we propose a framework to investigate the underlying mechanisms by which llms merge retrieved and generated contexts. our results reveal a pronounced bias towards generated contexts in several llms (gpt 3.5/4 and llama2-7b/13b). we further identify two key factors that may contribute to this bias: higher similarity between generated contexts and questions, and the semantic incompleteness of retrieved contexts. our insights highlight the critical need for advanced integration methods that can validate and leverage information from both sources, moving beyond the current overreliance on generated contexts. while this study primarily focuses on integrating generated and retrieved contexts, the observed bias also highlights a critical risk of retrieval-augmented llms. with llm-generated content increasingly prevalent on the internet, retrieval results may include generated contexts @xcite . as this scenario becomes more common, the observed bias of llms toward generated contexts implies their susceptibility to misinformation and malicious attacks generated by llms. this raises concerns about the security of retrieval-augmented systems, a critical problem that is gaining attention in recent works @xcite . addressing the challenges posed by the widespread presence of generated content, such as the detection of such content, represents a promising direction for future research.