Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective (2025)

Abstract: Retrieval-Augmented Generation (RAG) systems have shown promise in enhancing the performance of Large Language Models (LLMs). However, these systems face challenges in effectively integrating external knowledge with the LLM’s internal knowledge, often leading to issues with misleading or unhelpful information. This work aims to provide a systematic study on knowledge checking in RAG systems. We conduct a comprehensive analysis of LLM representation behaviors and demonstrate the significance of using representations in knowledge checking. Motivated by the findings, we further develop representation-based classifiers for knowledge filtering. We show substantial improvements in RAG performance, even when dealing with noisy knowledge databases. Our study provides new insights into leveraging LLM representations for enhancing the reliability and effectiveness of RAG systems.

## introduction retrieval-augmented generation (rag) is a technique designed to enhance the outputs of large language models (llms) by incorporating relevant information retrieved from external knowledge sources. this approach has been applied to various domains and scenarios @xcite @xcite @xcite @xcite @xcite . it typically operates in two stages: retrieval and generation. in the retrieval stage, relevant knowledge from an external database is retrieved based on the user query. then, in the generation stage, the retrieved information is integrated with the query to form an input for llms to generate responses. in rag, two potential knowledge sources can be utilized to answer input queries: llm's internal knowledge and the external knowledge provided in the context. ideally, these external and internal knowledge sources should be effectively integrated. however, existing works have shown that llms often struggle to identify the boundaries of their own knowledge and tend to prioritize external information over their internal knowledge learned during pre-training @xcite @xcite @xcite @xcite . this characteristic can potentially degrade the generation quality of rag when the quality of external knowledge is low. on one hand, the external knowledge may be misleading @xcite . for instance, @xcite proposed the poisonedrag approach, demonstrating that llms can be easily manipulated into producing incorrect information simply by injecting false answers corresponding to targeted queries into the retrieval database. on the other hand, although some retrieved contexts are semantically similar to a query, they may only superficially related to the topic but lack the answer to the question @xcite . such contexts can distract llms and consequently hurt rag performance. thus, it is important to conduct knowledge checking in rag systems. to achieve this goal, we design the following critical tasks: a straightforward approach to tackle these tasks can directly prompt llms @xcite @xcite . alternatively, we could examine superficial indicators of llms, such as probability scores @xcite or perplexity @xcite . however, based on our evaluation in section 3, we find that none of these methods can effectively accomplish these tasks. recent studies @xcite @xcite have shown that llms' representations exhibit distinct patterns when encountering contrasting high-level concepts, such as harmful versus harmless prompts . this observation prompts us to investigate whether llms' representations also display distinct behaviors and can be leveraged in knowledge checking tasks? to answer this question, we conduct a comprehensive study and analysis of llm representation behaviors regarding the aforementioned tasks, including pcabased checking as well as contrastive-learningbased checking (section 3.1). our analysis reveals that positive and negative samples exhibit different behaviors in the representation space. consequently, representation-based methods demonstrate significantly superior performance in the aforementioned tasks. leveraging these findings, we utilize representation classifiers for knowledge filtering. results show that simple filtering of contradictory and irrelevant information substantially improves rag performance, even in scenarios with poisoned knowledge databases. ## representations for knowledge checking drawing on insights from cognitive neuroscience, previous studies @xcite @xcite have demonstrated the potential of using llms' representation to indicate contrast high-level concepts. in this subsection, we investigate whether llms' representations also show distinct patterns in knowledge checking tasks and can therefore be used to improve their performance. we begin by introducing our representationbased checking procedures in section 3.1, which includes both pca-based checking(rep-pca) and the contrastive-learning-based checking(rep-con). we then visualize and compare the performance of our representation-based methods against tradi-tional approaches across four knowledge checking tasks from section 3.2 to section 3.5 foot_1 . ## representation based context filtering in this section, we investigate how knowledge checking based on representations affect performance of rag systems. ## conclusions this study delves into the knowledge checking in rag systems. to achieve this goal, we identified and proposed four key tasks. through comprehensive analysis of llms' representation behaviors, we found that representation-based methods significantly outperform answer-based or probabilitybased approaches. leveraging these findings, we developed representation-based classifiers for knowledge filtering. results demonstrate that simply filtering of contradictory and unhelpful knowledge substantially improves rag performance.