Fine-grained Knowledge Enhancement for Retrieval-Augmented Generation (2025)

Abstract: Retrieval-augmented generation (RAG) effectively mitigates hallucinations in large language models (LLMs) by filling knowledge gaps with retrieved external information. Most existing studies primarily retrieve knowledge documents based on semantic similarity to assist in answering questions but ignore the fine-grained necessary information within documents. In this paper, we propose a novel fine-grained knowledge enhancement method (FKE) for RAG, where fine-grained knowledge primarily includes sentence-level information easily overlooked in the document-based retrieval process. Concretely, we create a disentangled Chain-of-Thought prompting procedure to retrieve fine-grained knowledge from the external knowledge corpus. Then we develop a decoding enhancement strategy to constrain the document-based decoding process using fine-grained knowledge, thereby facilitating more accurate generated answers. Given an existing RAG pipeline, our method could be applied in a plug-and-play manner to enhance its performance with no additional modules or training process. Extensive experiments verify the effectiveness and generality of our method.

## introduction large language models (llms) have achieved impressive advancements across various tasks @xcite and applications @xcite in recent years. however, llms still lack knowledge underrepresented in their training data, especially in up-to-date and domain-specific settings @xcite . to address these limitations, retrieval-augmented generation (rag) techniques have been widely adopted to retrieve external knowledge and enhance llms in diverse tasks, such as question-answering @xcite , infor- ## approach we propose a fine-grained knowledge enhancement method for rag (fke). in this method, we primarily create a fine-grained knowledge retrieval paradigm to retrieve sentence-level finegrained knowledge from the external corpus. subsequently, we develop a decoding enhancement strategy to optimize the decoding process with retrieved fine-grained knowledge. ultimately, these two components are integrated into the basic rag pipeline to further enhance its performance. we will first briefly formulate the rag technique and then elaborate on the two components. ## conclusion in this paper, we propose a novel fine-grained knowledge enhancement method for retrievalaugmented generation (rag) tasks. unlike previous methods primarily relying on document-level retrieved knowledge, we extract and utilize finegrained knowledge to enhance rag performance. concretely, we design a disentangled prompting procedure to retrieve fine-grained knowledge and develop a decoding enhancement strategy to constrain the document-based decoding process with fine-grained knowledge. our method requires no additional modules or training process and can be widely integrated into basic rag pipelines. experiments on two mainstream rag pipelines and four benchmark datasets demonstrate the efficacy and generality of our proposed method.