Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (2025)

Abstract: Retrieval-augmented generation (RAG) encounters challenges when addressing complex queries, particularly multi-hop questions. While several methods tackle multi-hop queries by iteratively generating internal queries and retrieving external documents, these approaches are computationally expensive. In this paper, we identify a three-stage information processing pattern in LLMs during layer-by-layer reasoning, consisting of extraction, processing, and subsequent extraction steps. This observation suggests that the representations in intermediate layers contain richer information compared to those in other layers. Building on this insight, we propose Layer-wise RAG (L-RAG). Unlike prior methods that focus on generating new internal queries, L-RAG leverages intermediate representations from the middle layers, which capture next-hop information, to retrieve external knowledge. L-RAG achieves performance comparable to multi-step approaches while maintaining inference overhead similar to that of standard RAG. Experimental results show that L-RAG outperforms existing RAG methods on open-domain multi-hop question-answering datasets, including MuSiQue, HotpotQA, and 2WikiMultiHopQA. The code is available in https://anonymous.4open.science/r/L-RAG-ADD5/.

## introduction large language models (llms) have made significant advancements in natural language processing, demonstrating exceptional performance across a variety of tasks @xcite , including complex multi-hop tasks. however, despite their success, llms often generate factually incorrect answers as they rely solely on their internal parametric memory, which struggles with tasks requiring external knowledge @xcite . to address this limitation, retrieval-augmented generation (rag) why did reginald le borg die? reginald le borg was ... died from a heart attack. heart attack. heart attack. intermediate representation bingo. why did the director of film the dalton girls die? bingo. but it is expensive and slow. the dalton girls is directed by reginald le borg ... reginald le borg was ... died from a heart attack. the dalton girls is directed by reginald le borg ... frameworks @xcite have been developed, utilizing semantic similarity metrics to retrieve relevant external documents, which are subsequently integrated into the generation context. nevertheless, rag still faces challenges, particularly in handling complex queries that require multi-hop document retrieval. multi-step methods, such as self-ask @xcite and iter-retgen @xcite , address the challenges of multi-hop reasoning and document retrieval by decomposing complex queries into manageable steps and iteratively refining the retrieval process @xcite @xcite . over multiple iterations, the response is refined until a final answer is produced. as illustrated in figure 1 , while multi-step methods significantly improve retrieval and reasoning capabilities, they often result in higher computational overhead due to their iterative nature. several variants aim to address this issue. adaptive-rag @xcite simplify the reasoning process for less complex queries. however, for more complex queries, multiple full inference iterations are still required. efficient rag @xcite get rid of relying on llm calls at each step, using the smaller model to generate the special token for information processing. despite these advancements, such methods still tend to address the problem in generating human understandable query, overlooking the nexthop information contained in intermediate representations during reasoning. there is a question we try to find out that how can we leverage the information contain to achieve performance comparable to multi-step approaches while maintaining a low inference overhead similar to vanilla rag? llms demonstrate the ability to extract entities and their relationships during inference @xcite , including implicit reasoning for bridge entities in multi-hop tasks @xcite @xcite . moreover, studies suggest that intermediate representations within llms often contain richer information than the final layer @xcite . through an analysis of llm weight matrices using singular value decomposition (svd), we identify the computational regime of llms: initial layers primarily perform raw information extraction, middle layers engage in contextual processing, and final layers focus on extracting information for token generation. this analysis suggests that intermediate layer representations, rather than those in the final layer, are more suitable for document retrieval. in light of this, we introduce layer-wise retrieval-augmented generation (l-rag), which leverages intermediate representations from the middle layers to retrieve external information. as illustrated in figure 2 , we train a contrieverbased representations retriever to effectively utilize these intermediate representations for retrieving higher-hop documents. our experimental results demonstrate that the proposed method achieves competitive performance while maintaining computational efficiency comparable to vanilla-rag systems. our key contributions include: ## analysis of llm reasoning mechanisms we systematically analyze how llms process information across layers, revealing distinct reasoning phases through singular value decomposition and empirical validation. ## method we present layer-wise retrieval-augmented generation (l-rag), an innovative framework that enhances retrieval-augmented llms by leveraging intermediate representations to improve the efficiency of multi-hop document retrieval, particularly at higher hops. in section 3.3, we discuss how llms extract valuable information from chaotic representations, process this information into comprehensive representations, and ultimately derive token-generation information for producing tokengenerating representations. based on this understanding, we proposed an overview of the l-rag framework as shown in figure 2 . this section outlines the pipeline structure in section 4.1, followed by the detailed implementation procedures in section 4.2.