PECAN : LLM -Guided Dynamic Progress Control with Attention-Guided Hierarchical Weighted Graph for Long-Document QA (2025)

Abstract: Long-document QA presents challenges with large-scale text and long-distance dependencies. Recent advances in Large Language Models (LLMs) enable entire documents to be processed in a single pass. However, their computational cost is significantly high. Retrieval-Augmented Generation (RAG) methods split text into smaller chunks, but they often yield inferior results and may lose global context. Recent approaches that integrate LLMs into RAG via iterative summarization either underutilize LLM capabilities or still incur high computational costs. In this paper, we combine the high accuracy of LLMs with the efficiency of RAG and propose LLM-Guided Dynamic Progress Control with Attention-Based Hierarchical Weighted Graph (PECAN). Our method introduces two key improvements: (1) LLM-Guided Dynamic Progress Control: We leverage LLMs to dynamically control the retrieval process, adjusting the amount of retrieved information based on different queries to achieve a better balance of effectiveness and efficiency. (2) Attention-Guided Retrieval: We propose a novel retrieval method that constructs a hierarchical graph where edges are derived by LLM attention weights. Experimental results demonstrate that PECAN achieves LLM-level performance while maintaining computational complexity comparable to that of RAG methods on two single-document and two multi-document QA datasets.

## introduction long-document question answering (qa) poses several challenges, including handling large-scale texts, uneven information distribution, and longdistance dependencies. while large language models (llms) @xcite @xcite can process entire documents after undergoing a context extension training stage, this approach is computationally intensive and inefficient, as queries often target only small segments of the text. in contrast, retrieval-augmented generation (rag) @xcite @xcite mitigate this by segmenting long documents into chunks and retrieving the most relevant segments. however, studies like longbench @xcite have demonstrated that the performance of rag methods is often inferior to directly feeding the full document into llms @xcite @xcite . rag methods may suffer from incomplete retrieval, lack of global context, and struggle to capture longdistance dependencies. recent methods, such as memwalker @xcite and raptor @xcite employ llms to iteratively summarize text into a tree. these methods use concise summaries generated by llm to capture the global context while leveraging rag to retrieve detailed information. however, memwalker still incurs a high computational cost since it repeatedly relies on the llm to determine retrieval strategies, while raptor only uses the llm for summarization and heavily depends on traditional rag. in this paper, we propose a new method, llm-guided dynamic progress control with attention-based hierarchical weighted graph (pecan), combining llm accuracy and rag efficiency. unlike existing approaches, our method dynamically structures and searches information using an attention-guided hierarchical graph. our approach comprises two stages: attention graph construction and dynamic graph search. in the attention graph construction stage, we prompt an llm to generate multiple information points (ips), each typically focusing on a single or a few events. these ips form a hierarchical weighted directed acyclic graph (hwdag), where llm attention weights between the generated nodes and the input nodes for example, node #11 is more closely related to nodes #5 and #6, while node #12 is more associated with nodes #6 and #7. in this instance, the llm extracts two events from three text chunks, with node #6 referencing both events. (for brevity, long text are truncated.) building on this graph, during the dynamic graph search stage, instead of retrieving fixedsize text chunks, we use llm attention to identify relevant high-level summaries and trace back to detailed information. this method effectively utilizes the llm's knowledge, capturing both the generated summarization and the dynamic flow of information between high-level summaries and their corresponding detailed content within the graph. in addition, previous rag methods retrieve a static number of top-ranked text chunks, which fails to account for the fact that different queries may require varying amounts of information. this fixed setting might lead to computational waste for simple queries and insufficient retrieval for more complex ones. we introduce a novel method called dynamic progress control, which allows the llm to adaptively determine the amount of information needed per query. this approach essentially reallocates computational resources based on the llm's knowledge to achieve a better balance of effectiveness and efficiency. meanwhile, previous inputs are kv-cached so that new documents can be appended without reprocessing the entire input. as a result, our method can effectively handle queries that require varying amounts of information, particularly when the necessary details are distributed across different parts of the graph. this avoids additional computational overhead and resolves the challenge of determining the optimal number of chunks to retrieve. in experiments, pecan achieves a good balance between effectiveness and efficiency, attaining llm-level performance while keeping computational costs on par with rag methods. in summary, our contributions in this paper include: ## related work long-context language models recent longcontext language models have focused on overcoming fixed context window limitations, primarily through positional interpolation and training on full-length texts. chen et al. (2023b); peng et al. (2024); fu et al. (2024) fine-tuned models on longer inputs and extended rotary position embedding (rope; su et al., 2023) for extended contexts. longrope @xcite performs direct extrapolation by rescaling rope with varied interpolation. longalign @xcite constructs a long-context dataset, adopting packing and sorted batching strategies. pose @xcite manipulates position indices by skipping bias terms. skipalign @xcite synthesizes long-range dependencies from the aspect of position indices. infini-transformer @xcite handles infinitely long inputs using compressive memory, masked local attention, and long-term attention mechanisms. our primary focus is on effectively leveraging llm capabilities, and the llms employed in these techniques can serve as the base models in our framework. rag traditional retrieval techniques, such as tf-idf (jones, 1972) and bm25 @xcite , rely on word-term matching. subsequently, deep learning-based retrieval methods quickly gained popularity @xcite @xcite @xcite . among these, dpr @xcite encodes queries and documents as dense embeddings. colbert @xcite produces multi-vector representations. dhr @xcite leverages both document-level and passage-level semantics. cpt-text @xcite utilizes contrastive pre-training on unsupervised data. nci @xcite directly generates relevant document identifiers. retro @xcite conditions on document chunks based on local similarity. hhr (arivazhagan et al., 2023) combines sparse and dense retrieval methods. dragon @xcite uses contrastive learning and data augmentation to train a model, achieving state-of-the-art retrieval performance. llm and rag additionally, with the rise of llms, several studies have explored combining llms with rag. genread @xcite prompts llms to generate contextual documents. recite @xcite retrieves from the llm's internal memory. kgp @xcite builds a knowledge graph with the llm navigating. recently, memwalker @xcite constructs tree-based summaries and uses llms to navigate. raptor @xcite creates tree-based summaries and leverages embedding similarities to select the most relevant nodes at each level for retrieval. in contrast, our approach makes greater use of llm knowledge by employing attention mechanisms to construct a graph and perform the search, allowing navigation along multiple paths and termination at any depth. ## methodology our method consists of two main steps: attention graph construction and dynamic graph search. in the attention graph construction stage, illustrated in figure 1 , we utilize the llm's attention weights to build a hierarchical weighted directed acyclic graph (hwdag) from documents. this is a one-time preprocessing step for each document, after which it can be reused for any query to that document. in the dynamic graph search stage, as illustrated in figure 2 , we dynamically control the volume of retrieved nodes and perform a search guided by the llm. problem setup the input consists of two parts: a document and multiple queries. the document is processed only by the attention graph construction stage, while the queries are handled by the dynamic graph search stage. initially, a graph @xmath0, e) is constructed from a document, where v denotes the collection of nodes and e denotes the collection of edges. each node v l @xmath1. each edge e i,@xmath2, derived from llm attention weights. each v i contains tokens {x i n } n i @xmath3, where n i denotes the number of tokens in node v i . the dynamic graph search stage then uses the graph g to generate a response for each query q. ## conclusion in this paper, we introduced pecan, a novel retrieval method that incorporates two key innovations. the llm-guided dynamic progress control adjusts the amount of information retrieved based on the query, achieving a better balance between effectiveness and efficiency. the attention-guided retrieval constructs a many-to-many hierarchical weighted directed acyclic graph using llm attention weights to guide the search. each node represents an information point that focuses on one or a few events, enabling the model to effectively track them. empirical results demonstrate that pecan achieves llm-level performance while maintaining rag-level computational complexity.