LONGAGENT : Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration (2024)

Abstract: Large language models (LLMs) have achieved tremendous success in understanding language and processing text. However, question-answering (QA) on lengthy documents faces challenges of resource constraints and a high propensity for errors, even for the most advanced models such as GPT-4 and Claude2.In this paper, we introduce _LongAgent_, a multi-agent collaboration method that enables efficient and effective QA over 128k -token-long documents. _LongAgent_ adopts a _divide-and-conquer_ strategy, breaking down lengthy documents into shorter, more manageable text chunks. A leader agent comprehends the user’s query and organizes the member agents to read their assigned chunks, reasoning a final answer through multiple rounds of discussion.Due to members’ hallucinations, it’s difficult to guarantee that every response provided by each member is accurate.To address this, we develop an _inter-member communication_ mechanism that facilitates information sharing, allowing for the detection and mitigation of hallucinatory responses.Experimental results show that a LLaMA-2 7B driven by _LongAgent_ can effectively support QA over 128k -token documents, achieving 16.42% and 1.63% accuracy gains over GPT-4 on single-hop and multi-hop QA settings, respectively.

## introduction nowadays, the capabilities of large language models (llms) have been rapidly advancing, driven by ever-increasing model sizes and data volumes @xcite . however, the prohibitively high training costs preclude these capability gains from extending to llms' understanding of lengthy texts @xcite . this poses a significant limitation on the practical applications of llms, such as querying information from books, analyzing legal documents or academic papers. researchers have primarily focused on two directions to address this issue. the first direction is to consider positional encoding as a crucial aspect @xcite @xcite , taking techniques like extrapolation or interpolation to enable the positional encoding to handle unseen positions during pre-training. the second category employs more complex mechanisms like recurrent structures @xcite , token selection @xcite , or sliding windows @xcite to enable the limited context window to process longer input texts. despite these efforts, effectively understanding lengthy texts and accurately answering user queries remains a challenging problem @xcite . models tailored for long text processing may compromise their innate capabilities for understanding shorter texts @xcite and tend to overlook critical information situated in the middle of long documents, a phenomenon known as lost in the middle @xcite . in this paper, we propose longagent, a novel approach for long-document qa. longagent employs a divide-and-conquer strategy, breaking down long documents into more manageable smaller text chunks. leveraging the characteristic of aligned llms to follow instructions, longa-gent adopts a multi-agent collaboration approach to effectively support qa on ultra-long documents (over 100k tokens). to alleviate conceptual confusion, we provide a detailed explanation in appendix a that distinguishes longagent from long-context llms and multi-turn retrieval systems. as shown in figure 1 , the agent team consists of a leader agent and multiple member agents. the leader is responsible for understanding the user's question and directing the members to gather clues from their assigned text chunks. depending on the complexity of the question, this process may involve one or more rounds of interaction. the interaction ends when the leader deems the clues sufficient to reason the final answer. managing a large number of members is non-trivial due to the model hallucination. we propose an intermember communication mechanism to identify the members in a hallucinatory state and mitigate their influence to the leader's decision-making. to comprehensively evaluate longagent, we propose needle-in-a-haystack plus. * compared with original needle-in-a-haystack, it can test the model's capability of handling multi-hop qa tasks. we have also tested longagent with all longdocument qa tasks of mainstream longbench @xcite and infinitebench @xcite . the experimental results have demonstrated the effectiveness of longagent. the contributions of this paper are as follows: ## related works several methods have been proposed to extend the positional encoding (pe) for handling longer sequences. initially, approaches like rope and pi @xcite attempted to interpolate position indices within pre-trained limits, but neglected frequency variations. recent advancements include "ntk-aware" (bloc97, 2023a) interpolation and "dynamic ntk" (bloc97, 2023b) interpolation, which address high-frequency component losses. additionally, "ntk-by-parts" (bloc97, 2023c) interpolation outperforms others when fine-tuned on longer-context data. another popular approach for managing longer sequences involves constraining global causal attention to local attention. rerope @xcite truncates context lengths during pretraining and lm-infinite @xcite restricts attention to a chevronshaped window. mohtashami and jaggi (2023) insert landmark tokens after text fragments, while @xcite propose beacon tokens for summarizing fragments. in contrast, our method effectively circumvents the risk of losing valuable contextual information while utilizing only a small amount (hundreds of agent interaction tracks) for fine-tuning, thereby reducing training costs. ## conclusions this paper proposes longagent, a novel longdocument qa approach based on multi-agent collaboration. the proposed inter-member communication mechanism alleviates the member hallucination when they reading documents, thus facilitating effective management by the leader of dozens to hundreds of members. we have also developed needle-in-a-haystack plus to facilitate a comprehensive assessment of the llm's understanding on long documents. our experimental results indicate that longagent offers a promising alternative for long-text qa.