More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG (2025)

Abstract: Retrieval-Augmented Generation (RAG) enhances the accuracy of Large Language Model (LLM) responses by leveraging relevant external documents during generation. Although previous studies noted that retrieving many documents can degrade performance, they did not isolate how the quantity of documents affects performance while controlling for context length. We evaluate various language models on custom datasets derived from a multi-hop QA task. We keep the context length and position of relevant information constant while varying the number of documents, and find that increasing the document count in RAG settings poses significant challenges for most LLMs, reducing performance by up to 20%. However, Qwen2 maintained consistent results across increasing document counts, indicating better multi-document handling capability. Finally, our results indicate that processing multiple documents is a separate challenge from handling long contexts. We will publicly release the datasets and code upon publication to facilitate further research in multi-document retrieval.

## introduction the rag approach enriches prompts with relevant documents, retrieved according to an input query @xcite . for example, given a question about a certain historical period, rag techniques can retrieve documents related to the time from a large historical corpus. recent work has noted a drop in rag performance when retrieving many documents. for example, in multi-hop qa, llms struggle when the number of retrieved documents grows, even when presented with all the needed information @xcite @xcite . such deficiencies were observed without controlling for the number of tokens figure 1 : more documents, same length. we create various sets containing the same questions but differing in the number of distractor documents. each set includes a multi-hop question, all of the supporting documents that contain the information to answer the question (pink), and varying distractor documents (blue).we begin with either 10 or 20 documents (depending on the dataset) as our full-doc version (left) and then reduce the number of documents while maintaining a fixed context size. when fewer documents are used, the remaining documents are extended (blue without text) so that concatenating them yields the same total length. in which the information is conveyed, i.e., when the number of documents grew, so did the number of overall tokens, thus conflating between the challenge of long context and multi document. in this work, we address the following question: assuming a fixed input length, how is llm performance affected by the number of retrieved documents? this disentangles the challenge of long context from the challenge in processing collections of related documents -which often contain redundancies, conflicting information, and implicit inter-document relations @xcite . from a practical perspective, answer-ing this question can help understand a breadth versus depth tradeoff -i.e., whether to strive to retrieve shorter context out of many documents or whether to aim to retrieve longer context out of fewer documents. an ideal experimental setup would have the exact information conveyed in the same number of tokens across varying number of documents, from a long and self-contained single document to a large, multi-document corpus. we find that the custom sets we constructed from musique @xcite and 2wikimulti-hopqa @xcite , a multi-hop qa datasets, serve as a convenient approximation, allowing us to explore the relationship between long-context and multi-document comprehension in a controlled environment with real-world texts. each instance in both datasets consists of a question and a set of documents, where each document is an excerpt from a wikipedia article retrieved according to the input question. each instance is constructed such that the question can be answered based on only a subset of the input documents, while the other documents serve as realistic distractors in retrieval settings, as they revolve around the question's topic but do not contain information required to answer the question. we vary the number of documents in the input by gradually removing the distractor documents. when removing a distractor document, we respectively extend each of the remaining documents with distracting content from their corresponding wikipedia article. importantly, the process preserves the position of the relevant information within the cotext. this process is illustrated in fig. 1 . if the context length is the sole challenge, we should expect the performance to remain similar regardless of the number of input documents. conversely, if processing multiple related documents presents an additional challenge, we would expect an inverse correlation between performance and the number of input documents. our evaluation of several state-of-the-art models (llama-3.3, qwen2.5, gemma2, and gpt-4o), presented in fig. 2 , indicates that in most cases, reducing the number of documents while keeping the amount of tokens improves performance by up to 10% in musique,and up to 20% in 2wmhqa. an exception is qwen2.5, which may indicate that it better handles multi-document collections. our work has several major implications and avenues for future work. first, from a practical perspective, rag systems should take the number of retrieved documents into consideration, as the introduction of additional documents into the prompt may hurt performance. second, future work should explore novel approaches for multi-document processing, which according to our findings presents a separate challenge from mere long context. such work can make use of our paradigm and data for training and evaluation. ## multi-document evaluation with controlled token count our goal is to understand how the number of retrieved documents affects llm performance when controlling the input length. to this end, we evaluate several models on multi-document multi-hop question answering, which requires models to find relevant information within a given context to answer a specific question. in particular, we make controlled adjustments to the number of documents in the input, while preserving the position of the key information needed to answer the questions, and keeping the context length consistent. our datasets are based on musique @xcite and 2wikimultihopqa @xcite , which we nickname as 2wmhqa. both datasets are multi-hop qa datasets that consist of questions associated with paragraphs (20/10 paragraphs for musique/2wmhqa) sampled from individual documents, retrieved from wikipedia according to the question. of these paragraphs, 2-4 contain the supporting information necessary to answer the question, while the remaining paragraphs serve as realistic distractors in a rag setup, as they are retrieved from related topics but do not contain relevant information to answer the question. fig. 1 shows an example query, and a list of retrieved documents, where three are relevant to the question (marked in pink), and the rest are distractors (marked in blue). we further elaborate on the dataset in section a in the appendix. leveraging musique's and 2wmhqa's structure, we constructed several data partitions to investigate the impact of the number of retrieved documents in a controlled manner. the process involved the following steps: ## conclusions we assess the challenges of multi-document retrieval tasks when varying the number of documents. our results indicate that input that includes more documents complicates the task in an environment of retrieval settings, highlighting the need for retrieval systems to balance relevance and diversity to minimize conflicts. future models could benefit from mechanisms to identify and discard conflicting information while leveraging document variety.