Can’t Remember Details in Long Documents? You Need Some R & R (2024)

Abstract: Long-context large language models (LLMs) hold promise for tasks such as question-answering (QA) over long documents, but they tend to miss important information in the middle of context documents [(Liu 2023)](https://arxiv.org/abs/2307.03172). Here, we introduce *R&R*—a combination of two novel prompt-based methods called *reprompting* and *in-context retrieval* (ICR)—to alleviate this effect in document-based QA. In reprompting, we repeat the prompt instructions periodically throughout the context document to remind the LLM of its original task. In ICR, rather than instructing the LLM to answer the question directly, we instruct it to retrieve the top k passage numbers most relevant to the given question, which are then used as an abbreviated context in a second QA prompt. We test R&R with GPT-4 Turbo and Claude-2.1 on documents up to 80k tokens in length and observe a 16-point boost in QA accuracy on average. Our further analysis suggests that R&R improves performance on long document-based QA because it reduces the distance between relevant context and the instructions. Finally, we show that compared to short-context chunkwise methods, R&R enables the use of larger chunks that cost fewer LLM calls and output tokens, while minimizing the drop in accuracy.

## introduction large language models (llms) have taken natural language processing (nlp) by storm and are increasingly being incorporated into user-facing applications. a user interacts with an llm via "prompting", where the user writes a free-form prompt that instructs the llm to perform some task, such as answering a question based on a document included in the prompt. the prompt, however, is usually limited to a context window with a set maximum number of "tokens" (subwords used as the vocabulary of the input language), which poses a challenge for tasks such as question-answering a for full prompts). in-context retrieval (blue) abbreviates the document to k passages max (based on the returned ids), and reprompting (red) every r tokens helps to mitigate the "lost in the middle" effect. entities in braces are substituted with text, with {. . . } replaced with the instructions nearly verbatim. qa is done with the abbreviated context in a second llm call. we have recently witnessed the release of very long-context llms such as gpt-4 turbo @xcite and claude-2.1 (anthropic, 2023) supporting context windows of 128k and 200k tokens respectively. while these llms support long contexts, the quality of their responses (e.g., qa accuracy) tends to deteriorate as the input prompt becomes very long. even at 16k-token context, @xcite found that an llm's accuracy on document-based qa significantly drops when relevant context is located in the middle of the document, as opposed to the beginning and end-a phenomenon termed the "lost in the middle" effect. there is thus a demand for techniques that can mitigate this loss of performance, thereby improving the efficacy of long-context llms. moreover, as llms such as gpt-4 turbo and claude-2.1 are proprietary, prompt-based approaches compatible with black-box models are particularly desirable as they can be put into practice immediately. in this paper, we propose a prompt-based method called r&r to alieviate the lost-in-the-middle effect in long document-based qa (fig. 1 ). r&r, in turn, is a combination of two novel prompt-based methods we call reprompting and in-context retrieval. in reprompting, the instructions to answer the question are repeated nearly verbatim periodically throughout the context document. our motivation for reprompting is the observation that in the "lost in the middle" experiments @xcite , the instructions to answer the question appear both before and right after the context document, leading us to wonder: could qa accuracy, in part, be related to the proximity of relevant context in the document to the task instructions? our hypothesis is that repeating the task instructions right before each piece of relevant context in the document will improve qa accuracy. since in practice, however, we do not know the locations of relevant context apriori, we propose to repeat the instructions or "reprompt" uniformly. in-context retrieval (icr) draws inspiration from retrieval-augmented generation (rag) and recent literature on context abbreviation @xcite . in icr, rather than having a long-context llm answer a question directly, we first prompt the llm to retrieve some number of passages from the context document that are most relevant to the question. we aggregate the retrieved passages into an abbreviated context document, and then do short-context qa in a second llm call. our underlying hypothesis is that passage retrieval is a simpler task than direct qa, as we are able to prioritize recall over precision. r&r is then a two-call method where we reprompt the instructions for retrieval during the first call in icr. in our experiments, we find r&r to indeed be beneficial and find evidence to support our motivating hypotheses. we also evaluate r&r in the setting of limitedcontext llms that must process long documents in chunks. in scenarios where relevant context is concentrated in the document and qa resembles information extraction, a chunkwise approach is likely to provide a strong baseline, thus raising the question: are long-context llms with reprompting really better than limited-context llms with chunk-based workarounds? as a strong baseline, we propose "chunked icr", where we first partition the context document into chunks and then perform icr within each chunk in independent short-context llm calls. we aggregate the retrieved passages across all chunks into an abbreviated document to do the final qa. we expect on extraction-like qa tasks, smaller chunks may facilitate finer retrieval at the cost of more llm calls, thus introducing an accuracy vs. cost trade-off. we find, however, that our method r&r softens the trade-off by boosting accuracy on larger chunks, often reducing the number of llm calls and output token usage with little to no drop in accuracy. our key contributions are the following: (1) we use gpt-4 turbo and claude-2.1 to show that reprompting and icr each independently improve performance on four long-context qa tasks, and our combined r&r approach improves performance further foot_0 ; (2) we analyze why reprompting works and show that the proximity between instructions and relevant context plays an important role in performance; and (3) we compare long-context qa with chunk-based approaches with shorter context windows and show that r&r can minimize accuracy loss at higher chunk sizes, thereby reducing llm usage cost in real-world use cases. ## related work long context there are a number of avenues in the literature that aim to achieve llms that perform well at long context. to the extent that long-context llms suffer from limited long samples during training, an avenue is to fine-tune llms on longcontext data so long as one can circumvent complexity challenges @xcite . alternatively, many avenues propose modifications to the llm architecture itself, including landmark attention @xcite , positional interpolation @xcite with rope embeddings @xcite , and parallel context windows @xcite . these methods, however, cannot be applied to boost the performance of existing black-box long-context llms such as gpt-4 turbo. there is little literature on prompt-based methods intended for such black-box models, but one example is "found in the middle" @xcite , where permutationequivariance of llms for document reranking is promoted with self-consistency, thereby reducing position bias in the context. however, this method likely suffers from complexity issues at very long contexts. in contrast to all these methods, we investigate what can be achieved with a set of simple prompt-templates with minimal complexity. reprompting there is literature to support the idea of reprompting in addition to the motivation given in sec. 1. for example, attention tends to focus on repeated tokens due in part to how it is trained @xcite , and it has been suggested that sycophancy (where the llm "seeks approval" in unwanted ways) is related to this problem of repetition @xcite . in reprompting, we seek to exploit this weakness by repeating the task instructions to ensure the llm does not "forget" its goal deep in the middle of the document. indeed, repeating the question has been previously shown to be beneficial @xcite , but in this previous work , the question is only repeated once and almost immediately following its initial statement. in contrast, we propose to repeat the question periodically throughout the context document and hypothesize that its mechanism is reducing distance between question and relevant context, as opposed to just repetition. retrieval our inspiration for in-context retrieval (icr) was the work of @xcite , who propose a prompt-based method to reduce sycophancy in qa where an llm is asked to extract the unbiased parts of the context document. in contrast, however, our motivation is to mitigate the "lost in the middle effect" and not reduce sycophancy per se. moreover, rather than instructing to do extraction in general, our prompt is more akin to traditional retrieval of chunks. retrieval in general has a long history in nlp (see e.g., the work of @xcite for a review from a text ranking perspective), but more recently retrieval and llms have come together in retrieval-augmented generation (rag), where e.g., a long context document can first be chunked, and a retriever is used to effectively abbreviate the document for a downstream llm @xcite . with long-context llms emerging, @xcite address the important question: how do long-context llms compare to rag? they find that a 4k-context-window llm enhanced with rag is able to achieve performance comparable to that of a 16k-context-window llm, thus demonstrating the power of chunk-based approaches. here, we ask a similar question but replace rag with icr as we are interested in a comparison to what short-context llms can achieve natively. moreover, we consider contexts reaching 80k tokens, where the question of long context vs. chunking is unclear. finally, the extraction of or focus on only key parts of a document is the basis for agent-based prompting methods such as memwalker @xcite , gist memory @xcite , and pearl @xcite , where an llm navigates, reads, and/or records chunks of long documents through a series of selected actions. in contrast, our proposed method r&r is very simple as it only chains together two prompts. we use icr because it is clear how we can apply reprompting to the icr prompt, and indeed one objective of this study is to test the simple idea of reprompting; how reprompting can be applied to the above agent-based methods is less clear and this is left for future work. ## conclusion we introduced the prompt-based method r&r to investigate how far we could push the performance of long-context llms on document-based qa. we found our nethod to be effective at mitigating the "lost in the middle effect" (see app. b for detailed results), and our analysis suggests that the mechanism underlying reprompting could be the minimization of distance between relevant context and the task instructions. for extraction-like qa tasks, chunkwise approaches provide a strong baseline, and indeed r&r can be performed within chunks. nevertheless, even in this setting, we found reprompting to be beneficial as it often allows larger chunks to be used (thus requiring fewer llm calls and less token usage) while reducing the drop in accuracy. r&r thus softens the accuracy vs. cost trade-off of chunkwise approaches and enables cost-savings in practical applications where accuracy is paramount. future directions of this work are numerous. we could combine r&r with other prompt-based methods to boost performance further; e.g., we could ask the llm to provide a brief justifica-tion of each page it retrieves, thus encouraging the llm to retrieve more wisely. perhaps more interestingly, we could consider "in-context chunking", where we ask the llm to retrieve 5 pages from every 50-page range, for example; this could help to temper the accuracy vs. cost trade-off further. as a different direction, we could investigate the utility of reprompting on tasks requiring a more wholistic understanding of the document such as summarization, where position bias has also been observed @xcite . finally, while these are only prompt-based methods, understanding their benefits and limitations could help to elucidate the complex behavior of long-context llms and could inspire architectural modifications to improve long-context llms.