Lost in the Middle: How Language Models Use Long Contexts (2024)

Abstract: While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.

## introduction language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing @xcite @xcite . these models perform downstream tasks primarily via prompting: all relevant task specification and data to process is formatted as a textual input context, and the model returns a generated text completion. these input contexts can contain thousands of tokens, especially when language models are * work partially completed as an intern at samaya ai. used to process long documents (legal or scientific documents, conversation histories, etc.) or when language models are augmented with external information (relevant documents from a search engine, database query results, etc.; @xcite @xcite @xcite . handling these use-cases requires language models to successfully operate over long sequences. existing language models are generally implemented with transformers @xcite , which require memory and compute that increases quadratically in sequence length. as a result, transformer language models were often trained with relatively small context windows (between 512 and 2048 tokens). recent improvements in hardware (e.g., faster gpus with more memory) and algorithms @xcite @xcite , inter alia) have resulted in language models with larger context windows (e.g., 4096, 32k, and even 100k tokens), but it remains unclear how these extended-context language models make use of their input contexts when performing downstream tasks. we empirically investigate this question via controlled experiments with a variety of state-ofthe-art open (mpt-30b-instruct, longchat-13b (16k)) and closed (openai's gpt-3.5-turbo and anthropic's claude-1.3) language models in settings that require accessing and using information within an input context. in particular, our experiments make controlled changes to the input context size and the position of the relevant information within the input context and study their effects on language model performance. if language models can robustly use information within long input contexts, then their performance should be minimally affected by the position of the relevant information in the input context. figure 1 : changing the location of relevant information (in this case, the position of the passage that answers an input question) within the language model's input context results in a u-shaped performance curvemodels are better at using relevant information that occurs at the very beginning (primacy bias) or end of its input context (recency bias), and performance degrades significantly when models must access and use information located in the middle of its input context. we first experiment with multi-document question answering, which requires models to reason over provided documents to find relevant information and use it to answer a given question; this task mimics the retrieval-augmented generation setup underlying many commercial generative search and question answering applications (e.g., bing chat). in this setting, we control (i) the input context length by changing the number of documents in the input context (akin to retrieving more or less documents in retrieval-augmented generation), and (ii) control the position of the relevant information within the input context by changing the order of the documents to place the relevant document at the beginning, middle or end of the context. we find that changing the position of relevant information in the input context can substantially affect model performance, indicating that current language models do not robustly access and use information in long input contexts. furthermore, we observe a distinctive u-shaped performance curve (figure 1 ); language model performance is highest when relevant information occurs at the very beginning (primacy bias) or end of its input context (recency bias), and performance significantly degrades when models must access and use information in the middle of their input context ( §2.3). for example, when relevant information is placed in the middle of its input context, gpt-3.5-turbo's performance on the multi-document question task is lower than its performance when predicting without any documents (i.e., the closed-book setting; 56.1%). furthermore, we find that models often have identical performance to their extended-context counterparts, indicating that extended-context models are not necessarily better at using their input context ( §2.3). given that language models struggle to retrieve and use relevant information in the multi-document question answering task, to what extent can language models even retrieve from their input contexts? we study this question with a synthetic key-value retrieval task, which is designed to be a minimal testbed for the basic ability to retrieve matching tokens from the input context. in this task, models are given a collection of json-formatted key-value pairs and must return the value associated with a specific key. similar to the multi-document qa task, the key-value retrieval task admits controlled changes to the input context length (adding more key-value pairs) and the position of relevant information. although some models perform the synthetic key-value retrieval task perfectly, other models struggle to simply retrieve matching tokens that occur in the middle of their input context and continue to exhibit a u-shaped performance curve. to better understand why language models struggle to robustly access and use information in their input contexts, we study the role of model architecture (decoder-only vs. encoder-decoder), query-aware contextualization, and instruction fine-tuning ( §4). we find that: our results indicate that prompting language models with longer input contexts is a trade-off-providing the language model with more information may help it perform the downstream task, but it also increases the amount of content that the model must reason over, potentially decreasing accuracy. to better understand this trade-off in practice, we perform a case study with retriever-reader models on open-domain question answering ( §5). in contrast to our controlled multi-document qa task, where the context always contains exactly one document that answers the question, none or many of the top k documents may contain the answer in the open-domain qa setting. when retrieving from wikipedia to answer queries from naturalquestions-open, we find that model performance saturates long before retriever recall saturates, indicating that current models fail to effectively use additional retrieved documents-using 50 documents instead of 20 retrieved documents only marginally improves performance (∼1.5% for gpt-3.5-turbo and ∼1% for claude-1.3). our analysis provides a better understanding of how language models use their input context and introduces new evaluation protocols for future long-context models; to claim that a language model can robustly use information within long input contexts, it is necessary to show that its performance is minimally affected by the position of the relevant information in the input context (e.g., minimal difference in best-and worst-case performance). to facilitate further work on understanding and improving how language models use their input context, we release our code and evaluation data. 1 2 multi-document question answering our goal is to better understand how language models use their input context. to this end, we analyze model performance on multi-document question answering, which requires models to find relevant information within an input context and use it to answer the question. in particular, we make controlled changes to the length of the input context and the position of the relevant information and measure changes in task performance. ## why are language models not robust to changes in the position of relevant information? our multi-document question answering and key-value retrieval results show that language models struggle to robustly access and use information in long input contexts, since performance degrades significantly when changing the position of relevant information. to better understand why, we perform some preliminary investigations into the role of model architecture (decoder-only vs. encoder-decoder), query-aware contextualization, and instruction fine-tuning. ## is more context is always better? our results indicate that prompting language models with longer input contexts is a trade-off-providing the language model with more information may help it perform the downstream task, but it also increases the amount of content that the model must reason over, potentially decreasing accuracy. even if a language model can take in 16k tokens, is it actually beneficial to provide 16k tokens of context? the answer to this question is ultimately downstream task-specific since it depends on the marginal value of the added context and the model's ability to effectively use long input contexts, but we perform a case study with open-domain question answering on naturalquestions-open to better understand this trade-off in existing language models. we use language models in a standard retriever-reader setup. a retrieval system (contriever, fine-tuned on ms-marco) takes an input query from naturalquestions-open and returns the k documents from wikipedia with the highest relevance score. to condition language models on these retrieved documents, we simply include them in the prompt. we evaluate retriever recall and reader accuracy (whether any of the annotated answers appear in the predicted output) as a function of the number of retrieved documents k. we use a subset of naturalquestions-open where the long answer is a paragraph (as opposed to a table or a list). figure 11 presents retriever recall and open-domain qa results. we see that reader model performance saturates long before retriever performance saturates, indicating that readers are not effectively using the extra context. using more than 20 retrieved documents only marginally improves reader performance (∼1.5% for gpt-3.5-turbo and ∼1% for claude-1.3), while significantly increasing the input context length (and thus latency and cost). these results, coupled with the observation that models are often better at retrieving and using information at the start or end of the input contexts, suggest that effective reranking of retrieved documents (pushing relevant information closer to the start of the input context) or ranked list truncation (retrieving fewer documents when appropriate; @xcite may be promising directions for improving how language-model-based readers use retrieved context. ## conclusion we empirically study how language models use long input contexts via a series of controlled experiments. we show that language model performance degrades significantly when changing the position of relevant information, indicating that models struggle to robustly access and use information in long input contexts. in particular, performance is often lowest when models must use information in the middle of long input contexts. we conduct a preliminary investigation of the role of (i) model architecture, (ii) query-aware contextualization, and (iii) instruction fine-tuning to better understand how they affect how language models use context. finally, we conclude with a practical case study of open-domain question answering, finding that the performance of language model readers saturates far before retriever recall. our results and analysis provide a better understanding of how language models use their input context and provides new evaluation protocols for future long-context models.