L oo GLE : Can Long-Context Language Models Understand Long Contexts? (2024)

Abstract: Large language models (LLMs) are typically limited to processing texts within context window size, which has spurred significant research efforts into enhancing LLMs’ long-context understanding as well as developing high-quality benchmarks to evaluate the ability. However, prior datasets suffer from short comings like short length compared to the context window of modern LLMs; outdated documents that might have data leakage problems; and an emphasis on short dependency tasks only. In this paper, we present LooGLE , a Long Context Generic Language Evaluation benchmark. It features documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning varying dependency ranges in diverse domains. Human annotators meticulously crafted over 1,100 high-quality question-answer (QA) pairs with thorough cross-validation for a most precise assessment of LLMs’ long dependency capabilities. We conduct a comprehensive evaluation of representative LLMs on LooGLE . The results indicate that most LLMs have shockingly bad long context ability and fail to capture long dependencies in the context, even when their context window size is enough to fit the entire document. Our results shed light on enhancing the “true long-context understanding” ability of LLMs instead of merely enlarging their context window.

## introduction the pursuit of enabling large language models (llms), such as chatgpt @xcite , to go beyond their limited context window size to process, comprehend, or even learn from long-context textual information @xcite @xcite is inevitable for next-generation of language intelligence attributed to its wide applications on realworld scenarios, such as domain-specific knowledge understanding, long-context conversational generation, long story or code generation, etc. meanwhile, there is an increasing need for highquality benchmarks with much longer text lengths and more challenging tasks to provide comprehensive evaluations. however, traditional benchmarks @xcite often fall short in text length with an average number of thousand words @xcite . besides, existing benchmarks automatically collect possibly outdated documents from existing datasets published a few years ago @xcite @xcite , which might lead to data leakage @xcite in pre-trained llms and make the evaluation inaccurate. further, the long texts are often restricted to domain-specific articles, making it hard to evaluate llms' ability on generic tasks and domains. finally, it is important to note that tasks in existing benchmarks are primarily short dependency tasks, which only require llms to retrieve answers from one specific sentence or paragraph, without really testing llms' ability to address long dependency questions. to mitigate the shortcomings of existing datasets, in this paper, we introduce a novel benchmark loogle to evaluate the long context understanding abilities of llms illustrated in fig. 1 . our benchmark has the following advantages: • extra-long documents. it contains 776 latest gathered and extremely long documents with an average of 19.3k words. there are over 6,448 test instances without distribution bias for a more generalized assessment. on one hand, they can better evaluate llms' capability on memorizing and understanding longer text that is far beyond their context window size. on the other hand, the excessive length is well suited to the common usage of long text scenarios. evaluations on loogle revealed: (i) commercial models outperformed open-sourced models; ## related work existing models for long context there are increasing research interests in extending llms' context window size such as utilizing recurrent memory, efficient attention, external memory and etc. @xcite @xcite . improved transformer architectures @xcite @xcite are developed to decrease memory complexity to efficiently model longer texts. for recurrent transformer @xcite , history information of previous segments will be leveraged when the subsequent segment is fed into the model without fragmentation problem. fine-tuned models on long documents @xcite are also explored, but they are often effortcosting and face difficulties in collecting ground truth fine-tuning data for long text tasks. there are also works incorporating external memory, using compression techniques and task-oriented process optimization. ## the loogle benchmark our loogle benchmark consists of 3 sources: scientific papers, wikipedia articles, movie and tv scripts, covering various topics and categories. all the documents in loogle are after 2022 and filtered by a length of over 10k words. by replicating the methodology proposed in this paper, they can be collected easily and periodically. based on that, we generate two main types of tasks: short dependency and long dependency tasks. the generation of short dependency tasks can be seen in appendix b. statistics of the source documents and generated tasks can be found in ## benchmark llms on loogle commercial models gpt4-32k, gpt4-8k, gpt3.5-turbo-16k (chen et al., 2023b) are models developed by openai foot_2 . we use the models of version 0613 by default. open-source models there has also been a great effort from the open-source community to address the long context problem and we chose representative instruct-tuned models as baselines. llama2-7b-32k-instruct @xcite is finetuned from meta's original llama2-7b @xcite using position interpolation @xcite . chatglm2-6b-32k @xcite integrates @xcite extending to 32k context length. longllama-3b-instruct, derived from openllama, has been finetuned using focused transformer @xcite . rwkv-4-14b-raven @xcite is notable for its architectural fusion of recurrent neural networks (rnns) to accommodate 8k context length. based on the conclusion from @xcite , we artificially truncate the input document to certain sizes (all not larger than above mentioned models) by concatenating the head and tail of the input. retrieval-based method @xcite augments the llm by incorporating external memory, allowing relevant information to be retrieved using a specific query. in our experiment, we employed the data framework llamaindex foot_3 with default embedding model text-embedding-ada-002 and language model textdavinci-003 @xcite . here we employ the same truncation strategy mentioned above. ## discussion in this paper, we propose a new dataset, loogle to enable the evaluation of long context understanding of llms. loogle addresses the limitations of previous datasets, including much longer text size, up-to-date documents without pre-training, multisources and categories and most remarkably well designed and annotated tasks with diverse context dependency. our comprehensive evaluations reveal the huge limitations of existing llms in understanding and reasoning inter-dependency among long texts even with much longer context windows. besides, there are significant gap between commercial models with open sourced models whereas they both fail in long dependency tasks in our benchmark. the results show that our dataset functions as a valuable reference for long context assessment and poses potential explorations for llms improve-ment. limitation and future work permissible lengthy input text is inevitable in wide applications and extensive efforts have been devoted to enlarging the supported context length of llms. current long context llms have spent huge effort to "memorize" extremely long sequences with efficient attention methods, long-term memory mechanism, length extrapolation mining and etc. @xcite @xcite . however they still suffer from tremendous information loss with low precision. in addition, our extensive evaluations unveil substantial limitations in the capacity of existing llms to understand and reason about the intricate interdependencies in lengthy texts, even when provided with considerably extended context windows. a few complex capabilities including mathematical computation, temporal awareness of the central storyline, intricate comprehensive understanding, and multi-hop reasoning are essentially needed for current llms in long dependency tasks. in the future, we will further explore the development of effective long context llms with practically optimized performance. despite our thorough experiments to evaluate the long-context llms, it may not be precisely reflected due to several factors, including the limitation of the size and diversity of existing models, a lack of adequate inference resources, and the potential for more efficient prompts and techniques to further stimulate the model's abilities. furthermore, since there is an increasing number of long-context llms designed to solve the problem, we will keep updating our long dependency dataset periodically and involving the up-to-date llms for more comprehensive evaluations.