L ong B ench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024)

Abstract: Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs’ long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Upon comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial model (GPT-3.5-Turbo-16k) outperforms other open-sourced models, but still struggles on longer contexts. (2) Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding. (3) Context compression technique such as retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability.

## introduction the field of nlp has long sought to endow machines with the ability to understand and reason † corresponding authors over a long context. tasks such as summarization and question answering based on books, reports, and documents, and code generation at the repository level demand the ability to model long context sequences that span thousands or even tens of thousands of tokens in length. however, many of today's large language models can only comprehend and generate texts a few thousand tokens long, leaving room for potential improvements in processing longer contexts. more recently, there has been an increasing effort to improve large language models' capabilities on long context understanding. these methods include extending the context window @xcite , utilizing recurrent memory @xcite , using sparsed attention @xcite , and augmenting with an external memory @xcite . however, unlike in short context, where a multitude of multi-task benchmarks are available for a multi-aspect evaluation @xcite , there is no such benchmark on longer context. to facilitate further research in this direction, we propose longbench, the first bilingual, multitask benchmark tailored for long context understanding. longbench is composed of 6 major task categories and 21 different tasks, covering key long-text application scenarios including multidocument qa, single-document qa, summarization, few-shot learning, code completion, and synthetic tasks. in addition, longbench includes different languages (chinese and english) to provide a more comprehensive evaluation of the large models' bilingual capabilities on long contexts. with overview statistics shown in figure 1 , long-bench contains 4,750 test instances, with an average length of 6,711 words and 13,386 characters for english and chinese instances, respectively. all 21 datasets in longbench are standardized into a unified format, among which 6 are directly extracted from the original datasets provided by previous studies, 10 are built based on the original datasets and processed to be suitable for long context evaluation, and 5 are created and annotated by us. we are fully aware of the potentially high costs involved in the model evaluation process, especially in the context of long context scenarios (such as manual annotation costs or api call costs). therefore, we adopt a fully automated evaluation method, where we utilize automatic metrics such as rouge-l and f1 to measure the similarity of the outputs to the groundtruth answers. we conduct a comprehensive evaluation of 8 models on longbench. the empirical results provide insightful conclusions about the multi-task capability of current models in terms of long context comprehension. additionally, to better disentangle the models' long context ability from their task ability, we construct longbench-e that features a more even length distribution, thus suited for gauging each model's capability across various context lengths. results on longbench-e reveal that although some models are trained or fine-tuned on longer contexts, they still experience a significant decline in performance as the context length increases. we also investigate the effect of retrievalbased and summarization-based context compression techniques. our results demonstrate that these approaches are beneficial only to models that exhibit weaker capability on long contexts. ## related work long context modeling techniques. we first discuss some popular lines of methods that aim to tackle long context understanding. these studies are mainly aimed at solving two key challenges in long text modeling, including the high runtime overhead on longer context, and the catastrophic forgetting phenomenon when processing long sequence. a series of studies focus on how to make transformers more efficient and unforgetful @xcite , with designs such as sparse and efficient computation @xcite @xcite @xcite @xcite , recurrent and memory modules @xcite @xcite @xcite @xcite . more recently, several methods @xcite @xcite have been proposed to enable length extrapolation of transformers, and have been adopted in the training process of long context llms such as chatglm2-32k @xcite and longchat-32k @xcite . evaluation for long context understanding. many previous works on long text modeling rely on the perplexity metric for evaluation @xcite @xcite . however, as suggested in @xcite , the perplexity metric may not necessarily reflect the model's performance on sequence-level tasks in real applications meanwhile, some works assess long text modeling through artificial tasks such as retrieval @xcite @xcite , which may also fall short in mirroring real-world scenarios. concurrently, zeroscrolls @xcite and l-eval @xcite are proposed as evaluation benchmarks for long text modeling. nonetheless, they encompass a restricted range of task types, thereby limiting the diversity of long text modeling patterns required in the bench-marks, and consequently, the comprehensiveness of the evaluation results. recently, agentbench @xcite also mentions the challenge of llmas-agent's handling long interaction trajectories, but fails to incorporate it as a dedicated evaluation dimension. in contrast, longbench includes six major task categories, with each category featuring sequences of varying lengths, languages, and domains. we believe it provides a more holistic evaluation of long text modeling ability of large language models across a spectrum of lengths, distributions, as well as long dependency patterns. ## conclusion in this paper, we introduce longbench, a multitask bilingual benchmark tailored for gauging long context understanding abilities of llms. long-bench covers six key categories and a total of 21 tasks, with data lengths extending from thousands of tokens up to tens of thousands of tokens. we also develop longbench-e which features a more evenly data length distribution. we conduct extensive experiments on longbench and longbench-e, yielding insightful conclusions about the capabilities of current llms on long context understanding. moreover, our analysis suggests that longbench and longbench-e serve as ideal testbeds for future research in long context modeling.