M 4 LE : A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models (2024)

Abstract: Managing long sequences has become an important and necessary feature for large language models (LLMs). However, assessing their ability to handle long contexts remains a challenge. This paper introduces M ^4 LE, a \textbf{M} ulti-ability, \textbf{M} ulti-range, \textbf{M} ulti-task, \textbf{M} ulti-domain benchmark for \textbf{L} ong-context \textbf{E} valuation. It encompasses 36 NLP datasets, covering 11 types of tasks and 12 domains, providing a comprehensive test bed. To address the lack of tasks featuring naturally long sequences, we propose an automatic approach to convert short-sequence tasks into long-sequence scenarios. These scenarios evaluate LLMs’ long-context understanding across five key abilities: understanding of single or multiple relevant spans in long contexts based on explicit or semantic hints, and global context understanding. This automatic approach allows us to create instances evenly distributed from 1k to 8k input length. Our evaluation of 11 prominent LLMs reveals that 1) Current LLMs struggle to understand long context, particularly when tasks require multiple-span attention. 2) Semantic retrieval is more difficult for competent LLMs. 3) Models fine-tuned on longer text with position interpolation have comparable performance to those using Neural Tangent Kernel (NTK) aware scaling methods without fine-tuning. We make our benchmark publicly available to encourage future research in this challenging area.

## introduction large language models (llms) are gaining traction in addressing diverse nlp challenges. llms, mostly transformer-based models @xcite * work done during an internship at huawei noah's ark lab. 1 the released benchmark would contain samples up to 32k words. even longer samples and other types of tasks can be constructed using our method. 2 code and data are available at https://github.com/ kwanwaichung/m4le . despite numerous efforts to develop benchmarks for assessing the knowledge or reasoning ability of llms @xcite @xcite , comprehensive evaluation of their long-context understanding ability has received limited attention. recent concurrent works, such as l-eval @xcite and @xcite , primarily rely on existing long-sequence nlp datasets which usually limit the task diversity and flexibility in conducting length-control experiments. they lack an objective and comprehensive understanding of the model's capability across different dimensions of long sequences. in this study, we aim to maximize the diversity of constructed tasks and analyze the long-context capabilities of llms from a user's practical perspective. we discovered that when processing instructions based on long sequences, the essential components for task completion can be classified as single-span, multiple-span, or global, based on relevance. building on this and considering how to locate the relevant information, we categorize long-context understanding into five distinct abilities and introduce an automated method to transform short-sequence tasks into a comprehensive long-sequence scenario encompassing all these ca-figure 1 : the illustration of m 4 le. m 4 le covers multiple task types, domains and length ranges, and introduces five long-context understanding abilities, each of which is exemplified with a summarization instance, to facilitate the long-context evaluation. we conducted a systematic evaluation over 11 well-known llms, especially those claimed to support long inputs, with m 4 le. this involves evaluating their long-context understanding ability across different length ranges and their performance in our proposed five different abilities. we also delve into the factors influencing long-context understanding capability, including llms performance under different languages and the positioning of relevant information @xcite . we find that current llms still struggle to understand long-context inputs, especially when multiple-span attention is required. while semantic retrieval is considered more complex than explicit, the consistent performance drop in this scope can only be observed on competent models. a more effective fine-tuning approach deserves exploration, as current methods show no significant improvement over simple neural tangent kernel (ntk) aware scaling methods. we also observe that language differences and the positioning of relevant information impact the long-context understanding capabilities. ## m 4 le this section outlines the m 4 le benchmark's rationale, design principles, data sources, and task construction methodologies. m 4 le is designed to comprehensively evaluate large language models' (llms) abilities in understanding long contexts. it covers a wide range of tasks, domains, and context lengths, ensuring a thorough assessment of llms' competencies in this crucial area. ## conclusion in this paper, we propose m 4 le for llms assessing their capability of long-context understanding. to establish a benchmark with diverse nlp tasks, rather than just those that are inherently lengthy, we propose a systematic method to convert short nlp task instances into long context inputs, encompassing five distinct abilities. we collect and construct in total of 36 tasks from different sources and domains covering multiple length ranges to maximize the diversity of the tasks in benchmark, with our customized construction methods which enable flexibility to extend arbitrary context lengths. we evaluate 11 well-known llms with our benchmark and find that current models struggle to understand long-context inputs and the corresponding performance related to ability types, data used when finetuning, and positions of the relevant information.