A da- LE val: Evaluating long-context LLM s with length-adaptable benchmarks (2024)

Abstract: Recently, the large language model (LLM) community has shown increasing interest in enhancing LLMs’ capability to handle extremely long documents. As various long-text techniques and model architectures emerge, the precise and detailed evaluation of models’ long-text capabilities has become increasingly important. Existing long-text evaluation benchmarks, such as L-Eval and LongBench, construct long-text test sets based on open-source datasets, focusing mainly on QA and summarization tasks. These datasets include test samples of varying lengths (from 2k to 32k+) entangled together, making it challenging to assess model capabilities across different length ranges. Moreover, they do not cover the ultralong settings (100k+ tokens) that the latest LLMs claim to achieve. In this paper, we introduce Ada-LEval, a length-adaptable benchmark for evaluating the long-context understanding of LLMs. Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable a more reliable evaluation of LLMs’ long context capabilities. These benchmarks support intricate manipulation of the length of test cases, and can easily produce text samples up to 128k tokens. We evaluate 4 state-of-the-art closed-source API models and 6 open-source models with Ada-LEval. The evaluation results demonstrate the limitations of current LLMs, especially in ultra-long-context settings. Our code is available at https://github.com/open-compass/Ada-LEval.

## introduction large language models (llms), typically based on large transformers trained on vast corpus, have shown exceptional abilities in memorization, comprehension, and reasoning @xcite @xcite . a critical factor * the work was done during an internship at shanghai ai laboratory; † project lead; ‡ corresponding author. document n text segments shuffling question: what is the correct order of the segments? answer with the list of segment indices. tsort task bestanswer task python q 1 q 2 q 3 a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 java linux ruby q 1 a 11 a 12 a 13 q 1 a 11 a 12 a 13 q 1 a 11 a 12 a 13 a ij golden answer, tagged by the question asker a ij intra-question distractors a ij inter-question distractors question building question: which is the best answer for question ? choices are: . q 1 a 1 a 2 a 3 a 4 a 5 a n llm answer: a3. that affects llm performance is the 'context window' -the number of tokens an llm can process simultaneously. this window's size is pivotal in handling lengthy texts. since the debut of chatgpt with a 2,000-token window in november 2022, significant efforts have been made in this domain, including more efficient attention mechanisms @xcite @xcite , scalable position embeddings @xcite , and quantization techniques @xcite . as of december 2023, several llms claim to achieve context windows up to hundreds of thousands of tokens. this includes both proprietary models like gpt-4 turbo (128,000 tokens), claude-2.1 (200,000 tokens), and moonshot ai (200,000 chinese characters), and open-source models such as chatglm-32k @xcite and longchat-32k @xcite . this expansion significantly enhances the potential for processing extensive documents. nevertheless, the effectiveness of these long-context llms in managing long texts remains an area ripe for exploration and assessment. alongside the evolution of llms, a wide range of benchmarks have emerged for capability assessment @xcite @xcite . most of those benchmarks utilize short questions or instructions, making them unsuitable for evaluating llms' long-context capabilities. while a few benchmarks do focus on assessing specific long-context abilities like summarization, questionanswering (qa), and continue writing @xcite @xcite , comprehensive long-document evaluations have been limited. recent benchmarks such as scrolls @xcite , l-eval @xcite and @xcite have started to address this gap by including a suite of long-document tasks, aiming for a more holistic assessment of llms' long-context understanding. despite these advancements, three significant limitations persist in existing benchmarks: firstly, the ultra-long setting (32,000 tokens or longer) is scarcely represented, limiting insights into llm performance in extreme context lengths. secondly, the integration of test samples of varying lengths within these benchmarks complicates the evaluation of llms across different length ranges. lastly, the focus on traditional tasks such as questionanswering and summarization often does not necessitate comprehensive content understanding by the llms, as many questions in these tasks do not require full-text comprehension. this highlights the need for more targeted benchmarks that can rigorously evaluate the deep and complete understanding of long-form content by llms. to this end, we introduce ada-leval, a pioneering benchmark to assess the long-context capabilities with length-adaptable questions. ada-leval comprises two challenging tasks: tsort, which involves arranging text segments in the correct order, and bestanswer, which requires choosing the best answer of a question among multiple candidates. both tasks feature the following advantages: 1. controllable test cases: the length of each test case can be finely tuned -by adjusting the number and length of text segments in tsort and altering the number of distractor options in bestanswer. 2. necessity for full-text comprehension: successful completion of both tasks mandates complete reading and understanding of the provided text. ## conclusion in this paper, we introduce ada-leval, a lengthadaptable dataset to assess long-context capability of llms. we conduct comprehensive experiments on multiple llms and find that all open-source models still lag significantly behind state-of-the-art proprietary models in terms of long context capability. when the input length scales to 4,000 tokens, most open-source models rapidly deteriorates to random guess level. in the meanwhile, the capability of proprietary models is also severely limited, when it comes to the ultra-long setting (32,000+ tokens), no proprietary model notably outperforms the random baseline. ada-leval is the first benchmark that evaluates llms under the ultra-long setting, and we hope that the limitations pointed out by this benchmarks can serve as valuable references for future developments of long-context llms. acknowledgement. this project is supported by the national key r&d program of china no.2022zd0161600 and the shanghai postdoctoral excellence program (no.2023023).