CL ong E val: A C hinese Benchmark for Evaluating Long-Context Large Language Models (2024)

Abstract: Developing Large Language Models (LLMs) with robust long-context capabilities has been the recent research focus, resulting in the emergence of long-context LLMs proficient in Chinese. However, the evaluation of these models remains underdeveloped due to a lack of benchmarks. To address this gap, we present CLongEval, a comprehensive Chinese benchmark for evaluating long-context LLMs. CLongEval is characterized by three key features: (1) Sufficient data volume, comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability, accommodating to models with context windows size from 1K to 100K; (3) High quality, with over 2,000 manually annotated question-answer pairs in addition to the automatically constructed labels. With CLongEval, we undertake a comprehensive assessment of 6 open-source long-context LLMs and 2 leading commercial counterparts that feature both long-context abilities and proficiency in Chinese. We also provide in-depth analysis based on the empirical results, trying to shed light on the critical capabilities that present challenges in long-context settings. The dataset, evaluation scripts, and model outputs will be released.

## introduction large language models have demonstrated impressive performance across a wide range of natural language processing (nlp) tasks, including machine translation @xcite , fact checking @xcite ), text style transfer @xcite and other generation tasks @xcite . to enable llms to support more intricate and diverse applications, an increasing number of studies focus on extending the context window these models can handle. consequently, many long-context partial context full context abstraction extraction • long story qa • long conversation memory •stacked news labeling •stacked typo detection •key-passage retrieval •table querying llms that support chinese have emerged, including both commercial models @xcite and open-source ones @xcite @xcite , of which the context lengths span from 32k to 200k. despite these developments, the efficacy of models in long-context settings remains underexamined, primarily due to the lack of a robust evaluation benchmark. recently, a few benchmarks have been proposed for the evaluation of english long-context llms @xcite . as for the chinese domain, there exists only a bilingual benchmark @xcite , wherein only 5 out of 21 test tasks are designed for chinese. this benchmark offers only 1k instances in total, with an average token length capped at approximately 13k 2 , rendering it inadequate for a comprehensive evaluation. therefore, there is an urgent need for a high-quality chinese benchmark for long-context llms. considering the considerable advancements in this field, the establishment of such a benchmark facilitates a thorough investiga-tion of existing models, which might bring insights to the research community. in this paper, we present clongeval, a benchmark designed for evaluating chinese long-context llms. prior to the construction of datasets, we conduct a systematic analysis of the key capabilities requisite for handling long context, to ensure a thorough assessment of the model's functionality ( § 2). analogous to the human problem-solving paradigm, the basic functionalities of long-context llms can be conceptualized as: (1) the capacity to precisely identify and acquire the key information framing in either partial or full context; and (2) the competence to reason out the answer based on the given information in either an extractive or abstractive manner. these key abilities establish the evaluation framework behind clongeval, which is illustrated in figure 1 . to accommodate models with varying spans of context windows, we consider three subsets within clongeval: small set (1k-16k), medium set (16k-50k), and large set (50k-100k) ( § 3.1). in dataset construction, we select test tasks that correspond to the capabilities outlined in the evaluation framework. moreover, we ensure that primary test tasks are highly aligned with real-life user scenarios so that the benchmark can accurately reflect models' capability in practical applications @xcite ( § 3.2). overall, we craft 7 distinct tasks in clongeval: 2 tasks are human-annotated, 1 task is gpt-4-annotated and 4 tasks are re-constructed from public datasets. with clongeval, we evaluate 8 long-context llms proficient in chinese, including two commercial models known for their powerful longtext processing capability: moonshot-v1-128k and gpt-4-turbo-128k ( § 4.3). we highlight several key findings in the long-context setting: (1) the commercial models consistently outperform opensource models across tasks, and the performance gap is particularly evident in tasks that primarily involve straightforward information extraction. (2) extraction with full context is the most challenging setting. gpt-4-turbo displays a more significant decline in performance as context length increases, compared to other settings. (3) for tasks requiring an understanding of partial context, the answer's position within a long context does not consistently lead to significant performance fluctuations. more analysis and observations are elaborated in § 4.4 and appendix a. ## evaluation framework in clongeval to offer a thorough and systematic evaluation, we analyze the key capabilities necessary for the efficacy of long-context llms. generally speaking, the capacity for a long-context llm in interpreting human textual instructions largely depends on its ability of information acquisition. moreover, an indispensable ability for these models extends beyond mere information collection to encompass reasoning based on the assimilated information. ## conclusion we presented clongeval, a benchmark for chinese long-context llms, which contains 7 tasks and 7,267 examples. to the best of our knowledge, clongeval is the first benchmark in this setting. based on two basic capabilities for long-context llms, i.e., information acquisition and reasoning, we collected corresponding tasks and datasets for a comprehensive evaluation. we benchmarked 8 long-context llms and provided an in-depth analysis regarding each fine-grained capability.