L -Eval: Instituting Standardized Evaluation for Long Context Language Models (2024)

Abstract: Recently, there has been growing interest in long-context scaling of large language models (LLMs). To facilitate research in this field, we propose L-Eval to institute a more standardized evaluation for Long-Context Language Models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, and more than 2,000 human-labeled query-response pairs including diverse task types, domains, and input length (3k~200k tokens). On the other hand, we investigate the effectiveness of evaluation metrics for LCLMs and we show that Length-instruction-enhanced (LIE) evaluation and LLM judges can better correlate with human judgments. We conducted a comprehensive study of 4 popular commercial LLMs and 12 open-source counterparts using the L-Eval benchmark. Our empirical findings offer useful insights into the study of LCLMs and lay the groundwork for the development of a more principled evaluation of these models.

## introduction currently, extending the context length of large language models has emerged as a significant direction for the evolution of llms, with a considerable amount of effort being dedicated to this pursuit @xcite @xcite . as a consequence, these widely used benchmarks that mainly consist of short prompts @xcite , and whose majority of test samples contain fewer than 4k tokens, are becoming increasingly inadequate for evaluating these long-context language models (lclms). specifically designed evaluation suites, such as scrolls and ze-roscrolls @xcite have been proposed. however, they exhibit a limited range of task types and input lengths; tasks like summarization and abstractive qa are overwhelmingly predominant and usually have an average input length of less than 16k tokens. furthermore, these benchmarks mostly employ n-gram metrics such as rouge @xcite for automatic evaluation, while recent work underscored the limitations of such metrics in accurately evaluating the performance of llms @xcite @xcite . to address these issues, we propose l-eval, offering a more standardized long-context evaluation for llms through data construction and the investigation of evaluation metrics. on data construction, l-eval divides the tasks into two groups. tasks in the closed-ended group usually have clear answers and can be quickly and objectively scored. the open-ended group on the other hand, comprises tasks that allow for a wide range of acceptable responses, such as summarization, posing challenges for automatic evaluation @xcite . overall, l-eval consists of 20 subtasks, 4 of which with 378 query-response pairs are manually annotated from scratch to expand the coverage of domains, input lengths, and task types ( l-eval adopts diverse evaluation approaches including n-gram metrics, llm judges, and human evaluation, to better evaluate open-ended generation results. our experiments suggest that the dominant n-gram metrics in long-context tasks often cannot correlate well with human evalua-tion in the zero-shot setting. a crucial reason is that llms often struggle to generate responses of a similar length to the reference answers without being trained on domain-specific data (see we conducted a comprehensive study with 16 different llms in l-eval. some of our key findings are summarized below: (1) there is still a remarkable gap between open-source and commercial models, for both closed-ended tasks and open-ended tasks, while the performance gap is not accurately reflected by n-gram metrics. (2) although current extension approaches for opensource models significantly improve performance on closed-ended tasks, they fall short on openended tasks. this is largely due to the models' inadequate capacity to understand instructions in longer inputs. (3) experiments on gpt-3.5-turbo with both dense and sparse retrievers show that endto-end long-context models outperform traditional retrieval based systems (4) training-free scaled positional embeddings can enhance the retrieval capability of llms over longer input, while it can have a negative impact on reasoning ability. more findings are shown in §5.2 and §a.3. a detailed discussion with concurrent work can be found in §2.2. we hope l-eval and our findings contribute to a deeper understanding of current lclms as well as their evaluation metrics. ## data construction in this section, we detail the essential processes involved in constructing l-eval. specifically, we describe the pipeline for annotation, re-annotation, and post-processing, as well as provide statistics of l-eval. detailed explanation of tasks in l-eval is in appendix §b. ## evaluation metrics in this section, we present various evaluation metrics for text generation, including exam evaluation for close-ended tasks and different levels of openended evaluation, most of which are referencebased metrics. we also conduct experiments to study the correlation between automated metrics and human scoring. exam evaluation this is designed for closedended tasks, i.e., multiple-choice questions. the evaluation metric used for these tasks follows the exact match format (accuracy %), similar to grading exam papers. each question's score is calculated as 100 divided by the number of questions. human evaluation currently, automatic metrics for open-ended generations still cannot replace human evaluation @xcite . our human evaluation procedure is in appendix a.2. we engage human evaluators to score the outputs on a scale of 1 to 5, which signifies from poor output to excellent output. to reduce the cost of human evaluation, we select 12 long documents with 85 open-ended questions. human evaluation for longcontext tasks is extremely time-consuming. our human evaluation of 85 questions across 7 systems takes around 30 hours (4 days) for each annotator. llm judges in short-context settings, llm acting as a judge is commonly employed for openended tasks evaluation @xcite . in l-eval, we also incorporate llm judges and utilize a pairwise battle format following @xcite ; @xcite . we report the win rate versus gpt-3.5-turbo-16k. unlike short-context settings, long-context evaluation does not typically allow for feeding entire lengthy prompts into the judge. consequently, the results of the evaluation depend primarily on the reference answers and the instructions provided. llm evaluators have been reported to favor more detailed and lengthy responses @xcite . this bias becomes more pronounced in long context settings because the incomplete input makes it difficult to accurately determine the correctness of specific details and information. therefore, the judgment model must remember that details not corroborated by the reference answers should not be considered beneficial. we enhance the judgment prompt with additional details or information that are not mentioned in the reference answer cannot n-gram metrics assessing all tasks is still expensive for human/llm evaluators. l-eval also takes into account n-gram metrics like rouge and f-1, which have been widely used in previous text generation benchmarks. notice that n-gram metrics are susceptible to length bias. performing lexical matching between ground truth and predictions with significant length differences often can not yield accurate results. ## conclusion in conclusion, the much-needed rigorous benchmark l-eval introduced in this work provides a comprehensive suite of tasks and evaluation met-14395 rics to assess the capabilities of lclms. our analysis using l-eval offers valuable insights into the current state and limitations of lclms. we believe that with its focus on practical, long-form documents across domains, l-eval can serve as a challenging testbed to drive advances in modeling longer contexts.