L ong G en B ench: Long-context Generation Benchmark (2024)

Abstract: Current long-context benchmarks primarily focus on retrieval-based tests, requiring Large Language Models (LLMs) to locate specific information within extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark. Long-context generation refers to the ability of a language model to generate coherent and contextually accurate text that spans across lengthy passages or documents. While recent studies show strong performance on NIAH and other retrieval-based long-context benchmarks, there is a significant lack of benchmarks for evaluating long-context generation capabilities. To bridge this gap and offer a comprehensive assessment, we introduce a synthetic benchmark, LongGenBench, which allows for flexible configurations of customized generation context lengths. LongGenBench advances beyond traditional benchmarks by redesigning the format of questions and necessitating that LLMs respond with a single, cohesive long-context answer. Upon extensive evaluation using LongGenBench, we observe that: (1) both API accessed and open source models exhibit performance degradation in long-context generation scenarios, ranging from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of performance degradation, with the Gemini-1.5-Flash model showing the least degradation among API accessed models, and the Qwen2 series exhibiting the least degradation in LongGenBench among open source models.

## introduction long-context generation refers to the ability of a language model to generate coherent and contextually accurate text that spans across a lengthy passage or document. this capability involves maintaining the thematic continuity, logical flow, and consistency of details over extended sequences of text, which can include multiple paragraphs, pages, or even entire documents. to facilitate further research in this area, we propose the long-context generation benchmark (longgenbench), a new benchmark specifically designed to evaluate the long-context generation capabilities of llms, with a particular focus on consistency in logical flow. longgenbench synthesizes a dataset from current popular llm benchmarks, redesigns the input format, and includes multiple questions within a single query. the longgenbench requires llms to generate a comprehensive long-context response that sequentially addresses each question. to achieve better performance in longgenbench, llms need to maintain consistency regardless of whether the previous generation part is correct or incorrect. in longgen-bench, evaluating the quality of these long-context responses is straightforward: simply compare the generated answers with the ground truth. our study evaluates the performance of various language models using the longgenbench approach across different datasets, specifically longgenbench-mmlu, longgenbench-gsm8k, and longgenbench-csqa. figure 1 displays the performance of four powerful api accessed models tested in both the baseline scenario, which involves single-answer generation, and the longgen-bench scenario, which focuses on long-context generation. it is notable that the gemini-1.5-flash model exhibits the lowest performance degradation in long-context generation tasks, surpassing the gpt-4o. additionally, we conducted analyses on open-source models, revealing a general correlation between baseline performance and longgen-bench performance. models with higher baseline scores tend to show smaller declines in longcontext generation tasks. models like qwen2-72b-instruct and deepseek-v2-chat, both with high baseline scores, also exhibit minimal performance degradation. however, there are exceptions, such as llama-3-70b-instruct, which, de-spite its high baseline performance, experiences significant performance drops. moreover, model size influences performance, as larger models within the same series, such as the llama-3 and qwen2 series, demonstrate smaller declines. different architectures show varying trends in performance degradation; for example, llama-3-8b-instruct shows a performance degradation of 47.1% on gsm8k, while chatglm4-9b-chat only experiences a 10.8% drop, despite having similar baseline performances. consistency across tasks is observed, with models like llama-3-8b-instruct consistently showing significant drops on all datasets, whereas models such as qwen2-72b-instruct and deepseek-v2-chat maintain minimal declines across all datasets, underscoring their resilience in long-context generation tasks. these findings highlight the varying capabilities of different models to maintain accuracy over extended text generation and provide valuable insights for future model development and optimization. our contributions are as follows: • we introduce longgenbench, an effective approach for evaluating the long-context generation capabilities of language models across multiple datasets. our detailed experiments establish consistent trends in performance degradation across different longgenbench tasks, highlighting the importance of model resilience in long-context generation. ## longgenbench we propose longgenbench, a synthetic benchmark that is an efficient, low-cost approach focused on evaluating long-context generation in llms. ## expeirments setting in this section, we describe the details of the baseline models and the longgenbench approach, as well as their implementation in the subsequent experiments. all experiments were conducted three times, using the mean score to ensure robustness. ## ablation studies hyperparameters of longgenbench to gain deeper insights into longgenbench, we conducted an ablation study focusing on two critical hyperparameters: reconstructive processing and the ordering of k questions within a single query. for reconstructive processing, we compared the baseline format with the longgenbench format. the baseline format follows the cot @xcite setting, featuring eight question-answer pairs in sequence. in contrast, the longgenbench format presents eight questions in advance followed by the corresponding eight answers, with the k questions being addressed subsequently. regarding the order of k questions, we evaluated three sequences: the original order from the dataset, ascending order from shortest to longest, and descending order from longest to shortest. this ablation study was performed using gpt-3.5-turbo and gemini-1.5-flash on the gsm8k dataset, with the number of iterations set at @xmath0. ## conclusion in this study, we introduced longgenbench, an effective framework designed to evaluate the longcontext generation capabilities of language models (llms) across multiple datasets. our experiments included both api accessed and open source models, offering a comprehensive comparison of their performance in long-context generation tasks. the results indicate a correlation between baseline performance and longgenbench performance, with higher baseline models generally exhibiting smaller declines. additionally, model size and architecture significantly influence resilience, with larger models and specific architectures demonstrating greater robustness and consistent trends across different longgenbench tasks. these findings highlight the importance of considering both model architecture and size when evaluating llms in long-context generation tasks. the longgenbench framework effectively showcases the varying capabilities of different models, providing valuable insights for further model development and optimization.