Marathon: A Race Through the Realm of Long Context with Large Language Models (2024)

Abstract: With the advancement of large language models (LLMs) and the expansion of their context windows, existing long-context benchmarks fall short in effectively evaluating the models’ comprehension and reasoning abilities in extended texts. Moreover, conventional benchmarks relying on F1 metrics often inaccurately score responses: they may undervalue correct answers that differ from the reference responses and overvalue incorrect ones that resemble the reference texts. In response to these limitations, we introduce Marathon, a novel evaluation benchmark that adopts a multiple-choice question format. It is specifically designed to overcome the constraints of previous benchmarks and provide a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models. We conducted comprehensive evaluations on the Marathon benchmark with a range of state-of-the-art LLMs and assessed the effectiveness of various optimization strategies tailored for long-context generation. We anticipate that the Marathon benchmark and its associated leaderboard will enable a more precise and equitable evaluation of LLMs’ capabilities in understanding and reasoning over extended contexts.

## introduction in the rapidly evolving landscape of artificial intelligence technologies, the emergence of large language models (llms), as exemplified by chat-gpt @xcite , showcases notable capabilities. the influence of these models extends beyond the well-established chatgpt, gaining increasing prominence across diverse sectors. existing llms are typically built upon transformer † min yang is the corresponding author. 1 we also provide an online evaluation website: https: //openbenchmark.online/marathon architectures, which demand memory and computational resources that grow quadratically with sequence length. consequently, transformer language models have historically been trained with relatively modest predetermined context windows. for instance, llama @xcite employs a context size of 2048 tokens, while llama2 @xcite utilizes a context size of 4096 tokens. however, the pre-defined size imposes constraints on llms in various applications, such as summarizing extensive documents or addressing lengthy questions. significant research efforts have been devoted to extending the context length of llms. due to the prohibitive expense of training llms with extended context lengths from scratch, the predominant studies have endeavored to enhance the capabilities of llms to comprehend long contexts through fine-tuning. these methods encompass extending the context window @xcite , incorporating recurrent memory @xcite , employing sparse attention mechanisms @xcite , and augmenting with external memory @xcite . concurrently, an increasing multitude of benchmarks have been introduced to assess the long-context understanding capabilities of llms. longbench @xcite stands out as the first bilingual, multi-task benchmark specifically designed for the assessment of long-context understanding. this dataset continues to depend on the f1 score, which evaluates the responses of llms against a predefined set of possible answers. loogle @xcite encompasses intricate long dependency tasks, including event timeline reordering, comprehension/reasoning, and computation. nevertheless, the diverse nature of model-generated content introduces a challenge, as these predefined answers may not encompass all valid responses, thereby diminishing the precision of assessing model performance. there is a growing demand for high-quality benchmarks characterized by significantly longer text lengths and more challenging tasks, ensuring comprehensive evaluations. in this study, we introduce a novel benchmark named marathon, designed for long-context understanding and reasoning. in particular, this benchmark is constructed upon the foundations established by loogle @xcite and long-bench @xcite . the contextual lengths within this benchmark span from 2k to over 260k characters. for each extensive context provided, an associated question is paired with four meticulously crafted response options. these options have been carefully reviewed by humans and contain only one correct answer, with the remaining options designed to be highly misleading. this design makes the marathon benchmark a particularly challenging one. the task for the large language model is to discern the accurate response option based on the extensive context provided. the main contributions of this work are threefold: ## marathon present benchmarks for evaluating large language models primarily use a multiple-choice format, highlighted by studies such as mmlu @xcite and c-eval @xcite . this multiple-choice approach helps prevent situations where large language models produce correct answers but are scored low due to missing corresponding elements in the reference answers or when they produce incorrect answers that are scored high because they resemble the reference answers closely. therefore, influenced by loogle @xcite and longbench @xcite , we developed a multiple-choice, long-context benchmark to more accurately evaluate the ability of large language models to understand extended contexts. ## conclusion in this paper, we compared 10 open-source large language models, including variations in their parameter sizes and context windows, along with ope-nai's chatgpt and gpt-4. we assessed two prevalent optimization techniques, such as longllm-lingua and rag. the experimental results indicate that rag-based optimization enhances the performance of large language models within longcontext scenarios for qa-type tasks. however, the improvement is limited for tasks involving timeline reorder and computation. despite high accuracy in question-answering, these models show limited ability to follow instructions.