ETHIC : Evaluating Large Language Models on Long-Context Tasks with High Information Coverage (2025)

Abstract: Recent advancements in large language models (LLM) capable of processing extremely long texts highlight the need for a dedicated evaluation benchmark to assess their long-context capabilities. However, existing methods, like the needle-in-a-haystack test, do not effectively assess whether these models fully utilize contextual information, raising concerns about the reliability of current evaluation techniques. To thoroughly examine the effectiveness of existing benchmarks, we introduce a new metric called information coverage (IC), which quantifies the proportion of the input context necessary for answering queries. Our findings indicate that current benchmarks exhibit low IC; although the input context may be extensive, the actual usable context is often limited. To address this, we present ETHIC, a novel benchmark designed to assess LLMs’ ability to leverage the entire context. Our benchmark comprises 1,986 test instances spanning four long-context tasks with high IC scores in the domains of books, debates, medicine, and law. Our evaluations reveal significant performance drops in contemporary LLMs, highlighting a critical challenge in managing long contexts. Our benchmark is available at https://github.com/dmis-lab/ETHIC.

## introduction the field of natural language processing (nlp) has made remarkable progress in developing models that can manage much longer texts. while earlier transformer-based models @xcite could only process 512 tokens at a time @xcite , modern large language models (llm) have achieved a significant breakthrough, now capable of handling documents with up to two million tokens @xcite ., † corresponding authors. information coverage (%) model performance low performance on high-ic high performance on low-ic figure 1 example: who were the individuals involved in the incident on april 3, 2022? low-ic tasks example: list every unique character name that appears in the context. figure 1 : the variation in model performance with the level of information coverage (ic). unlike low-ic tasks, which focus on specific parts of the input context, our benchmark features new high-ic tasks that demand the full utilization of all available information, posing a significant challenge for long-context models. to this end, we propose ethic, a suite of long-context tasks specifically designed to assess whether llms can fully utilize the provided information. our benchmark encompasses four distinct domains: books, debates, medicine, and law, each containing a set of tasks that require the use of all relevant information in the context to arrive at a solution. to measure this capability, we introduce the concept of information coverage (ic), which quantifies the proportion of the input context required to answer a query. figure 1 shows that model performance significantly declines for high-ic tasks in our benchmark compared to low-ic tasks, even when the same input contexts are used. we evaluated the latest llms that support at least 128k tokens, along with several training-free methods using our benchmark. our findings revealed that recent models perform poorly across all tasks and domains, even when utilizing recent frameworks proposed for efficient long-context processing @xcite . this highlights a significant challenge for high-ic tasks and the need for further research in this area. we also conducted detailed analyses, comparing the performance gap between our task and traditional low-ic tasks, as well as identifying which parts of the input context the models typically fail to address. in summary, our contributions include: ## preliminaries in this section, we provide an overview of recent long-context llms and the benchmarks currently used to evaluate them (see sections 2.1 and 2.2). we also present a formal description of information coverage and explain how our benchmark differs from existing ones, emphasizing the new aspects of llms that we aim to evaluate (see section 2.3). ## ethic in this section, we outline the document collection process (see section 3.1) and the task construction process (see section 3.2) in detail. additionally, we describe the label annotation process for each task in section 3.3. ## experiments in this section, we outline different metrics used to evaluate each task (see section 4.1). we introduce baseline models and methods in section 4.2, and provide the results in section 4.3. ## analysis we conducted further analysis to better understand the models' limitations within our benchmark. we used gpt-4o mini as the main model throughout sections 5.1 to 5.3, and all models for section 5.4. ## conclusion in this study, we introduced ethic, a novel benchmark designed to evaluate llms' ability to fully process information in long-context settings. we also introduced a new metric, information coverage (ic), to quantify how much of the provided context is required to answer a query. compared to existing benchmarks, ethic has significantly higher ic values, which underscores its unique aspect. our experiments revealed that current commercial/open-source llms and long-context encoding methods perform poorly on our benchmark, emphasizing the need for future research to address long-context processing with high ic.