LCFO : Long Context and Long Form Output Dataset and Benchmarking (2025)

Abstract: This paper presents the Long Context and Form Output (LCFO) benchmark, a novel evaluation framework for assessing gradual summarization and summary expansion capabilities across diverse domains. LCFO consists of long input documents (5k words average length), each of which comes with three summaries of different lengths (20%, 10%, and 5% of the input text), as well as approximately 15 questions and answers (QA) related to the input content. Notably, LCFO also provides alignments between specific QA pairs and corresponding summaries in 7 domains. The primary motivation behind providing summaries of different lengths is to establish a controllable framework for generating long texts from shorter inputs, i.e. summary expansion. To establish an evaluation metric framework for summarization and summary expansion, we provide human evaluation scores for human-generated outputs, as well as results from various state-of-the-art large language models (LLMs). GPT-4o-mini achieves best human scores among automatic systems in both summarization and summary expansion tasks (≈ +10% and +20%, respectively). It even surpasses human output quality in the case of short summaries (≈ +7%). Overall automatic metrics achieve low correlations with human evaluation scores (≈ 0.4) but moderate correlation on specific evaluation aspects such as fluency and attribution (≈ 0.6).

## introduction robust long text generation capabilities are required to meet user demand for extensive content creation, including story writing and essay composition (xie and riedl, 2024), which is why recent models such as @xcite are expanding the output lengths from 4k tokens in gpt-4o to 64k in the latest versions. however, evaluating the performance of large language models (llms) on summarization and summary expansion tasks (see definitions in section 2.1) is particularly challenging, especially when it comes to summarizing very long input documents and generating either long summaries or long summary expansions. although there is a lot of work and interest in studying summarization evaluation (e.g. @xcite , evaluation of long text outputs is an emerging area @xcite . indeed, long-context input processing tasks (such as summarization or comprehension question answering applied to long documents) and longform output production both involve high cognitive loads for humans. this may be why evaluation work in these areas is not as mature as in others. to complement the summarization task, we include an "inverted" task of summary expansion: generating a longform text based on its shorter sketch. it requires more creativity than summarization (which can be approached extractively) but imposes more constraints than open-ended language modeling. a practical application for this task could be with a writing assistant expanding the text sketch created by the human writer into a longer output that the human subsequently revises. another potential use is "back-translation" for summarization, with a model training to summarize the synthetic expanded documents into their original short forms. the lcfo dataset that we are presenting in this paper is an exclusively human annotated benchmark and challenge dataset involving natural language understanding and generation across multiple domains in the aforementioned tasks. our dataset is carefully manually crafted with human revision from the selection of the documents to the final annotation, without relying on llms at any point. we provide detailed linguistic guidelines and abstractive qa. furthermore, we provide another set of linguistic guidelines to evaluate the tasks of summarization and summary expansion. together, we present the lcfo benchmark. given a source of long structured documents, we generate multiple long outputs and associated qa pairs, and we evaluate human and model outputs with human annotations. the main contributions of this work are the following (see figure 1 for a schematic representation of the dataset and tasks): if we base ourselves on the reported reading average time for first-language, secondary-education level readers (125 words per minute on average), we can consider that we will reach high cognitive load at around 1.5k words, and peak cognitive load at around 2.5k words. the quality of cognitive processing then starts to decrease significantly after. structured/hierarchical. the input and output documents are partitioned into sections and, if necessary, nested sub-sections. their structure is determined by task-and domain-specific guidelines. ## related work related work on long context and long form output comes in many flavors. we cover a summary on long context and long-form output datasets. long-context datasets. infinite length datasets such as niah, ruler (hsieh et al., 2024) work with distracting information. finite-length nondistractive-based datasets include: multi-lexsum (shen et al., 2022) is a collection of 9,280 expert-authored summaries drawn from single domain (civil rights litigation clearinghouse) writing the length of the source documents often exceeds two hundred pages per case and summaries are presented in two-sizes: a short and longer version; longbench (bai et al., 2024) and marathon (zhang et al., 2024b) that includes tasks with 5-25k context and, more recently, (kwan et al., 2024) build a dataset up to 8k tokens context length to evaluate llms' long-context understanding across five key abilities: understanding of single or multiple relevant spans in long contexts based on explicit or semantic hints, and global context understanding. loong is a multi-document qa dataset up to 200k context to assess rag abilities. hellobench (que et al., 2024) includes summarization of a selection of long-input documents (3k to 6k word length). ## conclusions lcfo provides gradual summaries references from 5k input documents with qa pairs for each of the documents and summaries. additionally, we provide human evaluation of human and modelgenerated summaries and model-generated summary expansions. overall, lcfo enables the evaluation of several tasks and metrics in the setting of long-context input documents and long-form output. while the main contribution of this paper is to present the freely available lcfo dataset foot_4 , we also evaluate model and human outputs, showing that llms are capable of surpassing human results when producing short summaries. current evaluation results question the usefulness of manually generating human references for short summarization of long documents. to confirm this, as further work, we plan to exploit the capabilities of lcfo by using qa as part of automatic evaluation (i.e. scoring how many questions are correctly answered in model-generated summaries).