L ong W eave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability (2025)

Abstract: Generating long, informative, and factual outputs remains a major challenge for Large Language Models (LLMs). Existing benchmarks for long-form generation typically assess real-world queries with hard-to-verify metrics or use synthetic setups that ease evaluation but overlook real-world intricacies. In this paper, we introduce LongWeave , which balance real-world and verifiable assessment with Target-Anchored Evaluation (TAE). TAE constructs tasks by first defining verifiable targets within real-world scenarios, then systematically generating corresponding queries, textual materials, and anchors based on these targets. This ensures that tasks are both realistic and objectively assessable, enabling rigorous assessment of model capabilities in meeting complex real-world constraints. LongWeave supports customizable input/output lengths (up to 64K/8K tokens) across seven distinct tasks. Evaluation on 23 LLMs show that even state-of-the-art models encounter significant challenges in long-form generation as real-world complexity and output length increase. Dataset will be publicly available.

## introduction large language models (llms) have significantly enhanced their capabilities to process long inputs @xcite @xcite through architectural design @xcite and data engineering @xcite . however, achieving robust longsequence generation remains highly challenging @xcite . several research efforts have attempted to optimize llms for long-form output generation @xcite @xcite figure 1 : the performance across the seven tasks in longweave. for better visualization, performance scores have been normalized to a range of 0.3 to 0.7. 2025). however, the generated content often lacks adequate informativeness, comprehensiveness, and factuality @xcite @xcite . the inherent complexity of long-form sequences further complicates accurate assessment of these qualities, highlighting the necessity for more reliable evaluation benchmarks. long-form generation with real-world queries is typically evaluated using similarity metrics (e.g., α-ndcg, self-bleu) or llm-as-a-judge @xcite . while straightforward to implement, direct evaluation struggles with the inherent longsequence complexity. to address this, another line of work breaks long-text evaluation into a set of verifiable sub-tasks, which can include factual claims (e.g., a statement like "the earth orbits the sun") or aspects (e.g., completeness, logical consistency). checklists are constructed through expert-curated guidelines @xcite or automated methods leveraging llms to extract claims from outputs for factual verification via search engines @xcite @xcite or fixed databases @xcite . a critical challenge lies in optimizing the degree of specificity scope: overly broad checklists produce vague claims that hinder verification, while overly detailed ones tend to overcomplicate verification processes by attempting to cover all corner cases. to enhance verifiability, some approaches use synthetic data rather than real-world data-for instance, combining short questions from datasets like mmlu @xcite into longer ones, and then checking each segment individually. other benchmarks conduct procedural simulation or utilize objective question-answering (qa) tasks where fixed answers are associated with precise constraints to limit the response scope @xcite . though these methods simplify verification, they generally sacrifice realism in real-world scenarios. to bridge real-world relevance with verifiability, we implement decomposition at the verification stage through a new constraint-verifier evaluation (cov-eval) mechanism, as illustrated in figure 2. rather than extracting checklists from raw materials, which is error-prone and hard to control, cov-eval reverses the test construction process: it begins with predefined verifiable checklist objectives (verifiers) grounded in real-world tasks, then synthesizes corresponding inference queries (including constraints and materials). the constraint acts as a constrained input that causally guides models toward generating the predefined verifier, enabling measurable verification and eval-uation. each constraint-verifier (cv) pair in cov-eval maintains a deterministic one-to-one relationship under structurally defined rules, systematically linked to source materials. cov-eval contains a series of cv pairs, where each pair is linked to the corresponding material. these pairs can take various forms, such as a question (c) and answer (v) in qa tasks, or a triplet (c) and corresponding sentence (v) in knowledge-to-text generation, as discussed in section 2.3. based on cov-eval, we introduce longweave, a new benchmark evaluating five challenge scenarios of long-form generation through seven realworld relevant tasks (figure 1 ). longweave supports customizable input lengths (up to 64k tokens) and output lengths of 1k, 2k, 4k, and 8k tokens, with adjustable difficulty settings for each task as detailed in our evaluation of 23 llms on longweave reveals critical limitations in long-form generation: even top models (deepseek-r1) reach a performance ceiling of 54.56%, with performance declining for 8k-token outputs (figure 1 ). furthermore, models exhibit input-output disconnect; while supporting inputs up to 64k tokens, they fail to effectively synthesize inputs into coherent long-form responses. expanding input context windows (e.g., to 1m tokens) does not fundamentally solve longgeneration challenges and may even degrade performance. moreover, large-scale reasoning-oriented llms consistently outperform general-purpose counterparts, but often suffer from failure to terminate the reasoning phase, leading to truncated outputs. our main contributions are: ## the longweave benchmark in this section, we first introduce the overall pipeline of longweave, followed by a detailed formulation of our constraint-verifier evaluation and a description of the individual tasks.