Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation (2025)

Abstract: This paper presents ICAT, an evaluation framework for measuring coverage of diverse factual information in long-form text generation. ICAT breaks down a long output text into a list of atomic claims and not only verifies each claim through retrieval from a (reliable) knowledge source, but also computes the alignment between the atomic factual claims and various aspects expected to be presented in the output. We study three implementations of the ICAT framework, each with a different assumption on the availability of aspects and alignment method. By adopting data from the diversification task in the TREC Web Track and the ClueWeb corpus, we evaluate the ICAT framework. We demonstrate strong correlation with human judgments and provide comprehensive evaluation across multiple state-of-the-art LLMs. Our framework further offers interpretable and fine-grained analysis of diversity and coverage. Its modular design allows for easy adaptation to different domains and datasets, making it a valuable tool for evaluating the qualitative aspects of long-form responses produced by LLMs.

## introduction the rapid advancement of large language models (llms) has revolutionized long-form text generation, enabling increasingly sophisticated applications from report writing to complex question answering. however, this progress has highlighted a critical challenge: how do we effectively evaluate not just the factual accuracy of generated content, but also its completeness and coverage of diverse perspectives? while recent work has made significant progress in developing comprehensive evaluation frameworks for llms @xcite , existing metrics often focus on isolated aspects like * equal contribution. recent approaches like factscore @xcite and veriscore @xcite have addressed the factuality challenge by evaluating atomic claims against reliable sources, but factual accuracy alone is insufficient. a unified metric that considers both factuality and coverage is crucial not only for evaluation but also for optimizing llm performance. such a metric could serve as a reward function in reinforcement learning to simultaneously improve both the factual accuracy and comprehensive coverage of llm outputs. consider a user asking "what are the health effects of coffee consumption?" while an llm response might present entirely factual claims about coffee's benefits, such as its role in improving alert-ness and potential protective effects against certain diseases, failing to address known health risks (like anxiety or sleep disruption) would present an incomplete and potentially misleading picture. this illustrates a critical gap in current evaluation approaches: the need to assess not just the accuracy of individual claims, but also the comprehensive coverage of diverse relevant information. this is particularly crucial for applications like medical information systems, policy analysis, or educational content where balanced, complete information is essential for informed decision making. this paper introduces icat, a novel evaluation framework that addresses this gap by measuring both factual accuracy and coverage of diverse factual information in long-form text generation. our key contributions include: 1) a modular evaluation framework that decomposes long-form text into atomic claims and evaluates both their factual accuracy and their alignment with expected aspects, 2) three implementations with varying degrees of automation, suitable for different evaluation scenarios, and 3) a comprehensive evaluation of various llms and demonstrating strong correlation with human judgments in a user study. icat first breaks down the generated long text into atomic claims. through retrieval from a (reliable) corpus c or the web, icat verifies each atomic claim to ensure its factuality. to measure completeness and coverage of diverse facts, icat requires a set of diverse aspects to compute an alignment between each atomic factual claim in the llm response and the set of diverse aspects. we study three implementations of the icat framework with different assumptions on the availability of aspects and alignment methods. in our experiments, we use clueweb (the lemur project, 2009) as the retrieval corpus. we solely focus on the english documents of the clueweb collection. in addition to custom open-source retrieval from clueweb, we explore web-based grounding using the brave search api. for experiments, we rely on the input queries from the trec web track @xcite @xcite . the argument for this decision is based on the fact that trec web track queries have also been used for search result diversification. this means that the queries include up to 7 aspects and documents are provided with aspect-level relevance annotations. our experiments show that there is relatively strong corelation of icat with human judgments, showcasing the utility of this framework for evaluating coverage of diverse factual information in llm responses without human input. by offering a modular and adaptable framework, icat enables researchers to tailor the evaluation process to specific needs, making it a valuable tool for assessing the qualitative aspects of long-form responses produced by llms. the decomposition of llm outputs into atomic claims and their alignment with specific topics makes the evaluation process highly interpretable -evaluators can trace exactly which claims support which topics and identify gaps in coverage. this granular analysis capability, combined with the framework's ability to evaluate both factual accuracy and topic coverage, provides a more comprehensive assessment compared to existing metrics that only measure one of these aspects. ## related work text generation evaluation traditional approaches to evaluating generated text have primarily focused on n-gram overlap metrics such as bleu @xcite , rouge @xcite , and meteor @xcite . while these metrics are effective for assessing local coherence and fluency, they fail to capture higher-level aspects such as topic coverage and diversity. recent work has introduced more sophisticated metrics like bertscore @xcite , bleurt @xcite , and unified multi-dimensional evaluators @xcite which leverage pretrained llms for more nuanced evaluation. ## icat queries that require a long-form response, e.g., complex non-factoid questions, are often associated with multiple aspects. the response to these queries often include multiple claims, some of which may be factually accurate, while others may be inaccurate. an ideal response to these queries should not only contain factually accurate claims, but should also leave no aspect or perspective unaddressed. for instance, an ideal answer to a question about a legislation should cover perspectives from all political parties. an ideal answer to a question about the impact of a food or a medication on health should cover both positive, neutral, and negative perspectives. however, no existing evaluation metric can evaluate both factual accuracy and aspect coverage in long-form text generation. to address these, given a long output y produced in response to an input x, the icat framework computes two main scores: factuality score and coverage score. factuality score. building upon prior work, such as factscore @xcite and veriscore @xcite , factuality score measures the ratio (or percentage) of factually accurate claims in y. to do so, it is crucial that the generated claims are accurate. let ac(y) be a function that extracts atomic claims from a generated response y. given the set of atomic claims @xmath0, we define the function c @xmath1; k) that verifies the factuality of claims in c using a given knowledge source k. therefore, c t ⊆ c denotes the set of factually verified claims in response y. factuality score is then defined as follows: coverage score. addressed in the generated response. formally, coverage score can be defined as: the icat β score. inspired by f-measure (van rijsbergen, 1979), we calculate the weighted harmonic average of these factuality and coverage scores, as follows: variants of icat we study three variants of icat implementations based on how they obtain query aspects and compute the alignment between atomic claims and aspects. the rest of this section provides details on how to develop models for generating factual claims (i.e., function ac), how to validate the factuality of claims (i.e., function cg), and how to obtain query aspects and compute an alignment between factual claims and all query aspects (i.e., functions t q and t o). the rest of this section describes the approaches at high level to introduce the generic icat, while section 4 provides the implementation details used in our experiments. ## experimental setup dataset. we conducted our experiments using the clueweb09 category b corpus-a large-scale web collection with over 50 million english documents @xcite . this corpus has been used in trec web track from 2009 to 2012 @xcite @xcite , consisting of 200 topics, derived from a commercial search engine's query log, balanced for pop-ularity. each topic includes a topic title (i.e., often used as the keyword search query), a description (i.e., detailed description of the information need), type, and subtopics (i.e., diverse aspects of the topic). the relevance judgments encompass 38,637 query-document pairs, with 19.06% (7366) marked as relevant. the dataset's unique advantage lies in its comprehensive coverage of internet content and human-annotated relevance judgments for topical diversity assessment. relevance was judged either binary or on a five-point scale (later converted to binary), with documents considered relevant when containing useful information for specific subtopics. in our experiments with this collection, we filtered out spam documents using the waterloo spam scorer @xcite with the threshold of 70%. we used bm25 to retrieve 1000 documents for each topic (given its title as the query string) and considered these documents for retrieval in our factual verification process. from the original 200 trec topics, we filtered out approximately 21 queries that were not suitable for long-form text generation evaluation (e.g., queries asking for images, urls, or specific data points), resulting in 179 valid queries. for human evaluation and detailed component analysis, we randomly sampled 50 queries from this filtered set due to cost constraints. these 50 queries were also used for llm evaluation to ensure consistency and enable direct comparison with human judgments. ## conclusions and future work we presented icat, a comprehensive framework for evaluating topic coverage in llm-generated text. through extensive experimentation using the clueweb09 dataset, we demonstrated the framework's effectiveness across different evaluation scenarios, with our best method achieving strong correlation with human judgments. the modular architecture of icat enables flexible adaptation to various evaluation requirements, from manual to automatic approaches for aspect identifications and alignment. our results highlighted several key findings: (1) the importance of sophisticated coverage models in improving evaluation accuracy, (2) the viability of automatic evaluation approaches that maintain comparable performance to methods requiring ground truth annotations, and (3) the framework's ability to provide meaningful assessments across different llm architectures and scales. in future work, individual components of our icat could be improved in terms of effectiveness and efficiency. additionally, exploring the relationship between model size, evaluation accuracy, and computational efficiency could provide valuable insights for practical applications. last but not least, the potential bias introduced by using the same or similar llm when generating ground-truth aspects should be investigated. by using our metric, other works can explore methods for optimizing llms to produce more comprehensive outputs.