Coverage-based Fairness in Multi-document Summarization (2025)

Abstract: Fairness in multi-document summarization (MDS) measures whether a system can generate a summary fairly representing information from documents with different social attribute values. Fairness in MDS is crucial since a fair summary can offer readers a comprehensive view. Previous works focus on quantifying summary-level fairness using Proportional Representation, a fairness measure based on Statistical Parity. However, Proportional Representation does not consider redundancy in input documents and overlooks corpus-level unfairness. In this work, we propose a new summary-level fairness measure, Equal Coverage, which is based on coverage of documents with different social attribute values and considers the redundancy within documents. To detect the corpus-level unfairness, we propose a new corpus-level measure, Coverage Parity. Our human evaluations show that our measures align more with our definition of fairness. Using our measures, we evaluate the fairness of thirteen different LLMs. We find that Claude3-sonnet is the fairest among all evaluated LLMs. We also find that almost all LLMs overrepresent different social attribute values. The code is available at https://github.com/leehaoyuan/coverage_fairness

## introduction multi-document summarization (mds) systems summarize the salient information from multiple documents about an entity, such as news articles about an event or reviews of a product. typically, such documents are associated with social attributes e.g. political ideology in news and sentiments in reviews. documents with different social attributes tend to have diverse information or conflicting opinions. a summary for them should fairly represent differing opinions across documents. fairness in mds measures whether a system can generate summaries fairly representing information from documents with different social attribute values. it can be measured at a summary-level-quantifying how fairly an individual summary represents documents with different social attribute values or at a corpus-level-quantifying how fairly a corpus of summaries as a whole represents documents with different social attribute values. previous works in this area measured fairness in extractive or abstractive settings @xcite @xcite . these works generally evaluate the fairness of a system as the aggregated summary-level fairness of its generated summaries. it is measured using proportional representation-a fairness measure based on statistical parity @xcite . it requires that a document sentence being selected as a summary sentence is independent of its originating document's social attribute value. therefore, the distribution of social attributes among information in a fair summary should be the same as the distribution of social attributes among input documents. the definition of proportional representation (pr) suffers from two key problems. the first problem is that proportional representation does not consider the redundancy in input documents, common in mds. for example, in reviews of a hotel with great views, multiple reviews could comment on the view. for input documents with much redundancy, it is difficult to generate a non-redundant, fair summary according to pr. for example, suppose 67% of input reviews are positive and mostly discuss topics a and b, while 33% of reviews are negative and evenly discuss topics d, e, f, and g (fig. 1a ). according to pr, summary 1 is fairer than summary 2. however, summary 2 is clearly better since, unlike summary 1, it avoids repetition (summary 1 repeats topic a twice) and almost equally cover the information from both positive and negative reviews. hence, there is a need to redefine fairness to consider redundancy in mds. besides, the idea behind pr can work for measuring fairness in other nlp tasks but not for abstractive summarization because recent llmbased summarization methods can use quantifiers like "most" and "a few" to indicate the amount of input information covered by a summary sentence. hence, there is a need to define fairness based on the proportion of input information covered by the summary. with this motivation, we propose a new summary-level coverage-based fairness measure, equal coverage (ec). unlike pr, ec requires a document being covered by a summary sentence to be independent of its social attribute value. since a summary sentence can cover multiple documents with similar contents, ec can address redundancy common in mds. the second problem is evaluating the fairness of an mds system only using summary-level fairness measures can overlook corpus-level unfairness. consider system 1 in fig. 1b . most of its summaries (blue dots) have a higher coverage probability for negative reviews than positive reviews. we observe that system 1 is unfair because its summaries overrepresent negative reviews. system 2 is fairer because its summaries (orange dots) have an equal chance of overrepresenting negative or positive reviews. since individual summaries from both systems overrepresent negative or positive reviews, their summary-level fairness scores may be comparable. hence, aggregated summary-level fairness scores cannot identify that system 2 is collectively fairer than system 1. to address this problem, we propose a new corpus-level fairness metric, coverage parity. coverage parity is based on the principle that documents with different social attribute values should have equal chances of being overrepresented or underrepresented. therefore, it can check whether the systems are equally (un)fair on different social attributes and identify which social attribute is overrepresented or underrepresented. our human evaluation shows that our measures align more with our definition of fairness than proportional representation. using our measures, we evaluate the fairness of thirteen different large language models (llms) @xcite in diverse domains: news, tweets, and reviews. for these domains, we consider social attributes with significant real-world impacts: ideologies and stances for news and tweets and sentiment for reviews. our experiments find that for llama2, claude3, mixtral, and gemma2, larger models are fairer, but this trend is inconsistent for gpts and llama3.1. our experiments also find that most llms tend to overrepresent certain social attribute values in each domain. it is an important finding that users can use to calibrate their perception before using llm-generated summaries. it can also be used by developers to build fairer llm-based summarization systems. to conclude, our contributions are three-fold: • we propose a new summary-level fairness measure, equal coverage, which incorporates redundancy of input information, common in mds; • we propose a new corpus-level fairness measure, coverage parity to detect corpus-level unfairness; • we evaluate the fairness of llms using these two measures in various domains. 2 related work shandilya et al. (2018, 2020); dash et al. (2019) propose to measure the summary-level fairness in mds under the extractive setting using proportional representation. they propose an inprocessing method to improve the fairness of extractive summaries by adding a fairness constraints to the optimization target. similarly, keswani and celis (2021) uses proportional representation to measure the fairness of summaries under different distributions of social attributes in input documents and proposes a post-processing method to improve fairness. olabisi et al. (2022) uses the same measure to measure fairness and proposes a clusteringbased pre-processing method to improve fairness. recently, zhang et al. (2023); huang et al. (2024) extend proportional representation to measure fairness under the abstractive setting. to estimate the distribution of social attributes in a summary, zhang et al. (2023) maps the summary back to the originating documents, while huang et al. (2024) uses a finetuned model to estimate the social attribute of each summary sentence. lei et al. (2024) measures the fairness similarly as huang et al. ( 2024 ) and proposes to improve the fairness of abstractive summaries using reinforcement learning with a polarity distance reward. however, these works generally measure fairness using proportional representation, which has the limitations discussed in the introduction. in our experiments, we use proportional representation as a baseline. ## fairness measures in this section, we describe notation (sec. 3.1) and our proposed measures, equal coverage (sec. 3.2) and coverage parity (sec. 3.3). ## experimental setup we now describe experimental setup to evaluate the fairness of llms using our measures. ## experiment results we now describe experiment results to evaluate the fairness of llms using our measures. ## conclusion we propose two coverage-based fairness measures for mds, equal coverage for measuring summarylevel fairness and coverage parity for measuring corpus-level fairness. using these measures, we find that claude3-sonnet is the fairest among all llms. we also find that most llms overrepresent certain social attribute values in each domain. future works can explore the effect of training data, especially instruction tuning and preference tuning data, on the fairness of llms. future works can also finetune llms based on our measures to develop fairer models.