Fair Abstractive Summarization of Diverse Perspectives (2024)

Abstract: People from different social and demographic groups express diverse perspectives and conflicting opinions on a broad set of topics such as product reviews, healthcare, law, and politics. A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. However, current work in summarization metrics and Large Language Models (LLMs) evaluation has not explored fair abstractive summarization. In this paper, we systematically investigate fair abstractive summarization for user-generated data. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people, and we propose four reference-free automatic metrics by measuring the differences between target and source perspectives. We evaluate nine LLMs, including three GPT models, four LLaMA models, PaLM 2, and Claude, on six datasets collected from social media, online reviews, and recorded transcripts. Experiments show that both the model-generated and the human-written reference summaries suffer from low fairness. We conduct a comprehensive analysis of the common factors influencing fairness and propose three simple but effective methods to alleviate unfair summarization. Our dataset and code are available at https://github.com/psunlpgroup/FairSumm.

## introduction different social and demographic groups of people hold diverse and potentially even conflicting perspectives and opinions, which are expressed in usergenerated text data in various domains and topics such as product reviews, social debates, healthcare, law, and politics @xcite @xcite @xcite . when a summarization system is faced with diverse review 1: this is the most stinky thing i've ever got ... looks nice but the whole room is smell so bad. i wonder if the bad smell will go. ... review 2: this chair is not red it's more like neon orange also footrest is unstable. chair, in general, is not good quality. i returned it and took a hit on the shipping which i'm not happy about. ... ## summarization of diverse perspectives with social attributes in this section, we first introduce social attributes, then formulate the task of summarization from diverse perspectives with social attributes. ## definition of fairness in summarization in this section, we provide definitions for fairness. figure 2 shows the logic of computation of fairness in abstractive summarization and corresponding evaluation metrics. the first stage is to produce the value distribution. given a sample pair of source and target text, we first split the source according to the annotated values of social attributes. then, we apply n-gram/bert/bart scores to compare target tokens with source tokens to obtain the value of each token in the summary. next, we count the number of tokens for each value in the source and target and obtain the value distribution of them. ## perspectivesumm to conduct a comprehensive analysis of fairness, we select and then process existing datasets to form our fair summarization benchmark perspec-tivesumm. we follow two main principles to choose six datasets from a large variety of existing datasets. first is the quality, the source texts need to be human-written and marked with clear, and precise social attributes. these attributes are existing metadata in the datasets, except for the sentiment attribute, which is derived from classifier predictions. second is diversity, the collected datasets need to cover various domains and perspectives. it covers six different domains with six attributes. overview of these six datasets are listed as follows (details in appendix a): claritin @xcite ) contains tweets about the effects of the drug claritin where @xmath0, female}. us election @xcite ) contains tweets posted during the 2016 us presidential election where @xmath1, pro-dem, neu}, meaning pro-republic, pro-democracy, or neutral. for claritin and us-election, we randomly sample tweets of different values with a certain ratio and then combine them as one input source. source text source dist. target dist. bart bert n-gram split target matching hard bur uer soft threshold auc sof produce value distribution compute final metrics attribute target text split by value value 0 value n ... count tokens figure 2: overview of our proposed metrics. dist. means value distribution. amazon and yelp are two datasets from the few-sum dataset @xcite containing product and business reviews. each sample in these two datasets consists of eight user-written reviews. in amazon dataset, @xmath2, and @xmath3, 2, 3, 4, 5} shows the ratings of each reviewer. in yelp dataset, @xmath4, and @xmath5, neu, neg} displays the sentiment of reviews. supremecourt (danescu-niculescu-mizil et al., 2012) contains a collection of conversations from the u.s. supreme court oral arguments, where @xmath6, v are the names of all participants. intelligence squared debate dataset (iq2) @xcite collects public debates that follow the oxford style and are recorded live, where @xmath7, v contains the names of all participants. for supremecourt and iq2, we truncate each transcript of the debate into shorter segments of k tokens. we include the whole text of each sample for the experiments of claude-100k. ## evaluating fairness in summarization as shown in figure 2 , we define four types of metrics to evaluate whether the generated summaries are fair: binary unfair rate (bur), unfair error rate (uer), area under curve (auc), and second-order fairness (sof). definition 5.1 binary unfair rate (bur). we define binary unfair rate (bur), a binary function that outputs 1 if the sample is unfair; and 0 otherwise. to be specific, we define the summary as fair if and only if p y (v k ) ≥ τ • p x (v k ) holds for each value @xmath8, 2, • • • , r, where 0 ≤ τ ≤ 1 is a tolerance hyperparameter that is chosen based on application scenarios @xcite . otherwise, if p y (v k ) < τ • p x (v k ), we say the value v k is underrepresented. thus this metric is either zero to one and it can be averaged over the dataset, demonstrating the proportion of summaries that are unfair in the dataset. however, this hard metric may not be able to describe which one is more unfair/fair if bur of two samples is the same. thus, we propose unfair error rate as a soft metric to compute the sum of the distance to source distribution of each value. definition 5.2 unfair error rate (uer). this metric defines a function f uer (x, y) measuring the distance between p x and p y . uer serves as the supplementary metric for bur as it shows more fined-grained details compared with bur. ## experiment results and analysis our results and analysis aim to answer the following research questions: we test nine llms as listed in appendix f. for analysis, we use gpt-turbo-3.5 on the claritin dataset with n-gramscore as the target matching method, if not specified. ## improving fairness of abstractive summarization we propose three methods to improve the fairness of summaries by llms: decoding temperature, summary length, and fairness instruction. ## conclusion and future work in this paper, we systematically investigate fair abstractive summarization. results of our metrics and human evaluation show that neither humans nor llms could maintain the fairness of summaries. further analysis shows that prompt engineering and careful choosing of the temperature of llms can significantly improve the performance of fairness. future work includes fine-tuning small or large language models to improve the fairness of abstractive summarization, extending metrics to other types of bias, and developing datasets that apply the concept of fairness in other generation tasks.