Bias in News Summarization: Measures, Pitfalls and Corpora (2024)

Abstract: Summarization is an important application of large language models (LLMs). Most previous evaluation of summarization models has focused on their content selection, faithfulness, grammaticality and coherence. However, it is well known that LLMs can reproduce and reinforce harmful social biases. This raises the question: Do biases affect model outputs in a constrained setting like summarization?To help answer this question, we first motivate and introduce a number of definitions for biased behaviours in summarization models, along with practical operationalizations. Since we find that biases inherent to input documents can confound bias analysis in summaries, we propose a method to generate input documents with carefully controlled demographic attributes. This allows us to study summarizer behavior in a controlled setting, while still working with realistic input documents.We measure gender bias in English summaries generated by both purpose-built summarization models and general purpose chat models as a case study. We find content selection in single document summarization to be largely unaffected by gender bias, while hallucinations exhibit evidence of bias.To demonstrate the generality of our approach, we additionally investigate racial bias, including intersectional settings.

## introduction pretrained large language models (llms) have increasingly found application across a wide variety of tasks, including summarization @xcite @xcite . while such models often evaluate favourably especially in human judgement for content @xcite , it is also well known that pretrained language models can often carry undesirable social biases @xcite @xcite . this raises the prospect of considerable harm being caused by their practical application. however, often these biases are studied in settings where model inputs are specifically crafted to reveal social biases @xcite @xcite . biases are also often observed in relatively unconstrained settings, such as dialog @xcite or the generation of personas @xcite . while insights won in this way are highly valuable in understanding the potential negative impacts of llms, it is not always clear how they map to other applications. summarization in particular is a highly conditional task. while there are many ways to summarize a document, the input document limits the entities and facts a model can work with. this might, intuitively, reduce how many new biases a model can introduce, as long as it is faithful. this leads us to ask: how can we study bias in text summarization? and to which extent do current models exhibit biases when applied to text summarization? we focus on gender bias in english since it is a well-known issue in llms @xcite @xcite @xcite and has grammatical indicators in english, making it an useful phenomenon to develop fundamental methodology for studying bias in text summarization. we run our experiments in a single document news summarization since it is a popular task @xcite @xcite which is performed well by current models @xcite and also a likely application of summarization models. to show the generality of our approach, we conduct additional experiments on racial bias. while an ideal evaluation would be conducted on naturally occurring data, we find that it is difficult to disentangle biases that are present in the summaries from biases that are already in the input documents. we thus propose a procedure that exploits high-quality linguistic annotations to generate mutations of real-world news documents with roger levin was named vice president and chief economist of this commodity futures and options exchange. he had been associate professor in the department of finance at seton hall university. melissa levin was named vice president and chief economist of this commodity futures and options exchange. she had been associate professor in the department of finance at seton hall university the chicago mercantile exchange (cme) has appointed a new chief economist. the chicago mercantile exchange (cme) has appointed its first female executive. 1. swap apparent gender of entities original document summarizer input summaries 2. summarize 3. observe differences jeffrey e. levin was named vice president and chief economist of this commodity futures and options exchange. he had been associate professor in the department of finance at seton hall university. figure 1 : schematic overview of our approach for summary gender bias evaluation with an example generated by bart xsum @xcite . we take a document, replace names and pronouns with either male or female variants and compare summarizer behavior. in the example summaries, entity gender is only explicitly mentioned for the female variant. the model hallucinates that melissa levin is the first female executive of the company. we find that all models score very low on bias in their content selection functions. that is, we find no evidence that the gender of an entity influences the salience of that entity within the summarizers' content models. where gender bias occurs, it is often linked to hallucinations. for racial bias we find largely comparable results. figure 1 shows a schematic overview of our approach, along with an example of a gender-biased hallucination. ## bias in text summarization bias in nlp is an overloaded term, which is not always used with a clear definition @xcite . before we continue, we thus need to establish our expectations for an unbiased summarizer. one approach chosen, for example, by @xcite is to require that all demographic groups receive equal representation in the generated summaries, following an equality of outcome paradigm @xcite . while a valid perspective, it requires models to actively counteract biases that might be present in the input documents. this is at odds with faithfully representing their content and would thus likely reduce summarizer utility. we instead expect summarizers to be faithful to the input but to not amplify their bias. we define three forms of bias under this setting and discuss their harms @xcite : inclusion bias, hallucination bias and representation bias. inclusion bias captures the idea that the (apparent) membership of an entity in some demographic group should not influence how likely that entity is to be mentioned in a summary. if we frame content inclusion in terms of a classification problem over the content units in a document, this corresponds to demanding equality of opportunity @xcite , as opposed to equality of outcome. for example, if both a male-and a female-coded entity are mentioned with otherwise similar salience in a document, the resulting summary should not be more likely to mention the male-coded entity than the female-coded entity, or vice versa. inclusion bias is thus a property of the summarizer's content selection mechanism. inclusion bias poses a form of allocative harm @xcite since it reduces visibility of members of certain groups if, for example, news is consumed through the filter of automatic summarization. abstractive summarization systems suffer from hallucinations @xcite , that is summary content unsupported by the input. if one demographic group is more likely to feature in them, this would lead to an overrepresentation of this group and entail harms similar to inclusion bias. we call this hallucination bias. the above measures can not capture all kinds of possible bias. as an additional canary, we fi-nally also measure representation bias, which intuitively measures any kind of summary deviation based on which groups are mentioned in the input. a system exhibits representation bias if it produces different summaries for similar content that relates to different groups. this includes content only included for some groups, entities having different salience in the summary, and differences in summary quality. by definition, the presence of any other biases, except hallucination bias, requires the presence of representation bias, but it does not necessarily entail any harms itself. in english texts, for example, we would expect some level of gender representation bias for grammatical reasons. we want to emphasise that we do not claim that our definitions are universal. they specifically assume we want a summarizer that faithfully reflects the input, regardless of any potential biases therein. ## bias measures we operationalize our bias measures for a set of demographic groups g. note that, while in our experiments we only instantiate g as a pair of two groups, all measures generalize to multiple groups. ## input documents are already biased all proposed measures, except hallucination bias, require us to isolate the effect of a particular demographic group in the input. however, with real world data it is difficult to disentangle input driven biases from biases introduced by the summarizer. to demonstrate, we investigate the frequency of gender identifiers from our inclusion score word lists w g on cnn and xsum inputs. we find that 62%/74% of identifiers are male for cnn/xsum, i.e. men are mentioned at a much higher rate. while this simple frequency issue would be mitigated by our formulation of inclusion bias that takes the input distribution into account (see section 3.1), we find that the underlying issue goes beyond just mention frequency. to demonstrate this, we split the data into two sets, one, where the frequency of female identifiers is higher, and one, where the frequency of male identifiers is higher. we then apply the fightin' words method @xcite with an uninformative dirichlet prior (α = 0.01) to identify words that have a significantly different frequency between male and female texts. 5 since all identifiers have paired male/female variants, we replace these pairs with special markers. this allows us to compare the frequency of the male/female variants (e.g. "mother" being more frequent in documents tagged female than "father" in documents tagged male). 6 we show results in ignoring the titles (mr./mrs./ms.), we see that a number of words have highly significant z-scores (z ≫ 1.96). specifically, in both corpora the male documents are much more likely to mention sportsrelated words foot_6 , while documents with more female identifiers have much higher occurrence of words related to family like husband, children etc. we demonstrate the consequences of biased input by examining word inclusion bias of clearly biased and unbiased summarizers. we consider two content-agnostic baselines: random selects three random sentences. lead selects the first three. we also study two content-aware summarizers. for this we heuristically classify every article as either mentioning more family or more sport based keywords or neither (unknown). details can be found in appendix b. topic randomly samples one, three or six sentences when the article is classified as family, unknown, or sport respectively. sexist selects three sentences to maximize the frequency of male identifiers for sport and of female identifiers for family articles, acting randomly otherwise. the latter is clearly the most biased, while neither random nor lead can, by construction, amplify bias. any bias in topic is a correlation of topics with gender in the input, not due to the algorithm. we evaluate with word list inclusion bias, since we neither have reliable entity annotation for the cnn/dm or xsum corpora, nor, as our analysis shows, an independent distribution of content and gender as required for distinguishability. results in ## summarizers we study both purpose-built summarizers and chat models. for purpose-built models we use bart @xcite and pegasus @xcite , both transformer models finetuned for summarization. we use the xsum and cnn/dm foot_11 models. for chat models we choose llama-2 chat @xcite 7b, 13b and 70b models with the standard system prompt. for the chat models, we randomly select one prompt per summary from a list of ten prompts designed to elicit summarizing behavior (see appendix g). we report statistics in appendix h. ## gender bias results remarkably, we find that all models carry a bias towards male entities in their hallucinations. we study this in more detail in section 9.1. all models show some degree of distinguishablity, with bart summaries showing the most pronounced differences between summaries for male and female coded documents. as noted in section 2, this is not in itself sufficient to establish whether this leads to harm to any particular group. we thus analyse this further in section 9.2. ## validating our measures to validate our approach, we conduct the following tests: 1) we check whether our modified input documents lead to degraded summary quality. 2) we test whether also altering content words in addition to names, pronouns and titles to conform to changed entity gender would impact results. 3) we test whether our method is capable of detecting inclusion bias in clearly biased summarizers. ## extension to racial bias our methods are applicable to any group-based bias where group membership can be indicated using names. we demonstrate this by investigating racial bias for stereotypically black and white names. we use the name dictionary of @xcite . we change both first and last names, since both are relevant in communicating race. entity gender is selected at random. due to the small name inventory, we can not generate instances for all documents. we thus only consider originals where we can generate a full set of 20 inputs, leaving us with 12,240 instances per dataset. since word lists for racial bias typically rely on last names, we only compute entity inclusion bias. we also opt not to compute hallucination bias, since we want to avoid constructing a classifier that attempts to identify entity race. ## related work while bias in llms is the subject of intense research @xcite @xcite , bias in summarization is underexplored. @xcite take only inclusion bias into account, measured by word lists, thus not respecting the prior distribution of groups in the input documents (see also section 2). their use of cnn/dm and xsum, both highly biased, makes it difficult to attribute the bias they find to amplification by models. @xcite study summarizers' gender bias on gpt-2-generated documents @xcite using word-embeddings. they find an overrepresentation of men in summaries. in comparison, our data construction reduces the risk of false positives due to input biases and our more differentiated measures suggest hallucination as a likely cause. @xcite find summarizers treat articles differently when replacing biden with trump and vice versa. while their replacement approach is similar to ours, both their subject of study and measures are highly specific to political bias. bias has also been observed in tweet and opinion summarization, where contributions by minority groups in the input are underrepresented @xcite @xcite @xcite . in contrast to our bias definition, which focuses on different treatment of groups, here bias is a failure to represent the full distribution of opinions and/or authorship. @xcite show models tend to hallucinate entity nationality in biographical summaries. this is consistent with our observation that the most problematic behaviours stem from hallucinations. our approach for generating inputs is related to approaches that generate context that ought to elicit equal behavior for perturbations to the input @xcite , although to the best of our knowledge we are the first to apply such modifications for bias in text summarization. ## conclusion we have introduced definitions that allow us to clearly formulate expectations for what constitutes bias in summarization, along with measures that allow us to detect these biases. we have shown that any measure of summarizer bias must account for bias in the input and proposed a rule-based method that allows us to create realistic data with controlled entity distribution for studying summarizer bias. our evaluation of seven models indicates that their content selection is not strongly affected by either gender bias or racial bias for black/white coded names. however, we caution that content selection in news summarization is known to be subject to easy heuristics like the lead "bias" @xcite . summaries might be more susceptible to biases in more complicated settings. we find significant gender bias in hallucinations revealing a connection between them and bias that suggests increasing faithfulness as a mitigation strategy.