Entity-Based Evaluation of Political Bias in Automatic Summarization (2023)

Abstract: Growing literature has shown that NLP systems may encode social biases; however, the *political* bias of summarization models remains relatively unknown. In this work, we use an entity replacement method to investigate the portrayal of politicians in automatically generated summaries of news articles. We develop an entity-based computational framework to assess the sensitivities of several extractive and abstractive summarizers to the politicians Donald Trump and Joe Biden. We find consistent differences in these summaries upon entity replacement, such as reduced emphasis of Trump’s presence in the context of the same article and a more individualistic representation of Trump with respect to the collective US government (i.e., administration). These summary dissimilarities are most prominent when the entity is heavily featured in the source article. Our characterization provides a foundation for future studies of bias in summarization and for normative discussions on the ideal qualities of automatic summaries.

## introduction automatic summarization aims to identify the most important information in input documents and produce a concise summary to save readers time and effort @xcite . however, the pursuit of efficiency may come with undesired biases, especially in a political context. bias of a political nature can contribute to issues like polarization and misinformation. it is also subjective and difficult to tell on the basis of a single article. in this work, we conduct the first empirical study to examine the political bias of automatic summarizers based on large-scale pre-trained language models. we investigate the portrayal of politicians in automatically generated summaries through an entity replacement experiment design; the advantage of this setup is the ability to control for the content of the source document. figure 1 shows an example: given a news article about joe biden, we replace all occurrences of biden's name with that of donald trump and examine the differences between the summary of the original article and that of the replaced article. what should an ideal summarization model output for these two versions of an article, which differ only by one entity name? this normative question is crucial to consider for responsible deployment and usage of summarization models. a first step towards addressing this query is to uncover whether state-of-the-art automatic summarizers actually encode any bias at all. in this paper, we focus on this empirical demonstration and leave the deliberation of desirable differences for future work (see §5). we demonstrate substantial differences across four summarization models. the consistent biases in the generated summaries suggest robustly different model representations of the two politicians. furthermore, entity-centric news articles, those that heavily feature the original entity, lead to more dissimilar summaries upon replacement. we acknowledge that "bias" is an overloaded term; in this paper, we use it to refer to significant differences with respect to entity representation. we do not endorse the interpretation that the summaries are "anti-trump/anti-conservative". in summary, our main contributions are: original article systematic and consistent differences in joe biden's inauguration, hope for truth and unity. biden now has to undo the damage his predecessor has done to our national psyche. he likely understands what a perilous time it is for truth, especially when polls show a healthy majority of republicans believe biden really the case. original summary ...[83]... joe biden has been inaugurated as the 46th president of the united states. the national nightmare that began on jan. 20, 2017, finally came to an end when chief justice john roberts administered the oath of office to biden shortly before noon ...[595]... ...[83]... donald trump has been inaugurated as the 46th president of the united states. the national nightmare that began on jan. 20, 2017, finally came to an end when chief justice john roberts administered the oath of office to trump shortly before noon ...[595]... replaced article entity replacement donald trump has been inaugurated as the 46th president of the united states. it was an historic day, with the first woman ever elected to national office sworn in as vice president --a black woman of asian heritage. in donald trump's inauguration, hope for truth and unity. replaced summary summarization model è emphasizes "undo[ing] the damage", republicans è emphasizes election of harris (black, asian, woman) ## related work there are two primary types of automatic summaries: extractive and abstractive @xcite . neural summarization models have demonstrated superior performance in benchmark datasets (e.g., zhang et al. ( 2020 )). to measure performance, rouge scores are typically used @xcite , but such metrics have limited ability to cohesively assess summary quality @xcite , including any potential political bias. there is strong resemblance between selecting the most important information in the summarization process and the phenomenon of media framing, where media outlets selectively report aspects of an issue or quotes of politicians @xcite . the selection process by media outlets has received substantial attention in computational research @xcite @xcite , our identified problem differs in that the content selection is done by models rather than media outlets and that the kind of bias may not align with the conservative vs. liberal biases as defined for media framing. a growing line of work investigates the biases of nlp systems to understand their potential harms if/when deployed @xcite . many studies have covered representation bias and social bias (e.g., gender and race) @xcite @xcite @xcite . in comparison, the harm of political bias is more subtle; it can be difficult to observe in an individual article, but can still have substantial impact (e.g., affecting elections in us swing states). while @xcite tracks political bias in language models for hate speech and misinformation detection, our work is the first study to reveal political bias in automatic summarization models. in addition, prior work has examined factual consistency in generated texts @xcite @xcite @xcite , fairness of extractive summaries with classic summarization methods @xcite , and generating neutral summaries to mitigate media bias @xcite . @xcite take an entity-based approach to evaluating factuality in abstractive summaries. replacement paradigms similar to our own have been used to uncover biases in other nlp applications, such as language modeling @xcite @xcite and neural machine translation @xcite . ## methodology in this section, we introduce our data, experimental design, and the models we evaluate. data and code are available at https://github.com/ chicagohai/entity-based-political-bias . data: for source documents, we use cleanlyparsed articles from the now corpus 1 from 1/1/2020 to 12/31/2021 that have at most 5,000 words. among these news articles that mention any us politician, about 62.9% include donald trump and/or joe biden. thus, we focus our investigation of bias on the entities trump and biden: well-known political figures that play important roles in us news. we separate the articles into two experimental groups: 157,648 articles that mention @xmath0: ↑↑↑↑ / ↓↓↓↓ @xmath1.001: ↑↑↑ / ↓↓↓ @xmath2.01: ↑↑ / ↓↓ @xmath3.05: ↑ / ↓ @xmath4.05: ## framework and results we present our results for similarity between summaries, and two feature functions: 1) entity name mentions in summaries and 2) temporal differences in status indicators. we focus on these two features in this short paper to zero in on the basic notion of representation in sensitivity to political entities. all results involve comparing frequencies via paired t-tests; we present results using the notation described in ## conclusion in this work, we conduct the first empirical study on political bias in summarization models. through experiments that replace the names of donald trump and joe biden with other presidential entities in us news articles, we identify several areas of variation in summaries when the source article differs by a single person's name. we show that summarization models learn different representations for different entities and provide consistently different summaries depending on the entities involved. our framework can be used for further analysis of biases in summarization models. politicians are often in similar situations, such as signing a bill or meeting a world leader. models with distinct representations for different entities will generate different summaries for such reports. these variations challenge the assumption that summaries are inherently neutral, that one can produce or seek a single gold-standard automatic summary. such differences and their downstream effects (e.g., on voter preferences) warrant further exploration. ultimately, we must consider: what constitutes an ideal automatic summary? this is an impor-tant normative query, which we purposefully do not answer in this paper. having distinct representations of politicians may indicate a nuanced model, but that does not automatically justify all differences. characterizing the systematic biases encoded in automatic summarizers is a first step towards responsible use of such models. our work aims to motivate discussion of this essential yet overlooked research area.