Evaluation of Summarization Systems across Gender, Age, and Race (2021)

Abstract: Summarization systems are ultimately evaluated by human annotators and raters. Usually, annotators and raters do not reflect the demographics of end users, but are recruited through student populations or crowdsourcing platforms with skewed demographics. For two different evaluation scenarios – evaluation against gold summaries and system output ratings – we show that summary evaluation is sensitive to protected attributes. This can severely bias system development and evaluation, leading us to build models that cater for some groups rather than others.

## introduction summarization -the task of automatically generating brief summaries of longer documents or collections of documents -has, so it seems, seen a lot of progress recently. progress, of course, is relative to how performance is measured. generally, summarization systems are evaluated in two ways: by comparing machine-generated summaries to human summaries by text similarity metrics (lin, 2004; nenkova and passonneau, 2004) or by human rater studies, in which participants are asked to rank system outputs. while using similarity metrics is controversial (liu and liu, 2008; graham, 2015; schluter, 2017), the standard way to evaluate summarization systems is a combination of both. both comparison to human summaries and the use of human raters naturally involve human participants, and these participants are typically recruited in some way. in liu and liu (2008), for example, the human subjects are five undergraduate students in computer science. undergraduate students in computer science are not necessarily representative of the population at large, however, or of the end users of the technologies we develop. in this work, we ask whether such sampling bias when we take steps toward evaluating the impact of the gender, age, and race of the humans involved in the summarization system evaluation loop: the authors of the summaries and the human judges or raters. we observe significant group disparities, with lower performance when systems are evaluated on summaries produced by minority groups. see §3 and ## experiments we present two evaluations in this short paper: an automated scoring against human summaries (exp. a) and a human rater study (exp. b). in both experiments, we use amazon mechanical turk to recruit annotators from different demographic groups, and the first paragraphs of biographies from english wikipedia as our input data, using the wikidata api for extraction. @xcite we create a dataset of biographies of women and men, obtain human summaries, and generate summaries of these biographies using two out-of-the-box extractive summarization systems. in exp. a, we compare the system summaries directly to the human summaries (from different groups); in exp. b, we let human raters compare and rate the two system summaries. to ensure differences between the two summarization systems, we use the 2004 graph-based textrank (mihalcea and tarau, 2004) and the 2020 state-of-the-art, bert-based match-sum (zhong et al., 2020). @xcite we follow the match-sum guidelines described in (zhong et al., 2020) and limit the length of the input biographies to a maximum 5 sentences and force the output summaries to be between 2-3 sentences long. our final dataset consists of the original 975 biographies (700 men and 275 women), along with two automatic summaries, as well as human 3 sentence summaries, and is made freely available. @xcite our evaluations rely on annotations and ratings from amazon mechanical turk. for quality control, we rely on a control question, as well as analyzing annotation time: if a task is completed faster than one standard deviation of the average time spent, the answers in that task are discarded. we 2 https://query.wikidata.org/ 3 we use the implementation of textrank by barrios et al. (2016) 4 and the original matchsum implementation. @xcite match-sum obtains state-of-the-art performance across a range of benchmarks by learning to produce summaries whose document encoding is similar to that of the input document. tex-trank is a much simpler extractive algorithm; it adopts pagerank to compute node centrality recursively based on a markov chain model. while matchsum obtains a rouge-1 score of .44 on cnn/daily mail, textrank obtains a rouge-1 score of .33 (zheng and lapata, 2019). we use both systems with recommended parameters, as was done in zheng and lapata (2019). note that textrank, in contrast to matchsum, is unsupervised. our rouge-1 scores below for wikipedia biographies are generally comparable. ## analysis in order to analyze the differences between the rating behavior of subdemographics, we learn which features are significant for each demographic by training a simple logistic regression text classifier trained on the summaries ranked by each of the subdemographics with significantly different ranking behavior. as task representation, we represent each ranking instance as a vector of 2*149 features, one 149-sized subspace for each summary. each subspace is made up of a one-hot vector of 145 frequent words (from the english stop words list in nltk 8 ), as well as four task specific features: the summary's average word length, whether the first sentence of the biography is included in the summary, the type/token ratio, and the text complexity of the summaries. we concatenate the 149 features from each system and scale them. we extract the top 20 most salient features for each demographic group and analyze them manually: the average word length of the matchsum system correlates positively to annotators preferring matchsum across several demographics, e.g., over 30 and male white, but this effect is absent with female annotators. since the inductive bias of textrank does not explicitly prohibit redundancy (mihalcea and tarau, 2004), this finding indicates that matchsum is preferred among older men, especially whites, when it is informative, introduces main entities, etc. however, other subdemographics seem less sensitive to this variation. matchsum is not generally rated more informative and useful across demographics ( our main observation is that female and black participants under 30 prefer textrank over match-sum. what drives this? the main predictors in our logistic regression analysis are a) textrank extracting the first sentence of the biography (twice as frequently than matchsum, in more than half of its summaries); and b) textrank sentences containing negation. the former suggests a need for anchoring or framing of the summary, as initial sentences tend to provide this; the latter could suggest that young female or black participants are less prone to the common bias of evaluating negated sentences as less important (kaup et al., 2013). ## conclusion our paper is, as far as we know, the first to evaluate summarization systems across different subdemographics. we did so in two different evaluation scenarios: automatic evaluation against gold summaries and system output ratings by human evaluators. we made the gold summaries and the ratings available for future research. what did we learn from our experiments? most importantly, of course, we learned that performance numbers differ when evaluated on summaries written by different subdemographics, and that the preferences of rathers from different subdemographics differ. in our experiments with automatic evaluation against gold summaries written by different subdemographics, we saw that summarization systems achieve higher performance scores when evaluated on summaries produced by white men, highlighting an unfortunate bias in these systems. in our rater studies, we also saw significant differences across subdemographics. most surprisingly, perhaps, we saw that a summarization system from 2004 was rated better than a state-of-the-art system from 2020 by some subdemographics, and effect that was found to relate to the occurrence of first sentences (providing anchoring or framing of summaries) and negation (often evaluated as less important by majority groups). for now, we can only speculate what a summarization system optimized to perform well across all subdemographics would look like, e.g., a system minimizing the worst-case loss across subdemographics rather than the average loss. our results show very clearly, however, the current state of the art in summarization is biased toward some demographics and therefore fundamentally unfair.