No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media (2022)

Abstract: News articles both shape and reflect public opinion across the political spectrum. Analyzing them for social bias can thus provide valuable insights, such as prevailing stereotypes in society and the media, which are often adopted by NLP models trained on respective data. Recent work has relied on word embedding bias measures, such as WEAT. However, several representation issues of embeddings can harm the measures’ accuracy, including low-resource settings and token frequency differences. In this work, we study what kind of embedding algorithm serves best to accurately measure types of social bias known to exist in US online news articles. To cover the whole spectrum of political bias in the US, we collect 500k articles and review psychology literature with respect to expected social bias. We then quantify social bias using WEAT along with embedding algorithms that account for the aforementioned issues. We compare how models trained with the algorithms on news articles represent the expected social bias. Our results suggest that the standard way to quantify bias does not align well with knowledge from psychology. While the proposed algorithms reduce the gap, they still do not fully match the literature.

## introduction social bias describes prejudices and stereotypical thinking towards certain groups in society, such as genders or ethnicities @xcite . media bias, by contrast, refers to the tendency of media entities (e.g., a news outlet) to favor certain facts, views, or framings of events over others @xcite . in this work, we focus on media bias induced by political orientations (henceforth, political bias). while social and political bias differ in appearance, it can be expected that they relate to and mutually influence each other. a particular political bias, for example, may transport ideas of stereotypes, manifesting as social bias, that strengthen specific political ideas in society @xcite … if we want to raise our daughters to be different kind of womennonconformists in a world run amok, insurgents for the gospel -we must be sure to give them strategic and specialized training. we must teach them both the beauty and the basics of biblical womanhood through our faithful (though flawed) example and our gracious teaching. we must also pluck the weeds of feminism that our culture sows and which can take root in our daughters' hearts. … … falling in love with a woman helped me move out of my marriage and into a new world of women. i discovered that intimate relationships with women were based on parity--there were no predetermined roles. both partners were women, born and raised with similar gender expectations. … 1999). vice versa, holding particular stereotypical views may make people more susceptible to a political view promoted by a news outlet @xcite . figure 1 shows excerpts of two news articles, conveying potential gender bias. the outlined kinds of bias are also relevant to nlp methods that employ news articles to train models @xcite or as a knowledge source @xcite . for example, bias present in the articles may be learned and amplified by word embeddings if not explicitly accounted for. this impacts generalization performance negatively @xcite and may have harmful consequences in practical applications @xcite . so far, one hurdle to mitigate these problems is the limited reliability of common measures of social bias present in a corpus @xcite , stemming from embedding training algorithms not tailored to low-resource situations @xcite . in this paper, we investigate how to assess social bias more reliably while empirically studying the interaction of social bias and political bias in us online news outlets. in particular, we identify low-resource settings and token frequency differences as two main issues with existing embeddingbased bias measures. we consider social bias towards genders, ethnicities and religions, and measure it with the widely used bias measure, weat @xcite . we restrict our political bias view to the unidimensional spectrum from left to right @xcite , ignoring objectivity and fairness aspects @xcite . in psychology literature, stereotypical views have been shown to coincide with political orientations (section 2), suggesting that the political views of news outlets coincide with social biases. under this premise, we aim to find out what word embedding algorithm best serves to reliably measure social bias. we investigate weaknesses of a standard algorithm that stem from the reliance on word lists, infrequent tokens in the data, and the quality of embeddings. we suggest (1) training frequency agnostic embeddings to compensate for lower quality of rare tokens, (2) a fine-tuned language model to account for smaller datasets, and (3) decontextualized embeddings to alleviate the "unnatural input" problem with contextualized models. for our experiments, we introduce a large-scale media bias corpus in section 4, covering more than 500,000 news articles from 47 english-language us online news outlets over 12 years @xcite @xcite @xcite @xcite @xcite @xcite . given the corpus, we evaluate each potential improvement and compare their capability to encode and represent social bias a text corpus (section 5). to this end, we systematically generate word embedding models from subsets of different political biases. in a second analysis, we explore the development of social bias in outlets over time in a respective manner. we can quantify the considered types of social bias for all models using weat. our results in section 5 provide evidence that the general embeddings quality improves notably over standard static embeddings. additionally, the proposed algorithms better model the expected social bias, though still not fully align with the literature. this work provides three contributions to computational research on bias in language: ## related work we consider social bias that manifests as stereotypes, that is, generalized beliefs about social outgroups based on experiences with single members @xcite . such beliefs may lead to prejudices and discrimination that cause lasting harm. stereotypes are usually transported through language, uttered either implicitly or explicitly @xcite . if entities with high public outreach, such as politicians and media outlets, spread stereotypes, this may therefore profoundly impact their audiences @xcite . psychology and political science literature study the relation of stereotypes with political aspects. as part of this, multiple layers of partisan biases have been evaluated @xcite @xcite . focusing on social values, @xcite find a general shift of public social values related to a shift in voting outcomes. other works compare social values of conservatives and liberals: while liberals seem more likely to reject "ingroup values", conservatives emphasize tradition and religion @xcite . accordingly, @xcite observe a higher level of self-reported prejudices towards social groups that "challenge or violate traditional social values" among conservative probands. @xcite , finally, report that liberals tend to value social equality, whereas conservatives justify social inequality with "the preservation of status quo". we use these connections between political bias and social values as a reference for our analyses. media bias can be evaluated from many angles, too. for instance, chen et al. (2020) explore media bias in political news, automatically detecting incomplete reporting and evaluating its linguistic manifestations. one of their results is that words expressing negative emotions are most correlated with selective biases. @xcite , in turn, focus on conservative and liberal news articles from the us, australia, and the uk to understand the construction of media frames. they find that the report framing of specific outlets aligns with their political bias. rather than unfairness or issue perception, our work targets the interaction of media bias with social bias in news articles. in particular, we quantify social bias in word embeddings with the widely used word embedding association test, weat @xcite . weat's main idea is to calculate the cumulative distance between groups of word vectors that describe a social group and attributes. similar measures exist, such as ect @xcite , rnsb @xcite , and mac @xcite , ripa @xcite , weatvec @xcite , the smoothed first-order co-occurrence @xcite and same @xcite but our goal is not to find the best measure. rather, we seek to learn how measures like weat behave for different embedding algorithms. we are not aware of works that have done similar. similar to the analysis we carry out, @xcite exploit the properties of word embeddings to evaluate temporal relationships between changes of social bias and empirical demographic changes in the us. they evaluate embedding models trained on texts from different decades, for example finding that gender bias decreased with the women's movement in the 1960's. in a comparable analysis, @xcite find that gender bias reduced in biomedical research over time for some areas, but not in others. in this work, we utilize weat to evaluate social bias in news articles. unlike previous work, however, we compare word embedding algorithms to model social bias in texts and their alignment with the literature reviewed above. closest to our work is the research of @xcite and @xcite . the former use weat to compare social biases present word embeddings trained on different ideological online wikis. all wikis are found to have similar biases for gender, race, and religion, but to varying degrees. @xcite collect us news articles from a liberal and a conservative media outlet. by training one embedding model for each outlet and measuring the differences of all words in the embedding spaces, they determine the most biased words. the underlying hypothesis is that words, for which the context varies more strongly, will also be more biased. we apply a data collection method similar to @xcite , but cover 47 outlets. additionally, instead of just focusing on two extreme communities, our corpus spans a wider spectrum of political opinions. our main goal is to deepen the understanding of the social bias in word embeddings for different training algorithms. ## method this paper studies how to best evaluate a text corpus for social bias, harnessing the ability of word embeddings to encode direct contexts. in particular, we quantify the social bias encoded in models trained on a corpus. the models are thus used as a proxy from which we derive the social bias in the original corpus. in the following, we present our evaluation method, discuss potential issues, and describe the employed embedding algorithms. ## data we now present the large-scale corpus that we acquired to study the existence of social bias in news articles across the political spectrum in the us. source data using media bias ratings from news aggregation platform allsides.com, we collected articles from liberal (left and lean-left labels) and conservative (right and lean-right labels), as well as neutral (center label) outlets. while this unidimensional view on the political spectrum is limited @xcite , it provides us with a clear distinction and makes results easier to interpret. we refer to news articles with liberal, neutral, and conservative labels as data subsets in section 5. similar to @xcite , we collected news articles from common crawl foot_2 . since the media bias rating history is not available, we mapped each outlet to its current rating. to extract the pure text from the collected files in warc format, we used the library news-please @xcite . for our experiments, we also extracted the articles' date of publication automatically as far as possible. orientation 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 preprocessing to filter out non-english articles, we classified the language of each text automatically using the langdetect library foot_3 . in contrast, we intentionally did not filter news categories (e.g., keeping only news articles about politics), in order to avoid selection bias. furthermore, the different embedding algorithms require varying preprocessing steps. for word2vec, sentence splitting is required. in order to train the frage model, we tokenized the data and replaced ultra-rare tokens with "<@xmath0, since the model expects the preprocessing of the wikitext-2 corpus @xcite . to do so, we used the huggingface tokenizer foot_4 and ended up with a vocabulary of around 39k tokens. ## experiments we now describe our experiments to evaluate embedding algorithms regarding their capabilities to accurately represent social bias in text corpora. to do so, we assess an algorithm's ability to generate a meaningful embedding space and to avoid the issues detailed in section 3 arising from sparse data. in particular, we systematically train models on all news articles with either political bias from our corpus (section 4), once with each of the four word embedding algorithms from section 3. to increase the data available for each bias, we aggregate news articles for lean-left and left as liberal as well as for lean-right and right outlets as conservative. ## conclusion in this paper, we have compared word embedding algorithms for the task of evaluating text corpora for social bias. to this end, we have introduced a us online news corpus that covers three political bias directions at five levels. our literature review has motivated that specific political bias coincides with social bias with respect to gender, ethnicity, and religion. we have taken advantage of this property to train three word embedding algorithms and evaluate them for social bias using weat. lastly, we present an example application, analyzing the development of social bias in news articles over a 12 year period. we find that the particularly frequency-agnostic and decontextualized embedding spaces are more meaningful and encode the social bias more accurately than word2vec. they fail, however, to do so consistently for all bias types. while the respective algorithms should be more reliable, especially when evaluating sparse datasets, the exact weat results should be considered with care. the values do not seem to quantify social bias in the same way for all embedding algorithms. future research should investigate the relation between weat values of an algorithm and the encoded bias. our findings give insights into the role of word embedding algorithms within the social bias evaluation of texts, and they demonstrate what type of embedding models work even in sparse data scenarios. thereby, we contribute to understanding social bias in texts and nlp applications in general.