Analyzing Political Bias and Unfairness in News Articles at Different Levels of Granularity (2020)

Abstract: Media is an indispensable source of information and opinion, shaping the beliefs and attitudes of our society. Obviously, media portals can also provide overly biased content, e.g., by reporting on political events in a selective or incomplete manner. A relevant question hence is whether and how such a form of unfair news coverage can be exposed. This paper addresses the automatic detection of bias, but it goes one step further in that it explores how political bias and unfairness are manifested linguistically. We utilize a new corpus of 6964 news articles with labels derived from adfontesmedia.com to develop a neural model for bias assessment. Analyzing the model on article excerpts, we find insightful bias patterns at different levels of text granularity, from single words to the whole article discourse.

## introduction reporting news in a politically unbiased and fair manner is a key component of journalism ethics and standards. "politically unbiased" means to report on an event without taking a political position, characterization, or terminology, and "fair", in this context, means to focus on original facts rather than on analyses or opinion statements based on false premises. 1 although it is known that biased and unfair news do exist in the media, spreading misleading information and propaganda @xcite , people are not always aware of reading biased content. when fighting bias and unfairness, only the understanding of how these emerge (and how to avoid them) can allow media organizations to maintain credibility and can enable readers to choose what to consume and what not. in pursuit of this goal, we recently studied how sentence-level bias in an article affects the political bias of the whole article @xcite . also other researchers have proposed approaches to automatic bias detection (see section 2 for details). however, the existing approaches lack an analysis of what makes up bias, and how it exposes in different granularity levels, from single words, to sentences and paragraphs, to the entire discourse. to close this gap, we analyze political bias and unfairness in this paper within three steps: we utilize two well-known websites that address media bias, allsides.com and adfontesmedia.com, in order to create a corpus with 6964 news articles, each of which is labeled for its topic, political bias, and unfairness. based on this corpus, we devise a recurrent neural network architecture to learn classification knowledge for bias detection. we choose this network class because of its proven ability to capture semantic information at multiple levels: taking the model output for whole texts, we conduct an in-depth reverse feature analysis to explore media bias at the word, the sentence, the paragraph, and the discourse level. at the word level, we correlate the most biased sentences with liwc categories @xcite . at the sentence and paragraph level, we reveal what parts of an article are typically most politically biased and unfair. at the discourse level, we reveal common sequential media bias patterns. the results show that our model can utilize highlevel semantic features of bias, and it confirms that they are manifested in larger granularity levels, i.e., on the paragraph level and the discourse level. at the word level, we find some liwc categories to be particularly correlated with political bias and unfairness, such as negative emotion, focus present, and percept. at levels of larger granularity, we observe that the last part of an article usually tends to be most biased and unfairest. ## related work computational approaches to media bias focus on factuality @xcite , political ideology @xcite , and information quality @xcite . bias detection has been done at different granularity levels: single sentences @xcite , articles @xcite , and media sources @xcite . recently, the authors of this paper studied how the two granularity levels "sentence" and "discourse" affect each other. an outcome of this research are insights and means of how sentence-level bias classifiers can be leveraged to better predict article-level bias @xcite . given these results, the paper in hand digs deeper by investigating how bias is actually manifested in a text at different granularity levels. our approach resembles techniques where the attention mechanism in a model is used to output weights (which indicate feature importance) for each text segment @xcite . @xcite , for instance, use word-level attention to focus on sentiment words, while @xcite use sentencelevel attention to select valid sentences for entity relation extraction. related to media bias, @xcite show that the learned attention can focus on the biased sentences when detecting political ideology. regarding "attention" at multi-level granularity, @xcite propose a hierarchical attention network for document classification that is used at both word and sentence level. yet, problems when using attention for such analyses are that the analysis unit (be it a word or a sentence) has to be defined before the training process, and that the set of classifiers which can output attention is limited. by contrast, our unsupervised reverse feature analysis can be used with any classifier and at an arbitrary semantic level, even after the training process. ## media bias corpus although bias detection is viewed as an important task, we could not find any reasonably sized labeled corpus that suits our study, which is why we decided to built a new one. we started with the corpus of @xcite in order to obtain news articles with topics and political bias labels. we extended the corpus by crawling the latest articles from allsides.com and by adding the fairness labels provided by adfontesmedia.com. the new corpus is available at https://github.com/ webis-de/nlpcss-20 . allsides.com this website is a news aggregator that collects news articles about american politics, starting from june 1st, 2012. each event comes with articles representing one of the three political camps: left, center, and right. @xcite crawled the website to extract 6447 articles. for our study, we extended their corpus by integrating all articles until march 15th, 2019, resulting in a total of 7775 articles. in addition to the political bias labels, we crawled the topic tags of each article. adfontesmedia.com since allsides.com focuses on political bias, we exploit adfontesmedia.com as another source for additional bias types. this portal maintains a "bias scale" quantifying the media bias of a broad set of us news portals. the bias assessments stem from media experts who annotate each portal with bias and fairness labels. @xcite show that the portals' labels are highly correlated to findings from social scientists. a new media bias corpus based on the labels from adfontesmedia.com, we define three media bias types for news portals: ## media bias analysis this section presents our approach to study bias and unfairness at different levels of text granularity. at first, we develop classifiers for political bias, unfairness, and non-objectivity. second, we do a reverse feature analysis study to explore the manifestation of bias at multiple levels of granularity and to identify different bias patterns. ## results and discussion in the following, we report and discuss the results of the experiments described in section 4. ## conclusion this paper has studied political bias and unfairness in news articles. we have trained sequential models for bias detection and have applied a reverse feature analysis to demonstrate that it is possible to reveal at what granularity level and how sequential patterns media bias is manifested. specifically, we find that the last quarter seems to be the most biased part. a significant "by-product" of our research is a new corpus for bias analysis. and we believe this corpus can help, for example, investigate how journalists convey bias in a news article. in the future, we will study how to utilize these results to detect bias or to rewrite the articles to remove and change bias. for example, based on the findings of this paper, the bias remover should pay more attention to the later paragraphs.