Detecting Media Bias in News Articles using G aussian Bias Distributions (2020)

Abstract: Media plays an important role in shaping public opinion. Biased media can influence people in undesirable directions and hence should be unmasked as such. We observe that feature-based and neural text classification approaches which rely only on the distribution of low-level lexical information fail to detect media bias. This weakness becomes most noticeable for articles on new events, where words appear in new contexts and hence their “bias predictiveness” is unclear. In this paper, we therefore study how second-order information about biased statements in an article helps to improve detection effectiveness. In particular, we utilize the probability distributions of the frequency, positions, and sequential order of lexical and informational sentence-level bias in a Gaussian Mixture Model. On an existing media bias dataset, we find that the frequency and positions of biased statements strongly impact article-level bias, whereas their exact sequential order is secondary. Using a standard model for sentence-level bias detection, we provide empirical evidence that article-level bias detectors that use second-order information clearly outperform those without.

## introduction media bias is discussed and analyzed in journalism research @xcite @xcite and nlp research @xcite @xcite . according to the study of @xcite , bias "has nothing to do with the honesty or accuracy", but it means "taste or preference". in fact, journalists may (1) report facts only in favor of one particular political side and thus (2) conclude with their own opinion. as an example, the following sentences from allsides.com reporting on the event "trump asks if disinfectant, sunlight can treat coronavirus" demonstrate media bias on the sentence level: the activists falsely claimed that trump "urged americans to inject themselves with disinfectant" and "told people to drink bleach." -the daily wire, right-oriented lysol maker issues warning against injections of disinfectant after trump comments -the hill, center-oriented "this notion of injecting or ingesting any type of cleansing product into the body is irresponsible and it's dangerous," said gupta. bias detection even becomes harder at the article level. for illustration, figure 1 shows two articles and their sentence-level bias from the used dataset. it becomes clear that the actual words in the biased sentences are not always indicative to distinguish biased from neutral articles, nor is the count of the biased sentences: bias assessments on sentence level do not "add up". in this regard, the position of biased sentences appears to be a better feature. the existing approaches to bias detection are transferred from other, less intricate text classification tasks. they largely model low-level lexical information, either explicitly, e.g. by using bag-ofwords @xcite , or implicitly via neural networks @xcite . such approaches tend to fail at the article level, particularly for articles on events not covered in the training data. the reason is that bias clues are subtle and rare in articles, especially event-independent clues. altogether, modeling low-level information at the 22 24 35 republicans are challenging a host of statements made by secretary of state hillary clinton and democratic allies during wednesday's heated libya testimony --claiming that complaints about a lack of funding are bogus and questioning the secretary's insistence she never saw urgent cables warning about the danger of an attack. @xcite one issue that may come up is the department's funding. assertions that state department posts are left vulnerable because congress has decided not to fully fund security requests pervaded wednesday's hearings. "shame on the house for ... failing to adequately fund the administration's request," rep. gregory meeks, d-n.y., said : asked wednesday about lamb's testimony, clinton noted that the review board that examined the libya attack found budget issues have played a role. "that's why you have an independent group like an (accountability review board); that's why it was created to look at everything," clinton said. but rep. dana rohrabacher, r-calif., said "any suggestion that this is a budget issue is off base, or political." [...] that cable is seen as one of the vital warnings sent out of libya in the months leading up to the attack. but, to the dismay of lawmakers, clinton repeatedly said she never saw it. : the secretary tried to explain that "1.43 million cables" come through the department every year. they are addressed to her but in many cases do not go to her. rather, they go through "the bureaucracy." republicans argue the aug. 16 cable was rather high priority. as sen. rand paul, r-ky., put it, "libya has to have been one of the hottest of hot spots around the world." he claimed that not knowing about their security requests "really, i think, cost these people their lives." "had i been president at the time, and i found that you did not read the cables from benghazi, you did not read the cables from ambassador stevens, i would have relieved you of your post. i think it's inexcusable," paul said. ## related work media bias detection has been studied with computers since the work of @xcite . as of then, media bias has been investigated in slight variations under different names, including perspective @xcite , ideology @xcite , truthfulness @xcite , and hyperpartisanship @xcite . to detect bias, early approaches relied on low-level lexical information. for example, @xcite used kill verbs and domain-relevant verbs to detect articles being pro israeli or palestinian perspectives. recasens et al. (2013) relied on linguistic cues, such as factoid verbs and implicatives, in order to assess whether a wikipedia sentence conveys a neutral point of view or not. besides the nlp community, also researchers in journalism have approached the measurement of media bias. e.g., @xcite used the preferences of phrases at each side (such as "war on terror" for republican but "war in iraq" for democratic). @xcite used the counts of think-tank citations to estimate the bias. with the rise of deep learning, nlp researchers have also used neural-based approaches for bias detection. @xcite used rnns to aggregate the polarity of each word to predict sentence-level bias based on parse trees. @xcite made use of headline attention to classify article bias. @xcite encoded social information in their graph-cnn. while deep learning is believed to capture deeper relations among its inputs, we show that extending a neural network from sentence-level to article-level bias detection does not "just work". one point of variation in media bias detection is the level of text being analyzed, which varies from tokens @xcite and sentences (bhatia and deepak, 2018) to articles @xcite , sources @xcite , and users @xcite . while the effectiveness of machine learning models on different levels helps understanding how media bias becomes manifest at different levels, @xcite are to our knowledge the only to discuss the difference between sentence-level and article-level bias detection. source-level and user-level bias can be seen as directly emerging from summing up bias in the associated texts. for example, @xcite averaged the feature vectors of articles as the feature vectors of a source. the relation between sentencelevel and article-level bias remains unstudied so far. the goal of this paper is not to discuss the difference between these levels. rather, we examine how to aggregate the sentence-level bias to generate second-order features, and then use these features to predict article-level bias. the use of low-level information to generate second-order features was studied in the context of product reviews by modeling patterns in the reviews' sentiment flow @xcite , by tuning neural network to capture important sentences @xcite , and by routing in aggregating sentence embeddings into document embedding @xcite . in particular, our usage of low-level information is inspired by @xcite , where we hypothesize that such flows exist in media bias as well. however, we do not limit our approach to entire sequences of sentencelevel information, but we also consider frequency, position, or only two to three continuous sentences. ## standard bias detection approaches standard approaches for bias detection, on both article and sentence level, mainly exploit the lowlevel lexical features to classify the texts as biased or not, neglecting bias-specific features. the two main low-level lexical feature types that are employed in such approach ares: (1) n-gram features, where n is typically one to three (i.e., unigram, bigram, or trigram), and (2) word embeddings, especially within pre-trained language models (i.e., transformers) such as bert. we propose two classification settings to answer research question q1, which addresses the importance of event information: in the first setting, called event overlapping, we form the training and test sets by randomly assigning examples to them, more specifically, without looking at event information. the setting allows texts of the same event to occur in both the training and the test set. the second setting is called event non-overlapping since the texts to be classified are first categorized according to the main event that they address. during the splitting in training set and test set we then ensure for each event that all its related texts are in exactly one of these sets. the difference in the effectiveness of the standard approaches on the two settings indicates whether and to what extent standard bias detection approaches rely on event information. ## second-order bias information for research question q2, we study the correlation between sentence-level and article-level bias. specifically, we examine whether article-level bias correlates with (a) the frequency of biased sentences, (b) their position in an article, and (c) their sequential order. for each correlation, we extract features and then train a respective machine learning model. the code is available at https: //github.com/webis-de/emnlp-20 . ## experiments this section presents the experiments that we designed to study research questions q1-q3 based on the media bias dataset basil. ## results and discussion to answer the three research questions of this paper, we report and discuss the results of the experiments described in section 5. ## conclusion in this paper we have given evidence that the exploitation of low-level lexical information is insufficient to detect article-level bias -especially, if the dataset is small. to provide a complete picture, we have formulated three research questions related to article-level bias detection, in order (1) to assess the state of the art of event-dependent and eventindependent bias prediction, (2) to learn about the relation between sentence-level and article-level bias, and (3) to study whether sentence-level bias can be leveraged to predict article-level bias. to tackle the detection of article-level bias, we have proposed and analyzed derived (second-order) bias features, including bias frequency, bias position, and bias sequence. as a main result of our research, we have shown that this new approach clearly outperforms the best approaches existing so far. if bias detection can be done sufficiently robust on article level, we envisage, as a line of future research, the development of "reformulation" strategies and algorithms for the task of neutralizing biased articles @xcite .