Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts (2021)

Abstract: Media coverage has a substantial effect on the public perception of events. Nevertheless, media outlets are often biased. One way to bias news articles is by altering the word choice. The automatic identification of bias by word choice is challenging, primarily due to the lack of a gold standard data set and high context dependencies. This paper presents BABE, a robust and diverse data set created by trained experts, for media bias research. We also analyze why expert labeling is essential within this domain. Our data set offers better annotation quality and higher inter-annotator agreement than existing work. It consists of 3,700 sentences balanced among topics and outlets, containing media bias labels on the word and sentence level. Based on our data, we also introduce a way to detect bias-inducing sentences in news articles automatically. Our best performing BERT-based model is pre-trained on a larger corpus consisting of distant labels. Fine-tuning and evaluating the model on our proposed supervised data set, we achieve a macro F1-score of 0.804, outperforming existing methods.

## introduction online news articles have, over time, started to replace traditional print and radio media as a primary source of information @xcite . a varying word choice may have a major effect on the public and individual perception of societal issues, especially since regular news consumers are mostly not fully aware of the degree and scope of bias @xcite . as shown in existing research @xcite , detecting and highlighting media bias might be relevant for media analysis and to mitigate the effects of biased reports on readers. also, the detection of media bias can assist journalists and publishers in their work @xcite . to date, only a few research projects focus on the detection and aggregation of bias @xcite . even though bias embodies a complex structure, contributions @xcite often neglect annotator background and use crowdsourcing to collect annotations. therefore, existing data sets exhibit low annotator agreement and inferior quality. our study holds both theoretical and practical significance. we propose babe (bias annotations by experts), a data set of media bias annotations, which is built on top of the mbic data set @xcite . mbic offers a balanced content selection, annotations on a word and sentence level, and is with 1,700 annotated sentences one of the largest data sets available in the domain. babe improves mbic, and other data sets, in two aspects. first, annotations are performed by trained experts and in a larger number. second, the corpus size is expanded considerably with additional 2,000 sentences. the resulting labels are of higher quality and capture media bias better than labels gathered via crowdsourcing. in sum, babe consists of 3,700 sentences with gold standard expert annotations on the word and sentence level. 1 to analyze the ideal trade-off between the number of sentences, annotations, and human annotation cost, we divide our gold standard into 1,700 and 2,000 sentences, which are annotated by eight and five experts, respectively. 2 lastly, we train and present a neural bert-based classifier that outperforms existing approaches such as the one by @xcite . even though neural network architectures have been applied to the media bias domain @xcite , their data sets created using crowdsourcing do not exhibit similar quality as our expert data set. in addition, we include five state-of-the-art neural models in our comparison and extend two of them in a distant supervision approach @xcite . leveraging large amounts of distantly labeled data, we formulate a pre-training task helping the model to learn bias-specific embeddings by considering bias information when optimizing its loss function. for the classification presented in this paper, we focus on sentence level bias detection, which is the current standard in related work (section 2) 3 . we address future work on word level bias in section 7. we publish all our code and resources on https://github.com /media-bias-analysis-group/neura l-media-bias-detection-using-dis tant-supervision-with-babe . ## related work media bias can be defined as slanted news coverage or internal news article bias @xcite . while there are multiple forms of bias, e.g., bias by personal perception or by the omission of information @xcite , our focus is on bias caused by word choice, in which different words refer to the same concept. for a detailed explanation of the types of media bias, we refer to @xcite . in the following, we summarize the existing literature on bias data sets and media bias classification. ## data set creation since media bias by word choice rarely depends on context outside the sentences @xcite , we focused on gathering sentences only. to tackle the weaknesses of existing bias data sets, we created a robust and diverse corpus containing bias annotations by experts (babe). ## methodology we propose the use of neural classifiers with automated feature learning capabilities to solve the given media bias classification task. a distant supervision framework, similar to @xcite , allows us to pre-train the feature extraction algorithms leading to improved language representations, thus, including information about a sample's bias. as obtaining large amounts of pre-training labeled data using humans is prohibitively expensive, we resort to noisy yet abundantly available labels that provide supervisory signals. ## results models trained and evaluated on sg2 generally perform better due to their bigger corpus size. the increase is around 0.02 points of the macro f 1score for all models except roberta + distant, where it is insignificant. overall, we believe the improvement indicates that extending the data set in the future will be valuable. results of the fourth block of with the added data, information on a sequence's bias is incorporated in the loss function, which is not the case in "general purpose" language models. model macro f 1 sg1 sg2 baseline 0.511 (0.008) 0.569 (0.008) bert 0.762 (0.019) 0.789 (0.011) distilbert 0.758 (0.029) 0.777 (0.009) roberta 0.775 (0.023) 0.799 (0.011) electra 0.742 (0.020) 0.760 (0.013) xlnet 0.760 (0.042) 0.797(0.015) bert + distant 0.778 (0.017) 0.804 (0.014) roberta + distant 0.798 (0.022) 0.799 (0.017) standard errors across folds in parentheses. the first model block shows the best results of feature-based models. the second block of models consists of bert and optimize variants. the models in the third block use new architectural or training approaches. the fourth block refers to models having learned biasspecific embeddings from the distantly supervised corpora. the best results are printed in bold. ## discussion employing annotators with domain expertise allows us to achieve an inter-annotator agreement of α = 0.40, which is higher than existing data sets @xcite . we believe domain knowledge and training alleviate the difficulty of identifying bias and are imperative to create a strong benchmark due to the complexity of the task. in future work, apart from improving the current data set and classifier, we will also explore why a text passage might be biased, not just its overall classification. currently, traditional machine learning models are interpretable @xcite but outperformed by recurrence and attention-based models. hand-crafted features like static dictionaries cannot adequately address the complexity and context-dependence of bias. we argue that standard metrics (e.g., accuracy and f 1 ) provide a limited perspective into a model's predictive power in case of a complex construct like media bias. further research needs to tackle these pitfalls to propose systems with better generalization capabilities. a promising starting point might be a more refined evaluation scheme that decomposes the bias detection task into multiple sub-tasks, such as presented in checklist @xcite . this scheme will also allow us to understand how our system performs on different types of bias (e.g., bias by context, by linguistics, by overall reporting). additionally, we believe that current research on explainable artificial intelligence might increase users' trust in neural-based classifiers. existing research already presents ways to visualize transformer-based models and make their results more accessible and interpretable @xcite . lastly, combining neural methods with advances in linguistic bias theory @xcite to explain a classifier's decision to users will also be part of our future work. for this work, we focused on sentence level bias, which is often used in the media bias domain. still, in addition to the 3,700 labeled sentences, we also include word level annotations in our data set to encourage solutions focusing on more granular characteristics. we believe that word level bias conveys strong explanatory and structural knowledge and see a detailed word level bias analysis and detection as a promising research direction. ## conclusion this work proposes babe, a new high-quality media bias data set. babe contains 3,700 labeled sentences, and enables us to compare crowdsourcing and expert annotations directly. additionally, we propose a sentence level bias classifier based on bert, which outperforms existing work in the domain. by deriving bias-specific word embeddings using distant supervision, we have improved our classifier even more, achieving a macro f 1 -@xmath0.804. we make all models, data, and code publicly available. 20