Media Bias, the Social Sciences, and NLP : Automating Frame Analyses to Identify Bias by Word Choice and Labeling (2020)

Abstract: Media bias can strongly impact the public perception of topics reported in the news. A difficult to detect, yet powerful form of slanted news coverage is called bias by word choice and labeling (WCL). WCL bias can occur, for example, when journalists refer to the same semantic concept by using different terms that frame the concept differently and consequently may lead to different assessments by readers, such as the terms “freedom fighters” and “terrorists,” or “gun rights” and “gun control.” In this research project, I aim to devise methods that identify instances of WCL bias and estimate the frames they induce, e.g., not only is “terrorists” of negative polarity but also ascribes to aggression and fear. To achieve this, I plan to research methods using natural language processing and deep learning while employing models and using analysis concepts from the social sciences, where researchers have studied media bias for decades. The first results indicate the effectiveness of this interdisciplinary research approach. My vision is to devise a system that helps news readers to become aware of the differences in media coverage caused by bias.

## introduction media bias describes differences in the content or presentation of news @xcite . it is a ubiquitous phenomenon in news coverage that can have severely negative effects on individuals and society, e.g., when slanted news coverage influences voters and, in turn, also election outcomes @xcite . potential issues of biased coverage, whether through the selection of topics or how they are covered, are compounded by the fact that in many countries only a few corporations control large parts of the media landscape-in the us, for example, six corporations control 90% of the media @xcite . even subtle changes in the words used in a news text can strongly impact readers' opinions @xcite @xcite . when referring to a semantic concept, such as a politician or other named entities (nes), authors can label the concept, e.g., "illegal aliens," and choose from various words to refer to it, e.g., "immigrants" or "aliens." instances of bias by word choice and labeling (wcl) frame the referred concept differently @xcite , whereby a broad spectrum of effects occurs. for example, the frame may change the polarity of the concept, i.e., positively or negatively, or the frame may emphasize specific parts of an issue, such as the economic or cultural effects of immigration @xcite . in the social sciences, research over the past decades has resulted in comprehensive models to describe media bias as well as effective methods for the analysis of media bias, such as content analysis @xcite and frame analysis @xcite . because researchers need to conduct these analyses mostly manually, the analyses do not scale with the vast amount of news that is published nowadays @xcite . in turn, such studies are always conducted for topics in the past and do not deliver insights for the current day @xcite ; this would, however, be of primary interest to people reading the news. revealing media bias to news consumers would also help to mitigate bias effects and, for example, support them in making more informed choices @xcite . in contrast, in computational linguistics and computer science, fewer approaches systematically analyze media bias @xcite . the models used to analyze media bias tend to be simpler @xcite compared to previously mentioned models. many approaches analyze media bias from the perspective of news consumers while neglecting both the established approaches and the comprehensive models that have already been developed in the social sciences @xcite @xcite @xcite @xcite . correspondingly, their results are often inconclusive or superficial, despite the approaches being technically superior. ## research question, tasks, and contributions t3. implement a prototype of a media bias identification system that employs the developed methods to demonstrate the applicability of the approach in real-world news article collections. the target group of the prototype are non-expert people. t4. evaluate the effectiveness of the bias identification methods with a test corpus and evaluate the effectiveness of using the prototype in a user study. combining the expertise of the social sciences and computational linguistics appears beneficial for research on media bias. thus, the main contribution of my ph.d. research will be an approach that combines models and methods from multiple disciplines. on the one hand, it will leverage established models from the social sciences to describe media bias and will follow currently manual methods to analyze media bias. on the other hand, it will take advantage of scalable methods for text analysis developed and used in computational linguistics. i need to employ and extend the state-of-the-art in two closely related nlp fields (cf. section 4): (1) cross-document coreference resolution (cdcr) as well as (2) target-dependent sentiment classification (tsc) including "sentiment shift" and identification of framing effects and causes (see section 4.2). i plan to embed both techniques into an approach that is inspired by the procedure of manually conducted, inductive frame analyses (cf. section 3.1). for the first technical contribution, a sieve-based cdcr approach was already devised that addresses characteristics of coreferences as they often occur in bias by wcl. the examples in the abstract show that even phrases that are usually considered contrary may be coreferential in a set of articles reporting on a specific event. for the second technical contribution, i.e., to estimate how a semantic concept may be perceived by people when reading a news article, i primarily plan to devise and test neural models that i will design specifically for the task. i also plan to implement a prototype that includes visualizations to reveal the identified instances of bias by wcl to users of the system. in the remainder of this document, i will give a brief overview of manual techniques for the analysis of bias by wcl and exemplary results from the social sciences as well as related, automated approaches (section 3). section 3 concludes with the current research gap, which motivates my ph.d. research. section 4 describes the tasks that i have already conducted as well as current and future tasks to complete my ph.d. research. section 5 describes a preliminary evaluation, which i already completed, as well as remaining tasks. ## related work the following summarizes an interdisciplinary literature review that i conducted as part of my ph.d. research (t1) @xcite . ## methodology research task t2 will be the main contribution of my ph.d. research; hence, this section focuses on completed and future tasks related to t2. technically, addressing the research question represents two main challenges. first, resolving coreferences of semantic concepts across a set of news articles. in bias by wcl, journalists often use coreferences in a broader, sometimes even contradictory, sense than the state-of-the-art in coreference resolution and cdcr is capable of @xcite @xcite . second, classifying how actors and other semantic concepts are framed due to their mentions and their mentions' contexts, for which i will use tsc. i plan to integrate the two tasks into the analysis shown in figure 1 (rt3). given a set of news articles reporting on the same event, the analysis will find subsets of articles and in-text phrases that similarly frame the concepts involved in the event. lastly, the system will visualize the results to news consumers. because rt3 is not directly related to nlp, it is described only briefly in section 4.3. ## evaluation i conducted preliminary evaluations of the two main methods described in section 4 (rt4). to measure the cdcr performance on broad coreferences as they occur in wcl bias (section 4.1), i created a test dataset named newswcl50. the dataset was created by manually annotating coreferential mentions of persons, actions, and also vaguely defined, abstract concepts across 50 news articles @xcite . the evaluation seems to confirm the research direction for this task. the approach currently achieves f 1 = 45.7, or 84.4 if evaluated only on technically feasible annotations, compared to 29.8, or 42.1, respectively, achieved by the best baseline. technically feasible refers to only comparing to annotations that the approach theoretically should be able to resolve, e.g., currently only nps while excluding vps. a future evaluation will include a comparison to state-of-the-art cdcr methods @xcite . for improved soundness, we plan to create a second dataset similar to the newswcl50 dataset but with more coders and more articles. to do so, we will crowdsource the annotations of concept mentions on mturk and use an improved codebook. the improvements will address issues of newswcl50's codebook, e.g., by making annotation types less ambiguous @xcite . further, we plan to use two additional datasets: ecb+ @xcite and nident @xcite . both datasets are commonly used to evaluate cdcr approaches and contain cross-document coreferences. to evaluate the second task, frame identification, i plan to create a comprehensive training and test set for the tsc method described in section 4.2. i already created a preliminary dataset of 3000 sentences, each including a target mention and a sentiment label agreed on by three coders. the dataset was created analogously to established tsc datasets @xcite @xcite . preliminary results seem to indicate that tsc on the news domain is in part more difficult than on tsc prime domains, such as product reviews, where authors often express their opinion explicitly. state-of-the-art tsc achieves average recall @xmath0.0 on news articles, whereas performances on common tsc test datasets range from @xmath1.6 (twitter dataset) to 82.2 (restaurant). other baselines, e.g., using dictionaries and semantic networks, such as conceptnet, perform very poorly (f 1 < 15.0), which seems to confirm that token-based approaches fail to catch the subtlety common to wcl bias. finally, we plan to evaluate the system's effectiveness regarding visualization of the identified biases to non-expert users. an already conducted pre-study confirmed the study design @xcite . i will revisit this task once the classification methods described in section 4 can be used within the study. ## conclusion and implications in summary, both everyday news consumers, as well as researchers in the social sciences, could benefit strongly from the automated identification of bias by word choice and labeling (wcl) in news articles. devising suitable methods to resolve broad coreferences across news articles reporting on the same event and estimating the frames of the found instances of wcl bias are at the heart of this research project. one primary result of the project will be the first automated approach capable of identifying instances of bias by wcl in a set of news articles reporting on the same event or topic. my vision is that at a later point in time, such methods might be integrated into popular news aggregators, such as google news, helping news readers to explore and understand media bias through their daily news consumption. also, i think that these methods could be integrated into the analysis workflow of content analyses and frame analyses, helping to automate further these currently mostly manual and thus time-consuming analysis concepts prevalent in the social sciences.