Detecting and understanding moral biases in news (2020)

Abstract: We describe work in progress on detecting and understanding the moral biases of news sources by combining framing theory with natural language processing. First we draw connections between issue-specific frames and moral frames that apply to all issues. Then we analyze the connection between moral frame presence and news source political leaning. We develop and test a simple classification model for detecting the presence of a moral frame, highlighting the need for more sophisticated models. We also discuss some of the annotation and frame detection challenges that can inform future research in this area.

## introduction while much attention has focused on the role of fake news in political discourse, comparatively little attention has been paid to the dissemination of news frames. framing in news coveragehighlighting certain aspects of an issue or eventcan have a significant impact on public opinion formation @xcite . framing theory posits that preference formation depends on which subset of relevant considerations or beliefs-"frame in mind"-are activated by a particular message-"frame in communication." scholars refer to the power of such a frame as a framing effect, a phenomenon widely reported in academic scholarship on domestic and foreign issues alike @xcite @xcite @xcite . if one-sided and morally charged, it can exacerbate polarization and post-truth politics. according to the most widely cited model used by social scientists @xcite , the essential components of a frame include problem definition, diagnosis of cause, moral judgment, and prescribed remedy. for example, obesity may be defined as a significant national health problem, diagnosed as the result of increasingly passive lifestyles judged as detrimental to the strength of society and individuals, and effectively treated by increased physical activity. such an emphasis on individual choice redirects attention from other possible causes such as genetic disposition or advertising campaigns for caloric rich foods. even though moral judgment is central to frame analysis, much of the frame analysis research neglects the moral dimension. our work responds to this gap by adapting moral foundations theory (mft) which proposes a set of five modalities-each with a virtue and vice binary partner-that underlie moral thinking @xcite . morallyinflected frames follow the contours of political ideology @xcite , are more likely to be shared on social media @xcite , and, most importantly, reinforce attitudes, making compromise more difficult @xcite . technology offers little solution to mitigate these framing effects at the scale and speed of modern information networks. our work responds to the need for cross-disciplinary frameworks that enable the early detection, propagation, and influence of moral frames in such networks. in this paper, we offer an initial analysis on the steps necessary for detecting and understanding the prominence of moral frames in news. we annotate a small corpus of news articles with moral frames and look into their connection to issue-specific frames and news source leaning, together with models for detecting them. ## related work in the last few years, a number of nlp approaches have been devised for frame identification in text: most focus on coarser-grained primary frame identification @xcite @xcite , possibly based on a probabilistic distribution @xcite ; few address finer-grained frame tagging at the paragraph level @xcite . most research relies on word-based approaches, from direct keyword matching to latent representations @xcite @xcite @xcite . few studies use rhetorical information such as discourse structure @xcite . additionally, work has been done on general frames, such as economy or law and order @xcite that can apply across issues, or on issue-specific (also called topical) frames, such as innocence as concerns capital punishment; than on identifying moral foundations. approaches to the latter mostly rely on moral foundation keyword dictionaries, again directly @xcite or via latent representations @xcite . ## datasets and annotation since there is no existing corpus with moral frame annotations for news articles, we put together a small initial dataset to help us understand the intricacies of moral frame annotation and analysis. our dataset contains 400 articles on four different issues. 300 articles are from a previously collected corpus @xcite ), 100 articles for each of immigration, smoking and same-sex marriage issues from 13 news sources. another set of 100 articles was collected on the racial unrest in baltimore from 16 national and local newspaper sources. three undergraduate student annotators were hired as summer interns for this project. each article was independently annotated with sentencelevel and article-level moral frames by all three annotators based on the 10 moral foundations (graham et al., 2013) -care/harm, fairness/cheating, loyalty/betrayal, authority/ subversion, sanctity/degradation -or with na. the annotators used the brat software to perform the annotations ( https://brat.nlplab.org/ ). the annotation process proceeded in two stages, each of which involved a detailed annotation man-ual, that was modified in the second stage foot_0 . stage 1. the annotation manual instructed the annotators to proceed with coding in 3 ordered steps: (1) to identify the moral frame type; (2) to decide whether the author supports or rejects the frame; and (3) to decide whether the author explicitly favors or opposes the specific issue the article is about. the annotators were also instructed to do so for both sentences and the whole article; at the article-level, the annotators were asked to evaluate the entire article and specify what they regarded as its main moral frame. the annotators were told to first annotate the sentences in an article and then the article as a whole, but no explicit written guidelines were provided in this regard. stage 2. after the initial set of annotations from stage 1 (which were discarded), the protocol was adjusted based on annotator feedback, with the goal of making the annotation process less ambiguous. first, a preliminary step was added to the three annotation steps, a.k.a step 0: annotators were instructed to identify the presence or absence of any moral frame before embarking in the subsequent three steps. second, the sentence and article annotation were clearly separated, and for the article annotation specific guidelines were provided: evaluate the entire article and specify what you regard is its main moral claim. keep in mind that it may or may not be the most frequent one (based on counting sentences with moral claims). another main adjustment was providing the annotators with a list of keywords associated with each moral foundation developed by @xcite . the annotators were instructed to use such sets as keywords as guidance, but were warned that (a) a moral frame may contain none of the keywords listed in the codebook, and that (b) the presence of a keyword does not necessarily indicate the presence of a moral frame. the annotators were provided with examples of both (a) and (b). we refer to the dataset with sentence-level annotations as mf-sent and with article-level annotations as mf-art. 116 articles have both an article-level frame and at least one sentence-level frame, as annotated by at least one annotator. the percentage of articles whose moral frame is different from the most frequent moral frame among the article's sentences is 14.3%. figure 2: coincidence matrix showing number of times annotators (dis)agree on sentence-level frame types. to study the connection between issue-specific frames and more general, moral frames, a domain expert annotated a subset of 48 baltimore unrest articles from mf-sent with issue-specific frames at the sentence level following @xcite : black criminality, police racism, rogue cops, and structural inequality. we refer to this dataset as ol-sent. ## moral frame analysis 4.1 issue-specific vs. moral frames we analyze the connection between issue-specific and moral frames in the baltimore unrest articles (ol-sent dataset). when a sentence is annotated with multiple frames, we consider the one with the (a) issue specific frame distribution (b) moral frame distribution figure 4: frame type distributions in ol-sent dataset based on news source type (conservative, liberal, neutral). highest agreement. the sentence-level distribution of issue-specific and moral frames is given in figure 3 . while issue-specific frames are more evenly distributed, moral frames have a skewed distribution with cheating and harm being the dominant frames. the most likely reason for this is that, the care/harm and fairness/cheating foundations are valued by conservatives and liberals alike and is therefore more likely to be present in news frames. we computed the pearson correlation between different frame labels, and their heat-map representation can be found in figure 3c . it is interesting to note that for most moral frames, there is a dominant corresponding issue-specific frame. for subversion, it is black criminality (blk crm), for fairness, it is rogue cops (rog cop), for cheating/injustice it is structural inequality (str inq). ## conclusion we presented a small-scale study of moral frames in news showing that moral frames have the potential to explain issue-specific frames and the biases of their news sources. in order to increase the analysis scale and reliability, we need to collect a larger dataset covering more issues and news sources. we also need to improve the annotation protocol and overcome the challenges associated with annotator subjectivity. future directions include improving on the machine learning models for predicting moral frames in news articles and studying their impact on opinions expressed in social media.