Detecting Stance in Media On Global Warming (2020)

Abstract: Citing opinions is a powerful yet understudied strategy in argumentation. For example, an environmental activist might say, “Leading scientists agree that global warming is a serious concern,” framing a clause which affirms their own stance (“that global warming is serious”) as an opinion endorsed ("[scientists] agree”) by a reputable source (“leading”). In contrast, a global warming denier might frame the same clause as the opinion of an untrustworthy source with a predicate connoting doubt: “Mistaken scientists claim [...]." Our work studies opinion-framing in the global warming (GW) debate, an increasingly partisan issue that has received little attention in NLP. We introduce DeSMOG, a dataset of stance-labeled GW sentences, and train a BERT classifier to study novel aspects of argumentation in how different sides of a debate represent their own and each other’s opinions. From 56K news articles, we find that similar linguistic devices for self-affirming and opponent-doubting discourse are used across GW-accepting and skeptic media, though GW-skeptical media shows more opponent-doubt. We also find that authors often characterize sources as hypocritical, by ascribing opinions expressing the author’s own view to source entities known to publicly endorse the opposing view. We release our stance dataset, model, and lexicons of framing devices for future work on opinion-framing and the automatic detection of GW stance.

## introduction ascribing opinions to other people is a powerful yet understudied strategy in argumentation. for example, an environmental activist might say, "leading scientists agree that global warming is serious," whereas a global warming denier could say, "mistaken scientists claim that global warming is serious." in both these examples, the embedded clause (that global warming is serious) is presented as an opinion belonging to a source entity (scientists). however, differences in the choice of predicate (agree vs. claim) and in how the source is described lead to very different interpretations. we henceforth refer to the use of such @xcite [express] [statement] sentences as opinionframing, and to the respective components as the source, predicate, and opinion (see fig. 1 ). despite its pervasiveness in argumentative discourse, opinion-framing is understudied as a persuasive strategy. this paper studies opinionframing in the media coverage of global warming (gw), an increasingly partisan issue in the united states (pew research center, 2020) that has received little attention in nlp despite its real world urgency. we focus on acts of opinion-framing representing self-affirming and opponent-doubting discourses, i.e., discourse affirming one's own opin-ions (embedded clauses ascribed to a source, as depicted in fig. 1 ) and discourse casting doubt on the other side's. studying such discourses requires a way to identify the stance of a given opinion with respect to gw, but this is a challenging task. to this end, we introduce gwsd: global warming stance dataset, a dataset for detecting and analyzing gw stance in text. we collect human judgments of gw stance for 2k sentences with amazon mechanical turk (amt) foot_2 and use our dataset to train a bert-based classifier that achieves 75% accuracy (competitive with human performance) for gw stance detection. extending prior work in nlp and linguistics, we develop lexicons of affirming and doubting framing devices with respect to the predicate that embeds the opinion (e.g., know vs. claim) and the source to which the opinion is ascribed (e.g., a peer-reviewed study vs. a misleading paper) (see fig. 1 ). we then apply our model and lexicons to study two questions about opinion-framing in argumentation: q1: do different sides of a debate (in this case, gw-accepting and gw-skeptical media) show symmetry in their use of self-affirming and opponent-doubting discourse? we might expect some similarities (e.g., the use of agree to frame opinions expressing one's own side's stance, or the use of claim to cast doubt on opinions from the opposing side), but given inherent asymmetries in the nature of the gw debate, it is not clear whether such strategies will be found across sides to equal extents. second, since opinion-framing is a way of putting words into someone's mouth, we also ask q2: in cases where opinions are ascribed to a named entity with a known (public) stance, does the stance of the opinion match the expected stance of the named entity? applying our model to a set of 500k opinions (op f ull ) extracted from 56k gw articles, we find that gw-skeptical media engages in comparatively more opponent-doubt, though both sides of the debate show more self-affirmation overall, and use similar sets of framing devices for each respective discourse type. we also find that opinion-framing does indeed ascribe opinions differing from the overt views of entities to those entities nonetheless, as part of a rhetorical strategy of ascribing hypocrisy: authors portray their own opinion as being held (in private) by figures who endorse the opposite opinion (in public). our contributions are the following: we release our dataset, model, and lexicons as part of this paper. foot_3 our work is related to social psychology research on persuasion @xcite and recent nlp research on argumentation, such as predicting argument convincingness @xcite and studying discourse-level and non-linguistic features predictive of persuasion @xcite . the latter's work on self-vs. opponent-coverage is particularly relevant to the gw debate and we apply a similar categorization to the stance of ascribed opinions. also relevant is the literature on factuality and speaker commitment @xcite @xcite @xcite , and relatedly, work studying how words can express subjectivity or bias @xcite @xcite . our current paper builds upon previous work by examining such triggers as opinion-framing devices in an argumentation context, where biases related to people's prior beliefs may interact with the lexical effects of these words. opinion-framing can be thought of as a special case of the broader phenomenon of framing as discussed in the communications and political science literatures @xcite @xcite , as well as in nlp @xcite @xcite . both phenomena serve to emphasize particular aspects of an issue, and are often used with the intent to influence perception of that issue. our attention to the component of source in instances of opinion-framing is also informed by communications research on the messenger effect (that people's perceptions of a message may depend heavily on the message source) @xcite @xcite . furthermore, our interest in predicates of opinion attribution is inspired by communications studies examining how the choice of predicate (say vs. assert) can encode journalist stance @xcite and bias audience perception of the quoted entity @xcite . finally, our dataset contribution builds on @xcite , who created the first climate change stance task and dataset. ## gwsd: a dataset for gw stance to enable our study of opinion-framing, and to facilitate further work on stance, we create a new publicly-available dataset of opinion spans extracted from gw news articles (described in §3.1) that we have annotated with stance judgements using amt ( §3.2). to investigate potential annotator biases, we study the impact of annotator characteristics on their perception of stance (with approval from our institutional review board) ( §3.3), and combine ratings so as to infer a distribution over stance labels for each span while accounting for bias ( §3.4), which we release along with the raw annotations. ## a model for gw stance classification in order to classify stance in op f ull , the full dataset of 500k opinions, we train a model using the set of 2k annotated examples. the goal of this task is to predict the stance of a sentence s toward the label count neutral 873 agree 777 disagree 400 here, we report on variations on a bert classifier @xcite , as well as a linear baseline, in order to provide a sense of relative performance in comparison to past work. to ensure comparison against a strong baseline, we perform a grid search over hyperparameters for both approaches, and choose the best model from each according to validation accuracy, evaluating only the best model of each type on the held-out test set. for our neural model, we use the generalpurpose bert base architecture, trained by minimizing cross-entropy loss. we use the transformers library foot_7 as the basis for the models that we develop and compare. as potential augmentations, we experiment with a) fine-tuning the base model as a language model to unlabeled data; b) including the text of the target opinion as an input to the model; and c) using label weights as opposed to simply using the most probable label. for the weighted version, we include a copy of each training instance with each label, along with an instance weight corresponding to the label probability estimated by our label aggregation model above. (full details of hyperparameter tuning in appendix h). the test-set performances of best models we obtain are shown in further inspection of the validation results reveals that training on the weighted data offers a statistically significant improvement on validation accuracy, but the expected performance is statistically indistinguishable with respect to fine-tuning and/or incorporating the target opinion as an input. the best linear model was a simple l 2 -weighted logistic regression classifier using unigrams and bigrams (details in appendix h). ## analyses in this section, we first describe the lexicons of framing devices we use for our analyses ( §5.1). we then present analyses that address our two research questions. in §5.2, we find that qualitatively-speaking, both sides leverage similar linguistic framing devices for self-affirmation and opponent-doubt, but quantitatively-speaking, gw-skeptical media engages in more opponent-doubt. in §5.3, we find that both sides use opinion-framing to ascribe opinions expressing their own stance to sources known to publicly endorse the opposing view, thereby depicting such sources as hypocritical.