Examining Political Rhetoric with Epistemic Stance Detection (2022)

Abstract: Participants in political discourse employ rhetorical strategies—such as hedging, attributions, or denials—to display varying degrees of belief commitments to claims proposed by themselves or others. Traditionally, political scientists have studied these epistemic phenomena through labor-intensive manual content analysis. We propose to help automate such work through epistemic stance prediction, drawn from research in computational semantics, to distinguish at the clausal level what is asserted, denied, or only ambivalently suggested by the author or other mentioned entities (belief holders). We first develop a simple RoBERTa-based model for multi-source stance predictions that outperforms more complex state-of-the-art modeling. Then we demonstrate its novel application to political science by conducting a large-scale analysis of the Mass Market Manifestos corpus of U.S. political opinion books, where we characterize trends in cited belief holders—respected allies and opposed bogeymen—across U.S. political ideologies.

## introduction political argumentation is rich with assertions, hypotheticals and disputes over opponent's claims. while making these arguments, political actors often employ several rhetorical strategies to display varying degrees of commitments to their claims. for instance, political scientists have studied the footing-shift strategy, where actors convey their own beliefs while claiming that they belong to someone else @xcite . sometimes they may attribute their beliefs to a majority of the population via argument from popular opinion @xcite . actors can also resort to hedging, stating their own beliefs, but qualified with a partial degree of certainty @xcite @xcite or express simple political disagreements, contradicting claims made by their opponents @xcite @xcite . traditionally, political scientists and other scholars have manually analyzed the impact of such strategies and argumentation on audience perception @xcite . recent advances in natural language processing (nlp) and digital repositories of political texts have enabled researchers to conduct large-scale analyses of political arguments using methods such as subjectivity analysis @xcite , argument mining @xcite @xcite , and opinion mining @xcite @xcite . while these approaches primarily concern argument structure and normative attitudes, we propose a complementary approach to analyze sources' epistemic attitudes towards assertions @xcite @xcite -what they believe to be true and the extent to which they commit to these beliefs. consider an example shown in figure 1 , where the author of the text (s1) quotes a speculation from the congressional quarterly (s2) about what mitch mcconnell (s3) said concerning obama (s4). in this example, while the author of the text believes that the congressional quarterly hinted something about mcconnell (thus, exhibiting a certainly positive (ct+) stance towards the event (e1), she remains uncommitted (uu) about the quoted event (e3) that mcconnell describes (edge omitted for visual clarity). of course, this event is asserted as certainly negative (ct-) by mcconnell, the speaker of the quote. the congressional quarterly suggests that mitch mcconnell made a statement (a probably positive (pr+) stance towards e2) while remaining uncommitted towards what he said. finally, obama's own beliefs about whether he paid attention to republican ideas are not expressed in this sentence; thus, s4 (obama) has a non-epistemic label toward the listening event (e3). to address this challenging problem of epistemological analysis, researchers within the nlp community have created several datasets and models in this paper, we seek to further how epistemic stance analysis can help computationally investigate the use of rhetorical strategies employed in political discourse. in particular, we use the theory, structure and annotations of factbank @xcite , an expert-annotated corpus drawn from english news articles, which distinguishes different types of epistemic stances expressed in text. factbank features annotations not just for the author, but also other sources (entities) mentioned in the text. such multi-source annotations allow us to disambiguate the author's own beliefs from the beliefs they attribute to others. our main contributions in this work are: ## epistemic stance from different perspectives the notion of epistemic stances has been studied under several scholarly areas, including linguistics, political science and nlp. in this section, we discuss various notions of epistemic stances and how they have been utilized in these different areas. ## an epistemic stance framework for analyzing political rhetoric this section formally introduces the task of epistemic stance detection and describes the details of the factbank dataset. we then explain how the epistemic stance framework relates to several rhetorical strategies often used in political discourse. ## model we present a simple and reproducible robertabased neural model for epistemic stance classification using a standard fine-tuning approach. foot_3 bert fine-tuning is effective for many nlp tasks @xcite , and recent work on pre-trained language models such as bert @xcite @xcite shows such models encode syntactic and semantic dependencies within a sentence, which is highly related to the epistemic stance task. recently, jiang and de marneffe (2021) use a fine-tuned bert model for author-only epistemic stance prediction, obtaining strong performance on several datasets. we extend their approach, developing a bert model (using the roberta @xcite pre-training variant) for the structurally more complex multi-source task, and give the first full comparison to the foundational multisource system, defacto @xcite . we leave the exploration of other advanced transformer-based models @xcite for further performance gains as future work. to develop a model suitable for multi-source predictions, we follow @xcite and @xcite 's architecture for semantic (proto-role) labeling, which they formulate as predicting labels for pairs of input embeddings. to predict the epistemic stance for an event-source pair (e, s) in a sentence, we first compute contextual embeddings for the sentence's tokens, [h l 1 , h l 1 , ...., h l n ], from a bert encoder's last (l th ) layer. we concatenate the source (h l s ) and event (h l e ) token embeddings (each averaged over bert's sub-token embeddings), and use a single linear layer to parameterize a final softmax prediction @xmath0, 1] c over the @xmath1, foot_4 which is trained with cross entropy loss over all tuples in the training set. we apply inverse frequency class weighting to encourage accurate modeling for comparatively rare classes like the ct-, pr+ and ps+ class. finally, to cleanly analyze the author source in the same manner as other mentioned sources, we augment the sentence with the prefix "author: " (following a dialogue-like formatting), foot_5 and use its index and embedding for inferences about the author source. the above epistemic stance classifier, like most previous modeling approaches @xcite , requires preidentified sources and events, which do not exist in real-world text. we use qian et al. (2018)'s twostep approach to first identify sources and events in the input text and then determine stances for every recognized (source, event) pair. source and event identification is through two roberta-based token classifiers, using a linear logistic layer for binary classification of whether a token is a source (or event), fine-tuned on the same training corpus. our source and event identification models achieve a macro-averaged f1 score of 81.8 ± 0.019 and 85.78 ± 0.007, respectively, slightly improving upon the only existing prior work of qian et al. ( 2018 ) by 1.85% and 1.29% respectively, with pairwise significance tests resulting in p-@xmath3.01. we also experimented with a joint model to identify sources and events; however, individual classifiers gave us better performance (appendix §b.1). ## case study: belief holder identification political discourse involves agreement and contention between the author and other belief-holding sources they cite. as a first step, we extract major belief holders mentioned in a text to allow analysis of ideological trends in u.s. political discourse. ## conclusion semantic modeling has exciting potential to deepen the nlp analysis of political discourse. in this work, we analyze the epistemic stance of various sources toward events, by developing a robertabased model, and using it for identifying major belief holders mentioned by political authors. we conduct a large-scale analysis of the mass market manifestos corpus of u.s. political opinion books, where we characterize trends in cited belief holders across u.s. political ideologies. in future, we hope to use this framework to help construct a database of beliefs, belief holders, and their patterns of agreement and disagreement in contentious domains. thus, analyzing hedging requires understanding of syntactic contexts within which claims are expressed, which our model can tackle. we establish the external validity of our proposed epistemic stance framework by computationally replicating the findings of jalilifar and alavi (2011)'s manual content analysis. to ensure the external validity of our proposed epistemic stance framework, we computationally replicate the findings of jalilifar and alavi (2011)'s manual content analysis. the study examines transcripts of topically similar television interviews of three political figures, george w. bush (at the time, incumbent u.s. president), jimmy carter (former u.s. president), and david coltart (founding member of zimbabwe's main opposition party). 14 for each interview transcript, we employ our epistemic stance classifier to predict the stance of the political figure (author source) towards all extracted events, and calculate each author's uncertainty level as the fraction of events with a pr+ or ps+ epistemic stance. we find the same ordering of commitment as the previous work: bush using the fewest uncertain pr+/ps+ stances (5.41%), with progressively more for carter (8.32%) and coltart (12.2%). this follows jalilifar and alavi's interpretation of commitment being correlated to power (bush being the highest status, for example).