Using Masked Language Model Probabilities of Connectives for Stance Detection in E nglish Discourse (2023)

Abstract: This paper introduces an approach which operationalizes the role of discourse connectives for detecting argument stance. Specifically, the study investigates the utility of masked language model probabilities of discourse connectives inserted between a claim and a premise that supports or attacks it. The research focuses on a range of connectives known to signal support or attack, such as because, but, so, or although. By employing a LightGBM classifier, the study reveals promising results in stance detection in English discourse. While the proposed system does not aim to outperform state-of-the-art architectures, the classification accuracy is surprisingly high, highlighting the potential of these features to enhance argument mining tasks, including stance detection.

## introduction the task this paper addresses is argument stance detection in english discourse. more concretely, based on the definition of argument following established terminology @xcite , where an argument consists of a claim, a controversial statement, and a premise, a statement supporting or attacking the claim, we want to automatically decide whether the premise supports (label: 1) or attacks (label: 0) the claim. this task has been modeled in a number of approaches already @xcite . in contrast to these approaches, we aim at operationalizing the role of connectives with the following simple idea: we insert one-word connectives, i.e., linking words such as because, but, so, or although, between the claim and the candidate premise and use a language model (lm) to quantify acceptability. connectives include coordinators (such as and, or but), subordinators (such as because, or while), as well as linking adverbs (such as therefore, or however; @xcite . they can express support, attack, or other types of relations. the underlying hypothesis is that features obtained from an lm's probability for inserting certain connectives between a claim and premise can improve stance detection. put differently, our research question is whether we can verify whether a premise is a support for or an attack against a given claim based on explicit discourse connectives. we show that using probabilities of connectives as features, we obtain a significant improvement in stance detection compared to a majority and a random baseline. this indicates that, although we do not aim at a competitive argument mining system in this paper, integrating these features into argument mining has the potential to improve existing approaches. we use english data but we assume that a similar approach should also work for other languages. 1 ## motivation and related work the expression of stance is linked closely to argumentative structures in discourse since arguments by definition involve stance, and stance markers are known to facilitate the processing of argumentative relations @xcite . besides a variety of other stance markers @xcite , connectives play a crucial role in that respect. work on various languages has shown that the discourse function of connectives is closely related to that of other linguistic elements expressing stance or subjectivity in their role for argumentative discourse. in particular, there seems to be a "division of labor," where the presence of stance markers makes an explicit connective less expected while fewer stance markers make the use of specific connectives more likely @xcite . such a trade-off between connectives and other cues for stance suggests that markers of one kind may be omitted if there are cues in the context that make the information of those markers 1 the code and results are available at https:// github.com/rstodden/stance-detection . already predictable (uniform information density hypothesis; torabi asr and demberg 2015), which motivates here our expectation that discourse connectives also mark argument stance. masked lms (mlms), e.g., bert @xcite or roberta @xcite , are bidirectional encoders which are mostly trained on massive data to solve the task of language modeling. the intention of language modeling is similar to a cloze test; the model is trained on extensive unlabeled data, wherein random tokens (at any position of a sequence) are masked, enabling the model to learn how to predict them @xcite . the pre-trained mlms return probabilities for any word of the vocabulary at the position of the masked token; the higher the probability the more suitable the word in the sequence. in recent years, mlms are also often used for stance detection. 2 following @xcite , the current state-of-the-art model (called mt-dn m dl ) across multiple stance detection datasets is a bert model (bert-largeuncased with an additional classification layer; devlin et al. 2019), initially fine-tuned on the glue benchmark @xcite and subsequently fine-tuned concurrently on several stance detection datasets . in contrast to mt-dn m dl and related models, in our approach we do not predict the stance based on the weights of an mlm but make use of the knowledge of mlms with respect to connectives as stance markers. methodologically, the present study builds on existing approaches which tackle the problem of classifying implicit discourse relations by using masked lms to explicitate the relations. specifically, the models predict how likely a given connective is in sentence pairs without an overtly expressed discourse relation. for example, kishimoto et al. (2020) experiment with additionally pre-training and fine-tuning mlms on texts with masked connectives (called connective prediction task), finding that only the first technique provides gain. @xcite use a pipeline approach to classify unlabeled, implicit discourse relations, where explicit data -a set of 65 candidate connectives -is concatenated with two sequences and then fed into an explicit discourse relation classifier. recently, @xcite have tackled the problem by using a prompt learning method. given a template that arises from natural language use (e.g. 'arg1: arg1. arg2: arg2. the conjunction between arg1 and arg2 @xmath0.'), they select the most frequent and least ambiguous predicted connective as the answer word to replace the mask token. we do not use prompting or causal lms as we are interested in the probabilities of the connectives. masked lms, in contrast to generative lms, are capable of giving the probabilities of a word at any position of a sequence based on the left and right context (and not only at the end of a sequence). to the best of our knowledge, our work is the first approach to use probabilities of discourse connectives of masked lms as features and to combine them with stance detection. ## methods our method comprises four components: 1. the concatenation of claims and premises with a masked token (see subsection 3.2), 2. an lm that estimates the likelihood of a given connective in the concatenated sequence (see subsection 3.1 and subsection 3.3), 3. a feature vector which comprises all the probabilities of the connectives (see subsection 3.3), 4. and a binary classifier which, based on the feature vector, learns whether the premise supports or attacks the claim (see subsection 3.4). we hypothesize that the lms have learned argumentative structures and the usage of connectives. therefore, we anticipate that the model will assign higher probabilities to support connectives and lower probabilities to attack connectives for support premises, and vice versa for attacks. for example, in example 1, the premise attacks the claim, and we expect lower probabilities for support connectives like because or since as they would render the argument incoherent. for attack connectives like but or although we expect higher probabilities as they are in line with the attack relation. ## results we first validated our main assumption by measuring spearman's correlation coefficient ρ between the probabilities of the connectives and the stance per each sample of each dataset. appendix b summarizes all correlations and significance levels. for all three datasets, we found that the probabilities of nearly all connectives significantly correlate with stance (p-level at @xmath1.1; all except with, if, and when). as expected, the probabilities of the attack connectives show a negative correlation, whereas those of the support connectives show a positive correlation, and the ambiguous connectives show a mixed picture. however, most correlations are weak (i.e., ρ < 0.3) except for five moderate (i.e., 0.3 ≤ ρ < 0.5; except, unless, until, yet, and three strong ones (i.e., ρ ≥ 0.5; although, though, but). to sum up, our assumption was validated across all three datasets. therefore, we can now turn to our results on stance detection based on the connectives' probabilities. all our models using all connectives (group 1) can outperform the two baselines. the best model with all probabilities (group 1) of only one lm is roberta-large (see bold row in the third part of as can be seen in the last six lines of also, the combination of attack and support connectives seems to be helpful for stance detection (see appendix c). for all datasets, the f1-score drops when removing support connectives (by less than 0.01 points) as well as, more noticeably, when removing attack connectives (between 0.01 and 0.35 points). when using only connectives which are in both lists (@xmath4, the score even drops by one more 0.01 point. this effect might be due to the decreasing number of features, as the analysis of the connectives of group 3 with the same number of features (i.e., connectives most often used for attack or support, @xmath5. an additional observation is that some connectives (e.g., and, when) appear in both groups, indicating that their interpretation as support or attack is inferred. this highlights that the role of connectives in signaling stance does not necessarily demand the explicit expression of the semantics of the claim-premise relation. ## conclusion and future work in this paper, we performed stance detection based only on the masked lm probabilities of discourse connectives that are assumed to indicate support or attack. the classifiers we trained on these features performed surprisingly well, given that the aim was not at all to develop a competitive argument mining in future work, we plan to also experiment with additional punctuation marks between the first part and the linking word. this is a promising avenue because some connectives occur more naturally at a sentence beginning and not between two clauses, e.g., therefore, or require a preceding comma, e.g., but. furthermore, we plan to integrate features based on the mlm probabilities of connectives, as used in this paper, with state-of-the-art approaches to stance detection that use input embeddings representing the actual text of claim and premise. finally, we will investigate whether additional preprocessing of the lms in the form of fine-tuning on argumentative data or data with explicit connectives before extracting the mlm probabilities increases stance detection performance.