Dynamic Stance: Modeling Discussions by Labeling the Interactions (2023)

Abstract: Stance detection is an increasingly popular task that has been mainly modeled as a static task, by assigning the expressed attitude of a text toward a given topic. Such a framing presents limitations, with trained systems showing poor generalization capabilities and being strongly topic-dependent. In this work, we propose modeling stance as a dynamic task, by focusing on the interactions between a message and their replies. For this purpose, we present a new annotation scheme that enables the categorization of all kinds of textual interactions. As a result, we have created a new corpus, the Dynamic Stance Corpus (DySC), consisting of three datasets in two middle-resourced languages: Catalan and Dutch. Our data analysis further supports our modeling decisions, empirically showing differences between the annotation of stance in static and dynamic contexts. We fine-tuned a series of monolingual and multilingual models on DySC, showing portability across topics and languages.

## introduction stance detection consists of identifying opinions, perspectives, or attitudes expressed in texts. the task has been found to be relevant as a component for fact verification, argument mining, and media analysis @xcite @xcite , but also on its own @xcite . the standard setting for stance detection is the adoption of a static perspective, where different texts are labeled with a three-class label approach, i.e., being in favour, neutral or against a given topic @xcite @xcite . one of the limits of this approach is its strong dependence on topics, resulting in poorly portable systems @xcite . other works have taken a different direction: rather than keeping a fixed topic, stance is in-vestigated between pairs of texts. key contributions in this direction are the rumoureval shared tasks @xcite , the fake news challenge @xcite and, more recently, the stanceosaurus corpus @xcite . although different with respect to the nature of the data, in all cases systems are required to determine the stance of a message with respect to a previous claim with the goal of annotating stance as a component of fact or rumor verification. consequently, the used labels focus on denying or supporting some facts instead of modeling the actual internal dynamics of an online exchange. ## related work stance has been modeled extensively as a static task in which a text is tagged with respect to its attitude towards a specific topic or target @xcite @xcite . methods for stance classification have evolved alongside the developments in natural language processing, moving away from feature-based approaches up to fine-tuning and prompting pretrained models @xcite @xcite @xcite . while performances on benchmark data have seen improvements, issues such as balance among classes and portability of systems across topics and languages remain open research questions. stance detection has only been approached as a relation between two texts on a few occasions. as already mentioned, remarkable contributions are the fake news challenge @xcite , rumoureval @xcite , and the stanceosaurus corpus @xcite . in the fake news challenge, the authors aim at using disagreements between news pieces to detect false information. they design an annotation scheme based @xcite to estimate the stance of a body text from a news article relative to a headline in english. the labels they define are agrees, disagrees, discusses, and unrelated. while the headline and the content of an article are very often in agreement, in online debates this is not frequent. therefore, more granular labels are needed to model messages engaging with each other in an exchange of opinions. in rumoureval, the goal is to verify a rumor in english by modeling the discourse around it. from this perspective, the authors designed an annotation scheme that focuses on making explicit whether the replies to a rumor on twitter support or deny the given claim. four labels are available: support, deny, comment, and query. the stanceosaurus corpus @xcite follows a similar approach. in this work, the authors provide specific claims and look for twitter messages related to those claims. the corpus is multilingual, covers multiple topics, and uses a more fine-grained set of labels when compared to rumoureval (supporting, refuting, discussing, querying, and irrelevant). stanceosaurus is the largest available corpus for stance with respect to claims. besides the size of the corpus, the authors obtain a macro f1 score of 0.53 when testing on unseen claims, foot_0 with performances dropping to 0.40 for zero-shot cross-lingual transfer in hindi and arabic. 2 the focus of these two studies was to use interactions between texts to verify claims. therefore, they use labels such as support and deny, which entail that the departing message must be a claim, which is the case in rumoureval and stancesaurus. however, in online interactions, most of the messages we encounter are not claims but rather opinions and perspectives, which can not always be supported or denied, but rather agreed upon or disagreed. when it comes to language-specific resources for stance detection in catalan and dutch, previous work is quite limited and focused on modeling static stance. for catalan, stance is investigated in twitter and only with respect to the "catalan indepedence" topic @xcite @xcite . for dutch, the only available dataset is @xcite , where stance is annotated targeting face masks and social distance measures. the corpus comprises messages from different platforms (twitter, reddit, and nu.nl). additionally, the contact corpus @xcite identifies stance towards vaccines using twitter and facebook posts, but their data is not publicly available. our work departs from previous studies by focusing on modeling online exchanges intrinsically. the goal of this approach is to be able to capture the relation of a text with respect to a disputed claim as well as any other dialogic interaction in online forums or micro-blogging platforms. this will be relevant for tasks such as fact verification, but also for controversy detection @xcite , argument mining @xcite @xcite , detecting previously verified claims @xcite @xcite , and discourse analysis. ## dynamic stance annotation inspired by this previous work, we hypothesize that the stance of a message towards a previous message (dynamic stance) can show more linguistic patterns than the static approach, as it will be less topic-dependent than modeling agreement over a topic. while dynamic stance is an independent task, for the purpose of this work, we also annotated static stance. having access to both annotation levels is essential to empirically investigate the interactions between these two schemes, showing similarities and differences. ## the dynamic stance corpus the dynamic stance corpus (dysc) is a multilingual, and multi-topic corpus in catalan and dutch. this section will illustrate the methods that were used for collecting and annotating the data in the two languages. we will also analyze the resulting labels. ## automatic stance detection we developed several baseline models for both static and dynamic stance detection. we investigate the abilities of pre-trained language models to model static and dynamic stance with two settings: one where the topic is seen at training time and another where the topic is excluded. lastly, we investigate the cross-lingual transfer abilities of a multilingual model on dust. we used a learning rate of 1-e5, a batch size of 8, a warmup of 0.06, and trained for 10 epochs with an adam optimizer and cross-entropy loss. we kept the best checkpoint on the development set. ## conclusion and future work in this work, we have investigated the task of stance detection with a focus on dynamic interactions between parent messages and their replies. our approach addressed the limitations of traditional static stance detection models, which heavily rely on specific topics and exhibit poor generalization capabilities. by introducing a novel annotation scheme and creating the dynamic stance corpus (dysc), we provided a new perspective on the categorization of textual interactions, which captures different insights from the data. through our data analysis, we demonstrated empirical differences between the annotation of stance in static and dynamic contexts. furthermore, dysc is fully developed for non-english languages, namely catalan and dutch. using a generic monolingual pre-trained model for catalan (roberta-ca-l) we achieve a macro f1 score of 0.65 for dynamic stance using all topics on training. we also show that, while dynamic stance models exhibit some learning on the zero-shot topic scenario (with an average score of 0.47), static stance models go from a macro f1 score of 0.71 with all topics on train, to an average score of 0.21 when in the zero-shot scenario. we therefore conclude that dynamic stance is easier to model in cross-topic scenarios. as for the cross-lingual experiments, the results show that there is some knowledge transfer between languages. while the results leave room for improvement, they are close to the ones obtained by similar previous work such as the stancesaurus (reported in section 2). this study represents an important initial step toward effectively modeling the task of stance detection in dynamic textual interactions. future work will address some of the pending issues by expanding the annotated data to other languages and investigating new methods (including the use of incontext learning) to improve the portability of the dynamic stance models.