Topic-specific social science theory in stance detection: a proposal and interdisciplinary pilot study on sustainability initiatives (2024)

Abstract: Topic-specificity is often seen as a limitation of stance detection models and datasets, especially for analyzing political and societal debates. However, stances contain topic-specific aspects that are crucial for an in-depth understanding of these debates. Our interdisciplinary approach identifies social science theories on specific debate topics as an opportunity for further defining stance detection research and analyzing online debate. This paper explores sustainability as debate topic, and connects stance to the sustainability-related Value-Belief-Norm (VBN) theory. VBN theory states that arguments in favor or against sustainability initiatives contain the dimensions of feeling power to change the issue with the initiative, and thinking whether or not the initiative tackles an urgent threat to the environment. In a pilot study with our Reddit European Sustainability Initiatives corpus, we develop an annotation procedure for these complex concepts. We then compare crowd-workers with Natural Language Processing experts’ annotation proficiency. Both crowd-workers and NLP experts find the tasks difficult, but experts reach more agreement on some difficult examples. This pilot study shows that complex theories about debate topics are feasible and worthwhile as annotation tasks for stance detection.

## introduction online platforms see people discussing politicians (i.e., emmanuel macron), political issues (i.e., immigration), and cultural debates (i.e., feminist messages in the movie barbie). stance models usually classify written arguments in such debates into whether they are in favor or against the topic under discussion @xcite . the task of stance detection is often conceptualized as topicindependent: in datasets and papers, a stance in favour of feminism is seen as conceptually similar as one in favour of immigration. however, it has been shown that stance models are in fact topic-dependent: transformer models trained on detecting different stances in one topic do not necessarily work on unseen topics @xcite . recent work @xcite attempts to tackle this limitation of topic-independent stance modelling by diversifying the number of topics in stance detection datasets, while @xcite update transformer models' access to knowledge of topic context to improve cross-topic stance detection. instead of seeing specialization into one topic as a limitation, we argue that this topic-specificity of debates can also be an asset for stance detection research. social science theories can play a crucial role in this challenge. such theories can be used to develop topic-specific stance data and models, which increases the impact of stance detection on socially relevant research questions. this approach also tackles limitations of work assuming stance is topic-independent, such as models not fully capturing the underlying socio-cultural dimensions of specific topics @xcite . social science theory can lead computational argumentation researchers to dimensions of stance that are unique for specific debate topics. these dimensions can then be annotated, and this knowledge of theories help models (and humans) navigate the unique dimensions of the debate. we argue that defining topic-specific aspects of the debate helps analyzing, modelling, and interpreting the stances in such debates. as a case study, we apply value-belief-norm (vbn) theory @xcite of environmental debates to stances in environmental debates. we develop an annotation framework and test-drive this by annotating a dataset of 91 reddit comments reacting to sustainability initiatives with stance, threat, and power. we then analyze the advantages and disadvantages of this approach for stance research on sustainability, and also on other debate topics. this paper has the following contributions: ## topic-dependence and theory in stance stance detection @xcite ) is a task in computational argumentation or argument mining (lawrence and reed, 2020) consisting of classifying arguments into pro, con, or neutral towards an idea or discussion topic. stance detection has been used to measure support on social media for topics @xcite . these topics are for instance vaccination, but also debate statements such as "we should abolish free speech." recent work has indicated that stance models are topic-dependent despite being designed as topicindependent. @xcite found that crosstopic capabilities of transformer stance models are dependent on topical cues, and that model errors are related to a lack of understanding of sociocultural dimensions in debates such as gun control and abortion. thorn jakobsen et al. @xcite found that these models learn topic-dependent signals, and use mostly topic-dependent words not related to stance as a topic-independent concept (e.g. word 'gun' rather than argumentation-related words). earlier work has claimed high cross-topic stance performance, but these performances have still been highly topic-dependent. some research uses topic dependence in stance for these results, by measuring similarity between two discussion topics, and using the most related topics for cross-topic stance detection. this obtains f1 scores between .67 and .80 on stance detection in unseen topics @xcite @xcite . however, @xcite do not consider topic-relatedness when modelling and obtain much more modest scores of @xmath0.49 and .54 on unseen topics. a similar result can be seen in @xcite , where strategies for few-shot cross-topic stance detection with transformer models lead to inconsistent performance (between @xmath1.344 and @xmath2.766) and are largely dependent on dataset choice rather than choices made in model design. approaches to improve these non-robust crosstopic capabilities of stance detection models go into two related, but distinct, directions. one is a data-centric approach that can be summarized as improving the debate topic diversity in datasets. earlier work already mentioned how claims and arguments as defined in datasets are topic and context-dependent @xcite . recently, @xcite have developed an ontology for defining diversity of debate topics in computational argumentation datasets. another direction is a more model-centric approach that can be summarized as improving the models' use of topic knowledge. earlier work has also looked into improving world knowledge use in stance models @xcite . @xcite recently designed a transformer model architecture that uses realworld knowledge for classification decisions, in the form of a context encoder that "injects" domainrelevant world knowledge into stance models. see @xcite for an overview of using knowledge in computational argumentation. while both directions have promising results, we argue there is another option for overcoming the weaknesses of topic-independent stance detection: designing datasets as well as models with relevant social science theory on the specific debates. this work argues a debate topic is broader than specific individual texts or statements (such as "climate change is bad"), but more narrow than what other works call domain (which often resorts to categories such as "legal", "social media", etc.). we define a topic as a specific area of socio-cultural discussion, with its own dimensions and aspects of debate such as "climate change", or "immigration". our definition of topic most closely responds to the level 1 and 2 topics in @xcite 's argument topic ontology. stances in such topics have unique, topic-specific aspects, that can be captured by social science theory on the debate topic in question. recently, stance detection work has attempted to include dimensions of opinion beyond simply support or reject, such as argument type @xcite and underlying values @xcite . these variables add underlying reasons why an idea is supported or rejected, often a neglected aspect of stance @xcite . however, theories on the individual debate topics are often neglected in this exploration of aspects related to stance and arguments. ## case study: sustainability initiatives a stance on sustainability initiatives can be defined as an argument in favor or against initiatives such as renewable energy in local communities @xcite or sustainable behavior at music festivals @xcite . other work within computational argumentation has looked into sustainability, for instance by annotating evidence that supports sustainability claims in scientific papers @xcite , or by detecting sustainable diet patterns in tweets @xcite . however, the tasks and annotation variables (such as stance, claim-evidence pairs, and argument units) in these earlier papers are very similar to other computational argumentation literature. we would like the annotation procedure to be influenced directly by the sustainability literature in social science. what can a social science theory about sustainability debates tell us about stances in this debate, and how to analyze debates on sustainability? ## data our dataset on european sustainability discussions, mostly in english, is obtained from the reddit.com sub-communities @xcite called europe, europeanunion, and europes. we identify any sustainability discussion posted from 2017 to 2022 to contain five years of comments foot_2 . our dataset consists of 2.073 discussions with 46.285 comments. nearly half (922) of these have one or more comments. we release the entire corpus, without annotations, as the reddit european sustainability initiatives corpus foot_3 , for non-commercial research-use only under cc-by-nc licence. foot_4 . ## discussion our pilot study gave several insights. firstly, we note that irony is a specific issue in argument annotation. this has been noted by earlier work integrating social science theory in computational argumentation studies, e.g. in a tutorial on the topic by @xcite . @xcite also find in annotation experiments for argument quality that even experts struggle with annotating and interpreting irony in arguments when annotating with complex theories. secondly, using theories in natural language processing can also help connecting a theory to a phenomenon, and finding gaps between these @xcite . responsibility is a dimension which is not part the vbn theory or of our annotation pilot, but in annotation we found it was a clear dimension in the debate: in multiple discussions, commenters mentioned that while they supported the initiative (e.g. nudging people to produce less waste), they felt others (either the rich elite, people in china or america, or companies) were mainly responsible for climate-related problems. these comments are in line with a different, but not mutually exclusive, theory about climate debate: that of social identity theory, were people feel pushed to blame outside groups @xcite . this connection may be interesting for future work on sustainability and stance. another question is whether the vbn theory applies to other debate topics. we note that 'power' and 'threat' may relate to stances in especially other policy-related debates. however, the two dimensions in this theory are also different, and it seems the threat dimension is more applicable to debates on debates that feel existential (e.g. is immigration a threat?). the power dimension (do we have power to restore the desired state?), is more related to feeling whether people have influence on the outcome with their own actions, which is more applicable to debates with a central role for individual action, i.e. donating money, or voting. topic-specific aspects also exist beyond sustainability. another debate topic is covid-19 policies, popular in stance detection research @xcite . topic-independent pro/con stances ignores the covid19-specific issue of whether people disagree because the measure is too strict, or not strict enough. without this topic-specific aspect, there are limitations to interpreting stances in this debate @xcite . ## conclusion we propose to integrate topic-specific social science theories in stance detection, improving some weaknesses of topic-independent conceptualizations of stance detection. as a case study, we use value-belief-norm theory @xcite for stances on sustainability, and apply this theory to a pilot annotation task on 91 comments in our reddit european sustainability initiatives corpus. the aspects are difficult to annotate, but experts annotate some difficult examples better than crowdworkers. topic-specific theories improve stance understanding -for both models and humans.