Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection (2023)

Abstract: The task of Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest. This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim. However, the framing of the task varies within these domains in terms of the data collection protocol, the label dictionary and the number of available annotations. Furthermore, these stance annotations are significantly imbalanced on a per-topic and inter-topic basis. These make multi-domain stance detection challenging, requiring standardization and domain adaptation. To overcome this challenge, we propose Topic Efficient StancE Detection (TESTED), consisting of a topic-guided diversity sampling technique used for creating a multi-domain data efficient training set and a contrastive objective that is used for fine-tuning a stance classifier using the produced set. We evaluate the method on an existing benchmark of 16 datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments. The results show that the method outperforms the state-of-the-art with an average of 3.5 F1 points increase in-domain and is more generalizable with an averaged 10.2 F1 on out-of-domain evaluation while using <10% of the training data. We show that our sampling technique mitigates both inter- and per-topic class imbalances. Finally, our analysis demonstrates that the contrastive learning objective allows the model for a more pronounced segmentation of samples with varying labels.

## introduction the goal of stance detection is to identify the viewpoint expressed by an author within a piece of text towards a designated topic @xcite . such analyses can be used in a variety of domains ranging from identifying claims within political or ideological debates @xcite , identifying mis-and disinformation @xcite , public health policymaking @xcite @xcite , news recommendation @xcite to investigating attitudes voiced on social media @xcite @xcite . however, in most domains, and even more so for crossdomain stance detection, the exact formalisation of the task gets blurry, with varying label sets and their corresponding definitions, data collection protocols and available annotations. furthermore, this is accompanied by significant changes in the topicspecific vocabulary @xcite ), text style @xcite and topics mentioned either explicitly @xcite or implicitly @xcite . recently, a benchmark of 16 datasets @xcite covering a variety of domains and topics has been proposed for testing stance detection models across multiple domains. it must be noted that these datasets are highly imbalanced, with an imbalanced label distribution between the covered topics, i.e. inter-topic and within each topic, i.e. per-topic, as can be seen in figure 2 and figure 3 . this further complicates the creation of a robust stance detection classifier. given the inherent skew present within the dataset and variances within each domain, we propose a topic-guided diversity sampling method, which produces a data-efficient representative subset while mitigating label imbalances. these samples are used for fine-tuning a pre-trained language model (plm), using a contrastive learning objective to create a robust stance detection model. these two components form our topic efficient stance detection (tested) framework, as seen in figure 1 , and are analysed separately to pinpoint the factors impacting model performance and robustness. we test our method on the multi-domain stance detection benchmark by @xcite , achieving state-of-the-art results with both in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics evaluations. note though that tested could be applied to any text classification setting. in summary, our contributions are: ## related work stance detection is an nlp task which aims to identify an author's attitude towards a particular topic or claim. the task has been widely explored in the context of mis-and disinformation detection @xcite @xcite , sentiment analysis @xcite and argument mining @xcite @xcite . most papers formally define stance detection as a pairwise sequence classification where stance targets are provided @xcite . however, with the emergence of differ-ent data sources, ranging from debating platforms @xcite @xcite to social media @xcite , and new applications @xcite , this formal definition has been subject to variations w.r.t. the label dictionary inferred for the task. previous research has predominantly focused on a specific dataset or domain of interest, outside of a few exceptions like multi-target @xcite and cross-lingual @xcite stance detection. in contrast, our work focuses on multi-domain stance detection, while evaluating in-and out-of-domain on a 16 dataset benchmark with state-of-the-art baselines @xcite . ## datasets our study uses an existing multi-domain dataset benchmark @xcite , consisting of 16 individual datasets split into four source groups: debates, news, social media, various. the categories include datasets about debating and political claims including arc @xcite , iac1 @xcite , perspectum @xcite , poldeb (somasundaran and wiebe, 2010), scd (hasan and ng, 2013), news like emergent @xcite , fnc1 @xcite , snopes @xcite , social media like mtsd @xcite , rumour @xcite , semeval2016t6 @xcite , semeval2019t7 @xcite , wtwt @xcite and datasets that cover a variety of diverse topics like argmin @xcite , ibmcs @xcite and vast (allaway and mckeown, 2020). overall statistics for all of the datasets can be seen in appendix c. ## methods our goal is to create a stance detection method that performs strongly on the topics known during training and can generalize to unseen topics. the benchmark by @xcite consisting of 16 datasets is highly imbalanced w.r.t the intertopic frequency and per-topic label distribution, as seen in figure 2 . these limitations necessitate a novel experimental pipeline. the first component of the pipeline we propose is an importance-weighted topic-guided diversity sampling method that allows the creation of supervised training sets while mitigating the inherent imbalances in the data. we then create a stance detection model by fine-tuning a pre-trained language model (plm) using a contrastive objective. ## results and analysis in this section, we discuss and analyze our results, while comparing the performance of the method against the current state-of-the-art @xcite and providing an analysis of the topic efficient sampling and the contrastive objective. ## conclusions we proposed tested, a novel end-to-end framework for multi-domain stance detection. the method consists of a data-efficient topic-guided sampling module, that mitigates the imbalances inherent in the data while selecting diverse examples, and a stance detection model with a contrastive training objective. tested yields significant performance gains compared to strong baselines on indomain experiments, but in particular generalises well on out-of-domain topics, achieving a 10.2 f1 point improvement over the state of the art, all while @xmath0. while in this paper, we have evaluated tested on stance detection, the method is applicable to text classification more broadly, which we plan to investigate in more depth in future work.