Cross-Domain Label-Adaptive Stance Detection (2021)

Abstract: Stance detection concerns the classification of a writer’s viewpoint towards a target. There are different task variants, e.g., stance of a tweet vs. a full article, or stance with respect to a claim vs. an (implicit) topic. Moreover, task definitions vary, which includes the label inventory, the data collection, and the annotation protocol. All these aspects hinder cross-domain studies, as they require changes to standard domain adaptation approaches. In this paper, we perform an in-depth analysis of 16 stance detection datasets, and we explore the possibility for cross-domain learning from them. Moreover, we propose an end-to-end unsupervised framework for out-of-domain prediction of unseen, user-defined labels. In particular, we combine domain adaptation techniques such as mixture of experts and domain-adversarial training with label embeddings, and we demonstrate sizable performance gains over strong baselines, both (i) in-domain, i.e., for seen targets, and (ii) out-of-domain, i.e., for unseen targets. Finally, we perform an exhaustive analysis of the cross-domain results, and we highlight the important factors influencing the model performance.

## introduction there are many different scenarios in which it is useful to study the attitude expressed in texts, e.g., of politicians with respect to newly proposed legislation @xcite , of customers regarding new products @xcite , or of the general public towards public health measures, e.g., aiming to reduce the spread of covid-19 @xcite . this task, commonly referred to as stance detection, has been studied in many different forms: not just for different domains, but with more substantial differences in the settings, e.g., stance (i) expressed in tweets @xcite @xcite vs. long news articles @xcite vs. news outlets @xcite vs. people @xcite , (ii) with respect to a claim @xcite vs. a topic, either explicit @xcite or implicit @xcite . moreover, there is substantial variation in (iii) the label inventory, in the exact label definition, in the data collection, in the annotation setup, in the domain, etc. the most crucial of these, which has not been investigated, currently preventing cross-domain studies, is that the label inventories differ between the settings, as shown in our goal in this paper is to design a common stance detection framework to facilitate future work on the problem is a cross-domain setting. to this end, we make the following contributions: finally, we release our code, models, and data. foot_0 the focus, however, has been on homogeneous text, as opposed to cross-platform or cross-domain. exceptions are @xcite , who worked on heterogeneous text, but limited to eight topics, and @xcite , who combined datasets from different domains, but used in-domain multi-task learning, and @xcite and @xcite , who used a cross-lingual setup. in contrast, we focus on cross-domain learning on 16 datasets, and out-of-domain evaluation. domain adaptation domain adaptation was studied in supervised settings, where in addition to the source-domain data, a (small) amount of labeled data in the target domain is also available @xcite @xcite @xcite , and in unsupervised settings, without labeled target-domain data @xcite @xcite @xcite . recently, domain adaptation was applied to pre-trained transformers @xcite . one direction therein are architectural changes (method-centric): @xcite proposed curriculum learning with domaindiscriminative data selection, wright and augenstein (2020) investigated an unsupervised multisource approach with mixture of experts and domain adversarial training @xcite . ## stance detection datasets in this section, we provide a brief overview of the 16 stance datasets included in our study, and we show their key characteristics in ## method we propose a novel end-to-end framework for cross-domain label-adaptive stance detection. our architecture (see figure 2 ) is based on input representations from a pre-trained language model, adapted to source domains using mixture-of-experts and domain adversarial training (section 4.1). we further use self-adaptive output representations obtained via label embeddings, and unsupervised alignment between seen and unseen target labels for out-of-domain datasets. unlike previous work, we focus on learning the relationship between datasets and their label inventories in an unsupervised fashion. moreover, our mixture-of-experts model is more compact than the one proposed by @xcite , as we introduce a parameter-efficient architecture with layers that are shared between the experts. finally, we explore the capability of the model to predict from unseen user-defined target. with this framework, we solve two main challenges: (i) training domain-adaptive models over a large number of datasets from a variety of source domains, and (ii) predicting an unseen label from a disjoint set of over 50 unique labels. ## experiments we consider three evaluation setups: (i) no training, random and majority class baselines; (ii) in-domain, training then testing on all datasets; and (iii) out-ofdomain, i.e., leave-one-dataset-out training for all datasets. the reported per-dataset scores are macroaveraged f 1 , which are additionally averaged to obtain per-experiment scores. ## discussion we further study the correlation between the scores for the best model in the out-of-domain setup mole w/ weak mappings and a rich set of quantitative and stance-related characteristics of the datasets (these are further discussed in section 3 and in appendix b). in particular, we represent each dataset as a set of features, e.g., fnc1 would have target -news, training set size of 42,476, etc., and then we measure the pearson correlation between these features and the model's f 1 scores per dataset. figure 3 shows the most important factors for out-of-domain performance. 7 we see positive correlations of f 1 with the training, and the development set sizes, and a negative one with the testing set size, which suggests that large datasets are indeed harder for the model. interestingly, if there is an overlap in the targets between the testing and the training sets, the model's f 1 is worse; however, this is not true for context overlap. unsurprisingly, the size of the vocabulary is a factor that negatively impacts f 1 , and its moderate negative correlation with the model's scores confirms that. the domain, the target and the context types are also important facets: the news domain has a sizable positive correlation with f 1 , which is also true for the related features headline target and article body. another positive correlation is for having a claim as the context. on the contrary, a key factor that hinders model performance is social media text, i.e., having a tweet as a context. ## conclusion and future work we have proposed a novel end-to-end unsupervised framework for out-of-domain stance detection with respect to unseen labels. in particular, we combined domain adaptation techniques such as mixture-of-experts and domain-adversarial training with label embeddings, which yielded sizable performance gains on 16 datasets over strong baselines: both in-domain, i.e., for seen targets, and out-of-domain, i.e., for unseen targets. moreover, we performed an exhaustive analysis of the crossdomain results, and we highlighted the most important factors influencing the model performance. in future work, we plan to experiment with more datasets, including non-english ones, as well as with other formulations of the stance detection task, e.g., stance of a person @xcite or of a news medium @xcite with respect to a claim or a topic.