Adversarial Learning for Zero-Shot Stance Detection on Social Media (2021)

Abstract: Stance detection on social media can help to identify and understand slanted news or commentary in everyday life. In this work, we propose a new model for zero-shot stance detection on Twitter that uses adversarial learning to generalize across topics. Our model achieves state-of-the-art performance on a number of unseen test topics with minimal computational costs. In addition, we extend zero-shot stance detection to topics not previously considered, highlighting future directions for zero-shot transfer.

## introduction stance detection, the problem of automatically identifying positions or opinions in text, is becoming increasingly important for social media (e.g., twitter), as more and more people turn to it for their news. zero-shot stance detection, in particular, is crucial, since gathering training data for all topics is not feasible. while there has been increasing work on zero-shot stance detection in other genres @xcite , generalization across many topics in social media remains an open challenge. in this work, we propose a new model for stance detection that uses adversarial learning to generalize to unseen topics on twitter. our model achieves state-of-the-art zero-shot performance on the majority of topics in the standard dataset for english stance detection on twitter @xcite and also provides benchmark results on two new topics in this dataset. most prior work on english social media stance detection uses the semeval2016 task 6 (semt6) dataset @xcite which consists of six topics. while early work trained using five topics and evaluated on the sixth (e.g., @xcite ; @xcite ; @xcite ), they used only one topic, 'donald trump' ⇤ ⇤ denotes equal contribution. in contrast, our work is a hybrid of these two settings: we train on five topics and evaluate on one other, but unlike prior work we do not assume a relationship between training and test topics and so we use each topic in turn as the test topic. this illustrates the robustness of our model across topics and additionally allows zero-shot evaluation of semt6 on two new topics that were previously ignored by cross-target models ('atheism' and 'climate change is a real concern'). recently, @xcite introduced a new dataset of news article comments for zero-shot stance detection. while this dataset evaluates generalization to many new topics when learning with many topics and only a few examples per topic, there are no datasets for social media with this setup. specifically, current datasets for stance detection on twitter @xcite @xcite @xcite have only a few topics but many examples per topic. therefore, zero-shot stance detection on social media is best modeled as a domain adaptation task. to model zero-shot topic transfer as domainadaptation, we treat each topic as a domain. following the success of adversarial learning for domain adaptation @xcite , we use a discriminator (adversary) to learn topic-invariant representations that allow better generalization across topics. although, @xcite also proposed adversarial learning for stance detection, their model relies on knowledge transfer between topics (domains) and so is only suited to the cross-target, not zero-shot, task. in contrast, our work adopts a successful crosstarget architecture into a domain adaptation model without requiring a priori knowledge of any relationship between topics. our contributions in this work are: 1) we propose a new model for zero-shot stance detection on twitter using adversarial learning that does not make assumptions about the training and test topics, and 2) we achieve state-of-the-art performance on a range of topics and provide benchmark zero-shot results for two topics not previously used in the zero-shot setting with reduced computational requirements compared to pre-trained language models. our models are available at: https: //github.com/malavikasrikanth16/ adversarial-learning-for-stance. ## methods we propose a new model, topic-adversarial network, for zero-shot stance detection, that uses the domain-transfer architecture from @xcite coupled with a successful stance model @xcite with an additional topicspecific attention layer, to produce topic-invariant representations that generalize to unseen topics (see figure 1 ). ## results as in prior work (e.g., zhang et al. ( 2020 )) we report f avg : the average of f1 on pro and con. our model toad achieves state-of-the-art results (see we also observe that toad is statistically indistinguishable from bert on three additional topics (hc, la, cc) while having only 0.5% as many parameters (600k versus 110mil). as a result of this small size, toad can be trained using only the cpu and, because of it's recurrent architecture, would gain less from the increased parallel computation of a gpu (compared to a transformer-based model). therefore, toad has a potentially much lower environmental impact than bert with similar (or better) performance on five 1 sekt code is not available for computing significance. out of six zero-shot topics. analysis since cross-target models (e.g., sekt) rely on assumptions about topic similarity, we first analyze the impact of topic similarity on stance performance (see figure 2 ). specifically, we compute the jensen-shannon divergence @xcite between word distributions for pairs of topics to examine the impact of topic similarity on stance performance (see a.4 for details). we use jensen-shannon divergence (d js ) because it has been shown to successfully distinguish domains @xcite . using the combined vocabulary of both topics in a pair (see figure 2a ), we observe that human notions of similarity (used to select pairs for crosstarget models) may be flawed. for example, while the cross-target pair dt @xmath0 la, fm is almost as similar to dt as to la. since zero-shot transfer methods use all non-test topics for training, they avoid difficulties introduced by flawed human assumptions about similarity (e.g., about the ideological similarity of fm and la). we then examine, whether distributional similarity between topics does actually relate to crosstarget (t 1 ! t 2 ) stance performance. using the vocabulary for only one topic (v t 1 ) per pair (see figure 2b ), we observe an inverse relationship between similarity and relative stance performance. specifically, relatively lower similarity (higher divergence) often leads to relatively higher stance performance. for example, d js (hc||dt) is higher than d js (dt||hc) suggesting that a model trained on hc has less information about the word-distribution for dt than a model trained on dt has about hc. however, the cross-target stance models trained in the hc ! dt setup (e.g., sekt) actually perform relatively better than those trained in the dt ! hc setup. this highlights a further problem in the cross-target setting: using similar topics may encourage models to rely on distributional patterns that do not correlate well with cross-topic stance labels. next, we examine how topic-invariant the representations from toad actually are, and the impact of this on stance classification. we extract representations from our models, apply k-means clustering with @xmath2, and compare the resulting clusters to the gold topic labeling (see finally, we conduct an ablation on the topicspecific components of toad ( ## conclusion we propose a new model for zero-shot stance detection on twitter that uses adversarial learning to produce topic-invariant representations that generalize to unseen topics. our model achieves state-ofthe-art performance on a number of unseen topics with reduced computational requirements. in addition, our training procedure allows the model to generalize to new topics unrelated to the training topics and to provide benchmark results on two topics that have not previously been evaluated on in zero-shot settings. in future work, we plan to investigate how to extend our models to twitter datasets in languages other than english.