Z ero- S hot S tance D etection: A D ataset and M odel using G eneralized T opic R epresentations (2020)

Abstract: Stance detection is an important component of understanding hidden influences in everyday life. Since there are thousands of potential topics to take a stance on, most with little to no training data, we focus on zero-shot stance detection: classifying stance from no training examples. In this paper, we present a new dataset for zero-shot stance detection that captures a wider range of topics and lexical variation than in previous datasets. Additionally, we propose a new model for stance detection that implicitly captures relationships between topics using generalized topic representations and show that this model improves performance on a number of challenging linguistic phenomena.

## introduction stance detection, automatically identifying positions on a specific topic in text @xcite , is crucial for understanding how information is presented in everyday life. for example, a news article on crime may also implicitly take a position on immigration (see there are two typical approaches to stance detection: topic-specific stance (developing topicspecific classifiers, e.g., @xcite ) and cross-target stance (adapting classifiers from a related topic to a single new topic detection, e.g., @xcite ). topic-specific stance requires the existence of numerous, well-labeled training examples in order to build a classifier for a new topic, an unrealistic expectation when there are thousands of possible topics for which data collection and annotation are both time-consuming and expensive. while cross-target stance does not require training examples for a new topic, it does require human knowledge about any new topic and how it is related to the training topics. as a result, models developed for this variation are still limited topic: immigration stance: against text: the jury's verdict will ensure that another violent criminal alien will be removed from our community for a very long period . . . in this work, we propose two additional variations of stance detection: zero-shot stance detection (a classifier is evaluated on a large number of completely new topics) and few-shot stance detection (a classifier is evaluated on a large number of topics for which it has very few training examples). neither variation requires any human knowledge about the new topics or their relation to training topics. zero-shot stance detection, in particular, is a more accurate evaluation of a model's ability to generalize to the range of topics in the real world. existing stance datasets typically have a small number of topics (e.g., 6) that are described in only one way (e.g., 'gun control'). this is not ideal for zero-shot or few-shot stance detection because there is little linguistic variation in how topics are expressed (e.g., 'anti second amendment') and limited topics. therefore, to facilitate evaluation of zero-shot and few-shot stance detection, we create a new dataset, varied stance topics (vast). vast consists of a large range of topics covering broad themes, such as politics (e.g., 'a palestinian state'), education (e.g., 'charter schools'), and public health (e.g., 'childhood vaccination'). in addition, the data includes a wide range of similar expressions (e.g., 'guns on campus' versus 'firearms on campus'). this variation captures how humans might realistically describe the same topic and con-trasts with the lack of variation in existing datasets. we also develop a model for zero-shot stance detection that exploits information about topic similarity through generalized topic representations obtained through contextualized clustering. these topic representations are unsupervised and therefore represent information about topic relationships without requiring explicit human knowledge. our contributions are as follows: (1) we develop a new dataset, vast, for zero-shot and few-shot stance detection and ( 2 ) we propose a new model for stance detection that improves performance on a number of challenging linguistic phenomena (e.g., sarcasm) and relies less on sentiment cues (which often lead to errors in stance classification). we make our dataset and models available for use: https://github.com/ emilyallaway/zero-shot-stance. ## related work previous datasets for stance detection have centered on two definitions of the task @xcite . in the most common definition (topicphrase stance), stance (pro, con, neutral) of a text is detected towards a topic that is usually a nounphrase (e.g., 'gun control'). in the second definition (topic-position stance), stance (agree, disagree, discuss, unrelated) is detected between a text and a topic that is an entire position statement (e.g., 'we should disband nato'). a number of datasets exist using the topicphrase definition with texts from online debate forums @xcite @xcite , information platforms @xcite , student essays @xcite , news comments @xcite and twitter @xcite @xcite . these datasets generally have a very small number of topics (e.g., abbott et al. (2016) has 16 ) and the few with larger numbers of topics @xcite @xcite still have limited topic coverage (ranging from 55 to 194 topics). the data used by @xcite , articles and comments from an online debate site, has the potential to cover the widest range of topics, relative to previous work. however, their dataset is not explicitly labeled for topics, does not have clear pro/con labels, and does not exhibit linguistic variation in the topic expressions. furthermore, all of these stance datasets are not used for zero-shot stance detection due to the small number of topics, with the exception of the semeval2016 task-6 (twitterstance) data, which is used for cross-target stance detection with a single unseen topic @xcite . in constrast to the twitter-stance data, which has only one new topic in the test set, our dataset for zero-shot stance detection has a large number of new topics for both development and testing. for topic-position stance, datasets primarily use text from news articles with headlines as topics @xcite . in a similar vein, @xcite use comments from news articles and manually construct position statements. these datasets, however, do not include clear, individuated topics and so we focus on the topic-phrase definition in our work. many previous models for stance detection trained an individual classifier for each topic @xcite @xcite @xcite @xcite or for a small number of topics common to both the training and evaluation sets @xcite . in addition, a handful of models for the twitterstance dataset have been designed for cross-target stance detection @xcite , including a number of weakly supervised methods using unlabeled data related to the test topic @xcite @xcite . in contrast, our models are trained jointly for all topics and are evaluated for zero-shot stance detection on a large number of new test topics (i.e., none of the zero-shot test topics occur in the training data). ## vast dataset we collect a new dataset, vast, for zero-shot stance detection that includes a large number of specific topics. our annotations are done on comments collected from the new york times 'room for debate' section, part of the argument reasoning comprehension (arc) corpus @xcite . although the arc corpus provides stance annotations, they follow the topic-position definition of stance, as in §2. this format makes it difficult to determine stance in the typical topic-phrase (pro/con/neutral) setting with respect to a single topic, as opposed to a position statement (see topic and arc stance columns respectively, therefore, we collect annotations on both topic and stance, using the arc data as a starting point. ## methods we develop topic-grouped attention (tga) net: a model to implicitly construct and use relationships between the training and evaluation topics without supervision. the model consists of a contextual conditional encoding layer ( §4.2), followed by topic-grouped attention ( §4.4) using generalized topic representations ( §4.3) and a feed-forward neural network (see figure 1 ). ## conclusion we find that our model tga net, which uses generalized topic representations to implicitly capture relationships between topics, performs significantly better than bert for stance detection on pro labels, and performs similarly on other labels. in addition, extensive analysis shows our model provides substantial improvement on a number of challenging phenomena (e.g., sarcasm) and is less reliant on sentiment cues that tend to mislead the models. our models are evaluated on a new dataset, vast, that has a large number of topics with wide linguistic variation and that we create and make available. in future work we plan to investigate additional methods to represent and use generalized topic information, such as topic modeling. in addition, we will study more explicitly how to decouple stance models from sentiment, and how to improve performance further on difficult phenomena. 2017) for sentiment words. we use wordnet to select synonyms with opposite polarity, ignoring word sense and part of speech. we show examples from each set type of swap, + ! (