O pen S tance: Real-world Zero-shot Stance Detection (2022)

Abstract: Prior studies of zero-shot stance detection identify the attitude of texts towards unseen topics occurring in the same document corpus. Such task formulation has three limitations: (i) Single domain/dataset. A system is optimized on a particular dataset from a single domain; therefore, the resulting system cannot work well on other datasets; (ii) the model is evaluated on a limited number of unseen topics; (iii) it is assumed that part of the topics has rich annotations, which might be impossible in real-world applications. These drawbacks will lead to an impractical stance detection system that fails to generalize to open domains and open-form topics. This work defines OpenStance: open-domain zero-shot stance detection, aiming to handle stance detection in an open world with neither domain constraints nor topic-specific annotations. The key challenge of OpenStance lies in open-domain generalization: learning a system with fully unspecific supervision but capable of generalizing to any dataset. To solve OpenStance, we propose to combine indirect supervision, from textual entailment datasets, and weak supervision, from data generated automatically by pre-trained Language Models. Our single system, without any topic-specific supervision, outperforms the supervised method on three popular datasets. To our knowledge, this is the first work that studies stance detection under the open-domain zero-shot setting. All data and code will be publicly released.

## introduction stance detection differentiates the attitude (e.g., support, oppose, or neutral) of a text towards a topic @xcite . the topic can be a phrase or a complete sentence. the same text can express the author's positions on many different topics. for example, a tweet on climate warm-ing may also express attitudes about environmental policies as well as the debate between electric or fuel cars. such compound expression can be seen on all online platforms, including news outlets, twitter, blogs, etc. therefore, stance detection can be a complicated task that is essential for developing the inference capability of nlp models as well as other disciplines such as politics, journalism, etc. since the textual expressions and the size of topics in the real world are unpredictable, zero-shot stance detection has become the mainstream research direction in this area: topics in the test set are unseen during training. for example, @xcite created a dataset semt6 based on tweets with six noun phrases as topics. one of the topics was reserved for testing and the remaining were used for training. @xcite extended the topic size on the domain of news comments by covering 4,000 topics in training and 600 unseen topics in testing. however, despite the change in the domain and topic size, there are three major limitations in previous studies which make the task not a real zero-shot task: (i) the dataset only contains texts from a single domain, such as news comments in vast @xcite and tweets in semt6 @xcite ; (ii) most literature studied only a limited size of topics with a single textual form (either noun phrases or sentential claims), e.g., @xcite ; (iii) rich annotation for at least part of the topics is always required, which is not possible in real-world applications because data collection can be very time-consuming and costly @xcite . those limitations lead to an impractical zero-shot stance detection system that cannot generalize well to unseen domains and open-form topics. in this work, we re-define what a zero-shot stance detection should be. specifically, we define openstance: an open-domain zero-shot stance detection, aiming to build a system that can work in the real world without any specific attention to the text domains or topic forms. more importantly, no task-specific supervision is needed. to achieve this, we propose to combine two types of supervision: indirect supervision and weak supervision. the indirect supervision comes from textual entailmentwe treat the stance detection problem as a textual entailment task since the attitude toward a topic should be inferred from the input text. therefore, the existing entailment datasets, such as mnli @xcite , can contribute supervision to the zero-shot setting. to collect supervision that is more specific to the openstance task, we design two mask choices (mask-topic and mask-text) to prompt @xcite to generate weakly supervised data. given an input text and a stance label (support, oppose, or neutral), mask-topic predicts what topic is appropriate based on the content; given a topic and a label, mask-text seeks the text that most likely holds this stance. the collection of weakly supervised data only needs the unlabeled texts and the set of topics that users want to include. the joint power of indirect supervision and weak supervision will be evaluated on vast, semt6 and perspectrum @xcite , three popular datasets that cover distinct domains, different sizes and diverse textual forms of topics. experimental results show that although no task-specific supervision is used, our system can get robust performance on all three datasets, even outperforming the task-specific supervised models (72.6 vs. 69.3 by mean f1 over the three datasets). our contributions are threefold: (i) we define openstance, an open-domain zero-shot stance detection task, that fulfills real-world requirements while having never been studied before; (ii) we design a novel masking mechanism to let gpt-3 generate weakly supervised data for openstance. this mechanism can inspire other nlp tasks that detect relations between two pieces of texts; (iii) our approach, integrating indirect supervision and weak supervision, demonstrates outstanding generalization among three datasets that cover a wide range of text domains, topic sizes and topic forms. zero-shot stance detection. recently, researchers started to work on zero-shot stance detection in order to build a system that can handle unseen topics. most work split the collected topic-aware annotations into train and test within the same domain. @xcite made use of topic similarity to connect unseen topics with seen topics. @xcite designed adversarial learning to learn domain-independent information and topic-invariant representations. similarly, @xcite applied adversarial learning to extract stance-related but domain-invariant features existed among different domains. @xcite utilized common sense knowledge from conceptnet @xcite to introduce extra knowledge of the relations between the texts and topics. most prior systems worked on a single domain and were tested on a small number of unseen topics. @xcite tried to test on various unseen datasets by jointly optimizing on multiple training datasets. however, they still assumed that part of the topics or domains has rich annotations. in contrast, our goal is to design a system that can handle stance detection in an open world without requiring any domain constraints or topic-specific annotations. ## problem definition openstance has the following requirements: openstance vs. prior zero-shot stance detection. prior studies of zero-shot stance detection worked on a single dataset d i in which all texts s comes from the same domain. topics t in the dataset are split into train, dev and test disjointly. the main issue is that a model that fits d i does not work well on a new dataset d j that may contain s of different domains and unseen t. for example, a model trained on vast can only get f1 49.0% on perspectrum, which is around the performance of random guess. openstance aims at handling multiple datasets of open domains and open-form topics without looking at their train and dev. openstance vs. textual entailment. stance detection is essentially a textual entailment problem if we treat the text s as the premise, and the stance towards the topic t as the hypothesis. this motivates us to use indirect supervision from textual entailment to deal with the stance detection problem. nevertheless, there are two distinctions between them: (i) even though we can match l of stance detection with the labels of textual entailment: @xmath0, @xmath1, whether a topic t in stance detection can be treated as a hypothesis depends on the text form of t. if t is noun phrases such as "gun control", t cannot act as a hypothesis alone as there is no stance in it; if t is a sentential claim such as "climate change is a real concern", inferring the truth value of this hypothesis is exactly a textual entailment problem. this observation motivates us to test openstance on topics of both phrase forms and sentence forms; ## methodology this section introduces how we collect and combine indirect supervision and weak supervision to solve openstance. indirect supervision. as we discussed in section 3, stance detection is a case of textual entailment since the stance l towards a topic t should be inferred from the text s. to handle the zero-shot challenge in openstance, textual entailment is a natural choice for indirect supervision. specifically, we first cast stance detection instances into the textual entailment format by combining l and t as a sentential hypothesis h, such as "it supports topic", and treating the s as the premise p; then a pretrained model on mnli @xcite , one of the largest entailment dataset, is ready to predict the relationship between the p and h. an entailed (resp. contradicted or neutral) h means the topic t is supported (resp. opposed or neutral) by the text s. unfortunately, the indirect supervision from textual entailment may not perform well enough in real-world openstance considering the widely known brittleness of pretrained entailment models and the open domains and open-form topics in openstance. therefore, in addition to the indirect supervision from textual entailment, we will collect weak supervision that is aligned better with the texts {x} and the topics {t}. weak supervision. for the next step, we would like to create some weakly supervised data using easily available resources to obtain a better understanding of the target task. we used gpt-3 (brown et al., 2020), a pre-trained autoregressive language model that can perform text completion at (arguably) a near-human level, to help us create some weakly labeled instances. we form incomplete sentences using prompts, and let the gpt-3 complete them. since a stance label l connects the text s and the topic t and such connection is unavailable in a zero-shot setting, the construction of incomplete sentences is driven by two questions: (i) given an input text s and a stance, e.g., support, what topics are supported by s? (ii) given a topic and a stance, for example, support, what texts support this topic? as a result, there are two kinds of prompts: mask-topic and mask-text. to implement the two masking mechanisms, we need to prepare three sets: the raw texts {s}, a set of topics {t}, and the known stance labels {support, oppose, neutral}. it is noteworthy that no topic-specific human annotations are used here. for any dataset of stance detection, we first collect the three sets (i.e., {s}, {t}, and {l}) from the label-free training set without peeking at any gold annotations, then use mask-topic and mask-text prompts to generate equal number of weakly supervised examples. we will study which masking scheme is more effective in experiments. in addition, to have a fair comparison with supervised methods that learn on the train of a task, we make sure our generated weakly supervised data has the same size as the train for any target task. although noise is common in weakly supervised data, gpt-3 performs badly on neutral completions for both mask-topic and mask-text tasks. this is not a surprise for the mask-topic since the gpt-3 is asked to provide a topic that the given text has a neutral attitude for, while most texts, obtained from unlabeled train and originally extracted from social networks, usually express a strong attitude. furthermore, in mask-text, even though the gpt-3 can output a text given the neutral label towards a topic, the response is very general and does not provide any insights. for example, when the template is "his attitude towards high school writing skills is neutral because he thinks [mask]", gpt-3 fills out the mask with "that they are important but not essential." obviously, it is much easier to generate text with a clear attitude compared to a neutral stance. on the one hand, gpt-3 may not really understand what a neutral stance is. on the other hand, even humans cannot easily write a neutral opinion towards a topic. since the quality of generated neutral instances is not very promising, we take the same approach as how vast @xcite collected its neutral samples: matching texts with random topics in the dataset. training strategy. to keep consistent format and make full use of the entailment reasoning framework, we convert all phrase-form topic in the weak supervision data into a sentence-form hypothesis with the positive stance, i.e., "he is in favor of topic" (note that this does not change the original label). then, we randomly split the weak supervision data as train (80%) and dev (20%). given the entailment dataset mnli as the indirect supervision data (d ind ) and weakly supervised data (d weak ) from gpt-3, we first pretrain a robertalarge @xcite on d ind , then finetune on d weak . in inference, we test the final model on the test of each task, checking the system's generalization ability on diverse domains without optimizing on any domain-specific train. 5 experiments ## conclusion in this work, we define openstance, a more realistic and challenging zero-shot stance detection problem in an open world. under such a setting, multiple domains and numerous topics can be involved, while no topic-specific annotations are required. to solve this problem, we proposed to combine indirect supervision from textual entailment and weak supervision collected from gpt-3. our system, without the help of any task-specific supervision, outperforms the supervised method on three benchmark datasets that cover various domains and free-form topics.