EZ - STANCE : A Large Dataset for E nglish Zero-Shot Stance Detection (2024)

Abstract: Zero-shot stance detection (ZSSD) aims to determine whether the author of a text is in favor, against, or neutral toward a target that is unseen during training. In this paper, we present EZ-STANCE, a large English ZSSD dataset with 47,316 annotated text-target pairs. In contrast to VAST, which is the only other large existing ZSSD dataset for English, EZ-STANCE is 2.5 times larger, includes both noun-phrase targets and claim targets that cover a wide range of domains, provides two challenging subtasks for ZSSD: target-based ZSSD and domain-based ZSSD, and contains much harder examples for the neutral class. We evaluate EZ-STANCE using state-of-the-art deep learning models. Furthermore, we propose to transform ZSSD into the NLI task by applying simple yet effective prompts to noun-phrase targets. Our experimental results show that EZ-STANCE is a challenging new benchmark, which provides significant research opportunities on English ZSSD. We publicly release our dataset and code at https://github.com/chenyez/EZ-STANCE.

## introduction the goal of stance detection is to automatically detect whether the author of a text is in favor of, against, or neutral toward a specific target @xcite @xcite , e.g., public education, mask mandate, or nuclear energy. the stance can reveal valuable insights relevant to events such as public policy-making and presidential elections. earlier research has concentrated on two types of stance detection tasks: in-target stance detection, in which models are trained and evaluated using data from the same target, e.g., both train and test sets contain data about "donald trump" @xcite @xcite , and cross-target stance detection, where the models are trained on source targets that are related to, but distinct from, the destination targets @xcite , which remain unseen during training (e.g., the destination target is "donald trump" whereas the source target is "hillary clinton"). however, it is unrealistic to incorporate every potential or related target in the training set. as such, zero-shot stance detection (zssd) has emerged as a promising direction @xcite to evaluate classifiers on a large number of unseen (and unrelated) targets. zssd is more related to real-world scenarios and has consequently started to receive significant interest recently @xcite @xcite . despite the growing interest in zssd, the task still exhibits several limitations. first, the vast dataset @xcite ) which is the only other large existing zssd dataset for english, contains only noun phrase targets. yet, in realworld scenarios, stance is often taken toward both noun phrases @xcite and claims @xcite . we notice that models trained on data with noun-phrase targets struggle to accurately predict the stance for claim-target data, and vice versa, due to the mismatch between the training and test data. the need to incorporate both types of targets for zssd has been relatively overlooked. second, vast is designed solely to detect the stance of unseen targets, but these unseen targets at the inference stage originate from the same domain as the training targets (in-domain), possessing similar semantics, which makes the task less challenging. third, despite being instrumental for the development of zero-shot stance detection, vast generates data for the neutral class by randomly permuting documents and targets, leading to a lack of semantic correlation between the two (we show an example from vast from the neutral class in in an effort to address the aforementioned limitations and spur research in zssd, we present ez-stance, a large english zero-shot stance detection dataset collected from twitter. in contrast with vast, ez-stance is, to our knowledge, the first large english zssd dataset that captures both noun-phrase targets and claim targets, covering a more diverse set of targets. by training a single model on our comprehensive dataset, we achieve comparable or superior performance than training separate models for each type of target. moreover, ez-stance includes two real-world scenarios for zero-shot stance detection, namely target-based and domain-based zssd. subtask a: target-based zero-shot stance detection. this subtask is the same as the traditional zssd task @xcite , where stance detection classifiers are evaluated using a large number of completely unseen (and unrelated) targets, but from the same domains (in-domain). subtask b: domainbased zero-shot stance detection. this subtask is our proposed zssd task where stance detection classifiers are evaluated using a large number of unseen targets from completely new domains @xcite . furthermore, in ez-stance, annotators manually extract targets from each tweet to form the neutral class, ensuring semantic relevance to the tweet content. we show an example from our dataset along with corresponding noun-phrase and claim targets for each stance (against, favor, and neutral) in in summary, our contributions are as follows: 1) we present ez-stance, a unique large zero-shot stance detection dataset, composed of 47,316 annotated english tweet-target pairs. ez-stance is 2.5 times larger than vast @xcite , which is the only large existing zssd dataset for english. we provide a detailed description and analysis of our dataset; 2) we consider a more diverse set of targets including both noun phrase and claim targets (see ## related work target-specific stance detection is the most prevalent type of stance detection (aldayel and magdy, 2021), whose goal is to determine the stance expressed in a text towards a target. usually, the target is an entity / short noun-phrase, e.g., a political figure or controversial topic @xcite @xcite , or a claim, e.g., an article's headline or a reply to a rumorous post @xcite @xcite @xcite . most earlier research is centered around in-target stance detection where a classifier is trained and evaluated on the same target @xcite @xcite @xcite @xcite @xcite . however, the challenge often arises in gathering enough annotated data for each specific target, and traditional models perform poorly when generalized to unseen target data. this spurred interest in investigating cross-target stance detection @xcite @xcite , where a classifier is adapted from different but related targets. however, cross-target stance detection still requires human knowledge of the destination target and how it is related to the training targets. thus, models developed for cross-target stance detection are still limited in their capability to generalize to a wide range of unseen targets @xcite . name authors source # tar-target type task size get(s) nounphrase claim semeval16 mohammad et al. (2016b) twitter 6 ✓ ✗ in-target 4,870 multitarget sobhani et al. (2017) twitter 4 ✓ ✗ cross-target 4,455 wtwt conforti et al. (2020b) twitter 5 ✓ ✗ in-target 51,284 vast allaway and mckeown (2020) comments 5,634 ✓ ✗ zssd 18,545 covid19 glandt et al. (2021) twitter 4 ✓ ✗ in-target 6,133 p-stance li et al. (2021) twitter 3 ✓ ✗ in-target 21,574 emergent ferreira and vlachos (2016) news articles 300 ✗ ✓ zssd 2,595 rumeval17 derczynski et al. (2017) twitter 325 ✗ ✓ cross-target 5,568 rumeval19 gorrell et al. (2019) twitter, reddit 446 ✗ ✓ zssd 8,574 ez-stance ours twitter 40,678 ✓ ✓ zssd 47,316 to detect the stance on a large number of unseen (and unrelated) targets represents a more realistic scenario, and has received significant interest in recent years @xcite @xcite @xcite @xcite . these works tackle stance detection where the target is an entity or noun phrase, whereas other works that follow a zssd scenario (training and test data are collected from different events) focus on stance classification towards rumours in which the target is a longer text or a claim @xcite @xcite . surprisingly, despite substantial interest in zssd, large-scale datasets for the evaluation of this task are limited. in ## dataset construction here, we detail the construction of ez-stance. ## methodology we now present our approach for converting zssd into the natural language inference (nli) task. ## baselines and models we introduce the zssd baselines and our approach of utilizing nli pre-trained models for zssd. ## results in this section, we first present results for subtask a ( §6.1) and subtask b ( §6.2). we then compare ez-stance with the vast dataset ( §6.3). next, we study the impact of different prompt designs ( §6.4). like prior works @xcite , we employ the macro-averaged f1 score across all classes as our evaluation metrics. ## conclusion in this paper, we present ez-stance, a large english zssd dataset. compared with existing english zssd datasets, our dataset is larger and more challenging. ez-stance covers both nounphrase targets and claim targets and also comprises two challenging zssd subtasks: targetbased zssd and domain-based zssd. we improve the data quality of the neutral class by extracting targets from texts. we evaluate ez-stance on zssd baselines and propose to transform zssd into the nli task which outperforms traditional baselines. we hope ez-stance can facilitate future research for varied stance detection tasks.