C - STANCE : A Large Dataset for C hinese Zero-Shot Stance Detection (2023)

Abstract: Zero-shot stance detection (ZSSD) aims to determine whether the author of a text is in favor of, against, or neutral toward a target that is unseen during training. Despite the growing attention on ZSSD, most recent advances in this task are limited to English and do not pay much attention to other languages such as Chinese. To support ZSSD research, in this paper, we present C-STANCE that, to our knowledge, is the first Chinese dataset for zero-shot stance detection. We introduce two challenging subtasks for ZSSD: target-based ZSSD and domain-based ZSSD. Our dataset includes both noun-phrase targets and claim targets, covering a wide range of domains. We provide a detailed description and analysis of our dataset. To establish results on C-STANCE, we report performance scores using state-of-the-art deep learning models. We publicly release our dataset and code to facilitate future research.

## introduction stance detection aims to automatically predict whether the author of a text is in favor of, against, or neutral toward a specific target @xcite @xcite , e.g., epidemic prevention, gasoline price, or equal rights. the stance can provide useful information for important events such as policymaking and presidential elections. early works focus on two types of stance detection tasks: in-target stance detection, where classifiers are trained and tested on data from the same set of targets @xcite @xcite and cross-target stance detection, where classifiers are trained on source targets that are related to destination targets @xcite , but destination targets are unseen during training. however, it is impractical to include 1 https://github.com/chenyez/c-stance all possible or related targets in the training set. more recently, zero-shot stance detection (zssd) has been identified as a promising direction @xcite to evaluate classifiers on a large number of unseen (and unrelated) targets. zssd is more similar to the situations in practice and has received a lot of attention @xcite @xcite . despite the growing interest in stance detection, the task has several limitations. first, most recent advances in stance detection are limited to english @xcite @xcite @xcite , and pay little attention to other languages such as chinese @xcite although large amounts of online data with expressions of stance are available in other languages. second, the current zssd task @xcite aims to detect the stance of unseen targets. however, these unseen targets come from the same domain of training targets with similar meanings, which makes the task less challenging. third, current stance detection datasets include targets either as noun phrases @xcite or as claims @xcite . however, in practice, stance is usually expressed toward both noun phrases and claims. models trained only on nounphrase targets do not necessarily work well for claim targets and vice versa. little attention is paid toward incorporating targets of both types. in an effort to minimize these drawbacks, we present c-stance, the first chinese zero-shot stance detection dataset. our dataset is collected from sina weibo, one of the most popular chinese social media sites (akin to twitter). we consider two practical scenarios for zero-shot stance detection, i.e., target-based and domain-based zssd. subtask a: target-based zero-shot stance detection. subtask a is similar to the previous zssd task, where stance detection classifiers are evalu-请赶紧打疫苗，勤洗手把口罩戴起来。随着新冠病例的剧增，医疗资源不足必然引起医患矛盾 的爆发，也请大家能互相体谅。 please get vaccinated quickly, wash hands frequently and put on your mask. with the sharp increase in covid-19 cases, the shortage of medical resources will inevitably lead to the outbreak of conflicts between doctors and patients. please understand each other. noun-phrase target/stance 1. 新冠疫苗 covid-19 vaccine / favor 2. 医患矛盾 conflict between doctors and patients / against claim target/ stance 1. 还是要去打疫苗，做好自我防护，尽量别阳，不去挤占医疗资源。 we should still get vaccinated and do self-protection, try not get covid and not to take medical resources. / favor 2. 那些戴好口罩，勤消毒的人都阳了，所以说防护没啥用的。 those who wear masks and disinfect frequently still get covid, so it is useless to defend. / against 3. 免疫力真的很重要，平时就要加强自身锻炼，增强免疫力。 immunity is really important, we need to strengthen our own exercise at ordinary times to enhance immunity / neutral 1 . as we can see from the table, the author of the microblog is in favor of the noun-phrase target "covid-19 vaccine" and against "the conflict between doctors and patients". the author also opposes claim target 2, whose main idea is to deny the necessity of self-protection. our contributions can be summarized as follows: 1) we present c-stance, the first large chinese zero-shot stance detection dataset. our dataset is composed of 48,126 annotated text-target pairs. c-stance is more than 2.5 times larger than the english zssd vast dataset @xcite ) and more than 16 times larger than the existing chinese stance detection dataset by @xcite . we provide detailed description and analysis of our dataset; 2) we include two challenging zssd subtasks: target-based zero-shot stance detection and domain-based zero-shot stance detection for c-stance; 3) we consider a more diverse set of targets including both noun phrases and claims in c-stance as well as multiple targets per input text (see ## related work most previous stance detection datasets are constructed for the english language @xcite @xcite . particularly, vast is the only dataset for zero-shot stance detection. even though recent years have witnessed an emerging trend of constructing stance detection datasets of other languages @xcite @xcite @xcite , chinese stance detection datasets are still very scarce. @xcite developed the first chinese stance dataset. the dataset focuses on in-target stance detection and only includes 3,000 examples from 6 targets. in contrast, we propose the first large-scale chinese dataset for zero-shot stance detection. our c-stance which includes 48,126 samples with 11,623 noun-phrase targets and 28,581 claim targets enables multiple stance detection tasks and covers a wide range of domains. besides classifying stance detection by target type (noun phrases or claims), we can also categorize the task as in-target, cross-target, and zero-shot stance detection. most previous works focused on in-target stance detection where a classifier is trained and evaluated on the same target @xcite @xcite @xcite @xcite @xcite . however, it is usually hard to obtain sufficient annotated data for each target and conventional models perform poorly when generalized to data of unseen targets. this motivated the research on cross-target stance detection @xcite @xcite @xcite , where a classifier is adapted from different but related targets. however, cross-target stance detection still requires prior human knowledge of the destination target and how it is related to the training targets. thus models developed for cross-target stance detection are still limited in their capability to generalize to a wide range of unseen targets. zero-shot authors source # target(s) target type language size ferreira and vlachos (2016) news articles 300 claim english 2,595 derczynski et al. (2017) twitter 305 claim english 5,568 gorrell et al. (2019) twitter, reddit 8,574 claim english 8,574 vamvas and sennrich (2020) political comments 194 claim english, french, germany, italian 67,000 xu et al. (2016) weibo 7 noun-phrase chinese 5,000 mohammad et al. (2016b) twitter 6 noun-phrase english 4,870 swami et al. (2018) twitter 1 noun-phrase english, hindi 3,545 conforti et al. (2020b) twitter 5 noun-phrase english 51,284 allaway and mckeown (2020) news comments 5,634 noun-phrase egnlish 18,545 glandt et al. (2021) twitter 4 noun-phrase english 6,133 li et al. (2021a) twitter 3 noun-phrase english 21,574 lai et al. (2020) twitter 6 noun-phrase english, spanish, catalonia, french, italian 14,440 c-stance (ours) weibo 40,204 noun-phrase, claim chinese 48,126 based on vast, many zssd models have been developed @xcite @xcite . in contrast to vast, we include two types of zssd subtasks in c-stance. the first subtask is the target-based zssd which is similar to the vast setting. the second subtask is the domain-based zssd where classifiers are evaluated on unseen targets from completely new domains, which is a more challenging task. target-specific stance detection is the most common stance detection task (aldayel and magdy, 2021), which aims to predict the stance label toward a target, which could be a figure or controversial topic @xcite @xcite . multi-target stance detection is another type of stance detection task that aims to jointly identify the stance toward two or more targets in the same text @xcite @xcite . unlike target-specific and multi-target stance detection where targets are usually noun phrases (phrasebased stance detection), claim-based stance detection aims to predict the stance toward a specific claim, which could be an article headline or a rumor's post @xcite @xcite @xcite . however, less attention has been paid to incorporating both noun-phrase targets and claim targets into one dataset. comparatively, our dataset supports data for both claim-based stance detection and phrase-based stance detection as well as captures multiple targets per input text (see examples from our dataset in appendix a). we compare our c-stance dataset with previous stance detection datasets in ## dataset construction in this section, we describe the creation and particularities of c-stance, a large comprehensive stance detection dataset composed of 48,126 annotated instances covering a wide range of domains. ## experimental settings in this section, we introduce the baselines in section 4.1 and the training settings in section 4.2. ## results in this section, we first perform experiments on subtask a and subtask b. we then conduct experiments on cross-lingual stance detection using c-stance and the previous english zssd vast dataset. we also study the impact of incorporating both nounphrase targets and claims targets into one dataset. lastly, we perform the spuriosity analysis for claim targets. each result is the average of 4 runs with different initializations. similar to prior works @xcite , we use the f1 for each class and the macro-averaged f1 of all classes as evaluation metrics. ## conclusion in this paper, we introduce c-stance, the first chinese zero-shot stance detection dataset. our dataset includes two challenging zssd subtasks: target-based zssd (evaluating classifiers using a large number of unseen targets) and domain-based zssd (evaluating classifiers using a large number of unseen targets from unseen domains). moreover, we consider both noun-phrase targets and claim targets. our dataset is larger and more challenging compared with the previous chinese stance detection dataset, consisting of 48,126 annotated microblog-target pairs. c-stance can serve as a new benchmark for zssd, along with vast, and can enable future research for other stance detection tasks. we conduct experiments using state-ofthe-art deep learning models. future work includes studying the multilingual zssd with the union of c-stance and other multi-lingual datasets.