Z ero S tance: Leveraging C hat GPT for Open-Domain Stance Detection via Dataset Generation (2024)

Abstract: Zero-shot stance detection that aims to detect the stance (typically against, favor, or neutral) towards unseen targets has attracted considerable attention. However, most previous studies only focus on targets from a single or limited text domains (e.g., financial domain), and thus zero-shot models cannot generalize well to unseen targets of diverse domains (e.g., political domain). In this paper, we consider a more realistic task, i.e., open-domain stance detection, which aims at training a model that is able to generalize well to unseen targets across multiple domains of interest. Particularly, we propose a novel dataset generation method ZeroStance, which leverages ChatGPT to construct a synthetic open-domain dataset CHATStance that covers a wide range of domains. We then train an open-domain model on our synthetic dataset after proper data filtering. Extensive results indicate that our model, when trained on this synthetic dataset, shows superior generalization to unseen targets of diverse domains over baselines on most benchmarks. Our method requires only a task description in the form of a prompt and is much more cost-effective and data-efficient than previous methods. We will release our code and data to facilitate future research.

## introduction the task of stance detection is to identify the attitude (e.g., favor, against or neutral, etc.) of a given text with respect to a specific target of interest @xcite @xcite @xcite @xcite . until recently, typical stance detection task was in-domain (mohammad et al., 2016; li and caragea, 2019; siddiqua et al., 2019; li et al., 2021b; upadhyaya et al., 2023; li and caragea, * both authors contributed equally to this research. 2023) in which the training and test sets share the same set of targets. most recent works began considering a cross-domain setup, i.e., crosstarget stance detection @xcite @xcite @xcite where models are trained on labeled data of a training target and tested on a destination target that is unseen during training. however, cross-target task requires human knowledge about any destination target and how it is related to the training target @xcite , which limits models' ability to generalize to a wide variety of unseen targets. hence, there has been an emerging trend to explore zero-shot stance detection @xcite @xcite , which aims to determine the stance towards unseen targets at the inference stage without requiring human knowledge about unseen targets or their relation to training targets. recent studies @xcite @xcite often conduct zeroshot evaluations with training and unseen destination targets originating from a single or limited domains, and thus zero-shot models perform poorly on out-of-domain data. for instance, the wt-wt dataset @xcite exclusively comprises targets within the financial domain for both training and evaluation sets, leading to poor performance of models trained on it when applied to political domain data. @xcite investigate open-domain stance detection which aims to train a model that can generalize well to unseen targets of multiple domains. however, their approach still relies on the texts and targets of existing datasets, and thus limits its applicability to a narrow spectrum of domains. to address the limitations of prior works, we propose a novel open-domain dataset generation approach, zerostance, which aims to promote model's generalization to unseen targets of diverse domains by using external knowledge from pow-erful large language models (llms). motivated by the remarkable success of llms such as chat-gpt in text generation @xcite @xcite , we present chatstance, a synthetic dataset created through zerostance that leverages chatgpt to generate high-quality, human-like targets (controversial claims) and texts spanning a variety of domains. we take into account the attribute diversity during the generation by incorporating desired attributes (including domains, geographical locations, and writing styles) as constraints in the prompts. by comparing the performance of zerostance with previous baselines, we observe a substantial underperformance of the latter, uncovering the effectiveness of our approach for open-domain stance detection. our contributions are summarized as follows: ## method in this section, we introduce the implementation of zerostance which includes three steps: target generation, text generation and data filtering. during target generation, chatgpt is used to generate relevant targets given the category information extracted from the debate forum. for text generation, generated targets and stance labels are fed to chatgpt in the form of prompts to generate corresponding texts. then, we perform data filtering to remove potentially noisy and less informative data. the overall framework of zerostance is shown in figure 1 . ## experimental settings in this section, we first introduce the baseline human-annotated datasets ( §4.1). then we discuss our evaluation setup ( §4.2). last, we describe baselines used in our experiments ( §4.3). wtwt the will-they-won't-they dataset @xcite ) consists of a large number of annotated tweet-target pairs from the financial domain, including five merger and acquisition operations (e.g., merger of cvs health and aetna). the dataset split are presented in appendix d. we also analyze the data contamination between the chatstance and the human-annotated datasets, as presented in appendix e. ## results and discussions in this section, we first compare our approach with previous unsupervised baselines and data augmentation methods ( §5.1). we then perform ablation studies to understand the effectiveness of data filtering and prompts that are used for text generation ( §5.2). we also investigate the impact of the data size to zerostance by training models on different sizes of the chatstance dataset ( §5.3). next, we explore the performance of augmenting existing datasets with our chatstance dataset ( §5.4) and utilizing each human-annotated dataset as the open-domain dataset ( §5.5). ## conclusion in this paper, we propose a novel dataset generation approach for open-domain stance detection, which aims to train a model that performs well on unseen targets from all domains of interest. we leverage the chatgpt to construct a synthetic open-domain dataset from scratch by generating controversial topics of diverse domains and the corresponding texts without training samples of existing datasets. experimental results indicate that the model trained on our synthetic dataset shows better generalization to unseen targets of diverse domains over baselines on most benchmark datasets. moreover, our proposed method is more cost-effective and dataefficient than previous methods.