Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection (2023)

Abstract: Data collection from manual labeling provides domain-specific and task-aligned supervision for data-driven approaches, and a critical mass of well-annotated resources is required to achieve reasonable performance in natural language processing tasks. However, manual annotations are often challenging to scale up in terms of time and budget, especially when domain knowledge, capturing subtle semantic features, and reasoning steps are needed. In this paper, we investigate the efficacy of leveraging large language models on automated labeling for computational stance detection. We empirically observe that while large language models show strong potential as an alternative to human annotators, their sensitivity to task-specific instructions and their intrinsic biases pose intriguing yet unique challenges in machine annotation. We introduce a multi-label and multi-target sampling strategy to optimize the annotation quality. Experimental results on the benchmark stance detection corpora show that our method can significantly improve performance and learning efficacy.

## introduction stance detection is one of the fundamental social computing tasks in natural language processing, which aims to predict the attitude toward specified targets from a piece of text. it is commonly formulated as a target-based classification problem @xcite : given the input text (e.g., online user-generated content) and a specified target, stance detection models are used to predict a categorical label (e.g., favor, against, none) @xcite . upon social networking platforms' growing impact on our lives, stance detection is crucial for various downstream tasks such as fact verification and rumor detection, with wide applications including analyzing user feedback and political opinions @xcite . data-driven approaches largely benefit from the available human-annotated corpora, which provide domain-specific and task-aligned supervision. developing state-of-the-art models via in-domain finetuning usually requires certain well-annotated training data, such as machine translation and text summarization. however, manual annotations are challenging to scale up in terms of time and budget, especially when domain knowledge, capturing subtle semantic features, and reasoning steps are needed. for instance, for computational stance detection, there are some corpora with target-aware annotation @xcite @xcite , but they are limited in sample size, target diversity, and domain coverage. learning from insufficient data leads to low generalization on out-of-domain samples and unseen targets @xcite , and models are prone to over-fit on superficial and biased features @xcite . recently, how the versatile large language models (llms) would contribute to data annotation raised emerging research interest @xcite . machine annotation is also one approach to knowledge distillation from large and general models @xcite , and student models can be lightweight and cost-effective to deploy. in this paper, we leverage and evaluate llms on automated labeling for computational stance detection. we empirically find that the general-purpose models can provide reasonable stance labeling, and show the potential as an alternative to human annotators. however, there are substantial variances from multiple aspects, and models exhibit biases on target description and label arrangement ( §3). such issues pose challenges regarding the quality and reliability of machine annotation. to optimize the annotation framework, we introduce the multi-label inference and multi-target sampling strategy ( §4). evaluation results on five benchmark stance detection corpora show that our method can significantly improve the performance and learning efficacy ( §5). ## related work computational stance detection aims to automatically predict polarities given the text and a stance target. the model design for this task has been shifted from the traditional statistical methods such as support vector machines @xcite , to neural approaches such as recurrent neural networks @xcite , convolutional neural networks @xcite @xcite , and transformer @xcite . to facilitate the development of data-driven approaches, human-annotated datasets for stance detection are constructed for model training and evaluation @xcite @xcite . while the transformer-based models, especially pre-trained language backbones @xcite , boost the performance on stance detection benchmarks @xcite , due to the limited training data and low labeling diversity, these supervised models showed degraded performance on unseen-target and out-of-domain evaluation @xcite . to address this problem, recent work introduced de-biasing methods @xcite , multitask learning @xcite , and the linguisticsinspired augmentation @xcite . ## machine annotation for computational stance detection based on large-scale pre-training and instruction tuning, language models are capable of solving various downstream tasks in a zero-shot manner @xcite . in many nlp benchmarks, instruction prompt a: classify a tweet stance on {target-placeholder} into "favor", "against", or "none". ## improving machine annotation via multi-label and multi-target sampling the aforementioned labeling variances pose a question mark on the reliability of machine annotation, and the noisy data cannot be readily used to train dedicated parameter-efficient models. previous work proved that capturing target referral information and simulating human reasoning steps are beneficial for stance detection @xcite . therefore, we introduce two methods to improve the machine annotation quality. in this section, we evaluate the vicuna-13b, as it is an open model and reaches a balance between performance and computational complexity. first, an off-the-shelf constituency parser is used to extract all noun phrases from the text. 1 we then use these phrases as targets for stance labeling; if one phrase is in a contrary stance label to the original target, we include it as an additional sample. this adversarial multi-target sampling helps models to condition their prediction more on the given target, as well as learn some correlation between explicit and implicit objects. in our experiment, we obtain 1.2k multi-target samples, where the original corpus size is 2914. ## leveraging machine-annotated data for model training based on our improvements in section 4, to compare the learning efficacy from machine-annotated and human-annotated data, we conduct experiments by training a supervised model. experimental setting as light-weight language backbones are more applicable to practical use cases, we select roberta @xcite which provides state-of-the-art performance in many previous works on stance detection @xcite . we train the model on the semeval-16 task-6 a corpus, and collect five representative benchmarks to build the in-domain, out-of-domain, and cross-target evaluation settings @xcite , including semeval-16 task-6 a and task-6 b @xcite , p-stance @xcite , vast (allaway and mckeown, 2020), and tweet-covid @xcite . model input is formulated as "<@xmath0, and the final hidden representation of the first token "</@xmath1. experimental details are shown in appendix ## conclusions in this work, we investigated the potential and challenges of leveraging large language models on automated labeling for computational stance detection. quantitative analyses demonstrated the feasibility, but their predictions are sensitive to taskspecific instructions, and llms inevitably present exposure biases on target description and label arrangement. we improved machine annotation by adopting a multi-label and multi-target sampling strategy. experimental results on several stance detection benchmarks showed the effectiveness of our proposed methods. this work can shed light on further extensions of machine annotation.