Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation (2021)

Abstract: Stance detection determines whether the author of a text is in favor of, against or neutral to a specific target and provides valuable insights into important events such as legalization of abortion. Despite significant progress on this task, one of the remaining challenges is the scarcity of annotations. Besides, most previous works focused on a hard-label training in which meaningful similarities among categories are discarded during training. To address these challenges, first, we evaluate a multi-target and a multi-dataset training settings by training one model on each dataset and datasets of different domains, respectively. We show that models can learn more universal representations with respect to targets in these settings. Second, we investigate the knowledge distillation in stance detection and observe that transferring knowledge from a teacher model to a student model can be beneficial in our proposed training settings. Moreover, we propose an Adaptive Knowledge Distillation (AKD) method that applies instance-specific temperature scaling to the teacher and student predictions. Results show that the multi-dataset model performs best on all datasets and it can be further improved by the proposed AKD, outperforming the state-of-the-art by a large margin. We publicly release our code.

## introduction people often express their stances toward specific targets (e.g., political figures, or abortion) on social media. these opinions can provide valuable insights into important events, e.g., legalization of abortion. the goal of stance detection is to determine whether the author of a text is in favor of, against or neutral toward a specific target @xcite @xcite . for example, for the tweet in ## related work stance detection task aims to identify the stance toward a specific target @xcite @xcite . the target is usually a political figure @xcite @xcite @xcite , a controversial topic such as marijuana legalization @xcite @xcite @xcite @xcite @xcite @xcite @xcite or a claim that could be a rumor's post @xcite @xcite @xcite @xcite . besides the in-target stance detection where the test topic has always been seen in the training stage, cross-target stance detection @xcite @xcite and zero-shot stance detection @xcite ) have also attracted a lot of attention in recent years. in this paper, we focus on the in-target stance detection. interestingly, despite significant progress on stance detection, the large-scale annotated datasets are limited and the number of training samples varies drastically between datasets. to make matters worse, previous studies @xcite @xcite @xcite @xcite adopted an ad-hoc training strategy, which means that the number of models that need to be trained is proportional to the number of targets. to address these issues, @xcite explored multitask learning for various stance detection tasks by fine-tuning the pre-trained bert @xcite on multiple datasets. different from @xcite , in this paper, we evaluate three different training settings on the datasets of diverse domains, showing the improvement brought by the joint training step by step. moreover, we investigate whether knowledge distillation can help further improve the performance of stance detection models. knowledge distillation @xcite aims to distill the knowledge from a teacher model into a student model and has been widely adopted and modified in computer vision @xcite @xcite @xcite and natural language processing @xcite @xcite @xcite @xcite . @xcite proposed self-distillation in which the teacher and student models have identical architectures. @xcite further extended selfdistillation to the multi-task setting to achieve supe-rior performance than standard multi-task training. zhang and sabuncu (2020) attributed the success of self-distillation to the increasing uncertainty and diversity in teacher predictions. despite recent progress in knowledge distillation, less attention has been paid to combining knowledge distillation with stance detection. @xcite distilled knowledge in a semisupervised manner for covid-19 stance detection. however, experiments have only been conducted on a small dataset and the test set only contains one target. motivated by recent works, we comprehensively investigate self-distillation in the stance detection under multi-target and multi-dataset training settings and evaluate the model performance on five datasets of different domains. moreover, we propose an instance-specific temperature scaling method to further improve the self-distillation and explore how to effectively distill knowledge to the student model in a holistic way. ## experimental settings in this section, we first describe stance detection datasets used for evaluation and introduce the evaluation metrics. then, we describe several baseline methods of knowledge distillation. ## results in this section, we thoroughly discuss the experimental results to answer our research questions presented in §1. first, we study the performance of models in three training settings in §5.1. then we explore the effectiveness of different distillation models on stance detection datasets and test the generalization ability of knowledge distillation models in §5.2. we finally compare the distillation models in different settings in §5.3. ## conclusion in this paper, we formulated three research questions for which evidence-based answers were unknown. we conducted extensive experiments on stance detection datasets and answer the questions as follows: 1) the performance of a stance detection model can be significantly improved by training on all targets of each dataset and on multiple datasets. moreover, the model trained on datasets of diverse domains shows superior performance than the model trained on each dataset, indicating that the multi-dataset model benefits from learning with more training data and the multi-target model might still overfit the training data. 2) selfdistillation can further improve the stance detection in both training settings. our proposed akd benefits stance detection the most and shows better generalization abilities over other knowledge distillation methods. 3) we explore different distillation settings and observe that @xmath0, which indicates that distilling knowledge from a well-trained teacher is more beneficial to the stance detection. future work includes further strengthening the multi-dataset model by incorporating more stance detection datasets. it would be also interesting to extend the knowledge distillation to more stance detection tasks such as rumour detection and multilingual stance detection. bowen zhang, min yang, xutao li, yunming ye, xiaofei xu, and kuai dai. 2020. enhancing crosstarget stance detection with transferable semanticemotion knowledge. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 3188-3197. zhilu zhang and mert sabuncu. 2020. self-distillation as instance-specific label smoothing. in advances in neural information processing systems, volume 33, pages 2184-2195. elena zotova, rodrigo agerri, manuel nuñez, and german rigau. 2020. multilingual stance detection in tweets: the catalonia independence corpus. in proceedings of the 12th language resources and evaluation conference, pages 1368-1375.