Distilling Calibrated Knowledge for Stance Detection (2023)

Abstract: Stance detection aims to determine the position of an author toward a target and provides insights into people’s views on controversial topics such as marijuana legalization. Despite recent progress in this task, most existing approaches use hard labels (one-hot vectors) during training, which ignores meaningful signals among categories offered by soft labels. In this work, we explore knowledge distillation for stance detection and present a comprehensive analysis. Our contributions are: 1) we propose to use knowledge distillation over multiple generations in which a student is taken as a new teacher to transfer knowledge to a new fresh student; 2) we propose a novel dynamic temperature scaling for knowledge distillation to calibrate teacher predictions in each generation step. Extensive results on three stance detection datasets show that knowledge distillation benefits stance detection and a teacher is able to transfer knowledge to a student more smoothly via calibrated guiding signals. We publicly release our code to facilitate future research.

## introduction the stance detection task aims to identify the position of a user toward a specific target @xcite @xcite . the target is usually a controversial topic @xcite , a public figure @xcite or a claim that could be a rumor's post @xcite . for example, for the sentence in model miscalibration can be widely observed in modern neural networks @xcite @xcite . if a model is well calibrated, then the probability associated with the predicted label should reflect its ground-truth correctness. according to our empirical observations, teacher models of akd trained on stance detection datasets are not well-calibrated, producing peaked distributions of confidence in stance detection. @xcite showed that the teacher that provides less peaked training signals makes it possible for the student to learn better signals from interclass similarity and can potentially reduce overfitting. therefore, we associate the performance of knowledge distillation with the calibration of teacher models in stance detection and propose to further improve the task performance by calibrating teacher predictions. specifically, we recalibrate teacher predictions using a post-processing method called temperature scaling @xcite and the student model is trained based on both hard labels and calibrated teacher predictions. further, born-again networks @xcite , in which the teacher and student models have identical model architectures, have achieved additional improvements with multiple students generations. at each consecutive step, a student model is taken as a new teacher to transfer knowledge to a new fresh student model. in this paper, we explore the born-again networks for stance detection and propose to calibrate teacher predictions in each generation. extensive experiments on stance detection datasets show that a teacher can transfer knowledge to a student more smoothly via calibrated soft labels generated by the teacher and training student models over multiple generations helps improve the task performance. our contributions are summarized as follows: ## related work previous works for stance detection mainly focus on the in-target setting @xcite @xcite @xcite where the test target has always been seen in the training stage. recently, cross-target stance detection @xcite @xcite and zero-shot stance detection @xcite @xcite @xcite have also attracted a lot of attention. in this paper, we mainly focus on the in-target stance detection. an ad-hoc training strategy that trains one model for one target has been widely used in previous works @xcite @xcite @xcite . however, it only considers one target during training and thus it fails to exploit the potential of all training data. recent works @xcite have shown that multi-target training that trains one model on all targets of a dataset can benefit the stance detection task. in our work, we adopt this multitarget training strategy and conduct extensive experiments in the multi-target training setting on three stance detection datasets @xcite @xcite . knowledge distillation, initially proposed by @xcite , has been widely used in natural language processing to distill external knowledge into a model @xcite @xcite @xcite @xcite . interestingly, despite that recent works @xcite @xcite @xcite have made impressive progress in knowledge distillation in which the teacher and student have the same model architecture, not much attention has been paid to using knowledge distillation for stance detection. one exception is the work by @xcite who proposed an adaptive knowledge distillation method (akd) that applies instance-specific temperature scaling to the teacher predictions in one generation. in contrast to this prior work which uses only a single generation step for knowledge distillation, we explore born-again networks on three stance datasets and observe further improvements in performance of stance detection models when they are trained over multiple generations. motivated by recent advances in calibration of neural networks @xcite @xcite @xcite , we hypothesize that the miscalibration of teacher predictions has a negative impact on the student model and further test this hypothesis by calibrating teacher's predictions in born-again networks. in this work, we conclude that calibrating teacher's predictions in each generation benefits the student model by providing smoother supervision signals in stance detection task. @xmath0 x h e n y 9 f 6 5 q t z 5 z a h z i f p 5 s e k c @xmath1 x m f j c 2 t 7 p 7 h b 2 t s / o d w q h 5 + 0 j u o 1 o s 2 i u n l d c b v k m a q t y y y n 3 u r t l c j o o 9 h k z u 5 3 h q k 2 t m l 7 o 0 1 o k p b i s p g r b j 3 u t g a t 6 t p l o f z x a / 4 c a j 0 e o a l a j u a g / n u f k p i k k i 3 h 2 j h e 4 c c 2 z l c 2 j h a 6 k / x m f j c 2 t 7 p 7 h b 2 t s / o d w q h 5 + 0 j u o 1 o s 2 i u n l d c b v k m a q t y y y n 3 u r t l c j o o 9 h k z u 5 3 h q k 2 t m l 7 o 0 1 o k p b i s p g r b j 3 u t g a t 6 t p l o f z x a / 4 c a j 0 e o a l a j u a g / n u f k p i k k i 3 h 2 j h e 4 c c 2 z l c 2 j h a 6 k / x m f j c 2 t 7 p 7 h b 2 t s / o d w q h 5 + 0 j u o 1 o s 2 i u n l d c b v k m a q t y y y n 3 u r t l c j o o 9 h k z u 5 3 h q k 2 t m l 7 o 0 1 o k p b i s p g r b j 3 u t g a t 6 t p l o f z x a / 4 c a j 0 e o a l a j u a g / n u f k p i k k i 3 h 2 j h e 4 c c 2 z l c 2 j h a 6 k / x m f j c 2 t 7 p 7 h b 2 t s / o d w q h 5 + 0 j u o 1 o s 2 i u n l d c b v k m a q t y y y n 3 u r t l c j o o 9 h k z u 5 3 h q k 2 t m l 7 o 0 1 o k p b i s p g r b j 3 u t g a t 6 t p l o f z x a / 4 c a j 0 e o a l a j u a g / n u f k p i k k i 3 h 2 j h e 4 c c 2 z l c 2 j h a 6 k / figure 1 : the calibrated-knowledge distillation training procedure over multiple generations. ## experimental settings in this section, we first describe the stance detection datasets used for evaluation and introduce the evaluation metrics. then, we describe several stance algorithm 1: calibrated-knowledge distillation with dynamic temp. scaling require :train set d @xmath2, y i )} n @xmath3, y i ) 2 do temperature scaling of teacher predictions with updated τ of temperature scaling that minimizes the ece of teacher predictions on the d val 3 train a student model by minimizing the sum of cross-entropy loss and the kl-divergence loss of d tr l @xmath4: use the student as a new teacher and go back to step 2. ## results in this section, we first compare the proposed ckd with strong baselines of stance detection. then we present an ablation study to show that dynamically updating the temperature helps improve the performance in generations. in addition, we compare the performance of different bin sizes of our proposed ckd on stance detection datasets. next, we compare our ckd with the best baseline akd and visualize the relationship between the confidence and the evaluation metric (f micro ) in the form of reliability diagrams. at last, we discuss the performance of our ckd and other kd baselines in ece on stance detection datasets. ## conclusion in this paper, we study the problem of knowledge distillation in generations on stance detection. we show that knowledge distillation in multiple generations can be beneficial to stance detection. moreover, based on the existing works, we provide a new perspective that a well-calibrated teacher can benefit the student by providing smoother training signals and make it possible for the student to learn from inter-class similarity. our proposed ckd produces calibrated teacher predictions by dynamically updating the temperature used for scaling in each generation. experimental results show that our proposed method consistently outperforms the best-performing baseline on different stance detection datasets. future work includes extending our proposed method to a broader range of nlp tasks such as emotion classification.