Calibrating Student Models for Emotion-related Tasks (2022)

Abstract: Knowledge Distillation (KD) is an effective method to transfer knowledge from one network (a.k.a. teacher) to another (a.k.a. student). In this paper, we study KD on the emotion-related tasks from a new perspective: calibration. We further explore the impact of the mixup data augmentation technique on the distillation objective and propose to use a simple yet effective mixup method informed by training dynamics for calibrating the student models. Underpinned by the regularization impact of the mixup process by providing better training signals to the student models using training dynamics, our proposed mixup strategy gradually enhances the student model’s calibration while effectively improving its performance. We evaluate the calibration of pre-trained language models through knowledge distillation over three tasks of emotion detection, sentiment analysis, and empathy detection. By conducting extensive experiments on different datasets, with both in-domain and out-of-domain test sets, we demonstrate that student models distilled from teacher models trained using our proposed mixup method obtained the lowest Expected Calibration Errors (ECEs) and best performance on both in-domain and out-of-domain test sets.

## introduction it has been shown that transferring knowledge from a teacher model with desired high performance to a student model, through knowledge distillation, can lead to better performance of student models distilled @xcite . however, little is known about the impact of the distillation process on the calibration of the student model. evaluating the uncertainty of a model's predictions is crucial, specifically in applications where the cost of an error is high. for instance, in a computer-assisted therapy session, an accurate and calibrated emotion or empathy detection model can inform the doctor when a model's predictions should (or should not) be trusted, which is helpful for them in deciding the preferred treatment for patients. in this work, we aim to shed light on the impact of knowledge distillation on the calibration of the student models on emotion-related tasks. calibration measures the discrepancy between the correctness of the prediction (i.e., accuracy) and the (empirical) probability that a model assigns to a prediction (i.e., confidence). a well-calibrated model knows how often it is correct or wrong; predicting an event with p confidence shall empirically be true p of the time @xcite . recently, a large body of work has investigated why neural networks have become miscalibrated @xcite @xcite @xcite @xcite . more recent attention, however, has focused on methods to alleviate this problem. specifically on natural language processing tasks, @xcite proposed a simple extension of platt scaling @xcite that softens the softmax by a learned scalar parameter which can effectively calibrate probabilistic models. @xcite ; @xcite ; @xcite also showed that regularization techniques such as label smoothing could prevent over-confident predictions and result in better model calibration. along these lines, in this paper, we empirically examine the impact of the mixed sample data augmentation technique, mixup @xcite , on the performance and calibration of the student models in a distillation setup and propose a simple yet effective mixup strategy to attain more accurate and better-calibrated models. mixup @xcite is a popular data augmentation and regularization technique that generates a weighted combination of random input pairs from the training data. it has been empirically shown that mixup can hone the accuracy and calibration of the pre-diction emanating from the desirable regularization effects it induces @xcite @xcite . by employing mixup, the goal is to provide the teacher models with more useful information and impart the student models with better supervision signals during the distillation of the emotion-related models. in addition, mixup may introduce some noise to the training data (as real-world emotion-related datasets), which enables us to gain additional information about relatively similar data. this, in turn, makes teacher models more robust, helping the student models to be more accurate and produce better-calibrated predictions. while mixup is making significant inroads in a broad range of tasks ranging from computer vision @xcite @xcite to natural language processing @xcite @xcite @xcite , there has hitherto been a limited number of works focusing on its effectiveness on model calibration specifically in nlp @xcite . with that caveats, what is not yet studied is using mixup for calibrating the student model predictions on the knowledge distillation setting; that is what this paper focuses on. in this paper, we study, for the first time to our knowledge, the impact of the mixup data augmentation technique on the distillation objective and propose a simple yet effective mixup strategy that is informed by training dynamics @xcite for calibrating the student models. to this end, we first characterize data instances based on their contributions to the model's learning, which yields distinct regions in the data, presenting easy-to-learn, ambiguous, or hard-to-learn instances. then, we generate mixup samples by interpolating easy-to-learn with ambiguous samples as a regularization technique to promote generalization to both in-domain (id) and out-of-domain (ood) test sets and improve the student model calibration. while ambiguous/hard-to-learn instances are intuitively the most challenging yet informative for learning, easy-to-learn instances are essential for convergence @xcite . therefore, interpolating samples from different regions (e.g., easy-to-learn with ambiguous) in the teacher model can potentially result in a better-calibrated student model with improved id and ood perfor-mance. to contextualize examples in our datasets based on training dynamics, we utilize data maps @xcite . data maps is a modelbased tool that characterizes datasets based on the model's behavior on each of the instances. by leveraging training dynamics, data maps estimates two measures, i.e., confidence and variability, the mean and standard deviation of the ground-truth probabilities, predicted for each instance across training epochs. we further experimentally explore the effect of popular regularization techniques like temperature scaling @xcite and label smoothing @xcite along with our informed mixup on the calibration of the student models in a teacher-student training set-up. we carried out extensive experiments to evaluate the proposed informed mixup data augmentation technique by creating teacher networks on two pre-trained models, bert @xcite , and roberta @xcite . student networks will then be distilled and evaluated against several data sets on three different text classification tasks, including emotion detection @xcite , sentiment analysis @xcite , and empathy detection @xcite . our contributions are thus summarized as follows: mixup: mixup @xcite was first proposed to improve the generalization of deep neural networks in computer vision. since then, many studies have explored mixup in natural language processing tasks @xcite @xcite @xcite . @xcite proposed a data-agnostic distillation framework that leverages mixup to confer the student model with better generalization ability. @xcite examined bert calibration using mixup by generating augmented samples based on a cosine distance of extracted features. park and caragea (2022) also improved pre-trained language models calibration by leveraging area under the margins @xcite along with saliency maps @xcite to generate mixup samples. in contrast to these works, we study the impact of mixup data augmentation technique on the distillation objective. calibration: calibration and uncertainty of the models have been investigated on several natural language processing tasks, including question answering @xcite , neural machine translation @xcite @xcite , language understanding (desai and durrett, 2020), estimating proportions from annotations @xcite , and coreference resolution @xcite . @xcite provided a benchmark of models on image and text classification tasks and explored the influence of distributional shift on accuracy and calibration. focusing on the pre-trained models, desai and durrett (2020) examined calibration over three tasks of paraphrase detection, natural language inference, and commonsense reasoning. unlike these works, we study the calibration of emotion-related tasks through knowledge distillation and propose a mixup strategy to enhance the performance and calibration of the transformer-based student models. ## conclusion in this paper, we showed that the dark knowledge of a pre-trained language teacher model could act as a regularization process, helping to calibrate the student model's confidence in its predictions. we further proposed an informed mixup process and demonstrated that using training dynamics to guide the interpolation process in the mixup data augmentation on a teacher model can effectively improve the calibration of the student model in a distillation setting. we showed that student models distilled from such teacher models trained using our proposed mixup method not only achieved the best performance but also obtained the lowest expected calibration errors (eces) on both in-domain and out-of-domain test sets on emotion-related tasks.