Model Calibration for Emotion Detection (2025)

Abstract: In this paper, we propose a unified approach to model calibration for emotion detection that exploits the complementary strengths of knowledge distillation and the MixUp data augmentation technique to enhance the trustworthiness of emotion detection models. Specifically, we use a MixUp method informed by training dynamics that generates augmented data by interpolating easy-to-learn with ambiguous samples based on their similarity and dissimilarity provided by saliency maps. We use this MixUp method to calibrate the teacher model in the first generation of the knowledge distillation process. To further calibrate the teacher models in each generation, we employ dynamic temperature scaling to update the temperature used for scaling the teacher predictions. We find that calibrating the teachers with our method also improves the calibration of the student models. We test our proposed method both in-distribution (ID) and out-of-distribution (OOD). To obtain better OOD performance, we further fine-tune our models with a simple MixUp method that interpolates a small number of OOD samples with ambiguous ID samples.

## introduction emotion detection in written content @xcite is a text classification task that entails the analysis of textual data in order to discern the emotional state expressed in it. text-based emotion detection has applications in many fields, ranging from customer service @xcite to mental health support @xcite , where it is important not only to correctly predict an emotion but also to know how trustworthy the prediction is. this is where model calibration comes in, which measures the discrepancy between the confidence of the model in a prediction (that is, the probability output assigned to the prediction) and the correctness of that prediction (i.e., accuracy). a well-calibrated model can inform us through its confidence when it is uncertain about a prediction (i.e., its confidence is low) and there is a chance that the prediction is wrong. however, modern neural networks tend to suffer from miscalibration @xcite . recently, @xcite studied the calibration of the pre-trained models bert and roberta across three emotion-related tasks: emotion detection, sentiment analysis, and empathy detection. they obtained better calibrated models both in and out of domain by training a model with mixup @xcite , which has been found to improve model calibration across diverse tasks @xcite , and then applied knowledge distillation @xcite , which involves transferring knowledge from a teacher model to a student model. the specific mixup method they used interpolates the top 33% easy-tolearn samples with the top 33% most ambiguous samples. they also experimented with two popular regularization techniques widely used to reduce miscalibration: temperature scaling @xcite and label smoothing @xcite . in contrast to the above study, which focused on calibrating the student models in a single generation of knowledge distillation, our study aims to calibrate both the teacher and student models in multiple generations using mixup and dynamic temperature scaling. we also used a different mixup method to train the teacher in the first generation. our mixup method uses 100% of the data to generate more informative samples based on training dynamics and saliency maps @xcite . specifically, we first leveraged the training dynamics to separate the training data into two equally sized sets: easy-to-learn and ambiguous samples (which include hard-to-learn samples). we then used mixup to combine samples from the two sets with the most similar and dissimilar samples from the other set according to their saliency signals. our contributions are summarized as follows: ## results the test performance and expected calibration errors of the trained models are summarized in tables 1 (id results) and 2 (ood results). we report the mean values across 5 training runs with the standard deviation shown in subscript. the best results are shown in bold. for the models trained using kd we report the values from the best generation. for the results of all generations see tables 11 and 12 in appendix. main results: firstly, we can see in secondly, we can observe in we can see that our proposed method that combines sim-mixup with csd brings further improvement to the calibration in the id setting, as well as similar or better performance to the baselines. in the ood setting, after finetuning the models on 3% of the ood training data, the performance of our model comes close to or, in the case of the ssec and emotion-stimulus datasets, exceeds the performance of the llms, without a significant increase in the calibration error. ablation study: our approach shows the best id macro/weighted f1 scores on two out of three datasets (goemotions and isear) and the best ece across all three datasets. in the ood setting, our model achieves the best ece on two datasets (dailydialog and ssec) and is only slightly worse than sim-mixup on emotion-stimulus. it also obtains the best ood weighted f1 score on all datasets and the best ood macro f1 score on ssec and emotion-stimulus. thus, combining csd with sim-mixup provides clear gains over both random and ea-mixup variants, in both id and ood settings, confirming the benefit of mixing the most similar and dissimilar examples. this suggests that mixup guided by training dynamics and saliency yields more informative synthetic examples than arbitrary or partially informed pairings. ## error analysis in this section we analyze the emotions that are commonly misclassified in all 5 training runs. note that all mean confidences reported have the standard deviation in subscript. ## conclusion in this work, we proposed a novel calibration method for pre-trained language models that combines mixup with knowledge distillation in order to calibrate teacher models for the emotion detection task, which in turn helps train better students. our method calibrates the first teacher using an informed mixup method that interpolates easy-tolearn with ambiguous samples guided by saliency signals. this way we provide the teacher models with more useful information that is then imparted to the student models during the distillation of the emotion-detection models. during the distillation process, we calibrate the teacher predictions in each generation by dynamically updating the temperature used for scaling. we empirically validated that our method achieves competitive performance and calibrates the pre-trained model roberta on various emotion-detection datasets, both in-domain and out-of-domain.