Knowledge Distillation with Noisy Labels for Natural Language Understanding (2021)

Abstract: Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications. However, one neglected area of research is the impact of noisy (corrupted) labels on KD. We present, to the best of our knowledge, the first study on KD with noisy labels in Natural Language Understanding (NLU). We document the scope of the problem and present two methods to mitigate the impact of label noise. Experiments on the GLUE benchmark show that our methods are effective even under high noise levels. Nevertheless, our results indicate that more research is necessary to cope with label noise under the KD.

## introduction large-scale pre-trained language models @xcite @xcite have shown remarkable abilities to match and even surpass human performances on many natural languages understanding (nlu) tasks @xcite @xcite . however, the deployment of these models in dynamic commercial environments come with challenges, including: large model size, and low training data quality. knowledge distillation @xcite is a compression technique of choice that has proven to be effective to fit a cumbersome nlu model on edge devices @xcite @xcite . meanwhile, numerous methods were developed to combat noisy (corrupted) labels, mainly for computer vision @xcite @xcite and more recently for nlu (ardehaly and culotta, ˚this work has been done while shivendra bhardwaj was at huawei. despite its success, kd has mostly been studied with the availability of massive amount of high quality labeled data. in practice, however, it is costly and impractical to produce such data (ghaddar and langlais, 2019), and noisy labels are commonly encountered. in this paper, we consider the problem of kd when noisy labels are provided for training the main (teacher) and compressed (student) models. to our knowledge, this is the first time kd is studied under a noisy setting in nlu. we conduct experiments on 7 tasks from the glue benchmark @xcite and observe a drastic drop of performance of distilled models when we increase the level of noise. in response, we propose 2 distillation training methods, namely co-distill and label refining, that are specifically designed to handle noise. experiments show that our methods lead to improvements over fair baselines, and that it combination also performs the best. yet, our analysis indicates that the problem is far from solved, and that there is much room for research. ## related work the vanilla kd framework @xcite consists in training a small student model to mimic the output of a large teacher model. recent years have seen a wide array of methods that leverage intermediate layer matching @xcite @xcite , data augmentation @xcite @xcite , or adversarial training @xcite @xcite in order to reduce the teacher-student performance gap. instead, our proposed methods are designed to handle label noise during kd. nevertheless, they can be easily fused with the aforementioned methods to further boost performance. label noise (corruption) is a common problem in real-world datasets, and it has been well studied in the literature @xcite @xcite . methods to combat noise build on the idea that samples with small training loss at early epochs are more likely to be clean @xcite . in co-teaching @xcite , two networks of different capacity teach each other to reject wrong labels. at each forward pass, each network keeps only small-loss samples and sends them to its peer network for updating the parameters. the main idea is that the error flow can be reduced, as networks of different learning abilities have different views on the data. self-distillation was proposed by @xcite , where the model is trained to mimic its own prediction from the previous training epoch. the goal is to prevent the model from memorizing wrong labels, as the model has less tendency to fit noise at early epochs. in addition, @xcite showed improvements when distillation at early epochs is used to refine noisy labels. another line of works is the learning to weight approach @xcite @xcite ) that aims to learn per-sample loss weights in order to discount noisy samples. the proposed methods use an auxiliary meta-learner to re-weight training samples of the main model. however, all aforementioned works mainly focus on computer vision. recently, @xcite utilize a noise detection model to cluster, then score the training samples for text classification in an attempt to guide the main model to focus on samples that are most likely to be correct. ## methodology we first introduce our method, co-distill (cd), which jointly trains the teacher and the student. next, we incorporate label refinement (lr) which is motivated by the algorithms of @xcite , @xcite and @xcite for noise mitigation in regular (no kd) training framework. ## conclusion we present the first study on knowledge distillation when learning from noisy labels in nlu, and show that the problem is extremely challenging. future work involves conducting a comparative study on the robustness of state-of-the-art kd techniques against noisy labels, and merging them within our methods. we hope that our study will encourage future research on kd in the noisy label setting, a genuine setting in real world applications.