Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model (2022)

Abstract: In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a teacher is considered a subject that holds inter-class relations which send a meaningful supervision to a student; hence, much effort has been put to find such knowledge to be distilled. In this paper, we explore a question that has been given little attention: “when to distill such knowledge.” The question is answered in our work with the concept of model calibration; we view a teacher model not only as a source of knowledge but also as a gauge to detect miscalibration of a student. This simple and yet novel view leads to a hard gate knowledge distillation scheme that switches between learning from a teacher model and training data. We verify the gating mechanism in the context of natural language generation at both the token-level and the sentence-level. Empirical comparisons with strong baselines show that hard gate knowledge distillation not only improves model generalization, but also significantly lowers model calibration error.

## introduction in recent years, the deep learning community has achieved marked performance gains across a variety of tasks @xcite . in the meantime, some deep learning models have become excessively large, limiting their applicability in some scenarios. to cope with the issue, @xcite proposed knowledge distillation (kd), in which knowledge of a large network, called a teacher network, is transferred to a relatively small model, called a student model. the benefits of kd have been widely witnessed across multiple domains @xcite this work was done while dongkyu was an intern at nvidia † corresponding author @xcite . recently, it has been observed that kd can be used in both reducing model size and improving model generalization @xcite . @xcite argue that a distribution, defined by a teacher, holds inter-class relations, commonly referred to as the dark knowledge, and that such distribution brings a meaningful supervision to a student. therefore, a large body of research in kd has viewed a teacher as a source of knowledge and has focused on finding a meaningful knowledge to be transferred @xcite @xcite @xcite . in this work, we focus on when to distill knowledge of a teacher. this is a central question to ask, as a model can benefit from the adaptive control of supervision between ground truth and a teacher; when a model is trained to increase the predictive score of a prediction, a one-hot encoded supervision, without incorporating teacher model, sends a direct signal in increasing the score @xcite . in another case, when a model is trained to learn knowledge of a teacher, a teacher's output without fusing a ground truth sends more direct signal in minimizing the knowledge gap between the student and the teacher. however, the question of "when" has not been answered. for this reason, previous works choose to learn from both of the supervisions. we give an answer to the question from the perspective of model calibration. model calibration refers to how well a predicted probability of a model reflects the true accuracy. therefore, a well-calibrated predictive score represents the likelihood of correctness of a prediction @xcite . in this light, such score can be viewed as a gauge to detect a miscalibration of a student in training; when a student makes a prediction with a probability mass that is higher than the expected accuracy of the prediction (overconfidence), a student model is trained with only supervision from a teacher. in the case of underconfidence, a student is trained with only supervision from ground-truth. switching supervision is supported by two widely accepted ideas: 1) the close link between miscalibration and overfitting, and 2) the regularization effect of kd. @xcite empirically find that a model overfits to negative log likelihood (nll) training, leading to miscalibration, and @xcite further support the claim. therefore, we utilize the regularization effect held in kd training @xcite . aside from the inter-class relations held in knowledge, recent findings suggest that kd is a form of adaptive regularization @xcite , where a teacher enforces a student to distribute probability mass on output space more evenly. taking all these factors into account, we present a simple, yet novel kd method, called hard gate knowledge distillation (hkd). given a calibrated teacher model, the teacher gates supervisions between knowledge and observation for each instance/time step, selecting which objective the student should be optimized to. we introduce two levels of hard gates: the token-level and the sentencelevel which are instance-specific hard gates computed on the fly during forward propagation. our work validates the proposed idea on a task in the natural language generation (nlg) domain, as there is an inseparable relation between the quality of an output and model calibration @xcite . the contributions of the proposed method are as follows: ## approach in this section, we first discuss a new interpretation of a teacher under kd training and introduce methods that switch supervision between knowledge and observations with an instance-specific hard gate. ## conclusion in this study, we present hard gate knowledge distillation, a mechanism that switches supervision between knowledge and ground truth at either the sequence-level or the token-level. this originates from the novel view and role of a teacher in kd. the proposed method is simple yet effective in improving model generalization and calibration, achieving superior performances compared to those of the strong baselines.