Knowledge Distillation \approx Label Smoothing: Fact or Fallacy? (2023)

Abstract: Originally proposed as a method for knowledge transfer from one model to another, some recent studies have suggested that knowledge distillation (KD) is in fact a form of regularization. Perhaps the strongest argument of all for this new perspective comes from its apparent similarities with label smoothing (LS). Here we re-examine this stated equivalence between the two methods by comparing the predictive confidences of the models they train. Experiments on four text classification tasks involving models of different sizes show that: (a) In most settings, KD and LS drive model confidence in completely opposite directions, and (b) In KD, the student inherits not only its knowledge but also its confidence from the teacher, reinforcing the classical knowledge transfer view.

## introduction knowledge distillation (kd) was originally proposed as a mechanism for a small and lightweight student model to learn to perform a task, e.g., classification, from a higher-capacity teacher model @xcite . in recent years, however, this view of kd as a knowledge transfer process has come into question, with a number of studies finding that it has features characteristic of a regularizer @xcite @xcite . perhaps the most direct argument for this new position comes from @xcite , who show that kd has interesting similarities with label smoothing (ls) @xcite . one of their main conclusions, that "kd is a type of learned ls regularization," has subsequently been reiterated by many @xcite @xcite and remains an important part of our understanding of kd to this day. this is a rather curious suggestion, however, given the starkly different purposes for which the two methods were originally developed: ls was designed "for encouraging the model to be less confident" @xcite by imposing a uniform prior on the target distribution; the resulting new distribution is intended to simply prevent overfitting without revealing any new details about the task. kd, on the other hand, was designed to explicitly teach the student better inter-class relationships for classification, so that "a bmw, for example, may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot" @xcite . such different motivations of the two methods prompt us to re-examine the claim of their equivalence in this paper, where we find the underlying argument to be essentially flawed. with a growing dependence in key areas like nlp on increasingly large models, kd and other model compression methods are becoming more relevant than ever before @xcite @xcite . it is therefore crucial that we also interpret such methods correctly to help steer their future research in the right direction, which is a primary goal of this work. kd and ls do indeed share the common property that they both substitute one-hot training labels with soft target distributions. the key question for us, however, is whether the soft teacher estimates in kd simply regularize training like the smoothed labels in ls do, or if they present a detailed and finergrained view of the task, as originally intended. @xcite argue in favor of the former based on observations such as (a) similarities in the cost functions of kd and ls, and (b) strong generalization of distilled image classifiers independent of the level of expertise of their teachers. here we first analyze their arguments and point out why they may be inadequate for a "kd ≈ ls" assertion ( §2, 3). then we present empirical evidence on four text classification tasks from glue @xcite that kd lacks the defining prop-erty of ls in most settings. concretely, relative to standard empirical risk minimization (erm) over one-hot expert labels, ls by design increases the uncertainty (entropy) in the trained model's posterior over the associated categories. in other words, it always reduces the model's confidence in its predictions, as expected. the effect of kd on the student's posterior, on the other hand, is a function of the relative capacities of the teacher and the student: in the more common setting where a larger teacher is distilled into a smaller student and also in self-distillation, kd almost always reduces entropy over erm, while also improving accuracy. it is only when the student is larger than its teacher that we see a small increase in entropy-still much less than in ls-along with a smaller gain in accuracy. these results clearly show that unlike ls, kd does not operate by merely making the trained model less confident in its predictions. the following is a summary of our contributions: kd, on the other hand, is defined directly as minimizing the following cost function: a second argument stems from their observation on a number of image classification datasets that a "teacher-free kd" method, which employs a smoothed target distribution as a virtual teacher, has similar performance as standard kd. finally, some of their empirical observations, where kd improves student accuracy even when the teacher is a weak or poorly trained model, provide support for the more general view of kd as a form of regularization. investigation of this general question can be found in others' work as well, e.g., in the context of multi-generational self-kd (zhang and r. sabuncu, 2020; @xcite and early stopping @xcite . finally, a related line of work view kd as "instance-specific label smoothing", where the amount of smoothing is not uniform across categories, but is a function of the input approximated using a pre-trained network @xcite . we note that this is a fundamentally different assertion than kd being a form of ls with a uniform prior, since such methods do in fact bring new task-specific knowledge into the training process, with the pre-trained network essentially serving as a teacher model. ## a closer look at the arguments as already stated, the focus of this paper is on the specific claim by @xcite that kd is a form of ls with a uniform prior @xcite , a primary argument for which is that their cost functions in eqs. 1 and 2 have a similar form. a careful examination of the two functions, however, reveals a key difference. to elaborate, let h(.) represent the entropy of a probability distribution; it then follows that ∀@xmath0, where ∆ n is a probability simplex, u is the uniform distribution and p a different distribution on ∆ n . accordingly, minimizing d kl (u, p) in eq. 1 unconditionally increases h(p), training higher-entropy models than erm, a fact that we also confirm empirically in §4. the same is not true for d kl (p t , p) in eq. 2, however, since p t is dependent on the specific selection of the teacher model t, and h(p t ) can take on any value between 0 and h(u). crucially, this means that h(p t ) can be both higher or lower than h(p) if the two respective models were trained separately to minimize l erm . unlike in ls, it therefore cannot be guaranteed in kd that minimizing d kl (p t , p) would increase h(p). in §4, we show how the selection of the teacher model affects the predictive uncertainty of the student, for different combinations of their sizes. based on the same results, we also comment on the teacher-free kd framework of @xcite . ## experimental results setup. we use four text classification tasks from glue @xcite in our experiments: • mrpc (dolan and brockett, 2005) asks for a yes/no decision on if two given sentences are paraphrases of each other. to examine the effect of kd in depth, like @xcite , we run experiments under three different conditions: standard kd distills from a finetuned bert-base @xcite (bert henceforth) classifier into a distilbert-base @xcite (distilbert henceforth) student; self-kd distills from a distilbert teacher into a distilbert student; reverse kd distills from a distilbert teacher into a bert student. in all cases, the teacher is trained using erm with a cross-entropy loss. for every model evaluated under each condition, we perform an exhaustive search over the hyperparameter grid of figure 1 illustrates how each training method regulates model uncertainty at real time during training. both kd variants cause a rapid drop in entropy early in training, and an entropy lower than that of both erm and ls is maintained throughout. the comparisons are also very similar between the training and the test set, indicating that the smoothing effects (or lack thereof) of each of these methods generalize well from training to inference. 0 20 40 60 80 100 % training 0.0 0.1 0.2 0.3 0.4 0.5 0.6 entropy erm ls reverse kd (a) mrpc 0 20 40 60 80 100 % training 0.1 0.2 0.3 0.4 0.5 0.6 0.7 entropy erm ls reverse kd (b) qnli 0 20 40 60 80 100 % training 0.0 0.1 0.2 0.3 0.4 0.5 0.6 entropy erm ls reverse kd (c) sst-2 0 20 40 60 80 100 % training 0.2 0.4 0.6 0.8 1.0 entropy erm ls reverse kd (d) mnli figure 2: change in posterior entropy of different bert classifiers as training progresses. in next we turn our attention to reverse kd from distilbert into bert. in the above results have key implications for the general question concerning the true nature of kd, namely, is it knowledge transfer or regularization? as a student learns to mimic its teacher in kd, we see in the above results that it also inherits its confidence-an integral aspect of the data-driven knowledge of both models. this is clearly unlike methods such as ls or teacher-free kd in (yuan et al., 2020), which unconditionally reduce confidence in a data-agnostic way to limit overfitting. with a weaker teacher model, reverse kd yields a smaller overall gain in accuracy than before over erm and ls (see ## conclusion knowledge distillation is a powerful framework for training lightweight and fast yet high-accuracy models. while still a topic of active interest, any misinterpretation, including false equivalences with other methods, can hinder progress for the study of this promising approach going forward. in this paper, we present evidence against its interpretation as label smoothing, and more generally, regularization. we hope that our work will inspire future efforts to understand distillation better and further improve its utility.