Original: P SEUDO
Revised: P SEUDO

Original: K NOWLEDGE
Revised: K NOWLEDGE

Original: D
Revised: D

Original: ISTILLATION : T OWARDS
Revised: ISTILLATION : T OWARDS

Original: L EARNING
Revised: L EARNING

Original: O PTIMAL
Revised: O PTIMAL

Original: I NSTANCE - SPECIFIC
Revised: I NSTANCE - SPECIFIC

Original: L ABEL
Revised: L ABEL

Original: S MOOTHING R EGULARIZATION
Revised: S MOOTHING R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training.
Revised: Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training.

Original: Although a complete understanding of KD is elusive, a growing body of work has shown that the success of both KD and label smoothing comes from a similar regularization effect of soft targets.
Revised: Although a complete understanding of KD is elusive, a growing body of work has shown that the success of both KD and label smoothing comes from a similar regularization effect of soft targets.

Original: In this work, we propose an instance-speciﬁc label smoothing technique, Pseudo-KD, which is efﬁciently learnt from the data.
Revised: In this work, we propose an instance-speciﬁc label smoothing technique, Pseudo-KD, which is efﬁciently learnt from the data.

Original: We devise a two-stage optimization problem that leads to a deterministic and interpretable solution for the optimal label smoothing.
Revised: We devise a two-stage optimization problem that leads to a deterministic and interpretable solution for the optimal label smoothing.

Original: We show that Pseudo-KD can be equivalent to an efﬁcient variant of self-distillation techniques, without the need to store the parameters or the output of a trained model.
Revised: We show that Pseudo-KD can be equivalent to an efﬁcient variant of self-distillation techniques, without the need to store the parameters or the output of a trained model.

Original: Finally, we conduct experiments on multiple image classiﬁcation (CIFAR-10 and CIFAR-100) and natural language understanding datasets (the GLUE benchmark) across various neural network architectures and demonstrate that our method is competitive against strong baselines.
Revised: Finally, we conduct experiments on multiple image classiﬁcation (CIFAR-10 and CIFAR-100) and natural language understanding datasets (the GLUE benchmark) across various neural network architectures and demonstrate that our method is competitive against strong baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks, predominantly, pre-trained language models have grown massively in scale over the past few years (Fedus et al., 2021; Brown et al., 2020; Zeng et al., 2021).
Revised: Neural networks, predominantly, pre-trained language models have grown massively in scale over the past few years (Fedus et al., 2021; Brown et al., 2020; Zeng et al., 2021).

Original: Some researchers have gone so far as to term these over-parameterized models as Foundation Models (Bommasani et al., 2021).
Revised: Some researchers have gone so far as to term these over-parameterized models as Foundation Models (Bommasani et al., 2021).

Original: Although larger models trained on more data tend to dominate various benchmarks (Wang et al., 2019b;a; Deng et al., 2009), deploying them on edge devices is a challenge.
Revised: Although larger models trained on more data tend to dominate various benchmarks (Wang et al., 2019b;a; Deng et al., 2009), deploying them on edge devices is a challenge.

Original: Training smaller networks is an option but empirical evidence suggests that training larger models and then compressing them can help achieve a higher accuracy per unit computation (Li et al.).
Revised: Training smaller networks is an option but empirical evidence suggests that training larger models and then compressing them can help achieve a higher accuracy per unit computation (Li et al.).

Original: Knowledge distillation (KD) (Hinton et al., 2015; Bucilu ˇ a et al., 2006) which is a knowledge transfer technique, from a single or an ensemble of neural networks to another, is a widely applied neural model compression technique across domains ranging from computer vision (He et al., 2019; Xu et al., 2020; Jafari et al., 2021) to natural language processing (Jiao et al., 2020; Sanh et al., 2019).
Revised: Knowledge distillation (KD) (Hinton et al., 2015; Bucilu ˇ a et al., 2006) which is a knowledge transfer technique, from a single or an ensemble of neural networks to another, is a widely applied neural model compression technique across domains ranging from computer vision (He et al., 2019; Xu et al., 2020; Jafari et al., 2021) to natural language processing (Jiao et al., 2020; Sanh et al., 2019).

Original: This empirical success is attributed to a few factors such as a) deeper teacher networks learn better representations, b) the teacher is able to transfer dark knowledge (Hinton et al., 2014) in its predictions and c) the regularization effect of the soft labels in the KD loss (Yuan et al., 2020).
Revised: This empirical success is attributed to a few factors such as a) deeper teacher networks learn better representations, b) the teacher is able to transfer dark knowledge (Hinton et al., 2014) in its predictions and c) the regularization effect of the soft labels in the KD loss (Yuan et al., 2020).

Original: Nonetheless, it is unclear how exactly student networks beneﬁt from these soft labels.
Revised: Nonetheless, it is unclear how exactly student networks beneﬁt from these soft labels.

Original: Recent work suggests that with KD, the student is unable to match the teacher on the training distribution but improves generalization on the test set (Stanton et al., 2021).
Revised: Recent work suggests that with KD, the student is unable to match the teacher on the training distribution but improves generalization on the test set (Stanton et al., 2021).

Original: In trying to understand KD, a series of investigations have looked at regularization and have demonstrated that the success of both KD and label smoothing is due to a similar regularization effect of soft targets (Yuan et al., 2020; Zhang & Sabuncu, 2020).
Revised: In trying to understand KD, a series of investigations have looked at regularization and have demonstrated that the success of both KD and label smoothing is due to a similar regularization effect of soft targets (Yuan et al., 2020; Zhang & Sabuncu, 2020).

Original: Label smoothing (LS) (Szegedy et al., 2016) is a popular technique for enhancing the generalization of a wide range of models without incurring additional computational costs.
Revised: Label smoothing (LS) (Szegedy et al., 2016) is a popular technique for enhancing the generalization of a wide range of models without incurring additional computational costs.

Original: It encourages the model to treat each non-target class as equally likely for classiﬁcation by using a uniform distribution to smooth the distribution of the hard labels.
Revised: It encourages the model to treat each non-target class as equally likely for classiﬁcation by using a uniform distribution to smooth the distribution of the hard labels.

Original: Although combining the uniform distribution with the original hard label is beneﬁcial for regularisation, LS does not take into account the true relationships between different label categories like the way KD does.
Revised: Although combining the uniform distribution with the original hard label is beneﬁcial for regularisation, LS does not take into account the true relationships between different label categories like the way KD does.

Original: Zhang & Sabuncu (2020) demonstrate the importance of an instance speciﬁc LS regularization.
Revised: Zhang & Sabuncu (2020) demonstrate the importance of an instance speciﬁc LS regularization.

Original: They demonstrate better performance compared to LS, but use a trained model to infer prior knowledge on the label space and thereby sacriﬁce some of the efﬁciency of LS.
Revised: They demonstrate better performance compared to LS, but use a trained model to infer prior knowledge on the label space and thereby sacriﬁce some of the efﬁciency of LS.

Original: In this work, we investigate whether we can bridge the performance gap between LS and KD, while maintaining the efﬁciency of LS .
Revised: In this work, we investigate whether we can bridge the performance gap between LS and KD, while maintaining the efﬁciency of LS .

Original: Motivated by this, we ﬁrst revisit the conventional LS technique and generalize the uniform LS method to an instance-speciﬁc smoothing regularization framework.
Revised: Motivated by this, we ﬁrst revisit the conventional LS technique and generalize the uniform LS method to an instance-speciﬁc smoothing regularization framework.

Original: Within this framework, we demonstrate that both LS and KD can be interpreted as instances of a smoothing distribution.
Revised: Within this framework, we demonstrate that both LS and KD can be interpreted as instances of a smoothing distribution.

Original: Next, we propose to learn the optimal smoothing function along with the model training, by devising a two-stage optimization problem.
Revised: Next, we propose to learn the optimal smoothing function along with the model training, by devising a two-stage optimization problem.

Original: We solve the ﬁrst-stage problem by giving a deterministic and interpretable solution and apply gradient based optimization to solve the second stage.
Revised: We solve the ﬁrst-stage problem by giving a deterministic and interpretable solution and apply gradient based optimization to solve the second stage.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • Our method, Pseudo-KD, provides a general framework, that improves upon conventional LS without an additional training cost.
Revised: • Our method, Pseudo-KD, provides a general framework, that improves upon conventional LS without an additional training cost.

Original: • We provide a plausible explanation for the relationship between LS and KD.
Revised: • We justify our framework by demonstrating the new formulation uniﬁes the conventional LS and KD objectives, which conﬁrm their relationship from a different perspective.

Original: • We demonstrate that Pseudo-KD can be equivalent to self-distillation without the need for training an additional model.
Revised: • Based on our framework, we further propose to learn the smoothing distribution along with training by formulating it as a bi-level optimization problem and we derive a deterministic and interpretable solution.

Original: • We conducted extensive experiments on image classiﬁcation (CIFAR10 and CIFAR100) and natural language understanding (the GLUE benchmark) and show that our method outperforms label smoothing, and performs on par with KD while being signiﬁcantly faster in training (approximately up to 3x for some settings).
Revised: • We conducted extensive experiments on image classiﬁcation (CIFAR10 and CIFAR100) and natural language understanding (the GLUE benchmark) and show that our method outperforms label smoothing, and performs on par with KD while being signiﬁcantly faster in training (approximately up to 3x for some settings).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We will provide a brief overview of KD and LS and teacher-free methods which reconcile these two.
Revised: We will provide a brief overview of KD and LS and teacher-free methods which reconcile these two.

Original: Knowledge Distillation.
Revised: Knowledge Distillation.

Original: KD came into prominence after the work of Hinton et al.
Revised: KD came into prominence after the work of Hinton et al.

Original: Given access to a trained teacher model, assume that we want to train a student.
Revised: Given access to a trained teacher model, assume that we want to train a student.

Original: Denote by P T ( x ) and p θ ( x ) the teacher’s and student’s predictions respectively.
Revised: Denote by P T ( x ) and p θ ( x ) the teacher’s and student’s predictions respectively.

Original: For a classiﬁcation problem, the total KD loss is deﬁned as:
Revised: For a classiﬁcation problem, the total KD loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where q ( x ) is the one-hot label, H ( q ( x ) , p ( x )) is the cross-entropy loss between the student’s output and the labels, D KL is the KL divergence and α is the scaling parameter.
Revised: where q ( x ) is the one-hot label, H ( q ( x ) , p ( x )) is the cross-entropy loss between the student’s output and the labels, D KL is the KL divergence and α is the scaling parameter.

Original: Note that we assume a temperature of 1 and omit it without loss of generality.
Revised: Note that we assume a temperature of 1 and omit it without loss of generality.

Original: KD training is equivalent to training with a smoothed label
Revised: KD training is equivalent to training with a smoothed label

Original: ˆ P ( x ) such that:
Revised: ˆ P ( x ) such that:

Original: [Equation]
Revised: [Equation]

Original: Label Smoothing We observe that the loss in equation 2 is similar in form to the LS loss (Yuan et al., 2020).
Revised: Label Smoothing We observe that the loss in equation 2 is similar in form to the LS loss (Yuan et al., 2020).

Original: In LS the KL divergence is between the student output and a uniform distribution
Revised: In LS the KL divergence is between the student output and a uniform distribution

Original: U ( k ) =
Revised: U ( k ) =

Original: 1 K where K is the number of classes.
Revised: 1 K where K is the number of classes.

Original: Therefore we can deﬁne label smoothened label as:
Revised: Therefore we can deﬁne label smoothened label as:

Original: [Equation]
Revised: [Equation]

Original: Both LS and KD incorporate training a model with a smoothened label.
Revised: Both LS and KD incorporate training a model with a smoothened label.

Original: KD tends to perform better as it leads to a more accurate student (M¨uller et al., 2019;
Revised: KD tends to perform better as it leads to a more accurate student (M¨uller et al., 2019;

Original: Shen et al., 2021).
Revised: Shen et al., 2021).

Original: However, LS is more efﬁcient since it does not need a pre-trained teacher network.
Revised: However, LS is more efﬁcient since it does not need a pre-trained teacher network.

Original: We note from equation 3 that a higher α can lead to a smoother label distribution and a lower α to a peakier label.
Revised: We note from equation 3 that a higher α can lead to a smoother label distribution and a lower α to a peakier label.

Original: Yuan et al.
Revised: Yuan et al.

Original: (2020) exploited this to propose a better LS loss to bridge the performance gap between KD and LS.
Revised: (2020) exploited this to propose a better LS loss to bridge the performance gap between KD and LS.

Original: Chen et al.
Revised: Chen et al.

Original: (2021) note that training on high entropy label distributions can also lead to better robustness under adversarial attacks. 
Revised: (2021) note that training on high entropy label distributions can also lead to better robustness under adversarial attacks. [Figure]

Original: Self Distillation We can take advantage of distillation without necessarily using a larger teacher.
Revised: Self Distillation We can take advantage of distillation without necessarily using a cumbersome teacher.

Original: A student can improve generalization by learning for the output of a trained, identically parameterized[Figure] model Furlanello et al.
Revised: A student can improve generalization by learning for the output of a trained, identically parameterized model Furlanello et al.

Original: ; Yuan et al.
Revised: ; Yuan et al.

Revised: Zhang et al.

Original: If we denote the output of  the trained model as ¯ p ( x ) then the label is smoothened for self-distillation as follows:
Revised: (2018) propose to train an ensemble of students without a static teacher, which beneﬁts from learning and teaching among different students collaboratively. Another self-boosting direction is to apply regularization on the intermediate layers instead of predictive distributions. During the training stage, Zhang et al. (2019) introduce auxiliary classiﬁers on top of intermediate layers and penalize the difference between the predictions of these classiﬁers and the penultimate layer.

Original: [Equation]
Revised: Along this direction, Liu et al. (2020) further propose a top-down feature fusion process to handle the capacity gap between the intermediate layers with different depths. However, all these methods require extra parameterized modules for training which are abandoned at inference. In contrast, some recent works focus on boosting the performance with explicitly regularized soft targets.

Original: Zhang & Sabuncu (2020) interpret student-teacher training as an amortized maximum a posteriori estimation and derive an equivalence between self-distillation and instance-speciﬁc label smoothing.
Revised: Zhang & Sabuncu (2020) interpret student-teacher training as an amortized maximum a posteriori estimation and derive an equivalence between self-distillation and instance-speciﬁc label smoothing.

Original: This analysis helped them to devise a regularization scheme, Beta Smoothing, which performs comparably to KD.
Revised: This analysis helped them to devise a regularization scheme, Beta Smoothing.

Original: However, they still use an extra model to infer a prior distribution on their smoothing technique.
Revised: However, they still use an extra model to infer a prior distribution on their smoothing technique during the training.

Revised: Yun et al.

Original: M ETHODOLOGY
Revised: (2020) introduce an additional regularization to penalize the predictive output between different samples of the same class. Ding et al. (2021) propose to capture the relation of classes by introducing decoupled labels table which increases O ( K × K ) space complexity. The concurrent work (Kim et al., 2021) utilizes the model trained at i -th epoch as the teacher to regularize the training at ( i +1) -th epoch along with annealing techniques.

Revised: The difference emerges as it still requires a separated model as teacher, or saving all soft labels generated at i -th epoch for the next epoch training which introduces O ( N × K ) space complexity.

Revised: Our work has a different basis: we develop Pseudo-KD motivated by a principled starting point: to generalize the smoothing distribution to a general form with neither time nor space complexity increases during the training and inference.

Revised:  M ETHODOLOGY

Original: In this section, we derive our Pseudo-KD framework from the two-stage optimization perspective.
Revised: In this section, we derive our Pseudo-KD framework from the two-stage optimization perspective.

Original: First, we revisit the conventional uniform LS method and introduce a closed-form solution to our optimal instance-speciﬁc label smoothing.
Revised: First, we revisit the conventional uniform LS method and introduce a closed-form solution to our optimal instance-speciﬁc label smoothing.

Original: Then, we describe a teacher-free KD implementation under this formulation.
Revised: Then, we describe a teacher-free KD implementation under this formulation.

Original: 3. R EVISITING L ABEL S MOOTHING
Revised: 3. R EVISITING L ABEL S MOOTHING

Original: Suppose that we have a K -class classiﬁcation task to be learned by a neural network, S θ ( · ) .
Revised: Suppose that we have a K -class classiﬁcation task to be learned by a neural network, S θ ( · ) .

Original: Given a training set of { x i , y i } Ni =1 samples where x i is a data sample and y i is the ground-truth label, the model S θ ( · ) outputs the probability p θ ( j | x i ) of each class j ∈ { 1 , . .
Revised: Given a training set of { x i , y i } N i =1 samples where x i is a data sample and y i is the ground-truth label, the model S θ ( · ) outputs the probability p θ ( j | x i ) of each class j ∈ { 1 , . .

Original: , K } :
Revised: , K } :

Original: [Equation]
Revised: [Equation]

Original: 5) where z i,j = [ S θ ( x i )]
Revised: where z i,j = [ S θ ( x i )]

Original: j is the logit for j -th class of input x i .
Revised: j is the logit for j -th class of input x i .

Original: A general label smoothing can be formally written as:
Revised: A general label smoothing can be formally written as:

Original: [Equation]
Revised: [Equation]

Original:  k is the ground-truth class
Revised: , where k is the ground-truth class

Original: [Equation]
Revised: [Equation]

Original: where ˆ P ls is a smoothed label, P ls is a smoothing distribution, and α is a hyperparameter that determines the amount of smoothing.
Revised: where ˆ P ls is a smoothed label, P ls is a smoothing distribution, and α is a hyperparameter that determines the amount of smoothing.

Original: If α = 0 , we obtain the original one-hot encoded label.
Revised: If α = 0 , we obtain the original one-hot encoded label.

Original: For the original label smoothing method, the probability P ls ( ·| x ) is independent on the sample x and is taken to be the uniform distribution P ls ( j )
Revised: For the original label smoothing method, the probability P ls ( ·| x ) is independent on the sample x and is taken to be the uniform distribution P ls ( j )

Original: =
Revised: =

Original: U ( j ) ,
Revised: U ( j ) ,

Original: However, the training can beneﬁt from instancespeciﬁc regularization.
Revised: However, the training can beneﬁt from instancespeciﬁc regularization.

Original: (Yuan et al., 2020; Zhang & Sabuncu, 2020).
Revised: (Yuan et al., 2020; Zhang & Sabuncu, 2020).

Original: In this work, we consider the general form of LS ˆ P ls ( ·| x i ) which is instance-speciﬁc and not necessarily uniform.
Revised: In this work, we consider the general form of LS ˆ P ls ( ·| x i ) which is instance-speciﬁc and not necessarily uniform.

Original: Let us consider the Cross Entropy (CE) loss with the smoothed labels:
Revised: Let us consider the Cross Entropy (CE) loss with the smoothed labels:

Original: [Equation]
Revised: [Equation]

Original: Note that computing the weighted sum of negative log-likelihood of each probability of label can be viewed as taking expectation of the negative log-likelihood over label space under a certain distribution
Revised: Note that computing the weighted sum of negative log-likelihood of each probability of label can be viewed as taking expectation of the negative log-likelihood over label space under a certain distribution

Original: ˆ P ls .
Revised: ˆ P ls .

Original: We modify this loss by adding a Kullback–Leibler (KL) divergence term
Revised: We modify this loss by adding a Kullback–Leibler (KL) divergence term

Original: D KL ( P ls ( ·| x i ) (cid:107) U ( · )) into Eq. 7 which encourages the sample-wise smoothing distribution P ls ( ·| x i ) to be close to the uniform distribution.
Revised: D KL ( P ls ( ·| x i ) (cid:107) U ( · )) into Eq. 6 which encourages the sample-wise smoothing distribution P ls ( ·| x i ) to be close to the uniform distribution.

Original: [Equation]
Revised: [Equation]

Original: where β is a hyper-parameter.
Revised: where β is a hyper-parameter.

Original: The instance-speciﬁc smoothing P ls ( ·| x i ) can be viewed as the prior knowledge over the label space for a sample x i and the KL divergence term can be understood as a measure of the ‘smoothness’ or ‘indecisiveness’ of each training label.
Revised: The instance-speciﬁc smoothing P ls ( ·| x i ) can be viewed as the prior knowledge over the label space for a sample x i . This KL divergence term can be understood as a measure of the ‘smoothness’ or ‘overconﬁdence’ of each training label.

Revised: We choose a Uniform distribution to constrain P ls in the KL term because we would like P ls contain more information on non-target classes.

Original: It plays a role similar to temperature in KD and controls the sharpness of the distribution P ls .
Revised: It plays a role similar to temperature in KD and controls the sharpness of the distribution P ls .

Original: On the one hand side, in case of KD, it regulates the amount of prior knowledge we inject in the soft label.
Revised: On the one hand side, in case of KD, it regulates the amount of prior knowledge we inject in the soft label.

Original: On the other hand side, it improves the calibration of the trained network by increasing the entropy of soft labels (a phenomenon studied by Zhang & Sabuncu (2020)).
Revised: On the other hand side, it reduce the overconﬁdence of the trained network by increasing the entropy of soft labels (a phenomenon studied by Zhang & Sabuncu (2020)).

Revised: Next we show the post-hoc interpretability of our new formulation.

Original: The following two Remarks discuss the relationship of our objective with LS and KD.
Revised: The following two Remarks discuss the relationship of our objective with LS and KD.

Original: Remark.
Revised: Remark.

Original: When P ls ( ·| x i ) is taken to be the uniform distribution U ( · ) for any x i , the objective in
Revised: When P ls ( ·| x i ) is taken to be the uniform distribution U ( · ) for any x i , the objective in

Original: Eq.
Revised: Eq.

Original: 8 reduces back to the one in Eq.
Revised: 7 reduces back to the one in Eq.

Original: 7 since D KL (cid:0) U ( · ) (cid:107) U ( · ) (cid:1)
Revised: 6 since D KL (cid:0) U ( · ) (cid:107) U ( · ) (cid:1)

Original: = 0 .
Revised: = 0 .

Original: Remark.
Revised: Remark.

Original: Considering the conventional KD with temperature τ = 1 and P T ( ·| x i )  to be  the output probability of a teacher model T ( · ) for any x i , the objectiveof KD can be rewritten as an expectation of negative log-likelihood w.r.t.
Revised: When P ls ( ·| x i ) is taken to be P T ( ·| x i ) which is the output probability of a teacher model T ( · ) for any x i , our objective can be converted to the conventional KD objective with temperature τ = 1 and plus a constant α log K

Original: a transformed distribution ˆ P T plus a KL term between the teacher output and uniform distribution

Original: [Equation]
Revised: [Equation]

Original: where ˆ P T ( j | x i ) =
Revised: where ˆ P T ( j | x i ) =

Original: (1 − α + α · P T ( k | x i )) for j = k
Revised: (1 − α + α · P T ( k | x i )) for j = k

Original: the ground truth, and ˆ P T ( j | x i ) =
Revised: the ground truth, and ˆ P T ( j | x i ) =

Original: α · P T ( j | x i ) for j (cid:54) =
Revised: α · P T ( j | x i ) for j (cid:54) =

Original: k .
Revised: k .

Original: Our objective function Eq.
Revised: Our objective function Eq.

Revised: 7 demonstrates that the both objective of KD and LS can be rewritten as an expectation of negative log-likelihood w.r.t.

Revised: a transformed distribution ˆ P T or

Revised: U ( · ) plus a KL term between the smoothing distribution and uniform distribution.

Revised: This is not only to conﬁrm previous ﬁndings from a different perspective but also justify why our framework is reasonable.

Revised: Our objective gives a uniﬁed formulation for instance-speciﬁc label smoothing in a general form.

Original: 8 bridges the objective of label smoothing and knowledge distillation, since it is easy to convert our objective to KD or LS by replacing the P ls ( ·
Revised: It is easy to bridge the objective of label smoothing and knowledge distillation by converting our objective to KD or LS via replacing the P ls ( ·| x i ) in Eq.

Original: | x i ) in Eq.with P T ( ·| x i ) or U ( · ) , respectively.
Revised: 7 with P T ( · | x i ) or U ( · ) , respectively.

Original: There is an inherent relation among these methods.
Revised: There is an inherent relation among these methods.

Original: KD and LS, in particular, can be interpreted as instances of our method with a ﬁxed smoothing distribution.
Revised: KD and LS, in particular, can be interpreted as instances of our method with a ﬁxed smoothing distribution.

Original: 3. P SEUDO -KD VIA TWO - STAGE O PTIMIZATION
Revised: 3. P SEUDO -KD VIA TWO - STAGE O PTIMIZATION

Original: The choice of the distribution P ls determines the label regularization method.
Revised: The choice of the distribution P ls determines the label regularization method.

Original: As described above, in KD the smoothing distribution is an output of the pre-trained teacher model, and in LS it is a uniform distribution.
Revised: As described above, in KD the smoothing distribution is an output of the pre-trained teacher model, and in LS it is a uniform distribution.

Original: Generally speaking, the optimal smoothing distribution is unknown, and ideally, we would like to learn the optimal P ls .
Revised: Generally speaking, the optimal smoothing distribution is unknown, and ideally, we would like to learn the optimal P ls .

Original: In this regard, we set up the following two-stage optimization problem:
Revised: In this regard, we set up the following two-stage optimization problem:

Original: [Equation]
Revised: [Equation]

Original:  This optimization setting, also called bilevel optimization (Colson et al., 2007), is strongly NP hard (Jeroslow, 1985) so getting an exact solution is difﬁcult.
Revised: subject to [Equation] [Equation] This optimization setting, also called bilevel optimization (Colson et al., 2007), is strongly NP hard (Jeroslow, 1985) so getting an exact solution is difﬁcult.

Original: Luckily, in our case the inner optimization loop has a closed-form solution.
Revised: Luckily, in our case the inner optimization loop has a closed-form solution.

Original: The following Theorem 1 gives the explicit formulation of R θ ( P ∗ ls ) .
Revised: The following Theorem 1 gives the explicit formulation of R θ ( P ∗ ls ) .

Original: In particular, we will show that R θ ( P ls ) is a convex function w.r.t P ls , and we will obtain the optimal solution P ∗ ls = arg min P ls R θ ( P ls ) by using a Lagrangian multiplier.
Revised: In particular, we will show that R θ ( P ls ) is a convex function w.r.t P ls , and we will obtain the optimal solution P ∗ ls = arg min P ls R θ ( P ls ) by using a Lagrangian multiplier.

Original: For the details of this derivation please refer to Appendix A.
Revised: For the details of this derivation please refer to Appendix A.

Original: Theorem 1.
Revised: Theorem 1.

Original: The solution to the inner loop of optimization in Eq.
Revised: The solution to the inner loop of optimization in Eq.

Original: 10 is given by:
Revised: 9 is given by:

Original: [Equation]
Revised: [Equation]

Original:  where p θ ( j | x i ) is the output probability of j -th class of model S θ ( · ) , α is the smoothing coefﬁcient deﬁned in Eq. 6, and β is deﬁned in Eq.
Revised: [Equation] where p θ ( j | x i ) is the output probability of j -th class of model S θ ( · ) , α is the smoothing coefﬁcient deﬁned in Eq. 5, and β is deﬁned in Eq.

Original: As a result, we reduced the two-stage optimization problem in Eq.
Revised: As a result, we reduced the two-stage optimization problem in Eq.

Original: 10 to a regular single-stage minimization:
Revised: 9 to a regular single-stage minimization:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: P ∗ ls ( j | x i ) is given in Theorem 1, and ˆ P ∗ ls ( j | x i ) is deﬁned in Eq.
Revised: P ∗ ls ( j | x i ) is given in Theorem 1, and ˆ P ∗ ls ( j | x i ) is deﬁned in Eq.

Original: Note that the solution P ∗ ls is deterministic and interpretable.
Revised: Note that the solution P ∗ ls is deterministic and interpretable.

Original: Moreover,  the two remarks below demonstrate the relation of our Pseudo-KD with LS and self-distillation methods.
Revised: Moreover, we show the post-hoc interpretability of the solution in the two remarks below by demonstrating the relation of the solution with LS and self-distillation methods.

Original: Remark.
Revised: Remark.

Original: When β is extremely large, P ∗ ls will be close to the Uniform distribution, and our objective function will be equivalent to optimizing the CE loss with a Uniform LS regularizer.
Revised: When β is extremely large, P ∗ ls will be close to the Uniform distribution, and our objective function will be equivalent to optimizing the CE loss with a Uniform LS regularizer.

Original: When β is extremely small, the smoothing distribution collapses to the one-hot vector.
Revised: When β is extremely small, the smoothing distribution collapses to the one-hot vector.

Original: Remark.
Revised: Remark.

Original: There is an intrinsic connection between the P ∗ ls distribution and generating softmax outputs with a temperature factor.
Revised: There is an intrinsic connection between the P ∗ ls distribution and generating softmax outputs with a temperature factor.

Original: Speciﬁcally, when β = α · τ , we could have
Revised: Speciﬁcally, when β = α · τ , we could have

Original: [Equation]
Revised: [Equation]

Original: Since the smoothing distribution in this case becomes the temperature smoothed output of the model, our Pseudo-KD technique can be considered as a generalized version of the self-distillation method 1 .
Revised: Since the smoothing distribution in this case becomes the temperature smoothed output of the model, our Pseudo-KD technique can be considered as a generalized version of the self-distillation method 1 .

Original: To summarize, our method can be expressed as an alternating two-stage process.
Revised: To summarize, our method can be expressed as an alternating two-stage process.

Original: We generate optimal smoothed labels P ∗ ls using Theorem 1 in the ﬁrst stage.
Revised: We generate optimal smoothed labels P ∗ ls using Theorem 1 in the ﬁrst stage.

Original: Then, in the second phase, we ﬁx the
Revised: Then, in the second phase, we ﬁx the

Original: P ∗ ls to compute loss R
Revised: P ∗ ls to compute loss R

Original: θ ( P ∗ ls ) and update the model parameters.
Revised: θ ( P ∗ ls ) and update the model parameters.

Original: We execute our method in an online manner, which eliminates the need for additional memory or storage for the parameters or outputs of the pre-trained model.
Revised: We execute our method in an online manner, which eliminates the need for additional memory or storage for the parameters or outputs of the pre-trained model.

Original: The online training process is shown in Algorithm 1.Input
Revised: The online training process is shown in Algorithm 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the performance of our proposed Pseudo-KD method.
Revised: In this section, we evaluate the performance of our proposed Pseudo-KD method.

Original: Experiments are conducted on both the image classiﬁcation tasks CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) for the ResNet-based Model with of various sizes (parameters) and the General Language
Revised: Experiments are conducted on both the image classiﬁcation tasks CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) for the ResNet-based Model with of various sizes (parameters) and the General Language

Original: Understanding Evaluation (GLUE) tasks (Wang et al., 2019b) with the RoBERTa-based model (Liu et al., 2019).
Revised: Understanding Evaluation (GLUE) tasks (Wang et al., 2019b) with the RoBERTa-based model (Liu et al., 2019).

Original: 4. E XPERIMENTS ON I MAGE C LASSIFICATION
Revised: 4. E XPERIMENTS ON I MAGE C LASSIFICATION

Original: Data.
Revised: Data.

Original: CIFAR is a benchmark dataset for image classiﬁcation (Krizhevsky et al., 2009).
Revised: CIFAR is a benchmark dataset for image classiﬁcation (Krizhevsky et al., 2009).

Original: We use both CIFAR-10 and CIFAR-100, which have images 50,000 training samples and 10,000 test samples divided into 10 and 100 different object classes, respectively.
Revised: We use both CIFAR-10 and CIFAR-100, which have images 50,000 training samples and 10,000 test samples divided into 10 and 100 different object classes, respectively.

Original: The validation set is made up of 10% of the training data.
Revised: The validation set is made up of 10% of the training data.

Original: Experimental settings.
Revised: Experimental settings.

Original: We evaluated our method on different model architectures including MobileNetV2 (Sandler et al., 2018), ShufﬂeNetV2 (Ma et al., 2018), ResNeXt29(8×64d)
Revised: We evaluated our method on different model architectures including MobileNetV2 (Sandler et al., 2018), ShufﬂeNetV2 (Ma et al., 2018), ResNeXt29(8×64d)

Original: (Xie et al., 2017), ResNet18, ResNet50 and ResNet101 (He et al., 2016).
Revised: (Xie et al., 2017), ResNet18, ResNet50 and ResNet101 (He et al., 2016).

Original: We follow standard data augmentation schemes: random crop and horizontal ﬂip to augment the original training images.
Revised: We follow standard data augmentation schemes: random crop and horizontal ﬂip to augment the original training images.

Original: The models are trained for 200 epochs, with a batch size of 128 for MobileNetV2, ShufﬂeNetV2 and ResNet18, and 64 for ResNeXt29 (8×64d), ResNet50 and ResNet101.
Revised: The models are trained for 200 epochs, with a batch size of 128 for MobileNetV2, ShufﬂeNetV2 and ResNet18, and 64 for ResNeXt29 (8×64d), ResNet50 and ResNet101.

Original: For optimization we used stochastic gradient descent with a momentum of 0.9, and weight decay set to 5e-4.
Revised: For optimization we used stochastic gradient descent with a momentum of 0.9, and weight decay set to 5e-4.

Original: The learning rate starts at 0.1 and is then divided by 5 at epochs 60, 120 and 160.
Revised: The learning rate starts at 0.1 and is then divided by 5 at epochs 60, 120 and 160.

Original: All experiments are repeated 5 times with random initialization.
Revised: All experiments are repeated 5 times with random initialization.

Original: For the KD experiments (KD-Shufﬂe and KD-29), we used ShufﬂeNetv2 and
Revised: For the KD experiments (KD-Shufﬂe and KD-29), we used ShufﬂeNetv2 and

Original: ResNeXt29 as the teachers.
Revised: ResNeXt29 as the teachers.

Original: All teacher models are trained from scratch and picked based on their best accuracy.
Revised: All teacher models are trained from scratch and picked based on their best accuracy.

Original: To explore the best hyper-parameters, we conduct a grid search over parameter pools.
Revised: To explore the best hyper-parameters, we conduct a grid search over parameter pools.

Original: We explore { 0.1, 0.2, 0.5, 0.9 } for α and { 5, 10, 20, 40 } for KD temperature.
Revised: We explore { 0.1, 0.2, 0.5, 0.9 } for α and { 5, 10, 20, 40 } for KD temperature.

Original: For Pesudo-KD, we also perform explore { 1.5, 2.5, 4 } for β .
Revised: For Pesudo-KD, we also perform explore { 1.5, 2.5, 4 } for β .

Original: The best hyper-parameters can be found in Appendix C.
Revised: The best hyper-parameters can be found in Appendix C.

Original: We ﬁrst compare our proposed Pseudo-KD method with uniform label smoothing and conventional
Revised: We ﬁrst compare our proposed Pseudo-KD method with uniform label smoothing and conventional

Original: KD with different teachers.
Revised: KD with different teachers.

Original: All experiments are repeated 5 times with different random initialization
Revised: All experiments are repeated 5 times with different random initialization.

Original: [Table] [Table] Results Tab. 4.1 shows the test accuracy for each method using various network architectures.
Revised: Results Tab. 4.1 shows the test accuracy for each method using various network architectures.

Original: Our method signiﬁcantly improves classiﬁcation performance than conventional LS on all six models with over 0.85 or 0.35 points higher accuracy on average for CIFAR100 and CIFAR10, respectively. 
Revised: Our method signiﬁcantly improves classiﬁcation performance than conventional LS on all six models with over 0.85 or 0.35 points higher accuracy on average for CIFAR100 and CIFAR10, respectively. [Table]

Original: This indicates its adaptability to different networks.
Revised: This indicates its adaptability to different networks.

Original: Compared to KD methods, our method can give comparable or better performance, while being much more efﬁcient in training.
Revised: Compared to KD methods, our method can give comparable or better performance, while being much more efﬁcient in training.

Original: This is because our method could be implemented in an online manner, which only need one forward pass as Base or
Revised: This is because our method could be implemented in an online manner, which only need one forward pass as Base or

Original: LS methods.
Revised: LS methods.

Original: Furthermore, our method does not necessitate the use of a pre-trained teacher, which saves computational overhead.
Revised: Furthermore, our method does not necessitate the use of a pre-trained teacher, which saves computational overhead.

Original: We also observe that lightweight models like ShufﬂeNetv2 and MobileNetv2 beneﬁt more from LS,
Revised: We also observe that lightweight models like ShufﬂeNetv2 and MobileNetv2 beneﬁt more from LS,

Original: KD and our method than complex models (ResNeXt29 and ResNet101).
Revised: KD and our method than complex models (ResNeXt29 and ResNet101).

Original: Furthermore, as a teacher, using a model with a low capacity can provide comparable performance for improvements for small models to using a large model.
Revised: Furthermore, as a teacher, using a model with a low capacity can provide comparable performance for improvements for small models to using a large model.

Original: Speciﬁcally, by distilling three low-capacity models (ShufﬂeNetv2,
Revised: Speciﬁcally, by distilling three low-capacity models (ShufﬂeNetv2,

Original: MobileNetv
Revised: MobileNetv

Original: [Table]
Revised: [Table]

Original: and ResNet18) and using ShufﬂeNet as the teacher, all three models can achieve comparable performance, with ShufﬂeNet outperforming the others.
Revised: and ResNet18) and using ShufﬂeNet as the teacher, all three models can achieve comparable performance, with ShufﬂeNet outperforming the others [Table]

Original:  Next, we conduct a series of experiments on three models to compare our approach with other methods without requiring any pre-trained models (except for KD-29 reported as a reference).All [Table]
Revised: [Table] Next, we conduct a series of experiments on three models to compare our approach with other methods without requiring any pre-trained models (except for KD-29 reported as a reference).

Original:  experiments are repeated 5 times with different random initialization.
Revised: All experiments are repeated 5 times with different random initialization.

Original: The baseline methods include,
Revised: The baseline methods include,

Original: CS-KD which distill the soft output between different samples of the same class to student (Yun et al., 2020).
Revised: CS-KD which distill the soft output between different samples of the same class to student (Yun et al., 2020).

Original: TF-reg which regularize the training with a manually designed teacher (Yuan et al., 2020) and Beta-LS which leveraging a model with same capacity to learn instance-speciﬁc prior over the smoothing distribution (Zhang & Sabuncu, 2020).
Revised: TF-reg which regularize the training with a manually designed teacher (Yuan et al., 2020) and Beta-LS which leveraging a model with same capacity to learn instance-speciﬁc prior over the smoothing distribution (Zhang & Sabuncu, 2020).

Original: We follow their best hyper-parameter settings.
Revised: We follow their best hyper-parameter settings.

Original: The time for all baselines is measured based on its original implementations.
Revised: The time for all baselines is measured based on its original implementations.

Original: Tab. 2 shows the test accuracy and training time for 200 epoch of different methods on three models.
Revised: Tab. 2 shows the test accuracy and training time for 200 epoch of different methods on three models.

Original: It can be seen that our method consistently improves the classiﬁcation performance on both lightweight and complex models, which indicates its robustness to different networks.
Revised: It can be seen that our method consistently improves the classiﬁcation performance on both lightweight and complex models, which indicates its robustness to different networks.

Original: Besides, it shows the training time of our method is equivalent to Base or LS methods, which show its stable efﬁciency over other baselines such as Beta-LS, which still requires a separated model to output a learned prior over the smoothing distribution. 
Revised: Besides, it shows the training time of our method is equivalent to Base or LS methods, which show its stable efﬁciency over other baselines such as Beta-LS, which still requires a separated model to output a learned prior over the smoothing distribution. We further compare two recent works KR-LS (Ding et al., 2021) and PS-KD (Kim et al., 2021). Our method achieves comparable or better performance than PSKD and consistently better than KR-LS.

Revised: We have to mention the implementation of KR-LS and PS-KD increase space complexity

Original:  4. R ESULTS ON N ATURAL L ANGUAGE U NDERSTANDING
Revised: O ( K × K ) and O ( N × K ) for class-wise and sample-wise smoothing labels, respectively. However, our method neither increases time nor space complexity during the training. 4. R ESULTS ON N ATURAL L ANGUAGE U NDERSTANDING

Original: Data.
Revised: Data.

Original: We run experiments on 7 classiﬁcation tasks from the GLUE benchmark (Wang et al., 2019b).
Revised: We run experiments on 7 classiﬁcation tasks from the GLUE benchmark (Wang et al., 2019b).

Original: These datasets can be broadly divided into 3 families of problems.
Revised: These datasets can be broadly divided into 3 families of problems.

Original: Single set tasks which include linguistic acceptability (CoLA) and sentiment analysis (SST-2), similarity and paraphrasing tasks (MRPC and QQP), and inference tasks which include Natural Language Inference (MNLI and
Revised: Single set tasks which include linguistic acceptability (CoLA) and sentiment analysis (SST-2), similarity and paraphrasing tasks (MRPC and QQP), and inference tasks which include Natural Language Inference (MNLI and

Original: RTE) and Question Answering (QNLI).
Revised: RTE) and Question Answering (QNLI).

Original: Following prior works (Jafari et al., 2021; Rashid et al., 2021), we report Spearman correlation for STS-B, Matthews correlation for CoLA, and accuracy for the other tasks for the development set.
Revised: Following prior works (Jafari et al., 2021; Rashid et al., 2021), we report Spearman correlation for STS-B, Matthews correlation for CoLA, and accuracy for the other tasks for the development set.

Original: We report “F1” for the QQP and MRPC and the corresponding test sets.
Revised: We report “F1” for the QQP and MRPC and the corresponding test sets.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: We choose the pre-trained DistilRoBERTa (6-layer)
Revised: We choose the pre-trained DistilRoBERTa (6-layer)

Original: (Sanh et al., 2019) as the base model for all our experiments.
Revised: (Sanh et al., 2019) as the base model for all our experiments.

Original: For KD experiments, we use a RoBERTa-Large model (24-layer)
Revised: For KD experiments, we use a RoBERTa-Large model (24-layer)

Original: (Liu et al., 2019) ﬁne-tuned on each of the tasks as the corresponding teacher model.
Revised: (Liu et al., 2019) ﬁne-tuned on each of the tasks as the corresponding teacher model.

Original: All models are trained with the AdamW optimizer with the default settings Loshchilov & Hutter (2019).
Revised: All models are trained with the AdamW optimizer with the default settings Loshchilov & Hutter (2019).

Original: To explore the best hyper-parameters, we conduct a grid search over the learning rate ∈ { 1 e − 5 , 2 e − 5 , 3 e − 5 } , batch size ∈ { 8 , 16 , 32 , 64 } , α ∈ { 0 . 1 , 0 . 2 , 0 . 5 } and KD temperature ∈ { 1 , 5 , 10 } .
Revised: To explore the best hyper-parameters [Table] we conduct a grid search over the learning rate ∈ { 1 e − 5 , 2 e − 5 , 3 e − 5 } , batch size ∈ { 8 , 16 , 32 , 64 } , α ∈ { 0 . 1 , 0 . 2 , 0 . 5 } and KD temperature ∈ { 1 , 5 , 10 } .

Original: Moreover, for Pesudo-KD, we perform a search over three β values in { 1.5, 2, 4 } .
Revised: Moreover, for Pesudo-KD [Table] we perform a search over three β values in { 1.5, 2, 4 } .

Original: The best hyper-parameters can be found in Appendix.
Revised: The best hyper-parameters can be found in Appendix.

Original: We repeat our experiments 3 times with different random initialization.
Revised: We repeat our experiments 3 times with different random initialization [Table] [Table] [Table]

Original:  Results Table 3 presents the dev set results on GLUE.
Revised: w/ KD 51. 80.2 w/ LS 49. 79.9 w/ Pseudo-KD (Ours) 51. Results Table 3 presents the dev set results on GLUE.

Original: We present the average result for each experiment, over three runs, as well as the standard deviation.
Revised: We present the average result for each experiment, over three runs, as well as the standard deviation.

Original: As is standard on the GLUE benchmark, we present an average result across the datasets as well.
Revised: As is standard on the GLUE benchmark, we present an average result across the datasets as well.

Original: We compare Pseudo-KD to simple ﬁnetuning, LS and KD.
Revised: We compare Pseudo-KD to simple ﬁnetuning, LS and KD.

Original: We observe that our method is better on all datasets compared to ﬁne-tuning 
Revised: We observe that our method is better on all datasets compared to ﬁne-tuning and at par (QQP only) or better on all datasets compared to LS. We are comparable on average to [Figure] [Figure]

Original: [Table]
Revised: [Figure]

Original: [Table]
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: ResNet beta/alpha=

Original: [Figure]
Revised: [Figure]

Revised: beta/alpha=2.

Original: [Figure]
Revised: [Figure]

Original: 9 w/ Pseudo-KD (Ours)
Revised: beta/alpha= LS 0. 0.6 alpha 78. A cc

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: ResNet beta/

Original: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: and at par (QQP only) or better on all datasets compared to LS. We are comparable on average to
Revised: LS

Original: KD, with KD doing better on MNLI and QQP and Pseudo-KD doing better o CoLA, MRPC, RTE and SST-2.
Revised: KD, with KD doing better on MNLI and QQP and Pseudo-KD doing better o CoLA, MRPC, RTE and SST-2.

Original: Furthermore, Table 4 shows the GLUE leaderboard test results.
Revised: Furthermore, Table 4 shows the GLUE leaderboard test results.

Original: We chose the model with the best dev results for each method and submitted it to the ofﬁcial GLUE benchmark to get the test results.
Revised: We chose the model with the best dev results for each method and submitted it to the ofﬁcial GLUE benchmark to get the test results.

Original: We see a similar trend and observe that Pseudo-KD is better than LS and ﬁne-tuning on all datasets except RTE.
Revised: We see a similar trend and observe that Pseudo-KD is better than LS and ﬁne-tuning on all datasets except RTE.

Original: On average the results are comparable to KD as before.
Revised: On average the results are comparable to KD as before.

Original: KD is better on CoLA, RTE,
Revised: KD is better on CoLA, RTE,

Original: QNLI and MNLI and Pseudo KD is better on MRPC, SST-2 and at par on QQP.
Revised: QNLI and MNLI and Pseudo KD is better on MRPC, SST-2 and at par on QQP.

Original: We observe that on average all the models are close in performance, with an average difference of 0.4 between the best and the worst model.
Revised: We observe that on average all the models are close in performance, with an average difference of 0.4 between the best and the worst model.

Original: 4. E FFECT OF H YPER - PARAMETER α AND β
Revised: 4. E FFECT OF H YPER - PARAMETER α AND β

Original: We conducted ablation tests to probe how the α and β hyper-parameters affects the performance of our method.
Revised: We conducted ablation tests to probe how the α and β hyper-parameters affects the performance of our method.

Original: We also compared our results to LS while choosing the same α for both.
Revised: We also compared our results to LS while choosing the same α for both.

Original: Specifically, we chose α ∈ { 0 . 1 , 0 . 2 , . . .
Revised: Specifically, we chose α ∈ { 0 . 1 , 0 . 2 , . . .

Original: For Pseudo-KD, we also veriﬁed the inﬂuence of β/α ∈ { 1 . 5 , 2 . 5 , 4 } .
Revised: For Pseudo-KD, we also veriﬁed the inﬂuence of β/α ∈ { 1 . 5 , 2 . 5 , 4 } .

Original: Following the discussion on equation 14 we observe that β/α can be interpreted as the temperature in self-distillation.
Revised: Following the discussion on equation 13 we observe that β/α can be interpreted as the temperature in self-distillation.

Original: We plot the accuracy on CIFAR100 for two architectures (ResNet18 and ResNet50) as shown in
Revised: We plot the accuracy on CIFAR100 for two architectures (ResNet18 and ResNet50) as shown in

Original: Fig.
Revised: Fig.

Original: The plot shows that our method can consistently outperform the LS method across different α values even when varying the β/α parameter.
Revised: The plot shows that our method can consistently outperform the LS method across different α values even when varying the β/α parameter.

Original: The results are consistent for both architectures.
Revised: The results are consistent for both architectures.

Original: As is expected, the accuracy of both LS and our method gradually decreases when α increases from 0.
Revised: As is expected, the accuracy of both LS and our method gradually decreases when α increases from 0.

Original: to 0.6.
Revised: to 0.6.

Original: It is interesting that even at an α of 0.6 the drop in accuracy is less than 1% of the peak value.
Revised: It is interesting that even at an α of 0.6 the drop in accuracy is less than 1% of the peak value.

Original: We observe that a temperature value between 1.5 and 2.5 is the best to obtain optimal performance.
Revised: We observe that a temperature value between 1.5 and 2.5 is the best to obtain optimal performance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our aim in this work is to ﬁll the accuracy gap between KD and LS techniques, while maintaining the training efﬁciency of LS.
Revised: Our aim in this work is to ﬁll the accuracy gap between KD and LS techniques, while maintaining the training efﬁciency of LS.

Original: We proposed learning an instance-speciﬁc label smoothing regularization simultaneously with training our model on the target.
Revised: We proposed learning an instance-speciﬁc label smoothing regularization simultaneously with training our model on the target.

Original: We began by revisiting the traditional
Revised: We began by revisiting the traditional

Original: LS method and introduced our objective function by substituting the uniform distribution with a general, instance-speciﬁc, discrete distribution.
Revised: LS method and introduced our objective function by substituting the uniform distribution with a general, instance-speciﬁc, discrete distribution.

Original: Within the new objective, we explained the relationship between the LS and KD.
Revised: Within the new objective, we explained the relationship between the LS and KD.

Original: Particularly, our method can be interpreted as an online version of self-distillation.
Revised: Particularly, our method can be interpreted as an online version of self-distillation.

Original: Then, using a two-stage optimization approach, we obtained an approximation for the optimal smoothing function.
Revised: Then, using a two-stage optimization approach, we obtained an approximation for the optimal smoothing function.

Original: We conducted extensive experiments to compare our model with
Revised: We conducted extensive experiments to compare our model with

Original: LS, KD and various teacher-free methods on popular CV and NLU benchmarks and showed the effectiveness and efﬁciency of our technique.
Revised: LS, KD and various teacher-free methods on popular CV and NLU benchmarks and showed the effectiveness and efﬁciency of our technique.

Original: We want to extend this research in two directions in the future.
Revised: We want to extend this research in two directions in the future.

Original: First we want to improve our algorithm further and explore replacing the alternating approach to two-stage optimization with a more complex algorithm to reach closer to the optimal smoothing.
Revised: First we want to improve our algorithm further and explore replacing the alternating approach to two-stage optimization with a more complex algorithm to reach closer to the optimal smoothing.

Original: Next we want to explore practical applications where our method can replace KD.
Revised: Next we want to explore practical applications where our method can replace KD.

Original: One interesting application is the pre-training of compressed language models.
Revised: One interesting application is the pre-training of compressed language models.

Original: Typically, the best compressed models are pre-trained using distillation from a much larger teacher which makes the training process cumbersome and computationally intensive.
Revised: Typically, the best compressed models are pre-trained using distillation from a much larger teacher which makes the training process cumbersome and computationally intensive.

Original: It would be interesting to explore if Pseudo-KD can achieve comparable performance to KD in that setting.
Revised: It would be interesting to explore if Pseudo-KD can achieve comparable performance to KD in that setting

Original: A D ERIVATION OF THE PROOF FOR T HEOREM
Revised: A D ERIVATION OF THE PROOF FOR T HEOREM

Original: Proof.
Revised: Proof.

Original: First, observe that the function R θ deﬁned in Eq.
Revised: First, observe that the function R θ deﬁned in Eq.

Original: 8 is a convex combination of N nonnegative functions
Revised: 7 is a convex combination of N nonnegative functions

Original: R i
Revised: R i

Original: =
Revised: =

Original: E
Revised: E

Original: ˆ P ls [ − log p θ ( j | x i )]
Revised: ˆ P ls [ − log p θ ( j | x i )]

Original: + βD KL
Revised: + βD KL

Original: ( P ls ( ·| x i ) (cid:107) U ( · )) , for i
Revised: ( P ls ( ·| x i ) (cid:107) U ( · )) , for i

Original: = 1 , . .
Revised: = 1 , . .

Original: N .
Revised: N .

Original: We will show that each of R i is a convex function of the components of simplex P ls ( ·| x i ) by computing the Hessian matrix of R i with respect to P ls ( ·| x i ) : 
Revised: We will show that each of R i is a convex function of the components of simplex P ls ( ·| x i ) by computing the Hessian matrix of R i with respect to P ls ( ·| x i ) : H

Original: [Equation]
Revised: [Equation]

Original: 
Revised: R i ( P ls

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 
Revised: P

Original: [Equation]
Revised: [Equation]

Original: When β is greater than zero, the Hessian is positive deﬁnite.
Revised: When β is greater than zero, the Hessian is positive deﬁnite.

Original: Therefore, each R i is a convex function of the componets of P ls ( ·| x i ) .
Revised: Therefore, each R i is a convex function of the componets of P ls ( ·| x i ) .

Original: As a result, R θ is a convex function of the collection of components of every simplex P ls ( ·| x i ) .
Revised: As a result, R θ is a convex function of the collection of components of every simplex P ls ( ·| x i ) .

Original: For simplicity, we derive the global optimum solution for each R i with a Lagrangian multiplier:
Revised: For simplicity, we derive the global optimum solution for each R i with a Lagrangian multiplier:

Original: L i ( P ls , λ L ) =
Revised: L i ( P ls , λ L ) =

Revised: K (cid:88) j (cid:48)

Revised: = (cid:20)

Original: [Equation]
Revised: [Equation]

Original: where we omit the dependency on x i to simplify the notation.
Revised: where we omit the dependency on x i to simplify the notation.

Original: We set the corresponding gradients equal to 0 to obtain the global optimum for j = 1 , . . .
Revised: We set the corresponding gradients equal to 0 to obtain the global optimum for j = 1 , . . .

Original: , K .
Revised: , K .

Original: [Equation]
Revised: [Equation]

Original: Since [Equation] Kj (cid:48)
Revised: Since (cid:80) Kj (cid:48)

Original: =
Revised: =

Original: 1 P ls ( j (cid:48) )
Revised: 1 P ls ( j (cid:48) )

Original: = 1 , we have
Revised: = 1 , we have

Original: [Equation]
Revised: [Equation]

Original: So the optimal P ∗ ls ( j ) is given by the formula:
Revised: So the optimal P ∗ ls ( j ) is given by the formula:

Original: [Equation]
Revised: [Equation]

Original: B D ERIVATION FROM OPTIMAL SMOOTHING TO SOFTMAX OUTPUT WITH TEMPERATURE
Revised: B D ERIVATION FROM OPTIMAL SMOOTHING TO SOFTMAX OUTPUT WITH TEMPERATURE

Original: When β = α · τ
Revised: When β = α · τ

Original: [Table]
Revised: [Equation]

Original: we could have
Revised: we could have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This is the commonly used soft label in KD approach
Revised: This is the commonly used soft label in KD approach

Original: [Table]
Revised: [Equation]

Original: C H YPER - PARAMETERS FOR P SEUDO -KD
Revised: C H YPER - PARAMETERS FOR P SEUDO -KD

Revised: Model

Revised: [Equation]

Original: [Table]
Revised: [Table]