A DAPTIVE L ABEL S MOOTHING WITH S ELF -K NOWLEDGE
Anonymous authors
Paper under double-blind review
A BSTRACT
Overconﬁdence has been shown to impair generalization and calibration of a neural network.
Previous studies remedy this issue by adding a regularization term to a loss function, preventing a model from making a peaked distribution.
Label smoothing smoothes target labels with a pre-deﬁned prior label distribution; as a result, a model is learned to maximize the likelihood of predicting the soft label.
Nonetheless, the amount of smoothing is the same in all samples and remains ﬁxed in training.
In other words, label smoothing does not reﬂect the change in probability distribution mapped by a model over the course of training.
To address this issue, we propose a regularization scheme that brings dynamic nature into the smoothing parameter by taking model probability distribution into account, thereby varying the parameter per instance.
A model in training self-regulates the extent of smoothing on the ﬂy during forward propagation.
Furthermore, inspired by recent work in bridging label smoothing and knowledge distillation, our work utilizes self-knowledge as a prior label distribution in softening target labels, and presents theoretical support for the regularization effect by knowledge distillation and the dynamic smoothing parameter.
Our regularizer is validated comprehensively on various datasets in machine translation and illustrates greater results in model performance and model calibration than when training with hard targets.
I NTRODUCTION
In common practice, a neural network is trained to maximize the expected likelihood of observed targets, and the gradient with respect to the objective updates the learnable model parameters.
With hard targets (one-hot encoded), the maximum objective can be approached when a model assigns a high probability mass to the corresponding target label over the output space.
That is, due to the normalizing activation functions (i.e. softmax ), a model is trained in order for logits to have a marked difference between the target logit and the other classes logits (M ¨ uller et al., 2019).
Despite its wide application and use, the maximum likelihood estimation with hard targets has been found to incur an overconﬁdent problem: the predictive score of a model does not reﬂect the actual accuracy of the prediction.
Consequently, this leads to degradation in model calibration (Pereyra et al., 2017), as well as in model performance (M¨uller et al., 2019).
Additionally, this problem stands out more clearly with a limited number of samples, as a model is more prone to overﬁtting.
To remedy such phenomenon, Szegedy et al.
(2016) proposed label smoothing, in which one-hot encoded targets are replaced with smoothed targets.
Label smoothing has boosted performance in computer vision (Szegedy et al., 2016), and has been highly preferred in other domains, such as
Natural Language Processing (Vaswani et al., 2017; Lewis et al., 2020).
However, there are several aspects to be discussed in label smoothing.
First, it comes with certain downsides, namely the static smoothing parameter.
The smoothing regularizer fails to account for the change in probability mass over the course of training.
Despite the fact that a model can beneﬁt from adaptive control of the smoothing extent depending on the signs of overﬁtting and overconﬁdence, the smoothing parameter remains ﬁxed throughout training in all instances.
Another aspect of label smoothing for consideration is its connection to knowledge distillation (Hinton et al., 2015).
There have been attempts to bridge label smoothing with knowledge distillation, and the ﬁndings suggest the latter is an adaptive form of the former (Tang et al., 2021; Yuan et al.,
However, the regularization effect on overconﬁdence by self-knowledge distillation is still poorly understood and explored.
To tackle the issues mentioned above, this work presents adaptive label smoothing with selfknowledge as a prior label distribution.
Our regularizer allows a model to self-regulate the extent of smoothing based on the entropic level of model probability distribution, varying the amount per sample and per time step.
Furthermore, our theoretical analysis suggests that self-knowledge distillation and the adaptive smoothing parameter have a strong regularization effect by rescaling gradients on logit space.
Our work validates the efﬁcacy of the proposed regularization method on machine translation tasks and achieves superior results in model performance and model calibration compared to other baselines.
P RELIMINARIES & R ELATED W ORK 2. L ABEL S MOOTHING
Label smoothing (Szegedy et al., 2016) was ﬁrst introduced to prevent a model from making a peaked probability distribution.
Since its introduction, it has been in wide application as a means of regularization (Vaswani et al., 2017; Lewis et al., 2020).
In label smoothing, one-hot encoded ground-truth label ( y ) and a pre-deﬁned prior label distribution ( q ) are mixed with the weight, the smoothing parameter ( α ), forming a smoothed ground-truth label.
A model with label smoothing is learned to maximize the likelihood of predicting the smoothed label distribution.
Speciﬁcally,
[Equation]
| C | denotes the number of classes, ( n ) the index of a sample in a batch, and P θ the probability distribution mapped by a model.
α is commonly set to 0.1, and remains ﬁxed throughout training (Vaswani et al., 2017; Lewis et al., 2020).
A popular choice of q is an uniform distribution ( q ∼ U ( | C | ) ), while unigram distribution is another option for dealing with an imbalanced label distribution (Vaswani et al., 2017; Szegedy et al., 2016; M¨uller et al., 2019;
Pereyra et al., 2017).
The pre-deﬁned prior label distribution remains unchanged, hence the latter cross-entropy term in
Equation 1 is equivalent to minimizing the KL divergence between the model prediction and the predeﬁned label distribution.
In line with the idea, Pereyra et al.
(2017) proposed conﬁdence penalty (ConfPenalty) that adds negative entropy term to the loss function, thereby minimizing the KL divergence between the uniform distribution and model probability distribution.
Ghoshal et al.
(2021) proposed low-rank adaptive label smoothing (LORAS) that jointly learns a noise distribution for softening targets and model parameters.
Li et al.
; Krothapalli & Abbott (2020) introduced smoothing schemes that are data-dependent.
2. K NOWLEDGE D ISTILLATION
Knowledge distillation (Hinton et al., 2015) aims to transfer the dark knowledge of (commonly) a larger and better performing teacher model to a student model (Buciluundeﬁned et al., 2006).
The idea is to mix the ground-truth label with the model probability distribution of a teacher model, the result acting an adaptive version of label smoothing (Tang et al., 2021).
[Equation]
φ and θ denote the parameters of a teacher model and a student model respectively.
Similar to label smoothing, φ remains unchanged in training; thus a student model is learned to minimize the KL divergence between its probability distribution and that of the teacher model.
When P φ follows a uniform distribution with temperature set to 1, the loss function of knowledge distillation is identical to that of uniform label smoothing.
Training a large teacher model can be computationally expensive; for this reason, there have been attempts to replace the teacher model with the student model itself, called self-knowledge distillation (Zhang et al., 2019; Yuan et al., 2020;
Kim et al., 2021; Zhang & Sabuncu, 2020).
TF-KD (Yuan cc | C | ⋯ ℝ | N | × d θ t − θ t − θ t − f
( X ; θ t ) f
( X ; ϕ ) ℝ | N | × | C | ⋮ ⋮ ℝ | N | × | C | g
Past Checkpoints Θ cc | C | α ⋮ α ( | N | )
= 0 .
8α ( 1 ) = 0 .α
α α ℝ | N | ×
L ask
Parameter Update Forward Propagation
Backward Propagation Closer View
H ′ (˜ y ( 1 ) )
H ′ (˜ y
( | N | ) )
⋮
H ′
̂ y ̂ y ̂ y ̂ y ˜ y ˜
y ˜ y
˜
y x x x x et al., 2020) trains a student with a pre-trained teacher that is identical to the student in terms of structure.
SKD-PRT (Kim et al., 2021) utilizes the previous epoch checkpoint as a teacher with linear increase in α . Zhang & Sabuncu (2020) incorporate beta distribution sampling (BETA) and self-knowledge distillation (SD), and introduce instance-speciﬁc prior label distribution. Yun et al. (2020) utilize self-knowledge distillation to minimize the predictive distribution of samples with the same class, encouraging consistent probability distribution within the same class.
M ETHODOLOGY
The core components of label smoothing are two-fold: smoothing parameter ( α ) and prior label distribution.
The components determine how much to smooth the target label using which distribution, a process that requires careful choice of selection.
In this section, we illustrate how to make the smoothing parameter adaptive.
We also demonstrate how our adaptive smoothing parameter and self-knowledge distillation as a prior distribution act as a form of regularization with theoretical analysis on the gradients.
A DAPTIVE α
An intuitive and ideal way of softening the hard target is to bring dynamic nature into choosing α ; a sample with low entropic level in model prediction receives a high smoothing parameter to further smooth the target label.
In another scenario, in which high entropy of model prediction (ﬂat distribution) is seen, the smoothing factor is decreased.
With the intuition, our method computes the smoothing parameter on the ﬂy during the forward propagation in training, relying on the entropic level of model probability distribution per sample, and per time step in case of sequential classiﬁcation. [Equation]
[Equation]
The entropy quantiﬁes the level of probability mass distributed across the label space; therefore, low entropy is an indication of overﬁtting and overconﬁdence (Pereyra et al., 2017; Meister et al., 2020).
Since entropy does not have a ﬁxed range between 0 and 1, one simple scheme is to normalize the entropy with maximum entropy ( log | C | ).
Hence, the normalization is capable of handling variable size of class among
[Equation]
With this mechanism, a sample with high entropy is trained with low α , and a sample with low entropy receives high α .
The computation for α is excluded from the computation graph for the gradient calculation, hence, the gradient does not ﬂow through adaptive α ( n ) .
There are two essential beneﬁts of adopting the adaptive smoothing parameter.
As the smoothing extent is determined by its own probability mass over the output space, the hyperparameter search on α is removed.
Furthermore, it is strongly connected to the gradient rescaling effect on selfknowledge distillation, which will be dealt in Section 3.3 in detail.
3. S ELF -K NOWLEDGE A S A P RIOR
Similar to (Kim et al., 2021; Liu et al., 2021), our regularizer loads a past student model checkpoint as teacher network parameters in the course of training.
The intuition is to utilize past selfknowledge which is a generalized supervision, thereby hindering the model from overﬁtting to observations in the training set.
[Equation]
Θ is a set of past model checkpoints in training, and function f is a speciﬁc task, which in our work is machine translation.
X (cid:48) and Y (cid:48) are sets of input and ground-truth samples from a validation dataset 2 , and the function g could be any proper evaluation metric for model generalization (i.e. accuracy).
Our work utilizes the n -gram matching score, BLEU (Papineni et al., 2002) being the function for ﬁnding the prior label distribution ( q ∼ P φ ).
Combining the adaptive smoothing parameter and self-knowledge as a prior distribution, our loss function is as follows:
[Equation]
3. G RADIENT A NALYSIS
Tang et al.
; Kim et al.
(2021) theoretically ﬁnd that the success of knowledge distillation is related to the gradient rescaling in the logit space; the difﬁculty of a sample determines the rescaling factor, and difﬁcult-to-learn samples receive higher rescaling factors than those of the easy-to-learn samples.
4 We further extend the gradient analysis in the perspective of regularization effect, and discuss the importance of the adaptive smoothing parameter.
Before dissecting the gradients, we ﬁrst set a hypothesis: teacher network makes a less conﬁdent prediction than that of the student.
In self-knowledge distillation with a past checkpoint as the teacher, the assumption holds true.
The expected predictive score on target label by the teacher model is lower than that of the current checkpoint in training (Kim et al., 2021).
The gradient with respect to the logit ( z ) by the cross entropy loss ( L ce ) is as follows 5 :
[Equation]
With knowledge distillation ( L kd ), the gradient on the logit is
[Equation]
The following compares the ratio of the gradient from knowledge distillation and with that of the cross entropy.
[Equation]
When i = j , with j being the index of the ground truth, it is worth noting that the denominator of the second term in Equation 9 has range P θ ( y i ) − 1 ∈ [ − 1 , 0] , and the range of the numerator is conﬁned to P θ ( y i ) − P φ ( y i ) ∈ [0 , 1] .
Therefore, the equation can be written as
[Equation]
The norm of the gradient drastically diminishes when there is a large difference between the predictions by the models, and when the predictive score of a student model is high, which is a sign of overconﬁdence.
In terms of the direction of the gradient, when the following is seen,
[Equation]
the direction of the gradients with respect to knowledge distillation becomes the opposite to that of the cross entropy, pushing parameters to lower the likelihood of the target index.
The same applies when i is not equal to the target index ( i (cid:54) =
j ).
From Equation 9, the following can be derived.
[Equation]
With the generalized teacher, the expected predictive score on the incorrect labels by the teacher model is higher than that of the student model.
Therefore, in addition to the shrinking norm effect, the direction of the gradient can be reversed when 1 < α P φ ( y i ) P θ ( y i ) , similar to Equation 11; as a result, it leads to updating model parameters to increase the likelihood on the incorrect classes , an opposite behavior to that of the cross entropy with hard targets.
Overall, in either case, the theoretical support depicts strong regularization effects with the generalized supervision by the teacher.
As label smoothing is also closely linked to the above, the theoretical support can be easily extended to label smoothing if P φ ( y i ) is replaced with α | C | in case of uniform label smoothing, and P ( c i ) in unigram label smoothing.
The adaptive α is another factor to be discussed regarding the gradient analysis.
As clearly demonstrated in Equation 11 and 12, a high α , an indication of peak probability distribution, not only leads to drastic decrease in the gradient norm, but it is likely to make the gradient go the opposite direction to that of the cross entropy.
It leads to updating the model parameters to lower the predictive score on the ground-truth target, as opposed to the effect of the cross entropy with hard targets.
Furthermore, as the parameters updates are performed by aggregating the losses of samples, adaptive smoothing acts as a gradient rescaling mechanism.
The gradient rescaling by adaptive α reweights the gradients in aggregating the losses, hence, the regularization effect is stronger on the samples with signs of overconﬁdence than on other samples.
The use of adaptive α is not only intuitive in terms of tackling overconﬁdence, but it also serves as an important aspect in the theoretical support.
E XPERIMENT 4. D ATASET & E XPERIMENT S ETUP
We validate the proposed regularizer on three popular translation corpora: IWSLT14 GermanEnglish ( DE-EN ) (Cettolo et al., 2014), IWSLT15 English-Vietnamese ( EN-VI ) (Cettolo et al., 2015), and Multi30K German-English pair (Elliott et al., 2016).
IWSLT14 DE-EN contains 160K sentence pairs in training, 7K in validation, and 7K in testing.
IWSLT15 EN-VI has 133K, 1.5K [Table] and 1.3K in training, validation [Table]
and testing dataset respectively. Lastly, 28K training, 1K validation, and 1K testing sentences are used in Multi30K dataset.
Byte pair encoding (Sennrich et al., 2016) is used to process words into sub-word units.
All of the experiments are conducted with transformer architecture (Vaswani et al., 2017) on a Telsa
V100.
For generation, beam size is set to 4 in the inference stage.
The training conﬁguration follows the instruction of fairseq (Ott et al., 2019).
For the quality of the generated outputs, we report the popular metrics for machine translation: BLEU (Papineni et al., 2002), METEOR (Banerjee &
Lavie, 2005), Word Error Rate (WER), ROUGE-L (Lin, 2004), and NIST (Doddington, 2002).
4. E XPERIMENTAL R ESULT & A NALYSIS
Automatic evaluation results on the three test datasets are shown in Table 1.
Though most of the methods achieve meaningful gain, the most noticeable difference is seen with our method.
Our regularization scheme shows solid improvements on all of the metrics on the datasets without any additional learnable parameter.
For example, the absolute gain in BLEU compared to the base method in Multi30K dataset is around 3.75, which is 9.2% relative improvement.
Not only does our method excel in n -gram precision score, but it shows superior performance in having the longest [Figure] [Figure] common subsequence with the reference text, as well as in the informativeness of the n -grams.
The empirical result demonstrates that our regularizer improves the base method across all the metrics by a large margin.
In Figure 2a, the changes in α during training are visualized.
As expected, the smoothing parameters start with a very small number, as the entropic level must be high due to the under-ﬁtted models.
As training continues, the predictive scores of the models increase, and accordingly, adaptive α increases to prevent overconﬁdence.
One notable aspect is the convergence at a certain level.
Each training of the corpora ends up with a different α , and the model in training self-regulates the amount of smoothing and the value converges.
Furthermore, our adaptive α affects the norm of the gradients as depicted in Figure 2b.
The gradient norm of our regularizer is considerably smaller than that of the other methods.
This empirical ﬁnding mainly conforms with the gradient analysis in Section 3.3, where the importance of adaptive α and generalized teacher model was discussed.
4.2. M ODEL C ALIBRATION
In addition to the automatic evaluation, in which the improved generalization is seen through the performance gains, we look into the calibrations of the models trained with the methods.
Figuredepicts how the cross entropy with hard targets tends to make a model overconﬁdent in prediction.
In the reliability diagram, the conﬁdence score in each bin is larger than that of accuracy, the gap of which is noticeable.
Label smoothing mitigates the problem to some extent, yet the gap between the accuracy and the conﬁdence score still remains clear.
On the other hand, the proposed regularizer signiﬁcantly reduces the gap, showing the enhanced model calibration.
[Table] The improvement in calibration is more clear with expected calibration error (ECE) and maximum calibration error (MCE) reported in Table 2.
For instance, on the IWSLT14 dataset, the errors with label smoothing drop signiﬁcantly in both metrics, which is around 6% absolute decrease in ECE and 10% in MCE.
Nonetheless, ECE of the proposed method results in 1.76% which is around 11% absolute decrease and 86% relative improvement.
In addition, our method achieves 3.64% in MCE, which is 81% relative improvement over the base method.
The improved calibration with our method is seen across the datasets, demonstrating the effectiveness of our method in model calibration.
One important ﬁnding is the gap between the performance in automatic evaluation and the calibration error.
Conﬁdence penalty (Pereyra et al., 2017) is highly competitive in n -gram matching scores (BLEU) on all of the dataset tested.
Nevertheless, the calibration error is the highest among the methods.
Similar to the ﬁnding in (Guo et al., 2017), the discrepancy between the performance and model calibration exists, and it calls for caution in training a neural network when considering model calibration.
4. A BLATION S TUDY
Table 3: (+) denotes adding the following components to the base method. α ( n ) denotes our adaptive α , and α ↑ indicates a linear increase in α in the course of training. SK and Uniform denote Self-Knolwedge and Uniform distribution as a prior label distribution respectively.
g NLL and g BLEU indicates g function set to negative log likelihood and
BLEU respectively.
[Table] Table 3 shows the change in performance when adding our core components to the base method on the IWSLT14 dataset.
When using the ﬁxed smoothing parameter with self-knowledge as a prior, the BLEU score increases by a small margin, and the ECE does not drop signiﬁcantly.
In another case where the smoothing parameter is adaptive, and the prior label distribution is set to uniform distribution, there is a meaningful increase in BLEU score.
However, it impairs the
ECE score noticeably.
We empirically ﬁnd that the result mainly comes from underconﬁdence of a model.
The conﬁdence score is largely lower than that of the accuracy.
In an experiment with linearly increasing α with self-knowledge prior, the BLEU score improves by around 1.6 score, yet the ECE score still has room for improvement. Since α value is shared among samples in this ablation experiment, there is no gradient rescaling by adaptive α which may explain ECE score being high compared to that of our adaptive α . We also look into a case with a different g function: BLEU and Negative Log Likelihood (NLL). We observe that both g BLEU and g NLL greatly improve the scores.
As g has the purpose of selecting a generalized teacher from the set of past checkpoints, a proper metric would serve the purpose.
In conclusion, while the adaptive smoothing parameter plays an important role in regularizing a model, both adaptive smoothing parameter and the choice of prior label distribution greatly affect model calibration.
C ONCLUSION & F UTURE W ORK
In this work, we propose a regularization scheme that dynamically smooths the target label with self-knowledge.
Our regularizer self-regulates the amount of smoothing with respect to the entropic level of the model probability distribution, making the smoothing parameter dynamic per sample, and per time step.
The given idea is theoretically supported by gradient rescaling and direction, and the ﬁnding is backed up by the empirical results, both in model performance and calibration.
The future work is expected to validate the proposed regularization on other domains outside of Natural Language Processing.
The scope of application is not conﬁned to natural language generation tasks, and it can be easily extended to other classiﬁcation tasks.
R EPRODUCIBILITY S TATEMENT
For reproducibility, we report the three random seeds tested: { 0000, 3333, 5555 } .
For all of the experiments, this work utilizes the transformer architecture (Vaswani et al., 2017).
Both the encoder and the decoder are composed of 6 transformer layers with 4 attention heads.
The hidden dimension size of the both is 512.
For training conﬁguration, the maximum tokens in a batch is set to 4,096.
For optimization, Adam (Kingma & Ba, 2015) is used with beta 1 and beta 2 set to 0.9 and 0.98 respectively.
We slowly increase the learning rate up to 0.005 throughout the ﬁrst 4,000 steps, and the learning rate decreases from then on.
The source code and training script are included in the supplementary materials.
A A PPENDIX A. B ASELINES A.1. C ONFIDENCE P ENALTY
Conﬁdence penalty (Pereyra et al., 2017) adds a negative entropy term to the loss function, hence the model is encouraged to maintain entropy at certain level.
[Equation]
For the regularization-speciﬁc hyperparameter, following (Meister et al., 2020), β was set to 0.78.
A.1. TF-KD
TF-KD (Yuan et al., 2020), similar to conventional knowledge distillation, trains a teacher model prior to training a student; but it is different in that the model architecture is same with that of the student.
For the hyperparameters used in this paper, we empirically ﬁnd that high smoothing parameter leads to better performance.
Thus, we set the smoothing parameter to 0.9 and temperature scaling to 20.
A.1. SKD-PRT
SKD-PRT (Kim et al., 2021) is a self-knowledge distillation method, where a student model (epoch t ) is trained with its own last epoch checkpoint (epoch t − 1 ) in the course of training.
Though the idea is similar to ours, yet there are two core differences.
The ﬁrst is that we ﬁnd the teacher model that generalizes well with a function g .
Another difference is that SKD-PRT linearly increases α throughout the training, and this practice inevitably adds two hyperparameters ( max α and max epoch).
Following the original work (Kim et al., 2021), we set that maximum α to 0.7 and maximum epoch to 150 in our experiments. A.1. LORAS
LORAS (Ghoshal et al., 2021) jointly learns a soft target and model parameters in training in the aim of increasing model performance and model calibration, with low rank assumption.
For hyperparameters, η , α , rank and dropout probability are set to 0.1, 0.2, 25 and 0.5 respectively.
A.1. BETA & SD
Zhang & Sabuncu (2020) propose amortized MAP interpretation of teacher-student training, and introduce Beta smoothing which is an instance-speciﬁc smoothing technique that is based on the prediction by a teacher network.
For SD-speciﬁc hyperparameters, this work sets α to 0.3 and temperature to 4.0.
For BETA-speciﬁc hyperparameters, α and a are set to 0.4, 4.0 respectively.