Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation (2022)

Abstract: Overconfidence has been shown to impair generalization and calibration of a neural network. Previous studies remedy this issue by adding a regularization term to a loss function, preventing a model from making a peaked distribution. Label smoothing smoothes target labels with a pre-defined prior label distribution; as a result, a model is learned to maximize the likelihood of predicting the soft label. Nonetheless, the amount of smoothing is the same in all samples and remains fixed in training. In other words, label smoothing does not reflect the change in probability distribution mapped by a model over the course of training. To address this issue, we propose a regularization scheme that brings dynamic nature into the smoothing parameter by taking model probability distribution into account, thereby varying the parameter per instance. A model in training self-regulates the extent of smoothing on the fly during forward propagation. Furthermore, inspired by recent work in bridging label smoothing and knowledge distillation, our work utilizes self-knowledge as a prior label distribution in softening target labels, and presents theoretical support for the regularization effect by knowledge distillation and the dynamic smoothing parameter. Our regularizer is validated comprehensively, and the result illustrates marked improvements in model generalization and calibration, enhancing robustness and trustworthiness of a model.

## introduction in common practice, a neural network is trained to maximize the expected likelihood of observed targets, and the gradient with respect to the objective updates the learnable model parameters. with hard targets (one-hot encoded), the maximum objective can be approached when a model assigns a high probability mass to the corresponding target label * this work was done while dongkyu was an intern at nvidia over the output space. that is, due to the normalizing activation functions (i.e. softmax), a model is trained in order for logits to have a marked difference between the target logit and the other classes logits @xcite . despite its wide application and use, the maximum likelihood estimation with hard targets has been found to incur an overconfident problem; the predictive score of a model does not reflect the actual accuracy of the prediction. consequently, this leads to degradation in model calibration @xcite , as well as in model performance @xcite . additionally, this problem stands out more clearly with a limited number of samples, as a model is more prone to overfitting. to remedy such phenomenon, @xcite proposed label smoothing, in which one-hot encoded targets are replaced with smoothed targets. label smoothing has boosted performance in computer vision @xcite , and has been highly preferred in other domains, such as natural language processing @xcite . however, there are several aspects to be discussed in label smoothing. first, it comes with certain downsides, namely the static smoothing parameter. the smoothing regularizer fails to account for the change in probability mass over the course of training. despite the fact that a model can benefit from adaptive control of the smoothing extent depending on the signs of overfitting and overconfidence, the smoothing parameter remains fixed throughout training in all instances. another aspect of label smoothing to be considered is its connection to knowledge distillation @xcite . there have been attempts to bridge label smoothing and knowledge distillation, and the findings suggest that the latter is an adaptive form of the former @xcite . however, the regularization effect on overconfidence by self-knowledge distillation is still poorly understood and explored. to tackle the issues mentioned above, this work presents adaptive label smoothing with selfknowledge as a prior label distribution. our regularizer allows a model to self-regulate the extent of smoothing based on the entropic level of model probability distribution, varying the amount per sample and per time step. furthermore, our theoretical analysis suggests that self-knowledge distillation and the adaptive smoothing parameter have a strong regularization effect by rescaling gradients on logit space. to the best of our knowledge, our work is the first attempt in making both smoothing extent and prior label distribution adaptive. our work validates the efficacy of the proposed regularization method on machine translation tasks, achieving superior results in model performance and model calibration compared to other baselines. ## approach the core components of label smoothing are twofold: smoothing parameter (α) and prior label distribution. the components determine how much to smooth the target label using which distribution, a process that requires careful choice of selection. in this section, we illustrate how to make the smoothing parameter adaptive. we also demonstrate how our adaptive smoothing parameter and self-knowledge distillation as a prior distribution act as a form of regularization with theoretical analysis on the gradients.