Generalized Entropy Regularization or: There’s Nothing Special about Label Smoothing (2020)

Abstract: Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i.e. over-confident) predictions, a common sign of overfitting. This class of techniques, of which label smoothing is one, has a connection to entropy regularization. Despite the consistent success of label smoothing across architectures and data sets in language generation tasks, two problems remain open: (1) there is little understanding of the underlying effects entropy regularizers have on models, and (2) the full space of entropy regularization techniques is largely unexplored. We introduce a parametric family of entropy regularizers, which includes label smoothing as a special case, and use it to gain a better understanding of the relationship between the entropy of a model and its performance on language generation tasks. We also find that variance in model performance can be explained largely by the resulting entropy of the model. Lastly, we find that label smoothing provably does not allow for sparsity in an output distribution, an undesirable property for language generation models, and therefore advise the use of other entropy regularization methods in its place.

## introduction when training large neural networks with millions of parameters, regularization of some form is needed to prevent overfitting, even when large amounts of data are used; models for language generation are no exception. in probabilistic modeling, e.g. when the final layer of the neural network is a softmax, overfitting often manifests itself in overconfident placement of most of the probability mass on a few candidates, resulting in peaky (low-entropy) probability distributions over the vocabulary. specifically for language generation tasks, this behavior leads to the output of repetitive or frequently occurring but unrelated text, which is detrimental to the generalization abilities of the model @xcite . a natural regularizer to consider is, therefore, one that penalizes overconfidence, encouraging higher entropy in the learned distribution. indeed, the literature has ascribed gains of ≈ 1 bleu point in machine translation to label smoothing, one such technique @xcite . despite the clear relationship between low entropy and overfitting, only a handful of distinct entropy regularizers have been explored. to fill this gap, we introduce generalized entropy regularization (ger), a unified framework for understanding and exploring a broad range of entropyinducing regularizers. ger is based on the skew-jensen family of divergences j α,g @xcite and thus may be generalized to any bregman divergence through the choice of generator function g. for the negative entropy generator function, ger recovers label smoothing @xcite as α → 1, and the confidence penalty @xcite as α → 0. we provide formal properties of ger in §3, proving these special-case equivalences among other characteristics of ger. we then use ger to examine the relationship between entropy and the evaluation metrics in two language generation tasks: neural machine translation (nmt) and abstractive summarization. ger encompasses a large family of regularizers, which allows us to directly compare label smoothing to other forms of entropy regularization. by studying the relationship between different regularizers on the performance of natural language generation (nlg) systems, we can better understand not just when but also why label smoothing aids language generation tasks. through our analysis, we gain the following insights: in summary, entropy-inducing regularizers are a boon to probabilistic nlg systems, which benefit from higher entropy output distributions. label smoothing works because it forces the model towards a higher-entropy solution, but we recommend the confidence penalty and other entropy regularizers (α < 1) for reasons (i) and (ii) above. ## preliminaries in this work, we consider conditional probability models p θ (y | x) for natural language generation; such models assign probability to a target sequence @xmath0. specifically, our target sequence @xmath1, . . . , y n of arbitrary length n is a sequence of target words 1 y i from our vocabulary y . the set of all complete target sequences, which are padded with distinguished beginning-and end-of-sentence symbols, bos and eos, is then defined as y := {bos • y • eos | @xmath2. for language generation tasks, p θ (y | x) is typically a neural network with parameters θ; this network is often trained to approximate p(y | x), the empirical distribution (i.e. the distribution of the data). here, we focus on locally normalized models; in such models p θ (y | x) is factored as: one way of selecting the parameters θ is to minimize the kl-divergence between the empirical 1 targets yi may also be characters or subwords; our experiments use byte-pair encoding @xcite distribution and the model. this yields the crossentropy loss (plus an additive constant): 2 however, fitting a model that perfectly approximates the empirical distribution is, in general, fraught with problems @xcite . the goal of learning is to generalize beyond the observed data. exactly fitting the empirical distribution, often termed overfitting, is therefore not an ideal goal and for language generation models specifically, does not go hand-in-hand with the ability of a model to generate desirable text @xcite . consequently, it is advisable to minimize a regularized objective to prevent overfitting: ## generalized entropy regularization the positive effect of both label smoothing and the confidence penalty on model performance in language generation tasks motivates further exploration of entropy-promoting regularizers. to this end, we construct a parameterized family of regularizers with label smoothing and the confidence penalty as special cases. we discuss the formal properties of a subset of this family, providing upper and lower bounds for it. we show divergence only occurs in one case for this subset (α → 1), which directly implies that no sparse solution exists when label smoothing is used as a regularizer. ## experiments we evaluate our family of entropy regularizers on two language generation tasks: machine translation and abstractive summarization. we then analyze trends in model performance as a function of α and model entropy 9 and explore how this entropy affects other properties of language generation models. in the following experiments, each model is trained using eq. ( 4 ) where r(θ) = d jα (p || p θ ). we conduct searches over α and β using bayesian optimization @xcite to find the combination of regularizer d jα and strength coefficient β that lead to the lowest loss on the development set for the respective task. 10 we additionally do a more fine-grained grid search over β for j 0 (confidence penalty) and j 1 (label smoothing) for completeness. all other model hyperparameters are held constant. we run experiments on multiple architectures and across several data sets to ensure trends are general. ## discussion our experiments show entropy regularization has a number of beneficial effects on natural language generation models. clearly, low-entropy predictions, which are more aligned with the empirical distribution (figure 8 ), are a sign of overfitting in a model since they lead to poor generalization abilities (figure 3 ). in other words, we observe that closely approximating the empirical distribution is at odds with a well calibrated model, i.e. a model p θ (y | x) that matches the true, underlying probabilities p(y | x). 12 entropy regularization appears to alleviate this problem; namely, for more regularized models, figure 3 shows increased evaluation metric scores and figure 8 demonstrates an increase in the log-likelihood of the reference sequence relative to the highest probability sequence. decoding. overconfident predictions inhibit the ability to recover after a poor choice of words during decoding; chorowski and jaitly (2017) suggest that higher-entropy models p θ , like the ones resulting from regularization with label smoothing, would alleviate this problem. results throughout this paper support this hypothesis not just for label smoothing, but for the d jα family of entropy regularizers as well. ## related work entropy regularization has a long history in reinforcement learning @xcite @xcite where it has provided substantial improvements in exploration. such methods have since been adapted for supervised learning where they have proven to be reliable forms of regularization for various probabilistic modeling tasks @xcite . more recently, interpolating between exclusive and inclusive kl divergences has been explored in nmt by @xcite . however, this method was used for the objective function (i.e. between p and p θ ) and not as a regularization technique (i.e. between a baseline distribution q and p θ ). @xcite construct a baseline distribution q as a function of word embedding distances to to use in place of the uniform distribution u in the label smoothing equation. this work is complementary to ours, as q can similarly be used in place of u with ger. finally, our work is closest to that of @xcite , which attempts to find the circumstances under which label smoothing has a positive effect on model performance. however, they do not explore entropy regularization on the whole nor do they attempt to provide an explanation for why label smoothing works. we attempt to answer the "why" question through a quantitative analysis of label smoothing and empirical exploration of the relationship between model entropy and performance. ## conclusion we discuss the properties of generalized entropy regularization and provide empirical results on two language generation tasks. we find entropy regularization leads to improvements over baseline systems on evaluation metrics for all values of the parameter α with our regularizer d jα . theoretical and empirical evidence show label smoothing adds undesirable constraints to the model and is the hardest to tune of the regularizers tested. we therefore advocate the use of alternate forms of entropy regularization for language generation tasks.