Paper:
REGULARIZING NEURAL NETWORKS BY PENALIZING CONFIDENT OUTPUT DISTRIBUTIONS
We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT’14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.
We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT’14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.
1 INTRODUCTION
Large neural networks with millions of parameters achieve strong performance on image classification (Szegedy et al., 2015a), machine translation (Wu et al., 2016), language modeling (Jozefowicz et al., 2016), and speech recognition (Graves et al., 2013). However, despite using large datasets, neural networks are still prone to overfitting. Numerous techniques have been proposed to prevent overfitting, including early stopping, L1/L2 regularization (weight decay), dropout (Srivastava et al., 2014), and batch normalization (Ioffe & Szegedy, 2015). These techniques, along with most other forms of regularization, act on the hidden activations or weights of a neural network. Alternatively, regularizing the output distribution of large, deep neural networks has largely been unexplored.
To motivate output regularizers, we can view the knowledge of a model as the conditional distribution it produces over outputs given an input (Hinton et al., 2015) as opposed to the learned values of its parameters. Given this functional view of knowledge, the probabilities assigned to class labels that are incorrect (according to the training data) are part of the knowledge of the network. For example, when shown an image of a BMW, a network that assigns a probability of 10−3 to “Audi” and 10−9 to “carrot” is clearly better than a network that assigns 10−9 to “Audi” and 10−3 to carrot, all else being equal. One reason it is better is that the probabilities assigned to incorrect classes are an indication of how the network generalizes. Distillation (Hinton et al., 2015; Bucilu et al., 2006) exploits this fact by explicitly training a small network to assign the same probabilities to incorrect classes as a large network or ensemble of networks that generalizes well. Further, by operating on the output distribution that has a natural scale rather than on internal weights, whose significance depends on the values of the other weights, output regularization has the property that it is invariant to the parameterization of the underlying neural network.
∗Work done as part of the Google Brain Residency Program †Equal Contribution
In this paper, we systematically evaluated two output regularizers: a maximum entropy based confidence penalty and label smoothing (uniform and unigram) for large, deep neural networks on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT’14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters.
2 RELATED WORK
The maximum entropy principle (Jaynes, 1957) has a long history with deep connections to many areas of machine learning including unsupervised learning, supervised learning, and reinforcement learning. In supervised learning, we can search for the model with maximum entropy subject to constraints on empirical statistics, which naturally gives rise to maximum likelihood in log-linear models (see (Berger et al., 1996) for a review). Deterministic annealing Rose (1998) is a general approach for optimization that is widely applicable, avoids local minima, and can minimize discrete objectives, and it can be derived from the maximum entropy principle. Closely related to our work, Miller et al. (1996) apply deterministic annealing to train multilayer perceptrons, where an entropy based regularizer is introduced and slowly annealed. However, their focus is avoiding poor initialization and local minima, and while they find that deterministic annealing helps, the improvement diminishes quickly as the number of hidden units exceeds eight.
In reinforcement learning, encouraging the policy to have an output distribution with high entropy has been used to improve exploration (Williams & Peng, 1991). This prevents the policy from converging early and leads to improved performance (Mnih et al., 2016). Penalizing low entropy has also been used when combining reinforcement learning and supervised learning to train a neural speech recognition model to learn when to emit tokens (Luo et al., 2016). When learning to emit, the entropy of the emission policy was added to the training objective and was annealed throughout training. Indeed, in recent work on reward augmented maximum likelihood (Norouzi et al., 2016), this entropy augmented reinforcement learning objective played a direct role in linking maximum likelihood and reinforcement learning objectives.
Penalizing the entropy of a network’s output distribution has not been evaluated for large deep neural networks in supervised learning, but a closely related idea, label smoothing regularization, has been shown to improve generalization (Szegedy et al., 2015b). Label smoothing regularization estimates the marginalized effect of label-dropout during training, reducing overfitting by preventing a network from assigning full probability to each training example and maintaining a reasonable ratio between the logits of the incorrect classes. Simply adding label noise has also been shown to be effective at regularizing neural networks (Xie et al., 2016). Instead of smoothing the labels with a uniform distribution, as in label smoothing, we can smooth the labels with a teacher model (Hinton et al., 2015) or the model’s own distribution (Reed et al., 2014). Distillation and self-distillation both regularize a network by incorporating information about the ratios between incorrect classes.
Virtual adversarial training (VAT) (Miyato et al., 2015) is another promising smoothing regularizer. However, we did not compare to VAT because it has multiple hyperparameters and the approximated gradient of the local distributional smoothness can be computed with no more than three pairs of forward and back propagations, which is significantly more computation in grid-searching and training than the other approaches we compared to.
3 DIRECTLY PENALIZING CONFIDENCE
Confident predictions correspond to output distributions that have low entropy. A network is overconfident when it places all probability on a single class in the training set, which is often a symptom of overfitting (Szegedy et al., 2015b). The confidence penalty constitutes a regularization term that prevents these peaked distributions, leading to better generalization.
A neural network produces a conditional distribution pθ(y|x) over classes y given an input x through a softmax function. The entropy of this conditional distribution is given by
H(pθ(y|x)) = − ∑ i pθ(yi|x) log(pθ(yi|x)).
To penalize confident output distributions, we add the negative entropy to the negative log-likelihood during training
L(θ) = − ∑ log pθ(y|x)− βH(pθ(y|x)),
where β controls the strength of the confidence penalty. Notably, the gradient of the entropy term with respect to the logits is simple to compute. Denoting the ith logit by zi, then
∂H(pθ)
∂zi = pθ(yi|x) (− log pθ(yi|x)−H(pθ)) ,
which is the weighted deviation from the mean.
3.1 ANNEALING AND THRESHOLDING THE CONFIDENCE PENALTY
In reinforcement learning, penalizing low entropy distributions prevents a policy network from converging early and encourages exploration. However, in supervised learning, we typically want quick convergence, while preventing overfitting near the end of training, suggesting a confidence penalty that is weak at the beginning of training and strong near convergence. A simple way to achieve this is to anneal the confidence penalty.
Another way to strengthen the confidence penalty as training progresses is to only penalize output distributions when they are below a certain entropy threshold. We can achieve this by adding a hinge loss to the confidence penalty, leading to an objective of the form
L(θ) = − ∑ log pθ(y|x)− βmax(0,Γ−H(pθ(y|x)),
where Γ is the entropy threshold below which we begin applying the confidence penalty.
Initial experiments suggest that thresholding the confidence penalty leads to faster convergence at the cost of introducing an extra hyper-parameter. For the majority of our experiments, we were able to achieve comparable performance without using the thresholded version. For the sake of simplicity, we focus on the single hyper-parameter version in our experiments.
3.2 CONNECTION TO LABEL SMOOTHING
Label smoothing estimates the marginalized effect of label noise during training. When the prior label distribution is uniform, label smoothing is equivalent to adding the KL divergence between the uniform distribution u and the network’s predicted distribution pθ to the negative log-likelihood
L(θ) = − ∑ log pθ(y|x)−DKL(u‖pθ(y|x)).
By reversing the direction of the KL divergence, DKL(pθ(y|x)‖u), we recover the confidence penalty. This interpretation suggests further confidence regularizers that use alternative target distributions instead of the uniform distribution. We leave the exploration of these regularizers to future work.
4 EXPERIMENTS
We evaluated the confidence penalty and label smoothing on MNIST and CIFAR-10 for image classification, Penn Treebank for language modeling, WMT’14 English-to-German for machine translation, and TIMIT and WSJ for speech recognition. All models were implemented using TensorFlow (Abadi et al., 2016) and trained on NVIDIA Tesla K40 or K80 GPUs.
4.1 IMAGE CLASSIFICATION
4.1.1 MNIST
As a preliminary experiment, we evaluated the approaches on the standard MNIST digit recognition task. We used the standard split into 60k training images and 10k testing images. We use the last 10k images of the training set as a held-out validation set for hyper-parameter tuning and then retrained the models on the entire dataset with the best configuration.
We trained fully-connected, ReLu activation neural networks with 1024 units per layer and two hidden layers. Weights were initialized from a normal distribution with standard deviation 0.01. Models were optimized with stochastic gradient descent with a constant learning rate 0.05 (except for dropout where we set the learning rate to 0.001).
For label smoothing, we varied the smoothing parameter in the range [0.05, 0.1, 0.2, 0.3, 0.4, 0.5], and found 0.1 to work best for both methods. For the confidence penalty, we varied the weight values over [0.1, 0.3, 0.5, 1.0, 2.0, 4.0, 8.0] and found a confidence penalty weight of 1.0 to work best.
We also plotted the norm of the gradient as training progressed in Figure 2. We observed that label smoothing and confidence penalty had smaller gradient norms and converged more quickly than models regularized with dropout. If the output distributions is peaked on a misclassified example, the model receives a large gradient. This may explain why the regularized models have smaller gradient norms.
4.1.2 CIFAR-10
CIFAR-10 is an image classification dataset consisting of 32x32x3 RGB images of 10 classes. The dataset is split into 50k training images and 10k testing images. We use the last 5k images of the training set as a held-out validation set for hyper-parameter tuning, as is common practice.
For our experiments, we used a densely connected convolutional neural network, which represents the current state-of-the-art on CIFAR-10 (Huang et al., 2016a). We use the small configuration from (Huang et al., 2016a), which consists of 40-layers, with a growth rate of 12. All models were trained for 300 epochs, with a batch-size of 50 and a learning rate 0.1. The learning rate was reduced by a factor of 10 at 150 and 225 epochs. We present results for training without data-augmentation. We found that the confidence penalty did not lead to improved performance when training with data augmentation, however neither did other regularization techniques, including dropout.
For our final test scores, we trained on the entire training set. For label smoothing, we tried smoothing parameter values of [0.05, 0.1, 0.2, 0.3, 0.4, 0.5], and found 0.1 to work best. For the confidence penalty, we performed a grid search over confidence penalty weight values of [0.1, 0.25, 0.5, 1.0, 1.5] and found a confidence penalty weight of 0.1 to work best.
4.2 LANGUAGE MODELING
For language modeling, we found that confidence penalty significantly outperforms label noise and label smoothing. We performed word-level language modeling experiments using the Penn Treebank dataset (PTB) (Marcus et al., 1993). We used the hyper-parameter settings from the large configuration in (Zaremba et al., 2014). Briefly, we used a 2-layer, 1500-unit LSTM, with 65% dropout applied on all non-recurrent connections. We trained using stochastic gradient descent for 55 epochs, decaying the learning rate by 1.15 after 14 epochs, and clipped the norm of the gradients when they were larger than 10.
For label noise and label smoothing, we performed a grid search over noise and smoothing values of [0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5]. For label noise, we found 0.1 to work best. For label smoothing, we found 0.1 to work best. For the confidence penalty, we performed a grid search over confidence penalty weight values of [0.1, 0.5, 1.0, 2.0, 3.0]. We found a confidence penalty weight of 2.0 to work best, which led to an improvement of 3.7 perplexity points over the baseline.
For reference, we also include results of the existing state-of-the-art models for the word-level language modeling task on PTB. Variational dropout (Gal, 2015) applies a fixed dropout mask (stochastic for each sample) at each time-step, instead of resampling at each time-step as in traditional dropout. Note, that we do not include the variational dropout results that use Monte Carlo (MC) model averaging, which achieves lower perplexity on the test set but requires 1000 model evaluations, which are then averaged. Recurrent highway networks (Zilly et al., 2016) currently represent the state-of-the-art performance on PTB.
4.3 MACHINE TRANSLATION
For machine translation, we evaluated the confidence penalty on the WMT’14 English-to-German translation task using Google’s production-level translation system Wu et al. (2016). The training set consists of 5M sentence pairs, and we used newstest2012 and newtests2013 for validation and newstest2014 for testing. We report tokenized BLEU scores as computed by the multi-bleu.perl script from the Moses translation machine translation package.
Our model was an 8-layer sequence-to-sequence model with attention (Bahdanau et al., 2014). The first encoder was a bidirectional LSTM, the remaining encoder and decoder layers were unidirectional LSTMs, and the attention network was a single layer feed-forward network. Each layer had 512 units (compared to 1024 in (Wu et al., 2016)). The model was trained using 12 replicas running concurrently with asynchronous updates. Dropout of 30% was applied as described in (Zaremba et al., 2014). Optimization used a mix of Adam and SGD with gradient clipping. Unlike (Wu et al., 2016), we did not use reinforcement learning to fine-tune our model. We used a beam size of 12 during decoding. For more details, see (Wu et al., 2016).
For label smoothing, we performed a grid search over values [0.05, 0.1, 0.2, 0.3, 0.4, 0.5] and found 0.1 to work best for both uniform and unigram smoothing. For the confidence penalty, we searched over values of [0.5, 2.5, 4.5] and found a value of 2.5 to work best . For machine translation, we found label smoothing slightly outperformed confidence penalty. When applied without dropout, both lead to an improvement of just over 1 BLEU point (dropout leads to an improvement of just over 2 BLEU points). However, when combined with dropout, the effect of both regularizers was diminished.
4.4 SPEECH RECOGNITION
4.4.1 TIMIT
In the TIMIT corpus, the training set consists of 3512 utterances, the validation set consists of 184 utterances and the test set consists of 192 utterances. All 61 phonemes were used during training and decoding, and during scoring, these 61 phonemes were reduced to 39 to compute the phoneme error rate (PER).
As our base model, we used a sequence-to-sequence model with attention. The encoder consisted of 3 bidirectional LSTM layers, the decoder consisted of a single unidirectional LSTM layer, and the attention network consisted of a single layer feed-forward network. All layers consisted of 256 units. Dropout of 15% was applied as described in Zaremba et al. (2014). We trained the model with asynchronous SGD with 5 replicas. We used a batch size of 32, a learning rate of 0.01, and momentum of 0.9. Gradients were clipped at 5.0. For more details, see Norouzi et al. (2016).
For label smoothing, we performed a grid search over values [0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.8] and found 0.2 to work best. For the confidence penalty, we performed a grid search over values of [0.125, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0] and found a value of 1.0 to work best. Label smoothing led to an absolute improvement over the dropout baseline of 1.6%, while the confidence penalty led to an absolute improvement of 1.2%.
4.4.2 WALL STREET JOURNAL
For the WSJ corpus we used attention-based sequence-to-sequence networks that directly predicted characters. We used the SI284 subset for training, DEV93 for validation, and EVAL92 for testing. We used 240-dimensional vectors consisting of 80-bin filterbank features augmented with their deltas and delta-deltas with per-speaker normalized mean and variances computed with Kaldi Povey et al. (2011). We did not use text-only data or separate language models during decoding.
Network architecture details were as follows. The encoder of the network consisted of 4 bidirectional LSTM layers each having 256 units, interleaved with 3 time-subsampling layers, configured to drop every second frame (Bahdanau et al., 2016; Chan et al., 2015). The decoder used a single LSTM layer with 256 units. The attention vectors were computed with a single layer feedforward network having 64 hidden units and the convolutional filters as described in Chorowski et al. (2015). Weights were initialized from a uniform distribution [−0.075, 0.075]. All models used weight decay of 10−6, additive Gaussian weight noise with standard deviation 0.075, applied after 20K steps, and were trained for 650K steps. We used the ADAM optimizer asynchronously over 8 GPUs. We used a learning rate of 10−3, which was reduced to 10−4 after 400K and 10−5 after 500K steps.
We tested three methods of increasing the entropy of outputs: the confidence penalty and two variants of label smoothing: uniform and unigram. All resulted in improved Word Error Rates (WER), however the unigram smoothing resulted in the greatest WER reduction, and we found it to be least sensitive to its hyperparameter (the smoothing value). Furthermore, uniform smoothing and confidence penalty required masking network outputs corresponding to tokens that never appeared as labels, such as the start-of-sequence token.
Table 6 compares the performance of the regularized networks with several recent results. We observe that the benefits of label smoothing (WER reduction from 14.2 to 11) improve over the recently proposed Latent Sequence Decompositions (LSD) method (Chan et al., 2016) which reduces the WER from 14.7 to 12.9 by extending the space of output tokens to dynamically chosen character n-grams.
5 CONCLUSION
Motivated by recent successes of output regularizers (Szegedy et al., 2015b; Xie et al., 2016), we conduct a systematic evaluation of two output regularizers: the confidence penalty and label smoothing. We show that this form of regularization, which has been shown to improve exploration in reinforcement learning, also acts as a strong regularizer in supervised learning. We find that both the confidence penalty and label smoothing improve a wide range of state-of-the-art models, without the need to modify hyper-parameters.
ACKNOWLEDGMENTS
We would like to thank Sergey Ioffe, Alex Alemi and Navdeep Jaitly for helpful discussions. We would also like to thank Prajit Ramachandran, Barret Zoph, Mohammad Norouzi, and Yonghui Wu for technical help with the various models used in our experiments. We thank the anonymous reviewers for insightful comments.
6 GRADIENT NORMS

Review:
The authors propose a simple idea. They penalize confident predictions by using the entropy of the predictive distribution as a regularizer. The authors consider two variations on this idea. In one, they penalize the divergence from the uniform distribution. In the other variation, they penalize distance from the base rates. They term this variation "unigram" but I find the name odd as I've never seen multi-class labels described as unigrams before. What would a bigram be? 
The idea is simple,  and while it's been used in the context of reinforcement learning, it hasn't been popularized as a regularizer for improving generalization in supervised learning. 
The justifications for the idea still lacks analysis. And the author responses comparing it to L2 regularization have some holes. A simple number line example with polynomial regression makes clear how L2 regularization could prevent a model from badly overfitting to accommodate every data point. In contrast, it seems trivial to fit every data point and satisfy arbitrarily high entropy. Of course, the un-regularized optimization is to maximize log likelihood, not simply to maximize accuracy.  And perhaps something interesting may be happening at the interplay between the log likelihood objective and the regularization objective. But the paper doesn't indicate precisely what.
I could imagine the following scenario: when the network outputs probabilities near 0, it can get high loss (if the label is 1). The entropy regularization could be stabilizing the gradient, preventing sharp loss on outlier examples. The regularization then might owe mainly to faster convergence. Could the authors analyze the effect empirically, on the distribution of the gradient norms? 
The strength of this paper is its empirical rigor. The authors take their idea and put it through its paces on a host of popular and classic benchmarks spanning CNNs and RNNs. It appears that on some datasets, especially language modeling, the confidence penalty outperforms label smoothing. 
At present, I rate this paper as a borderline contribution but I'm open to revising my review pending further modifications. 
Typo:
In related work: "Penalizing entropy" - you mean penalizing low entropy