Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
Abstract
Modern deep learning is primarily an experimental science, in which empirical advances occasionally come at the expense of probabilistic rigor.
Here we focus on one such example; namely the use of the categorical cross-entropy loss to model data that is not strictly categorical, but rather takes values on the simplex.
This practice is standard in neural network architectures with label smoothing and actor-mimic reinforcement learning, amongst others.
Drawing on the recently discovered continuous-categorical distribution, we propose probabilistically-inspired alternatives to these models, providing an approach that is more principled and theoretically appealing.
Through careful experimentation, including an ablation study, we identify the potential for outperformance in these models, thereby highlighting the importance of a proper probabilistic treatment, as well as illustrating some of the failure modes thereof.
Introduction
The cross-entropy loss is one of the most commonly used loss functions for training deep neural network models, most notably in (multi-class) classiﬁcation problems.
When applied to categorical data, this loss function corresponds to a probabilistic log-likelihood, therefore resulting in favorable estimation properties.
On the other hand, several prominent methods in modern machine learning are concerned with ﬁtting data that is not quite categorical, but simplex-valued; key examples being the “soft targets” in label smoothing (LS) [27], and “expert policies” in actor-mimic reinforcement learning (AMN) [23], amongst others [16, 28].
In these methods, the deep learning community has defaulted to borrowing the same cross-entropy loss from the categorical case, despite the fact that it no longer deﬁnes a bona ﬁde probability model.
As well as highlighting this practice and putting it into question, our work proposes adjusting the LS and AMN objective functions by replacing the cross-entropy loss with the log-likelihood of the recently discovered continuous-categorical (CC) distribution [12].
Doing so amounts to incorporating a normalizing constant to our model, or in other words, adding the factor that scales the cross-entropy loss to a valid probability density function over the simplex.
As of yet, such an approach has only been considered in the context of knowledge distillation [12], although the one-dimensional special case (corresponding to the binary cross-entropy with [0 , 1] -valued data) has been studied more extensively [20].
Our inspiration draws from both of these works, although our focus is primarily on LS and AMN architectures instead.
Our exposition is organized as follows (note that our two main sections, 3 and 4, are based on the same idea, but are broadly independent of one another and can be read separately):
• In section 2 we detail the relevant background on the continuous-categorical distribution, highlighting its close connection to the cross-entropy loss.
• Section 3 focuses on label smoothing.
We propose a novel CC-LS model and perform an ablation study to isolate its potential as a regularizer for classiﬁcation networks, as well as a qualitative assessment of its learned representations.
• Section 4 focuses on actor-mimic reinforcement learning.
We recast the AMN model as the solution to a regression problem of simplex-valued data, we propose a novel CC-AMN model, and we provide an experimental evaluation thereof. • Section 5 concludes, combining insights from CC-LS and CC-AMN, and discussing potential directions for future research.
Background
We preface the introduction of the continuous-categorical distribution with a brief notational overview of the categorical cross-entropy loss, which will highlight the close connection between the two and will provide an orthogonal viewpoint to its original presentation in [12].
Categorical data refers to observations y that take values in a discrete sample space Ω formed by K distinct elements, which are typically expressed using the K one-hot vectors that form the standard basis of R K , namely Ω = { e 1 , . . .
e K } , where ( e k )
j = 1 ( k = j ) .
In this notation, the cross-entropy loss is equivalent to the negative log-likelihood of y ∈ Ω under a categorical distribution with parameter π :
[Equation]
In other words, the cross-entropy loss deﬁnes a coherent probabilistic model for discrete data over
K classes.
This elementary fact should not be overlooked; it provides the beneﬁts of the theory of maximum likelihood estimation, including frequentist consistency and asymptotic efﬁciency, as well as enabling efﬁcient Bayesian inference by specifying a conjugate prior.
2.1 From the Cross-Entropy to the Continuous-Categorical
So far so good.
However, what happens when the observation is not quite categorical, but instead takes values on the simplex, ∆ K = { y ∈ R K + : (cid:80)
Kk =
y k
= 1 } ?
Such data is called compositional , and is common in the sciences [1].
In the deep learning literature, while not explicitly referred to as such, compositional data plays a key role in label smoothing [27], actor-mimic reinforcement learning [23], knowledge distillation [16], and domain adaptation [28].
In all of these methods, neural networks are trained to target a simplex-valued outcome, y ∈ ∆ K , using the cross-entropy loss l ( λ ; y )
= − (cid:80)
Kk
=
y k log λ k , where λ represents the output of a neural network.
Crucially though, the change in sample space from Ω to ∆ K breaks the equivalence in (1) because the right-hand expression no longer deﬁnes a proper probability distribution; its integral over ∆ K does not normalize to 1.
Given the attractive properties of maximum likelihood estimation, there are still good reasons why a legitimate probability model is desirable (see [20] for a more detailed discussion).
The classical statistics literature offers some possibilities, notably the use of logratios [1, 2, 3, 9], or Dirichlet regression [6, 15].
However, we argue that the most natural probabilistic solution is to apply the recently discovered continuous-categorical distribution [12], since this corresponds to normalizing the cross-entropy loss directly so that it becomes a genuine log-likelihood model, namely:
[Equation]
where C ( λ ) is the normalizing constant:
[Equation]
This distribution was found to possess a number of attractive theoretical and empirical properties [12]; we highlight the closed form expression of its normalizing constant:
[Equation]
which enables the use of automatic differentiation for optimizing models with the continuouscategorical log-likelihood (2)
We also highlight that the continuous-categorical outperformed the
Dirichlet distribution in regression models of compositional data, including neural network models [12].
Continuous-Categorical Label Smoothing
Label smoothing [27] has enjoyed rapid growth and widespread use as a means to reduce overﬁtting and improve the out-of-sample accuracy of neural network classiﬁers across a range of tasks including computer vision [31, 24], speech recognition [8], and machine translation [30].
The mechanism is simple: given a neural network classiﬁer f θ : x → y , we replace our one-hot labels y ∈ Ω with “soft” targets:
[Equation]
where u = (1 /K, . . . , 1 /K )
(cid:62) is a uniform vector and ε > 0 is a constant.
The network weights θ are then trained to minimize the cross-entropy loss between the network output f θ ( x ) and the modiﬁed data y LS .
Equation 5 maps y ∈ Ω to y LS ∈ ∆ K , so that our targets are no longer categorical, but simplexvalued.
Thus, even though they are not continuously distributed, it is natural to consider labelsmoothed classiﬁcation through the lens of compositional regression.
Our proposal is therefore to use a continuous-categorical log-likelihood in lieu of the cross-entropy loss, and we refer to this model as
CC-LS.
Namely, we are interested in comparing the usual label smoothing loss:
[Equation]
against its continuous-categorical counterpart:
[Equation]
We remark that, strictly speaking, in order to make our targets continuous over the simplex, we would also have to add continuous noise to the labels, for example by drawing u uniformly at random on the simplex.
Such an approach produced little difference over using the ﬁxed value u
= (1 /K, . . .
, 1 /K )
(cid:62) , neither in LS nor CC-LS, and we will omit the results for clarity.
However, given the wealth of existing methods that achieve improved generalization error by adding noise at different stages in the training procedure [5, 26, 25], the idea of smoothing the labels with random noise may still hold potential, and we leave its further analysis for future work.
3.1 Experiments
Following the experimental setup of Muller et al [22], we train a CNN classiﬁer on CIFAR-10, with and without label smoothing as well as our novel CC-LS model (see appendix A.1 for the full details of our architecture).
This is an example in which label smoothing provided no signiﬁcant gain over the un-smoothed baseline, likely because the CNN is already regularized using dropout [26], weight decay [18], and batch normalization [17], which altogether are sufﬁcient to provide a good model of [Table] the data, given the level of complexity of CIFAR-10.
Likewise, we ﬁnd that the CC-LS model also performs no better than the baseline in this setting (top row of Table 1).
Driven by these observations
[Table]
we perform an ablation study over the different regularizers used in our network, and the results paint a more interesting picture (Table 1).
Notably, we ﬁnd that for the unregularized CNN (bottom row), CC-LS signiﬁcantly outperforms both LS and the baseline.
In the case where our network is partially regularized with dropout only, the baseline becomes equally good as LS, but the gap with CC-LS remains wide (penultimate row), and the gain from CC-LS persists after adding weight decay.
On the other hand, batch normalization (top half) was sufﬁcient to capture all the gain in test accuracy, with neither LS nor CC-LS outperforming the baseline in these cases.
Under weight decay without batch normalization (rows 5 and 6), training became less stable (as evidenced by the large standard deviations), but CC-LS was able to reduce the variability in model accuracy.
Overall, Table 1 indicates that the CC-LS loss function provides a different (and sometimes, signiﬁcantly better) regularization effect than that of vanilla LS, suggesting its potential for novel applications, particularly in the settings where batch normalization may be undesirable [10].
We note further that numerous existing works have been devoted to analyzing the interplay between dropout, weight decay, and batch normalization [29, 11, 7, 19, 14]; our focus is speciﬁcally on their relation to label smoothing and CC-LS.
[Table]
We end this section with a qualitative analysis of the learned representations from our trained classiﬁers.
Again following [22], we deﬁne the “template” vector of the k th class, w k , as the weight vector from the last CNN layer that is associated to the k th class, so that in other words:
[Equation]
where z is a vector containing the activations from the penultimate layer.
We then ﬁx three classes, and construct an orthonormal basis (consisting of two vectors) for the plane containing their three template vectors.
For each of the classes, we pick a random sample of input data belonging to that class and project their penultimate layer activations onto this plane.
The results are shown in Figure 1 for the (arbitrarily chosen) classes “airplane”, “automobile”, and “bird”.
As was noted by [22], while label smoothing can help the classiﬁer achieve better accuracy on the test set, it comes at the cost of a less informative learned representation, as can be seen from the more concentrated centroids in the second column relative to the ﬁrst.
On the other hand, CC-LS achieves a somewhat richer representation than vanilla LS, as can be observed from the greater within-cluster variances in the third column, which we quantify in Table 2.
This suggests that the CC may offer additional potential for combining LS with teacher models in the context of knowledge distillation, a setting in which the concentrated clusters enforced by LS proved detrimental to the training of a student model [22].
[Figure]
Probabilistic Actor-Mimic Reinforcement Learning
In this section we summarize the Actor-Mimic Reinforcement Learning framework [23] and recast it as a compositional regression problem, highlighting the potential for a probabilistic model with the continuous-categorical distribution.
Actor-Mimic Networks (AMN) provide a method for multitask and transfer reinforcement learning.
The goal of the AMN is to train a single agent to perform on several different “source games”, { G 1 , . .
, G L } , each of which corresponds to a Markov Decision Process deﬁned on a common state space and action set (shared across source tasks), but driven by a different set of transition probabilities and reward functions (speciﬁc to each task).
Formally, G i = ( S , A , T i , R i ) , where S is the set of states, A is the set of actions, T i ( s (cid:48) | s, a ) is the probability of transitioning from state s to state s (cid:48) when executing action a in game G i , and
R i is the reward function mapping states and actions to real-valued rewards representing the score of the i th game.
In practice, each G i corresponds to a different videogame from the Atari Learning Environment [4]; each game follows a different set of rules ( T i and R i ) while taking place on the same console display ( S ) and controller ( A ).
In order to train an AMN, we ﬁrst require access to a set of “experts” { E 1 , . . .
, E L } , each of which corresponds to an agent specialized in one of the source games.
Expert E i represents a policy π E i mapping states to distributions over actions, so that we can write π E i ( a k | s t ) for the probability that E i chooses action a k ∈ A when in state s t ∈ S .
Note that k indexes the action space, so that
A = { a 1 , . . .
, a K } , whereas t indexes time (i.e., frame number), so that S ⊇ { s 1 , s 2 , . . .
In our implementation, E i corresponds to a Deep Q-Network (DQN) [21] trained on game G i , though the fact that E i is a DQN is not necessary – any policy that performs well on G i will sufﬁce.
Given the set of expert policies, the AMN is trained to “mimic” the experts in their respective source games.
We can reformulate the method as a two-stage process.
First, we form an auxiliary dataset of “guidance vectors”, D aux
=
(cid:110)
y ( i ) t (cid:111) .
These vectors are obtained by generating, for each game G i , a sequence of states (cid:110)
s ( i ) t (cid:111)
n t =1 , and then feeding these states through the corresponding expert policies,
[Table]
in other words:
[Equation]
[Equation]
Second, the parameters of our Actor-Mimic Network, π AM θ , are learned by minimizing the categorical cross-entropy loss with respect to the auxiliary data:
[Equation]
In practice, we minimize this loss using minibatch stochastic gradient descent, running the gameplaygeneration in parallel with the gradient steps.
The effectiveness of the AMN approach is fundamentally computational; the expert policies can be trained independently in parallel, and the AMN is much faster to optimize via the cross-entropy loss (10) than using policy gradients, as it is able to leverage the rich information from the expert policies directly, (with an entire guidance vector of probabilities containing information for all classes at each time step, rather than learning from noisy and biased n -step bootstrap estimates).
Since y ( i ) t is a vector of probabilities over actions, our auxiliary data is simplex-valued rather than categorical.
It is therefore clear from Equation 10 that the AMN model solves a compositional regression problem, whence we propose replacing the cross-entropy loss with its probabilistic counterpart, the continuous-categorical log-likelihood:
[Equation]
We call this model CC-AMN, and we compare its performance against the AMN model, as well as the DQN baseline.
4.1 Experiments
We follow the experimental setup of Parisotto et al [23], choosing a subset of games from the Atari
Learning Environment in which the DQN model performed at super-human level.
For each game, we pre-train a DQN with the same network architecture; these are then used as the expert policies.
Our network architecture, described in appendix A.2, is taken directly from [21], and is also used for the
AMN and CC-AMN models.
First, we reproduce the results of [23] and compare with our novel CC-AMN model, as shown in
Table 3.
The evaluation scores of the CC-AMN are similar to those of the AMN, except for the game of Pong, where using the CC likelihood leads to unstable training, resulting in worse performance and higher variability in the evaluation score.
Note that both AMN and CC-AMN are generally able to achieve similar performance to the expert DQN.
Second, we focus speciﬁcally on the effect of the probabilistic objective (11) on network training by reducing the multi-task objective to a single-task objective, i.e., we no longer sum over i in Equation 10.
This corresponds to running AMN and CC-AMN against the expert DQN, E i , of a single game, and we do this separately for each game, as shown in Figure 2.
While both CC-AMN and AMN are able to train much faster than the DQN, converging in just a few epochs, CC-AMN fails to outperform
AMN, and can be slower to converge (Breakout) or worse overall (Pong).
The case of Pong highlights an important failure mode of CC-AMN, which also offers some insight as to why our model underperforms in the other games.
The issue originates in the normalizing constant
[Figure]
Breakout
[Figure]
Atlantis
[Figure]
SpaceInvaders
(4), which is numerically unstable when the parameter λ is close to uniform, due to the product of log-ratios vanishing in the in the denominator, as was noted in [12].
In the case of CC-AMN, our optimization hovers around this unstable region, since the guidance vectors tend to concentrate around the centroid of the simplex (this is because our expert policies correspond to the softmax of
Q-value functions, which don’t typically exhibit large variability across actions since, over small time steps, most actions are not individually critical to the outcome of the game).
In practice this is not necessarily a problem, as we zero out the unstable gradients during optimization, as in [12].
Doing so provides a reasonable approximation, since ∇ G ( λ ) = 0 for λ
= (1 /K, . . .
, 1 /K ) , by symmetry ( (cid:80)
k
λ k
= 1 is constrained by deﬁnition of the continuous-categorical).
Nevertheless, this behavior likely results in a worse optimization landscape overall (unlike in CC-LS where y , and hence λ , are far away from the centroid), which in the game of Pong, derails our gradient search altogether.
Further investigation may involve re-running our experiments with arbitrary-precision ﬂoating point, though we currently ﬁnd this to be computationally prohibitive.
Discussion and Future Work
Comparing our experiments on CC-LS and CC-AMN, it may come as a surprise that the former yields the more promising empirical results, in spite of its less rigorous theoretical underpinning (with targets that are simplex-valued, but not genuinely continuous).
This observation suggests that the continuous-categorical may provide useful modeling advances outside the realm of compositional data analysis and probabilistic modeling, for example in classiﬁcation problems.
It is also worth noting that, throughout our experiments, the cross-entropy loss may have beneﬁtted disproportionately from favorable network initialization, which has been developed for and become increasingly specialized toward networks with particular loss functions [13].
A similar argument can be made about network architecture, noting that a good architecture for the cross-entropy loss may not be equivalent to a good architecture for its continuous-categorical counterpart.
In fact, we have observed experimentally that additional architecture or hyperparameter search can lead to improved performance for the CC-based approaches. However we deliberately chose not to focus on such experimentation, as doing so could further entangle the effect of the loss function on our models; we leave such analyses to future work.
At a more theoretical level, as identiﬁed in [12], it follows from the properties of exponential families that optimizing the continuous-categorical log-likelihood results in an unbiased estimator for its mean parameter, which corresponds to a (local) average of the observed data.
This is, in fact, akin to the cross-entropy loss, which is also maximized at local average that approximates the conditional expectation of the outputs given the inputs.
The optimization landscapes deﬁned by the two loss functions could therefore be of similar nature, though this remains an open question.
Last, we highlight a computational limitation of our approach: the numerical instabilities noted in section 4.1 are exacerbated in high dimensions.
In fact, evaluating Equation 4 for much more than classes is problematic for all λ ∈ ∆ K (not only for λ around the centroid), since the K summands typically cancel out beyond numerical precision.
2 As a result, we constrained our experimentation to examples where K ≤ 10 , however, similar applications with K ∼ 100 or greater are also of interest.
Further advances in theory or numerical analysis will be needed to enable successful applications of the continuous-categorical at this scale.
We conclude by noting that, taken together with the theoretical and empirical results in [20] and [12], our work suggests that future methodological advances may be possible through a combination of careful probabilistic consideration of the cross-entropy loss and the use of the continuous-categorical distribution. Acknowledgments and Disclosure of Funding We thank Andres Potapczynski and the anonymous reviewers for helpful conversations and the Simons Foundation, Sloan Foundation, McKnight Endowment Fund, NSF and the Gatsby
Charitable Foundation for support.
A Experimental details A.1 Label Smoothing
We denote a convolutional layer by W × W × N-S, where W is the width of the convolution, N the number of ﬁlter maps, and S the stride.
Our architecture is: 3 × 3 × 32-1 → BatchNorm → 3 × 3 ×
→ BatchNorm → MaxPooling (2 × 2) →
Dropout (0.2) → 3 × 3 × 64-1 → BatchNorm → 3 × 3 ×
64- → BatchNorm → MaxPooling (2 × 2) →
Dropout (0.3) → 3 × 3 × 128-1 → BatchNorm → 3 × 3 × 1281 → BatchNorm → MaxPooling (2 × 2) →
Dropout (0.4) → 10 fully-connected units.
We use weight decay of 0.0001 in the ﬁnal fully-connected layer, which, together with dropout and batch normalization, was switched on and off in the different runs of our ablation study in Table 1.
Our models were trained for 500 epochs using a minibatch size of 128 and the Adam optimizer with a learning rate of 10 − 3 .
The label smoothing hyperparameter ε was set to 0.1 as per [22].
Note however, that we were unable to replicate the results of [22] exactly, as they did not share their code, nor did they describe their architecture in full.
A.2 Actor-Mimic Network
Our architecture is 8 × 8 × 32-4 → 4 × 4 × 64-2 → 3 × 3 × 64-1 → 7 × 7 × 1024-1 → 512 fully-connected units → 6 fully connected units (corresponding to 6 possible actions).
We used the Adam optimizer with a learning rate of 10 − 5 , and a minibatch size of 32.