S MOOTH L OSS F UNCTIONS FOR D EEP T OP - K
C
LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Human labeling of data constitutes a long and expensive process.
As a consequence, many classiﬁcation tasks entail incomplete annotation and incorrect labels, while being built on a restricted amount of data.
In order to handle the ambiguity and the label noise, the performance of machine learning models is usually assessed with top- k error metrics rather than top- 1 .
Theoretical results suggest that to minimize this error, various loss functions, including cross-entropy, are equally optimal choices of learning objectives in the limit of inﬁnite data.
However, the choice of loss function becomes crucial in the context of limited and noisy data.
Besides, our empirical evidence suggests that the loss function must be smooth and have non-sparse derivatives to work well with deep neural networks.
Consequently, we introduce a family of smoothed loss functions that are suited to top- k optimization via deep learning.
The widely used cross-entropy is a special case of our family.
Evaluating our smooth loss functions is computationally challenging: a na ¨ ıve algorithm would require O ( (cid:0) Ck (cid:1) ) operations, where C is the number of classes.
Thanks to a connection to polynomial algebra and a divideand-conquer approach, we provide an algorithm with a time complexity of O ( kC ) .
Furthermore, we present a novel and error-bounded approximation to obtain fast and stable algorithms on GPUs with single ﬂoating point precision.
We compare the performance of the cross-entropy loss and our margin-based losses in various regimes of noise and data size, for the predominant use case of k = 5 . Our investigation reveals that our loss is more robust to noise and overﬁtting than cross-entropy.
I NTRODUCTION
In machine learning many classiﬁcation tasks present inherent label confusion.
The confusion can originate from a variety of factors, such as incorrect labeling, incomplete annotation, or some fundamental ambiguity that obfuscate the ground truth label even to a human expert.
For example, consider the images from the ImageNet data set (Russakovsky et al., 2015) in Figure 1, which illustrate the aforementioned factors.
To mitigate these issues, one may require the model to predict the k most likely labels, where k is typically very small compared to the total number of labels.
Then the prediction is considered incorrect if all of its k labels differ from the ground truth, and correct otherwise.
This is commonly referred to as the top- k error.
Learning such models is a longstanding task in machine learning, and many loss functions for top- k error have been suggested in the literature.
In the context of correctly labeled large data, deep neural networks trained with cross-entropy have shown exemplary capacity to accurately approximate the data distribution.
An illustration of this phenomenon is the performance attained by deep convolutional neural networks on the ImageNet challenge.
Speciﬁcally, state-of-the-art models trained with cross-entropy yield remarkable success on the top- 5 error, although cross-entropy is not tailored for top- 5 error minimization.
This phenomenon can be explained by the fact that cross-entropy is top- k calibrated for any k (Lapin et al., 2016), an asymptotic property which is veriﬁed in practice in the large data setting.
However, in cases where only a limited amount of data is available, learning large models with cross-entropy can be prone to over-ﬁtting on incomplete or noisy labels.
To alleviate the deﬁciency of cross-entropy, we present a new family of top- k classiﬁcation loss functions for deep neural networks.
Taking inspiration from multi-class SVMs, our loss creates a margin between the correct top- k predictions and the incorrect ones.
Our empirical results show that in their original form, such loss functions do not perform well in combination with deep neural networks.
We believe that the reason for this is the lack of smoothness and the sparsity of the derivatives that are used in backpropagation.
In order to overcome this difﬁculty, we smooth the loss with a temperature parameter.
The evaluation of the smooth function and its gradient are challenging, as smoothing increases the na ¨ ıve time complexity from O ( C ) to O ( (cid:0) Ck (cid:1) ) .
With a connection to polynomial algebra and a divide-and-conquer method, we present an algorithm with O ( kC ) time complexity and training time comparable to cross-entropy in practice.
We provide insights for numerical stability of the forward pass.
To deal with instabilities of the backward pass, we derive a novel and inexpensive approximation for which we prove an error bound.
Our investigation reveals that our top- k loss outperforms cross-entropy in the presence of noisy labels or in the absence of large amounts of data.
We further conﬁrm that the difference of performance reduces with large correctly labeled data, which is consistent with known theoretical results.
R ELATED W ORK
Top- k Loss Functions.
The majority of the work on top- k loss functions has been applied to shallow models: Lapin et al.
(2016) suggest a convex surrogate on the top- k loss; Fan et al.
(2017) select the k largest individual losses in order to be robust to data outliers; Chang et al.
(2017) formulate a truncated re-weighted top- k loss as a difference-of-convex objective and optimize it with the Concave-Convex Procedure (Yuille & Rangarajan, 2002); and Yan et al.
(2017) propose to use a combination of top- k classiﬁers and to fuse their outputs.
Closest to our work is the extensive review of top- k loss functions for computer vision by Lapin et al.
The authors conduct a study of a number of top- k loss functions derived from cross-entropy and hinge losses.
Interestingly, they prove that for any k , cross-entropy is top- k calibrated, which is a necessary condition for the classiﬁer to be consistent with regard to the theoretically optimal top- k risk.
In other words, cross-entropy satisﬁes an essential property to perform the optimal top- k classiﬁcation decision for any k in the limit of inﬁnite data.
This may explain why cross-entropy performs well on top- 5 error on large scale data sets.
While thorough, the experiments are conducted on linear models, or pre-trained deep networks that are ﬁne-tuned.
For a more complete analysis, we wish to design loss functions that allow for the training of deep neural networks from a random initialization.
Smoothing.
Smoothing is a helpful technique in optimization.
In work closely related to ours, Lee & Mangasarian (2001) show that smoothing a binary SVM with a temperature parameter improves the theoretical convergence speed of their algorithm.
Schwing et al.
(2012) use a temperature parameter to smooth latent variables for structured prediction.
Lapin et al.
(2017) apply Moreau-Yosida regularization to smooth their top- k surrogate losses.
Smoothing has also been applied in the context of deep neural networks.
In particular, Zheng et al.
(2015) and Clevert et al.
both suggest to modify the non-smooth ReLU activation to improve the training.
Gulcehre et al.
(2017) suggest to introduce “mollifyers” to smooth the objective function by gradually increasing the difﬁculty of the optimization problem.
Chaudhari et al.
(2017) add a local entropy term to the loss to promote solutions with high local entropy.
These smoothing techniques are used to speed up the optimization or improve generalization.
In this work, we show that smoothing is necessary for the neural network to perform well in combination with our loss function.
We hope that this insight can also help the design of losses for tasks other than top- k error minimization.
T OP - K SVM 3. B ACKGROUND : M ULTI -C LASS SVM
In order to build an intuition about top- k losses, we start with the simple case k
= 1 , namely multi-class classiﬁcation, where the output space is deﬁned as Y = { 1 , ..., C } .
We suppose that a vector of scores per label s ∈ R C , and a ground truth label y ∈ Y are both given.
The vector s is the output of the model we wish to learn, for example a linear model or a deep neural network.
The notation 1 will refer to the indicator function over boolean statements (1 if true, 0 if false).
Prediction.
The prediction is given by any index with maximal score:
[Equation]
Loss.
The classiﬁcation loss incurs a binary penalty by comparing the prediction to the ground truth label.
Plugging in equation (1), this can also be written in terms of scores s as follows:
[Equation]
Surrogate.
The loss of equation (2) is not amenable to optimization, as it is not even continuous in s .
To overcome this difﬁculty, a typical approach in machine learning is to resort to a surrogate loss that provides a continuous upper bound on Λ .
Crammer & Singer (2001) suggest the following upper bound on the loss, known as the multi-class SVM loss:
[Equation]
In other words, the surrogate loss is zero if the ground truth score is higher than the largest other score by a margin of at least one.
Otherwise it incurs a penalty which is linear in the difference.
Rescaling.
Note that the value of 1 as a margin is an arbitrary choice, and can be changed to α for any α > 0 .
This simply entails that we consider the cost Λ of a misclassiﬁcation to be α instead of 1.
3. T OP - K C LASSIFICATION
We now generalize the above framework to top- k classiﬁcation, where k ∈ { 1 , ..., C − 1 } .
We use the following notation: for p ∈ { 1 , ..., C } , s [ p ] refers to the p -th largest element of s , and s \ p to the vector ( s 1 , ..., s p − 1 , s p
+1 , ..., s C ) ∈ R C − 1 (that is, the vector s with the p -th element omitted).
Y ( k ) is the set of k -tuples with k distinct elements of Y .
Note that we use a bold font for a tuple ¯ y ∈ Y ( k ) in order to distinguish it from a single label ¯ y ∈ Y .
Prediction.
Given the scores s ∈ R C , the top- k prediction consists of any set of labels corresponding to the k largest scores:
[Equation]
Loss.
The loss depends on whether y is part of the top- k prediction, which is equivalent to comparing the k -largest score with the ground truth score:
[Equation]
Again, such a binary loss is not suitable for optimization.
Thus we introduce a surrogate loss.
Surrogate.
As pointed out in Lapin et al.
, there is a natural extension of the previous multi-class case:
[Equation]
This loss creates a margin between the ground truth and the k -th largest score, irrespectively of the values of the ( k − 1) -largest scores.
Note that we retrieve the formulation of Crammer & Singer (2001) for k = 1 .
Difﬁculty of the Optimization.
The surrogate loss l k of equation (6) suffers from two disadvantages that make it difﬁcult to optimize: (i) it is not a smooth function of s – it is continuous but not differentiable – and (ii) its weak derivatives have at most two non-zero elements.
Indeed at most two elements of s are retained by the ( · )
[ k ] and max operators in equation (6).
All others are discarded and thus get zero derivatives.
When l k is coupled with a deep neural network, the model typically yields poor performance, even on the training set.
Similar difﬁculties to optimizing a piecewise linear loss have also been reported by Li et al.
(2017) in the context of multi-label classiﬁcation.
We illustrate this in the next section.
We postulate that the difﬁculty of the optimization explains why there has been little work exploring the use of SVM losses in deep learning (even in the case k = 1 ), and that this work may help remedy it.
We propose a smoothing that alleviates both issues (i) and (ii), and we present experimental evidence that the smooth surrogate loss offers better performance in practice.
3. S MOOTH S URROGATE L OSS
Reformulation.
We introduce the following notation: given a label ¯ y ∈ Y , Y ( k )
¯ y is the subset of tuples from Y ( k ) that include ¯ y as one of their elements.
For ¯ y ∈ Y ( k ) and y ∈ Y , we further deﬁne ∆ k (¯ y , y ) (cid:44)
( y / ∈ ¯ y ) .
Then, by adding and subtracting the k − 1 largest scores of s \ y as well as s y , we obtain:
[Equation]
We give a more detailed proof of this in Appendix A.1.
Since the margin can be rescaled without loss of generality, we rewrite l k as:
[Equation]
Smoothing.
In the form of equation (8), the loss function can be smoothed with a temperature parameter τ > 0 :
[Equation]
Note that we have changed the notation to use L k,τ to refer to the smooth loss.
In what follows, we ﬁrst outline the properties of L k,τ and its relationship with cross-entropy.
Then we show the empirical advantage of L k,τ over its non-smooth counter-part l k .
Properties of the Smooth Loss.
The smooth loss L k,τ has a few interesting properties.
First, for any τ > 0 , L k,τ is inﬁnitely differentiable and has non-sparse gradients.
Second, under mild conditions, when τ → 0 + , the non-maximal terms become negligible, therefore the summations collapse to maximizations and L k,τ → l k in a pointwise sense (Proposition 2 in Appendix A.2).
Third, L k,τ is an upper bound on l k if and only if k = 1 (Proposition 3 in Appendix A.3), but L k,τ is, up to a scaling factor, an upper bound on Λ k
(Proposition 4 in Appendix A.4).
This makes it a valid surrogate loss for the minimization of Λ k .
Relationship with Cross-Entropy.
We have previously seen that the margin can be rescaled by a factor of α > 0 .
In particular, if we scale ∆ by α → 0 + and choose a temperature τ = 1 , it can be seen that L 1 , 1 becomes exactly the cross-entropy loss for classiﬁcation.
In that sense, L k,τ is a generalization of the cross-entropy loss to: (i) different values of k ≥ 1 , (ii) different values of temperature and (iii) higher margins with the scaling α of ∆ .
For simplicity purposes, we will keep α = 1 in this work.
Experimental Validation.
In order to show how smoothing helps the training, we train a DenseNet 40-12 on CIFAR-100 from Huang et al.
(2017) with the same hyper-parameters and learning rate schedule.
The only difference with Huang et al.
(2017) is that we replace the cross-entropy loss with
L 5 ,τ for different values of τ .
We plot the top- 5 training error in Figure 2a (for each curve, the value of τ is held constant during training): [Figure]
[Figure] We conﬁrm that smoothing helps the training of a neural network in Figure 2a, where a large enough value of τ greatly helps the performance on the training set.
In Figure 2b, we observe that such high temperatures yield gradients that are not sparse. In other words, with a high temperature, the gradient is informative about a greater number of labels, which helps the training of the model.
We remark that the network exhibits good accuracy when τ is high enough (0.01 or larger).
For τ too small, the model fails to converge to a good critical point.
When τ is positive but small, the function is smooth but the gradients are numerically sparse (see Figure 2b), which suggests that the smoothness property is not sufﬁcient and that non-sparsity is a key factor here.
C OMPUTATIONAL C HALLENGES AND E FFICIENT A LGORITHMS 4. C HALLENGE
Experimental evidence suggests that it is beneﬁcial to use L k,τ rather than l k to train a neural network.
Moreover, at ﬁrst glance, L k,τ may appear prohibitively expensive to compute.
Speciﬁcally, there are summations over Y ( k ) and Y ( k ) y , which have a cardinality of (cid:0) Ck (cid:1) and (cid:0) Ck − 1 (cid:1) respectively.
For instance for ImageNet, we have k = 5 and C = 1 , 000 , which amounts to (cid:0) Ck (cid:1) (cid:39) 8 .
10 12 terms to compute and sum over for each single sample, thereby making the approach practically infeasible.
This is in stark contrast with l k , for which the most expensive operation is to compute the k -th largest score of an array of size C , which can be done in O ( C ) .
To overcome this computational challenge, we will now reframe the problem and reveal its exploitable structure.
For a vector e ∈ R C and i ∈ { 1 , .., C } , we deﬁne σ i ( e ) as the sum of all products of i distinct elements of e .
Explicitly, σ i ( e ) can be written as σ i ( e ) = (cid:80) 1 ≤ j 1 <...<
j i ≤ C e j 1 ...
e j i .
The terms σ i are known as the elementary symmetric polynomials.
We further deﬁne σ 0 ( e ) = 1 for convenience.
We now re-write L k,τ using the elementary symmetric polynomials, which appear naturally when separating the terms that contain the ground truth from the ones that do not:
[Equation]
[Equation]
[Equation]
) (cid:89) j ∈ ¯ y exp( s j /kτ )
(cid:21)
[Equation]
(cid:20) (cid:88)
¯ y ∈Y ( k ) y
(cid:89)
j ∈ ¯ y exp( s j /kτ ) (cid:21) , =
[Equation]
(cid:20)
[Equation]
¯ y ∈Y ( k ) y (cid:89)
j
[Equation]
¯ y
[Equation] [Equation]
[Equation]
) (cid:88) ¯ y ∈Y ( k ) \Y ( k ) y (cid:89) j ∈ ¯ y exp( s j /kτ )
(cid:21)
[Equation]
(cid:20) (cid:88)
¯ y ∈Y ( k ) y
(cid:89) j ∈ ¯ y
[Equation]
[Equation]
[Equation]
) (cid:21) , (10)
[Equation] [Equation]
[Equation] [Equation]
\ y [Equation]
[Equation]
exp
[Equation]
[Equation]
k (exp( s \ y /kτ ))
(cid:21)
[Equation]
Note that the application of exp to vectors is meant in an element-wise fashion.
The last equality of equation (10) reveals that the challenge is to efﬁciently compute σ k − 1 and σ k , and their derivatives for the optimization.
While there are existing algorithms to evaluate the elementary symmetric polynomials, they have been designed for computations on CPU with double ﬂoating point precision.
For the most recent work, see Jiang et al.
To efﬁciently train deep neural networks with L k,τ , we need algorithms that are numerically stable with single ﬂoating-point precision and that exploit GPU parallelization.
In the next sections, we design algorithms that meet these requirements.
The ﬁnal performance is compared to the standard alternative algorithm in Appendix D.
4. F ORWARD C OMPUTATION
We consider the general problem of efﬁciently computing ( σ k − 1 , σ k ) .
Our goal is to compute σ k ( e ) , where e ∈ R n and k (cid:28) n .
Since this algorithm will be applied to e = exp
( s \ y /kτ )
(see equation (10)), we can safely assume e i (cid:54)
=
for all i ∈ (cid:74) 1 , n (cid:75) .
The main insight of our approach is the connection of σ i ( e ) to the polynomial:
[Equation]
Indeed, if we expand P to α
+ α 1 X
+ ...
+ α n X n , Vieta’s formula gives the relationship:
[Equation]
Therefore, it sufﬁces to compute the coefﬁcients α n − k to obtain the value of σ k ( e ) .
To compute the expansion of P , we can use a divide-and-conquer approach with polynomial multiplications when merging two branches of the recursion.
This method computes all ( σ i ) 1 ≤ i ≤ n instead of the only ( σ i )
k − 1 ≤ i ≤ k that we require.
Since we do not need σ i ( e ) for i > k , we can avoid computations of all coefﬁcients for a degree higher than n − k .
For ImageNet, we have k = 5 and n = 1 , 000 , therefore we have to compute coefﬁcients up to a degree 995 instead of 1,000, which is a small improvement.
To turn k (cid:28) n to our advantage, we notice that σ i ( e ) = σ n ( e ) σ n − i (1 / e ) .
Moreover, σ n ( e ) = n (cid:81) i =1 e
i can be computed in O ( n ) .
Therefore we introduce the polynomial:
[Equation]
Then if we expand Q to β 0 + β 1 X
+ ... + β n X n , we obtain with Vieta’s formula again:
[Equation]
Subsequently, in order to compute σ k ( e ) , we only require the k ﬁrst coefﬁcients of Q , which is very efﬁcient when k is small in comparison with n .
This results in a time complexity of O ( kn ) (Proposition 5 in Appendix B).
Moreover, there are only O (log( n )) levels of recursion, and since every level can have its operations parallelized, the resulting algorithm scales very well with n when implemented on a GPU (see Appendix D.2 for practical runtimes).
The algorithm is described in Algorithm 1: step 2 initializes the polynomials for the divide and conquer method.
While the polynomial has not been fully expanded, steps 5-6 merge branches by performing the polynomial multiplications (which can be done in parallel).
Step 10 adjusts the coefﬁcients using equation (14).
We point out that we could obtain an algorithm with a time complexity of O ( n log( k ) 2 ) if we were using Fast Fourier Transform for polynomial multiplications in steps 5-6.
Since we are interested in the case where k is small (typically 5), such an improvement is negligible.
Algorithm 1 Forward Pass
Require: e ∈ ( R ∗ + ) n , k ∈ N ∗ 1: t ←2: P ( t ) i ← (1 , 1 /e i ) for i ∈ (cid:74) 1 , n (cid:75) (cid:46) Initialize n polynomials to X + 1 e i (encoded by coefﬁcients) 3: p ← n (cid:46) Number of polynomials 4: while p > 1 do (cid:46)
Merge branches with polynomial multiplications 5:
P ( t +1)← P ( t ) 1 ∗ P ( t )(cid:46) Polynomial multiplication up to degree k ...
P ( t +1)
( p − 1) //
2 ← P ( t ) p − 1 ∗ P ( t ) p
(cid:46) Polynomial multiplication up to degree k 7: t ← t +8: p ← ( p − 1) //(cid:46) Update number of polynomials 9: end while 10: P ( t +1) ← P ( t ) × n (cid:81) i
=1 e i (cid:46)
Recover σ i ( e ) = σ n − i (1 / e ) σ n ( e ) 11: return P ( t +1)
Obtaining numerical stability in single ﬂoat precision requires special attention: the use of exponentials with a temperature parameter potentially arbitrarily small is fundamentally unstable.
In
Appendix C.1, we describe how operating in the log-space and using the sum-log-exp trick alleviates this issue.
The stability of the resulting algorithm is empirically investigated in Appendix D.3.
4. B ACKWARD C OMPUTATION
A side effect of this procedure is that a large number of buffers are allocated for automatic differentiation: for each addition in log-space, we apply log and exp operations, each of which needs to store values for the backward pass.
This results in a signiﬁcant amount of time spent on memory allocations, which become the time bottleneck.
To avoid this, we exploit the structure of the problem and design a backward algorithm that relies on the results of the forward pass.
By avoiding the memory allocations and considerably reducing the number of operations, the backward pass is then sped up by one to two orders of magnitude and becomes negligible in comparison to the forward pass.
We describe our efﬁcient backward pass in more details below.
First, we introduce the notation for derivatives:
[Equation]
We now observe that:
[Equation]
In other words, equation (16) states that δ ( j ) i , the derivative of σ j ( e ) with respect to e i , is the sum of product of all ( j − 1)
-tuples that do not include e i .
One way of obtaining σ j − 1 ( e \ i ) is to compute a forward pass for e \ i , which we would need to do for every i ∈ (cid:74) 1 , n (cid:75) .
To avoid such expensive computations, we remark that σ j ( e ) can be split into two terms: the ones that contain e i (which can expressed as e i σ j − 1 ( e \ i ) ) and the ones that do not (which are equal to σ j ( e \ i ) by deﬁnition).
This gives the following relationship:
[Equation]
Simplifying equation (17) using equation (16), we obtain the following induction for the derivatives:
[Equation]
Since the ( σ j ( e )) 1 ≤ i ≤ k have been computed during the forward pass, we can initialize the induction with δ (1) i = 1 and iteratively compute the derivatives δ ( j ) i for j ≥ 2 with equation (18).
This is summarized in Algorithm 2.
Algorithm 2 Backward Pass
Require:
e , ( σ j ( e )) 1 ≤ j ≤ k , k ∈ N ∗ (cid:46) ( σ j ( e )) 1 ≤ j ≤ k have been computed in the forward pass 1: δ (1) i = 1 for i ∈ (cid:74) 1 , n (cid:75) 2: for j ∈ (cid:74) 1 , k (cid:75) do 3: δ ( j ) i =
σ j − 1 ( e ) − e i δ ( j − 1) i for i ∈ (cid:74) 1 , n (cid:75) 4: end for
Algorithm 2 is subject to numerical instabilities (Proposition C.2 in Appendix C.2).
In order to avoid these, one solution is to use equation (16) for each unstable element and therefore require additional forward passes.
To avoid this inefﬁciency, we derive a novel and error-bounded approximation, which simply requires us to compute the additional ( k + 1) -th coefﬁcient during the forward pass.
Note that the time complexity of the forward pass remains unchanged since O (( k + 1) C )
=
O ( kC ) .
Once this additional coefﬁcient is obtained from the forward pass, applying our approximation to each unstable element of the backward pass has a constant time complexity (exactly four ﬂoating-point operations) instead of
a full forward pass of time complexity O ( kC ) .
For details, we refer the reader to Appendix C.2.
E XPERIMENTS
Theoretical results suggest that Cross-Entropy (CE) is an optimal classiﬁer in the limit of inﬁnite data, by accurately approximating the data distribution.
In practice, the presence of label noise makes the data distribution more complex to estimate when only a ﬁnite number of samples is available.
For these reasons, we explore the behavior of CE and L k,τ when varying the amount of label noise and the training data size.
For the former, we introduce label noise in the CIFAR-100 data set (Krizhevsky, 2009), in a way that would not perturb the top- 5 error of a perfect classiﬁer.
For the latter, we vary the training data size on subsets of the ImageNet data set (Russakovsky et al., 2015).
In all the following experiments, the temperature parameter is ﬁxed throughout the training and is set to one, unless explicitly stated otherwise.
This choice is discussed in the next section.
The algorithms are implemented in PyTorch and will be made publicly available.
Experiments on CIFAR-100 are performed on a single Nvidia Titan Xp card, and the experiments on ImageNet make use of two
Nvidia Titan Xp cards.
5. CIFAR-100 WITH NOISE
Data set.
In this experiment, we investigate the impact of label noise on CE and L 5 , 1 .
The CIFAR100 data set contains 60,000 RGB images, with 50,000 samples for training-validation and 10,000 for testing.
There are 20 “coarse” classes, each consisting of 5 “ﬁne” labels.
For example, the coarse class “people” is made up of the ﬁve ﬁne labels “baby”, “boy”, “girl”, “man” and “woman”.
In this set of experiments, the images are centered and normalized channel-wise before they are fed to the network.
We use the standard data augmentation technique with random horizontal ﬂips and random crops of size 32 × 32 on the images padded with 4 pixels on each side.
We introduce noise in the labels as follows: with probability p , each ﬁne label is replaced by a ﬁne label from the same coarse class.
This new label is chosen at random and may be identical to the original label.
Note that all instances generated by data augmentation from a single image are assigned the same label.
The case p = 0 corresponds to the original data set without noise, and p =to the case where the label is completely random (within the ﬁne labels of the coarse class).
With this method, a perfect top- 5 classiﬁer would still be able to achieve 100 % accuracy by systematically predicting the ﬁve ﬁne labels of the unperturbed coarse label.
Methods.
To evaluate our loss functions, we use the architecture DenseNet 40-40 from Huang et al.
and we use the same hyper-parameters and learning rate schedule as in Huang et al.
The temperature parameter is ﬁxed to one.
When the level of noise becomes non-negligible, we empirically ﬁnd that CE suffers from over-ﬁtting and signiﬁcantly beneﬁts from early stopping – which our loss does not need.
Therefore we help the baseline and hold out a validation set of 5, images, on which we monitor the accuracy across epochs.
Then we use the model with the best topvalidation accuracy and report its performance on the test set.
Results are averaged over three runs with different random seeds.
[Table]
[Table]
Results.
As seen in Table 1, L 5 , 1 outperforms CE on the top- 5 testing accuracy when the labels are noisy, with an improvement of over 5% in the case p = 1 .
When there is no noise in the labels, CE provides better top- 1 performance, as expected.
It also obtains a better top- 5 accuracy, although by a very small margin.
Interestingly, L 5 , 1 outperforms CE on the top- 1 error when there is noise, although
L 5 , 1 is not a surrogate for the top- 1 error.
For p = 0 .
, L 5 , 1 still yields an accuracy of 55.85%, as compared to 35.53% for CE.
This suggests that when the provided label is only informative about top- 5 predictions (because of noise or ambiguity), it is preferable to use L 5 , 1 .
5. I MAGE N ET
Data set.
As shown in Figure 1, the ImageNet data set presents different forms of ambiguity and noise in the labels.
It also has a large number of training samples, which allows us to explore different regimes up to the large-scale setting.
Out of the 1.28 million training samples, we use subsets of various sizes and always hold out a balanced validation set of 50,000 images.
We then report results on the 50,000 images of the ofﬁcial validation set, which we use as our test set.
Images are resized so that their smaller dimension is 256, and they are centered and normalized channel-wise.
At training time, we take random crops of 224 × 224 and randomly ﬂip the images horizontally.
At testing time, we use the standard ten-crop procedure (Krizhevsky et al., 2012).
We report results for the following subset sizes of the data: 64k images (5%), 128k images (10%), 320k images (25%), 640k images (50%) and ﬁnally the whole data set (1.28M − 50k =
1.23M images for training).
Each strict subset has all 1,000 classes and a balanced number of images per class – except for the largest subset, which has a slight unbalance alike the full ImageNet data set.
Methods.
In all following experiments, we train a ResNet-18 (He et al., 2016), adapting the protocol of the ImageNet experiment in Huang et al.
Namely, we optimize the model with Stochastic
Gradient Descent with a batch-size of 256, for a total of 120 epochs.
We use a momentum of 0.9.
The learning rate is divided by ten at epochs 30, 60 and 90, and is set to an initial value of 0.1 for CE and 1 for L 5 , 0 .
For both methods, training on the whole data set takes about a day and a half (it is only 10% longer with L 5 , 0 . 1 than with CE).
As previously, the validation top- 5 accuracy is monitored at every epoch, and we use the model with best top- 5 validation accuracy to report its test error.
Probabilities for Multiple Crops.
Using multiple crops requires a probability distribution over labels for each crop.
Then this probability is averaged over the crops to compute the ﬁnal prediction.
The standard method is to use a softmax activation over the scores.
We believe that such an approach is only grounded to make top- 1 predictions.
The probability of a label ¯ y being part of the topprediction should be marginalized over all combinations of 5 labels that include ¯ y as one of their elements.
This can be directly computed with our algorithms to evaluate σ k and its derivative.
We refer the reader to Appendix E for details.
All the reported results of top- 5 error with multiple crops are computed with this method.
This provides a systematic boost of at least 0.2% for all loss functions.
In fact, it is more beneﬁcial to the CE baseline, by up to 1% in the small data setting.
[Table]
Results.
The results of Table 2 conﬁrm that L 5 , 0 .
1 offers better performance on top- 5 error than
CE when the amount of training data is restricted.
As the data set size increases, the difference of performance becomes very small, and CE outperforms L 5 , 0 .
1 by an insigniﬁcant amount in the full data setting.
C ONCLUSION
This work has introduced a new family of loss functions that allow for direct minimization of the top- k error.
We have experimentally shown that non-sparsity is essential for loss functions to work well with deep neural networks.
Thanks to a connection to polynomial algebra and a novel approximation, we have shown how to efﬁciently evaluate the smooth loss and its gradient.
The experiments have shown that our smooth top- 5 loss function is more robust to noise and overﬁtting than cross-entropy when the amount of training data is limited.
In the context of large data, we have shown that using a margin improves results for the error being optimized.
Furthermore, we have demonstrated that careful optimization of the non-smooth top- k surrogate leads to improved results.
Making this optimization more accurate with an annealing of the temperature is an interesting direction for future research.
We have argued that smoothing the surrogate loss function helps the training of neural networks. This insight is not speciﬁc to top- k classiﬁcation, and we hope that it will help the design of other surrogate loss functions.
In particular, structured prediction problems could beneﬁt from smoothed SVM surrogate losses.
How to efﬁciently compute such smooth functions could open interesting research problems.
A S URROGATE L OSSES
In this section, we ﬁx C the number of classes.
We let τ > 0 and k ∈ { 1 , ..., C − 1 } .
The results assume that l k has the same rescaling as in equation (8):
[Equation]
A. R EFORMULATION
Proposition 1.
We can equivalently re-write l k as:
[Equation]
Proof.
l k ( s , y )
=
max (cid:40)(cid:18)
[Equation]
max
[Equation]
max
[Equation]
A. P OINT - WISE C ONVERGENCE
Lemma 1.
Let n ≥ 2 and e ∈ R n .
Assume that the largest element of e is greater than its second largest element: e [1] > e [2] .
Then lim τ →
[Equation]
Proof.
For simplicity of notation, and without loss of generality, we suppose that the elements of e are sorted in descending order.
Then for i ∈ { 2 , ..n } , we have e i − e 1 ≤ e 2 − e 1 < 0 by assumption, and thus ∀ i ∈ { 2 , ..n } , lim τ →τ>exp(( e i − e 1 ) /τ )
= 0 .
Therefore: lim
[Equation]
>n (cid:88) i
[Equation]
[Equation]
n
[Equation]
i
[Equation]
lim
[Equation]
And thus:
[Equation]
The result follows by noting that:
[Equation]
Proposition 2.
Assume that s [ k − 1] > s [ k ] and that s [ k ] > s [ k +1] or 1 k s y > 1 + 1 k s [ k ] .
Then lim τ → 0 τ> 0 L k,τ ( s , y ) =
l k ( s , y ) .
Proof.
From s [ k ] > s [ k +1] or 1 k s y > 1 + 1 k s [ k ] , one can see that max ¯ y ∈Y ( k ) (cid:110)
∆ k (¯ y , y ) + 1 k (cid:80) j ∈ ¯ y s j (cid:111) is a strict maximum.
Similarly, from s [ k − 1] > s [ k ] , we have that max ¯ y ∈Y ( k ) y (cid:110) 1 k (cid:80)
j ∈ ¯ y s j (cid:111) is a strict maximum.
Since L k,τ can be written as:
[Equation] [Equation]
=
[Equation]
the result follows by two applications of Lemma 1.
A.
B OUND ON N ON -S MOOTH
F UNCTION
Proposition 3.
L k,τ is an upper bound on l k if and only if k = 1 .
Proof.
Suppose k = 1 .
Let s ∈ R C and y ∈ Y .
We introduce y ∗ =
arg max ¯ y ∈Y ∆ 1 (¯ y, y )
+ s ¯ y .
Then we have:
[Equation]
Now suppose k ≥ 2 .
We construct an example ( s , y ) such that L k,τ ( s , y ) <
l k ( s , y ) .
For simplicity, we set y = 1 .
Then let s
= α , s i = β for i ∈ { 2 , ..., k + 1 } and s i
=
−∞ for i ∈ { k + 2 , ..., C } .
Assuming inﬁnite values simpliﬁes the analysis, and by continuity of L k,τ and l k , the proof will hold for real values sufﬁciently small.
We further assume that 1 + 1 k ( β − α ) > 0 .
Then can write l k ( s , y ) as:
[Equation] [Equation]
Exploiting the fact
that exp( s i /τ ) =
for i ≥ k + 2 , we have:
[Equation]
And:
[Equation]
This allows us to write L k,τ as:
[Equation]
We introduce x = 1 + 1 k ( β − α ) .
Then we have:
[Equation]
And:
[Equation]
For any value x > 0 , we can ﬁnd ( α, β )
∈ R 2 such that x = 1 + 1 k ( β − α ) and our hypotheses are all veriﬁed.
Consequently, we only have to prove that there exists x > 0 such that:
[Equation]
We show that lim x →∞ ∆( x ) < 0 , which will conclude the proof.
[Equation]
A. B OUND ON P REDICTION L OSS
Lemma 2.
Let ( p, q ) ∈ N 2 such that p ≤ q − 1 and q ≥ 1 .
Then (cid:0) qp (cid:1) ≤ q (cid:0) qp +1 (cid:1) .
Proof.
[Equation]
This is a monotonically increasing function of p ≤ q − 1 , therefore it is upper bounded by its maximal value at p = q − 1 :
[Equation]
Lemma 3. Assume that y / ∈ P k ( s ) .
Then we have:
[Equation]
Proof.
Let j ∈ (cid:74) 0 , k − 1 (cid:75) .
We introduce the random variable U j , whose probability distribution is uniform over the set U j (cid:44) {
¯ y ∈ Y ( k ) y :
¯ y ∩ P k ( s ) = j } .
Then V j is the random variable such that
V j | U j replaces y from U j with a value drawn uniformly from P k ( s ) . We denote by V j the set of values taken by V j with non-zero probability. Since V j replaces the ground truth score by one of the values of P k ( s ) , it can be seen that:
[Equation]
Furthermore, we introduce the scoring function f : ¯ y ∈ Y ( k ) (cid:55)→ exp( 1 kτ
[Equation]
j ∈ ¯ y s j ) . Since P k ( s ) is the set of the k largest scores and y / ∈ P k ( s ) , we have that: [Equation]
Therefore we also have that:
[Equation]
This ﬁnally gives us:
[Equation]
Making the (uniform) probabilities explicit, we obtain:
[Equation]
To derive the set cardinalities
[Equation]
we rewrite U j and V j as:
[Equation]
Therefore we have that:
[Equation]
And:
[Equation]
Therefore:
[Equation]
by Lemma
[Equation]
[Equation]
Combining with equation (42), we obtain:
[Equation]
We sum over j ∈ (cid:74) 0 , k − 1 (cid:75) , which yields:
[Equation]
Finally, we note that {U j } 0 ≤ j ≤ k − 1 and {V j } 0 ≤ j ≤ k − 1 are respective partitions of Y ( k ) y and Y ( k ) \Y ( k ) y , which gives us the ﬁnal result:
[Equation]
Proposition 4.
L k,τ is, up to a scaling factor, an upper bound on the prediction loss Λ k :
[Equation]
Proof.
Suppose that Λ k ( s , y ) = 0 .
Then the inequality if trivial because L k,τ ( s , y ) ≥ 0 .
We now assume that Λ k ( s , y )
= 1 .
Then there exist at least k higher scores than s y .
To simplify indexing, we introduce Z ( k ) y
[Equation]
\Y
[Equation]
k ) y and T k the set of k labels corresponding to the k -largest scores.
By assumption, y / ∈ T k since y is misclassiﬁed.
We then write:
[Equation]
Thanks to Lemma 3, we have:
[Equation]
Injecting this back into (53): (cid:88)
[Equation]
And back to the original loss:
[Equation]
B T IME C OMPLEXITY
Lemma 4.
Let P and Q be two polynomials of degree p and q .
The time complexity of obtaining the ﬁrst r coefﬁcients of PQ is O
(min { r, p } min { r, q } ) .
Proof.
The multiplication of two polynomials can be written as the convolution of their coefﬁcients, which can be truncated at degree r for each polynomial.
Proposition 5.
The time complexity of Algorithm 1 is O ( kC ) .
Proof.
Let N = log 2 ( C ) , or equivalently C = 2 N .
With the divide-and-conquer algorithm, the complexity of computing the k ﬁrst coefﬁcients of P can be written as:
[Equation]
Indeed we decompose P = Q 1 Q 2 , with each Q i of degree C/ 2 , and for these we compute their k ﬁrst coefﬁcients in T ( C 2 ) .
Then given the k ﬁrst coefﬁcients of Q 1 and Q 2 , the k ﬁrst coefﬁcients of
P are computed in O (min { k, C } 2 ) by Lemma 4.
Then we can write:
[Equation]
By summing these terms, we obtain T ( k, C ) =
2 N T ( k, 1) + N − 1 (cid:80)
[Equation]
Let n 0 ∈ N such that
[Equation]
C 2 n 0 .
In loose notation, we have kn
C =
O (1) .
Then we can write:
[Equation]
Thus ﬁnally:
[Equation]
C N UMERICAL S TABILITY C. F ORWARD P ASS
In order to ensure numerical stability of the computation, we maintain all computations in the log space: for a multiplication exp( x 1 ) exp( x 2 ) , we actually compute and store x 1 + x 2 ; for an addition exp( x 1 ) + exp( x 2 ) we use the “log-sum-exp” trick: we compute m = max { x 1 , x 2 } , and store m +log(exp
x 1 − m )
+exp( x 2 − m )) , which guarantees stability of the result.
These two operations sufﬁce to describe the forward pass.
C. B ACKWARD P ASS
Proposition 6.
The backward recursion of Algorithm 2 is unstable when e j (cid:29) 1 and e j (cid:29) max p (cid:54) =
j { e p } .
Proof.
To see that, assume that when we compute ( n (cid:80) p =1 e p ) − e j , we make a numerical error in the order of (cid:15) (e.g (cid:15) (cid:39) 10 − 5 for single-precision ﬂoats).
With the numerical errors, we obtain approximate ˆ δ as follows:
[Equation]
Since e j (cid:29) 1 , we quickly obtain unstable results.
Proposition 7.
We introduce the following approximation to the gradient:
[Equation]
Computing
[Equation]
with equation (60) is numerically stable, and requires four ﬂoating point operations.
Proposition 8.
If the following hypothesis is veriﬁed:
[Equation]
then the relative error of using the approximation of Proposition (7) can be bounded by:
[Equation]
) when (cid:15) [Equation]
[Equation]
Intuition To begin with, let us see why hypothesis (61) makes sense.
this hypothesis means that the terms that do not contain e j are negligible in comparison to the other ones.
Assume that ∀ p (cid:54) =
j, e p e j
=
O ( ξ ) , with ξ = o (1) and e j ξ =
O (1) .
This is the case of the instability shown before.
Then we can write:
[Equation]
Subsequently, hypothesis (61) is satisﬁed in the instability case presented previously.
Furthermore, we can show with similar reasoning that σ k ( e ) e j
= O (1) and σ k +1 ( e ) e 2 j = O ( ξ ) , therefore the approximated gradient σ k
( e ) e j − σ k +1 ( e ) e 2 j is stable to compute.
This means that we simply compute one more coefﬁcient ( σ k +1 ( e ) ) during the forward pass, in order to obtain a fast and stable backward pass.
Proof.
We now prove that equation (60) is true when assuming hypothesis (61).
First, we note that:
[Equation]
Then we can write:
[Equation]
Therefore we have:
[Equation]
And ﬁnally:
[Equation]
Since x (cid:55)→ 11 − x is strictly increasing on [
[Equation]
1) and :
[Equation]
we have:
[Equation]
and thus:
[Equation]
[Equation]
D C OMPARISON WITH THE S UMMATION A LGORITHM D. S UMMATION A LGORITHM
The Summation Algorithm (SA) is an alternative to the Divide-and-Conquer (DC) algorithm for the evaluation of the elementary symmetric polynomials.
It is described for instance in (Jiang et al., 2016).
The algorithm can be summarized as follows:
Require: e ∈ R n , k ∈ N ∗ 1: σ 0 ,i ← 1 for 1 ≤ i ≤ n (cid:46) σ j,i =
σ j ( e 1 , . . .
, e i ) 2: σ j,i ← 0 for i < j (cid:46)
Undeﬁned: j distinct elements out of a vector of size i if i < j 3: σ 1 , 1 ← e(cid:46)
Initialize recursion 4: for i ∈ (cid:74) 2 , n (cid:75) do 5: m ← max { 1 , i + k − n } 6:
{+M ← min { i, k } 7: for i ∈ (cid:74) m, M (cid:75) do 8: σ j,i ← σ j,i − 
+}
+ e i σ j − 1 ,i − 1 9: end for 10: end for 11: return σ k,n
Implementation Note that the inner loop can be parallelized, but the outer one is essentially sequential.
In our implementation for speed comparisons, the inner loop is parallelized and a buffer is pre-allocated for the σ j,i .
D. S PEED
We compare the execution time of the DC and SA algorithms on a GPU (Nvidia Titan Xp).
We use the following parameters: k = 5 , a batch size of 256 and a varying value of C .
The following timings are given in seconds, and are computed as the average of 50 runs.
In Table 3, we compare the speed of Summation and DC for the evaluation of the forward pass.
In Table 4, we compare the speed of the evaluation of the backward pass using Automatic Differentiation (AD) and our Custom Algorithm (CA) (see Algorithm 2).
time (s) of the forward pass.
The Divide and Conquer (DC) algorithm offers nearly logarithmic scaling with C in practice, thanks to its parallelization.
In contrast, the runtime of the Summation Algorithm (SA) scales linearly with C .
[Table]
We remind that both algorithms have a time complexity of O ( kC ) .
SA provides little parallelization (the parallelizable inner loop is small for k (cid:28) n ), which is reﬂected in the runtimes.
On the other hand,
DC is a recursive algorithm with O (log( C )) levels of recursion, and all operations are parallelized at each level of the recursion.
This allows DC to have near-logarithmic effective scaling with C , at least in the range { 100 − 10 , 000 } .
[Table]
These runtimes demonstrate the advantage of using Algorithm 2 instead of automatic differentiation.
In particular, we see that in the use case of ImageNet ( C = 1 , 000 ), the backward computation changes from being eight times slower than the forward pass to being three times faster.
D. S TABILITY
We now investigate the numerical stability of the algorithms.
Here we only analyze the numerical stability, and not the precision of the algorithm.
We point out that compensation algorithms are useful to improve the precision of SA but not its stability.
Therefore they are not considered in this discussion.
Jiang et al.
(2016) mention that SA is a stable algorithm, under the assumption that no overﬂow or underﬂow is encountered.
However this assumption is not veriﬁed in our use case, as we demonstrate below.
We consider that the algorithm is stable if no overﬂow occurs in the algorithm (underﬂows are not an issue for our use cases).
We stress out that numerical stability is critical for our machine learning context: if an overﬂow occurs, the weights of the learning model inevitably diverge to inﬁnite values.
To test numerical stability in a representative setting of our use cases, we take a random mini-batch of 128 images from the ImageNet data set and forward it through a pre-trained ResNet-18 to obtain a vector of scores per sample.
Then we use the scores as an input to the SA and DC algorithms, for various values of the temperature parameter τ .
We compare the algorithms with single-precision (S) and double-precision (D) ﬂoating points.
[Table]
(cid:88)
[Table]
DC log
[Table]
(cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)
DC log (D) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)
By operating in the log-space, DC is signiﬁcantly more stable than SA.
In this experimental setting,
DC is stable in single-precision until τ = 10 − 36 .
E P ROBABILITY FOR T OP - K P REDICTION
We consider the probability of label i being part of the ﬁnal top- k prediction.
To that end, we marginalize over all k -tuples that contain i as one of their element.
Then the probability of selecting label i for the top- k prediction can be written as:
[Equation]
Proposition 9.
The unnormalized probability can be computed as:
[Equation]
Proof.
[Equation]
Finally we can rescale the unnormalized probability by σ k (exp( s )) since it is independent of i .
We obtain:
[Equation]
NB.
We prefer to use d log σ i (exp( s ))
ds i rather than dσ i (exp( s ))
ds i for stability reasons.
Once the unnormalized probabilities are computed, they can be normalized by simply dividing by their sum.
F E XPERIMENTS : S UPPLEMENTARY D ETAILS F. CIFAR-
In the main paper, we report the average of the scores for clarity purposes.
Here
[Table]
we also detail the standard deviation of the scores for completeness.
[Table]