A Uniﬁed DRO View of Multi-class Loss Functions with top- N Consistency
Abstract
Multi-class classiﬁcation is one of the most common tasks in machine learning applications, where data is labeled by one of many class labels.
Many loss functions have been proposed for multi-class classiﬁcation including two well-known ones, namely the cross-entropy (CE) loss and the crammer-singer (CS) loss (aka.
the
SVM loss).
While CS loss has been used widely for traditional machine learning tasks, CE loss is usually a default choice for multi-class deep learning tasks.
There are also top- k variants of CS loss and CE loss that are proposed to promote the learning of a classiﬁer for achieving better top- k accuracy.
Nevertheless, it still remains unclear the relationship between these different losses, which hinders our understanding of their expectations in different scenarios.
In this paper, we present a uniﬁed view of the CS/CE losses and their smoothed top- k variants by proposing a new family of loss functions, which are arguably better than the CS/CE losses when the given label information is incomplete and noisy.
The new family of smooth loss functions named label-distributionally robust (LDR) loss is deﬁned by leveraging the distributionally robust optimization (DRO) framework to model the uncertainty in the given label information, where the uncertainty over true class labels is captured by using distributional weights for each label regularized by a function.
Introduction
Multi-class classiﬁcation is one of the fundamental tasks in machine learning (ML), where data is labeled by one of many class labels.
In the past decades, multi-class classiﬁcation has been used in a variety of areas, including image recognition [1, 2, 3], handwritten recognition [4, 5], natural language processing [6] etc.
More training data could always be a possible solution to improve the model performance.
However, labeling efforts could be more expensive as the machine learning task becomes more complicated.
Furthermore, for multi-class classiﬁcation, the labels of data could form a hierarchical tree structure, which leads to multi-label or ambiguity issue.
As a consequence, for computer vision task or natural language processing task, one image or text could contain multiple objects or concepts, which raises challenges not only for human labeling jobs but also model learning consistency [7, 8, 9].
On the other hand, a loss function plays a central role that can be researched for learning a model to achieve the learning consistency.
As we cannot directly optimize the noncontinuous 0-1 loss, optizming some surrogate loss functions such as the cross entropy (CE) and
the
SVM loss becomes the main-stream machine learning solution for multi-class classiﬁcation task.
In this paper, we consider a machine learning solution based on optimizing a novel loss function and its top- k simplex constrained variant for improving the performance of the standard CE loss and the SVM loss.
The motivation of our work springs from an interesting observation and its explanation: we have seen a lot of studies in traditional ML literature using the SVM losses (e.g., the Crammer-Singer (CS) loss [10]), but almost all studies of deep learning (DL) on ImageNet uses the CE loss.
What is the underlying reason for this?
One could explain this from two perspectives: (i) the CE loss is smooth while the CS loss is non-smooth, which is not favorable for optimization; (ii) the CE loss is consistent for multi-class classiﬁcation (i.e., yielding Bayes optimal classiﬁer with inﬁnite number of samples), while the CS loss is generally not consistent unless in an exceptional condition that the maximum conditional probability of a class label given the input is larger than 0 .
The exceptional condition means that the data is associated with a single label with high probability.
However, such condition usually does not hold for real-world multi-class classiﬁcation tasks in DL applications where data in inherently associated with multiple class labels.
Although each image in ImageNet is labeled by only one class, the recent studies [7, 8] show that these images inherently belong to multiple class labels.
In particular, they observed that a signiﬁcant fraction of images—more than a ﬁfth—contains at least two objects from the 1000 classes.
Related Work
Multi-class Loss functions.
Multi-class loss functions have been studied extensively in conventional machine learning (ML) literature [13, 14, 11, 10].
Many loss functions are shown to be consistent (producing Bayes optimal classiﬁer for minimizing zero-one loss, i.e., top- 1 error)
(e.g., CE loss), while others are shown to be inconsistent (e.g., CS loss, WW loss).
Variants of these standard losses have been developed for minimizing top-
k error ( k > 1 ), such as (smoothed) top-
k SVM loss, top- k entropy loss [15, 16, 17, 18].
However, the relationship between these losses and the CE loss is still unclear, which hinders the understanding of their performance in different scenarios.
Hence, it is important to provide a uniﬁed view of these different losses, which could not only help us understand the existing losses but also bring new loss functions.
In particular, the proposed LDR-KL loss has arguably more advantages than the CE loss and our methods based on the LDR-KL loss yields consistently better performance than the CE loss on benchmark datasets.
Distributional robust optimization (DRO) Objective.
It is interesting to connect the proposed LDR loss to the DRO objective in previous works for handling noisy and imbalanced data [19, 20, 21, 22].
Different from the proposed LDR loss that assocaite the DW variables to class labels, the existing
DRO objective assigns DW variables to instances and deﬁnes an aggregate objective by maximizing over the DW variables in an uncertainty set.
But it is interesting to make the analogy between the LDR loss and the DRO objective for arguing the beneﬁt of the LDR loss on handling the incomplete and noisy label information (cf.
the discussion in section Label-Distributionally Robust (LDR) Losses).
The Proposed Method
Notations.
Let x ∈ R
d
denote an input data, y ∈
[ K ] := { 1 , . .
, K } denote its class label, and let f ( x ) ∈ R K denote the prediction scores of a model.
Without causing any confusion, we simply use the notation f
= ( f 1 , . .
, f K ) =
f ( x ) ∈ R K to denote the prediction for any data x .
Interchangeably, we also use one-hot encoding vector y ∈ R K to denote the label information of a data.
For any vector q ∈ R K , let q [ k ] denote the k -largest entry of q with ties breaking arbitrarily.
Let ∆ K = { p ∈ R K : (cid:80)
K k
=1 p k
= 1 , p k ≥ 0 } denote a K -dimensional simplex.
3.1 Label-Distributionally Robust (LDR) Losses
The motivation of the proposed LDR losses is that the given label y for a given data x is usually incomplete.
Hence, the label information encoded in y that is used to deﬁne a loss function is noisy, meaning that the zero entries in y are not necessarily true zero.
In addition, the given label y itself could be wrong due to the labeling error, meaning that the entry in y that is equal to 1 is not necessarily true one.
A straightforward idea for learning a good prediction function is to enforce that f y to be larger than any other coordinates of f with possibly a large margin.
However, this approach could mis-guide the learning process due to that the label information y is often noisy.
To handle such uncertainty in the label information, we propose to use a regularized DRO framework to capture the inherent uncertainty in the label information.
Speciﬁcally, the proposed family of LDR losses is deﬁned by:
[Equation]
where p is referred to as distributional weight (DW) vector, Ω ⊆ { p ∈ R K : p k ≥ 0 , (cid:80)
k p k ≤ 1 } is a convex constraint set for the DW vector p , c ( y, k ) =
c
y
I
( y (cid:54) =
k ) denotes the margin parameter, λ > 0 is the DW regularization parameter, and R ( p ) is is a regularization term of p which is set by a strongly convex function .
The strong convexity of R ( p ) makes ψ λ ( f , y ) a smooth function of f due to the duality between strong convexity and smoothness [23].
The LDR-KL Loss .
A special loss in the family of LDR losses is called the LDR-KL loss deﬁned by specifying Ω = ∆ K , and R ( p ) as the KL divergence between p and the uniform probabilities (1 /K, . . .
, 1 /K ) , i.e., R ( p ) =
(cid:80)
Kk =1 p k log( p k K ) .
In this case, we can derive a closed-form expression of ψ λ ( f , y ) , denoted by ψ KL λ ( f , y ) ,
[Equation]
It is notable that the DW regularization parameter λ is a key feature of this loss.
It is interesting to see that the LDR-KL loss covers several existing losses as special case or extreme cases by varying the value of λ .
We include the proof in the appendix.
Extreme Case I. λ
= 0 .
The loss function becomes: ψ CS λ [Equation] f [Equation]
max [Equation]
) which is known as Crammer-Singer loss (an extension of hinge loss) for multi-class SVM [10] .
Extreme Case II.
λ = ∞ .
The loss function becomes: ψ + λ ( f , y )
= 1 K (cid:80)
Kk
=1 ( f k − f y + c
( y, k )) .
This loss is similar to the Weston-Watkins (WW) loss [13] for multi-class SVM, which is given by (cid:80) k (cid:54) =
y max(0 , f k − f y + 1) .
Special Case.
With λ = 1 , the LDR-KL loss becomes the CE loss: ψ CE λ ( f , y )
=
log ( (cid:80) k exp ( f k + c ( y, k ) − f y )) .
Note that in standard CE loss, the margin parameter c y is set to be zero.
Adding a margin parameter makes the above CE loss enjoys the large-margin beneﬁt.
This also recovers the label-distributionaware margin loss proposed by [24] for imbalanced data classiﬁcation with different margin values c y for different classes.
3.2 Top- N Consistency
The LDR-KL loss: the top- N error is deﬁned as err N ( f , y )
=
I ( y (cid:54)∈ r N ( f )) , where r N is a top- N selector that selects the N indices of the largest entries of the input by breaking ties arbitrarily.
According to previous studies [18], a necessary condition for a multi-class loss function is top- N consistent is that the loss function is top- N calibrated deﬁne below.
Hence, we will only present the result of top- N calibration of the LDR-KL loss, and its implication on consistency can be found in previous studies (cf.
Let P N ( f , q ) denotes that f is top- N preserving with respect to q , i.e., if for all l ∈ [ K ] , q l > q [ N +1] =
⇒ f l > f
[ N +1] , and q l < q [ N ] =
⇒
f l < f
[ N ] .
For anys q ∈ ∆ K , let
L ψ ( f , q ) =
E y ∼ q ψ y ( f )
=
(cid:80) l q l ψ ( f , l ) .
Deﬁnition 1.
[18] A loss function ψ ( f , y ) :
R K × [ K ] → R is called top- N calibrated for all q ∈ ∆ K it holds that: inf f ∈ R K : ¬ P N ( f , q ) L ψ ( f , q ) > inf f ∈ R K L ψ ( f , q ) .
Lemma 1.
For any q ∈ ∆ K , if f ∗ = arg min f ∈ R K L ψ ( f , q ) is rank consistent with respect to q , i.e., satisfying that for any pair q i < q j it holds that f ∗ i < f ∗ j , then ψ y ( f ) is top- N calibrated for any
N ≥ 1 .
Theorem 1.
If c y = c , ∀ y ∈
[ K ] , then f ∗ = arg min f ∈ R K L ψ KL λ ( f , q ) is rank consistent for any λ > 0 and hence the LDR-KL loss ψ KL λ ( f , y ) with any λ > 0 is top- N calibrated for any N ≥ 1 .
We include the proof of the above lemma and theorem in the appendix.
Consistency of the general LDR losses: we present a sufﬁcient condition on the DW regularization function R such that the resulting LDR loss ψ λ ( f , y ) is consistent.
The result derived below will address an open problem in [17] regarding the consistency of smoothed top- k SVM loss.
Deﬁnition 2.
A function R ( p ) is symmetric if its value is the same no matter the order of its arguments.
A set Ω is symmetric if a point p is in the set then so is any point obtained by interchanging any two coordinates of p . [Table]
Theorem 2.
If c y = c, ∀ y ∈
[ K ] , R and Ω are symmetric, R is strongly convex satisfying ∂R (0) <
and [ ∂R ( p )] i > 0 = ⇒ p
i (cid:54) = 0 , then ψ λ ( f , y ) is top- N classiﬁcation calibrated for any N ≥ 1 .
Remark: Examples of R that satisfy the above conditions include R ( p ) =
(cid:80) i p i log
( Kp i ) and
R ( p ) = (cid:107) p − 1 /K (cid:107) 2 . Consistency of LDR- k loss .
As an application of our result above, we consider the LDR- k loss for a given value k ∈ [ K ] .
In particular, by specifying top- k simplex Ω = Ω( k ) =
{ p ∈ R K : (cid:80) Kl =1 p l ≤ 1 , p l ≤ 1 /k, ∀ l } and λ = 0 , the LDR- k loss becomes the top- k SVM loss [17] , i.e.,
[Equation]
where c y =
( c ( y, 1) , . . .
, c ( y, K )) .
This top- k SVM loss is not consistent [17].
But, we can make it consistent by adding a strongly convex regularizer R ( p ) satisfying the conditions in Theorem 2.
In particular, we deﬁne a LDR- k loss by
[Equation]
Theorem 2 implies ψ ky ( f ) is classiﬁcation calibrated and hence consistent.
This addresses the open problem raised in [17] regarding the consistency of smoothed top- k SVM loss for multi-class classiﬁcation.
3.3 Efﬁcient Computation of LDR- k -KL loss
As an interesting application of our general LDR- k loss, we restrict our attention to the use of KL divergence for the regularization term R ( p ) =
(cid:80) i =1 p
i log( Kp i ) in the deﬁnition of (2), to which we refer as LDR- k -KL loss.
In particular, we consider how to efﬁciently compute the LDR- k -KL loss.
We present an efﬁcient solution with O ( K log K ) complexity.
Due to the limited space, we include the details of the Theorem 3 and the proof in the appendix.
Experiments
In this section, we ﬁrst show the effectiveness of the proposed LDR approach on a synthetic dataset and then verify the effectiveness of our proposed LDR and LDR-k methods on some well studied benchmark datasets for multiclass classiﬁcation tasks.
We also conduct experiments on image datasets using pretrained deep learning based features for learning a linear model with details described at the appendix.
We compare our LDR-KL and LDR- k -KL losses with the CE loss (with margin parameters), the CS loss, the WW loss, the top- k focused hinge loss (SVM- k ) [18] , the top- k truncated CE loss (CE- k ) [25].
Among of them, CE, SVM- k , LDR-KL and LDR- k -KL are calibrated for top- k consistency.
We present partial results in Table 1 for top-1 accuracy; more details and full results are included in the appendix.
Conclusion and Future Work
In this paper, we have proposed a novel family of label distributional robust losses that covers the cross entropy and the SVM loss as special cases and also induces new smoothed losses.
We proved the consistency of the proposed losses.
Our empirical studies also demonstrate the usefulness of the proposed loss functions.
For future work, we will consider to learn individual λ for each example in order to capture the difference between different examples in the given label information.
A Proof for The LDR-KL Loss Formulation
Proof.
Let q = f − f y
+ c
y , we want to solve:
[Equation]
It is not difﬁcult to show that p ∗ i =
[Equation]
exp
[Equation]
q i /λ ) .
By plugging this back we obtain the formulation of
LDR-KL loss, i.e,
[Equation]
B Proof for Theorem
Proof.
For simplicity, we consider λ = 1 .
The extension to λ > 0 is trivial.
Let us consider the optimization problem:
[Equation]
By the ﬁrst-order optimality condition ∂F ( f ) /∂f j = 0 , we have:
[Equation]
Move the negative term to the right and add
[Equation]
to both sides:
[Equation]
Moreover:
[Equation]
Z where Z > 0 is independent of j .
Then
[Equation]
If c k = c , we have
[Equation]
Hence the larger q j , the larger f ∗ j .
For q i < q j , we have
[Equation]
It implies that f ∗ i < f ∗ j .
C Proof of Theorem
Deﬁnition 3.
For a function R : Ω → R ∩ {−∞ , ∞} taking values on extended real number line, its convex conjugate is deﬁned as
[Equation]
Lemma 2.
If R ( · ) and Ω are symmetric, then R ∗ is also symmetric.
Proof.
[Equation]
For any i, j , let (cid:98) u be a shufﬂed version of u such that (cid:98) u i = u j , (cid:98) u
j = u i , (cid:98) u k = u k , ∀ k
(cid:54)
=
i, j .
Then
[Equation]
where (cid:98) p is a shufﬂed version of p .
Proof.
Proof of Theorem 2 , without loss of generalization, we assume λ = 1 .
[Equation]
where c y = c (1 − e y ) , e y is the y -th column of identity matrix.
Let us consider the optimization problem:
[Equation]
Consider q such that q 1 < q 2 , we prove that f ∗ is order preserving.
We prove by contradiction.
Assume f ∗ 1 > f ∗ 2 .
Deﬁne (cid:98) f as (cid:98) f 1 = f ∗ 2 and (cid:98) f 2 = f ∗ 1 and (cid:98) f k = f ∗ k , k > 2 .
Then
F (
[Equation]
=
[Equation]
Then
F ( (cid:98) f ) − F ( f ) =
q 1 ( R ∗ ( (cid:98) f − f 2 1 + c 1 ) − R ∗ ( f − f 1 1 + c 1 ))
+ q 2 ( R ∗ ( (cid:98) f − f 1 1 + c 2 ) − R ∗ ( f − f 2 1 + c 2 ))
Since [ (cid:98) f − f 2 1 + c 1 ] 1 = 0 , [ (cid:98) f − f 2 1 + c 1 ] 2 = f 1 − f 2 + c , [ (cid:98) f − f 2 1 + c 1 ] k = f k − f 2 + c, k > 2 , similarly [ f − f 2 1 + c 2 ] 1 = f 1 − f 2 + c , [ f − f 2 1 + c 2 ] 2 = 0 , [ f − f 2 1 + c 2 ] k = f k − f 2 + c, k > 2 , hence R ∗ ( (cid:98) f − f 2 1 + c 1 ) =
R ∗ ( f − f 2 1 + c 2 ) .
Similarly R ∗ ( f − f 1 1 + c 1 ) =
R ∗ ( (cid:98) f − f 1 1 + c 2 ) .
Hence, we have
[Equation]
Deﬁne f (cid:93) =
f − f 1 1 + c 1 with [ f (cid:93) ] 1 = 0 , [ f (cid:93) ]
k = f
k − f 1 + c, k > 1 , and f † = f − f 2 1 + c 2 with [ f † ] 2 = 0 , [ f † ] k = f k − f 2 + c, k (cid:54) = 2 .
Let f ‡ be a shufﬂed version of f † with [ f ‡ ] 1 = 0 , [ f ‡ ]
k = f 1 − f 2 + c, k = 2 , [ f ‡ ] k
= f k − f 2 + c, k > 2 .
Under the assumption that f 1 > f 2 , we have f (cid:93) ≤ f ‡ .
Then
[Equation]
Note that
[Equation]
Below, we prove R ∗ ( f (cid:93) )
− R ∗ ( f ‡ ) < 0 .
We will consider two scenarios.
First, consider f 2 − f 1 + c > 0 .
Let p ∗ = arg max p ∈ Ω (cid:80) k> 1 p k
[ f (cid:93) ]
k − R ( p ) .
We have f (cid:93) ∈ ∂R ( p ∗ ) .
Since [ f (cid:93) ] 2 > 0 , we have p ∗ 2 > 0 .
Hence R ∗ ( f (cid:93) )
= max p ∈ Ω (cid:80) k> 1 p k
[ f (cid:93) ]
k − R ( p ) =
(cid:80) k> 1 p
[Equation] k [ f
[Equation] ] k − R ( p ∗ ) < (cid:80) k> 1 p ∗
k [ f ‡ ] k − R ( p ∗ ) ≤ R ∗ ( f ‡ ) .
Next, consider f 2 − f 1 + c ≤ 0 .
By the convexity of R ∗ , we have
R ∗ ( f (cid:93) ) −
[Equation]
f
[Equation]
≤
[Equation]
f
[Equation]
R
[Equation]
f [Equation] f [Equation]
f
[Equation]
(cid:80) k> [Equation] [Equation] ( f
[Equation]
] k ( f [Equation] − f 1 ) ≤ c ([ ∇ R ∗ ( f (cid:93) )] 2 − [ ∇ R ∗ ( f (cid:93) )] 1 ) + (cid:80) k
[ ∇ R ∗ ( f (cid:93) )]
k ( f 2 − f 1 ) < [-
-]0 due to that (i) [ ∇ R ∗ ( f (cid:93) )] 2 ≤ [ ∇ R ∗ ( f (cid:93) )] 1 due to [ f (cid:93) ] 2 ≤ [ f (cid:93) ] 1 ; (ii) and (cid:80) k
[ ∇ R ∗ ( f (cid:93) )]
k = (cid:80) k
p ∗ k > 0 .
This is because p ∗ cannot be all zeros otherwise increasing the ﬁrst coordinate of p ∗ will decrease the value of R ( p ) and therefore increase the value of Q .
However, such a solution p ∗ is impossible since for the function Q ( p ) =
p (cid:62) f (cid:93) − R ( p ) , which has a unique maximizer due to strong concavity of
Q ( p ) .
This will prove that R ∗ ( f (cid:93) )
− R ∗ ( f ‡ ) < 0 .
Next, we prove that f 1 < f 2 .
Assume f 1 = f 2 , we establish a contradiction.
By the ﬁrst-order optimality condition, we have
[Equation]
Hence
[Equation]
Next, we prove that for any f such that [ f ] 1 = [ f ] 2 , then [ ∇ R ∗ ( f )]
=
[ ∇ R ∗ ( f )] 2 .
To this end, we consider the equation, R ( p ) + R ∗ ( f ) =
p (cid:62) f , where p = arg max p ∈ Ω Q ( p ) :
= p (cid:62) f − R ( p ) satisfying p = ∇ R ∗ ( f ) ∈ Ω .
If [ ∇ R ∗ ( f )] 1 (cid:54) =
[ ∇ R ∗ ( f )] 2 , which means p 1 (cid:54) =
p 2 .
We can deﬁne another vector p (cid:48) =
( p 2 , p 1 , p 3 , . . .
, p K ) ∈ Ω , p (cid:48) (cid:54) = p , which satisﬁes Q ( p (cid:48) )
=
p (cid:48)(cid:62)
f − R ( p (cid:48) ) =
Q ( p ) , which contradicts to the fact that Q ( · ) has a unique maximizer.
Then for any f with [ f ] 1 =
[ f ] 2 , we have [ f − f l 1 − c l ] 1 =
[ f − f l 1 − c l ] 2 .
As a result,
[Equation]
It is clear that the above equation is impossible since 1 (cid:62) ∇ R ∗ ( f ) (cid:54) = 0 , ∀ f with [ f ] 1 = 0 and q 1 < q 2 .
D Theorem[-
-]
Theorem 3.
Consider the problem max p ∈ Ω( k ) (cid:80)
i p i q
i − λ
(cid:80) i =1 p
i log( Kp i ) .
To compute the optimal solution, let the q to be sorted and π ( i ) denote the index for its i -th largest value.
Given a ∈ [ K ] deﬁne a vector p ( a ) as
[Equation]
The optimal solution p ∗ is given by p ( a ) such that a is the smallest number in { 1 , . . .
, K } satisfying [ p ( a )]
π ( a ) ≤ 1 k .
The overall time complexity is O
( K log( K )) .
Given that the algorithm would be simple with the presented theorem 3, we do not present a formal algorithm box here.
Instead, we simply describe it as follows: i) scan (from largest to smallest) through vector q to get cumulative summation (cid:80)
Kj = a exp( q π ( j ) λ − 1) for all possible values of a . ii) apply the theorem 3 to calculate [ p ( a )] π ( a ) for all possible values of a .
It is obvious that the cost is linear with K .
It is worth noting that the bottleneck for computing the analytical solution is sorting for q , which usually costs O ( K log K ) .
Hence, the computation of the LDR- k -KL loss can be efﬁcient conducted.
Proof.
Apply Lagrangian multiplier:
[Equation]
Stationary condition of p :
[Equation]
Dual form:
[Equation]
Stationary condition for β and γ :
[Equation]
Checkpoint I: for the π ( a ) that γ ∗ π ( a ) = 0 and γ ∗ π ( a − 1) > 0 , where 2 ≤ a ≤ k + 1 , (or a = 1 , γ ∗ π ( a − 1) is undeﬁned):
• We ﬁrst consider the case when β ∗ = 0 , then the [ p ( a )] ∗ π ( i )
=
1 k for i < a , and [ p ( a )] ∗ π ( i )
= 1 K exp( q π ( i ) λ − 1) for i ≥ a .
• Otherwise, if β ∗ > 0 :
[Equation]
It is similar to show for any i that i > a :
[Equation]
For those i < a that γ ∗ π
( i ) > 0 :
[Equation]
So we can get the analytical solution as far as we know a subject to γ ∗ π ( a ) = 0 and γ ∗ π ( a − 1) > 0 .
The [ p ( a )] ∗ π ( i )
= 1 k for i < a , and [ p ( a )] ∗ π ( i ) = [Equation] [Equation]
[Equation]
) for i ≥ a .
Checkpoint II: next, we show that as far as we ﬁnd the smallest a such that γ ∗ π ( a ) = 0 and γ ∗ π ( a − 1) > 0 , then it is the optimal solution.
• Suppose a (cid:48) <
a : because ∀ a (cid:48)
s.t.
k + 1 ≥ a > a (cid:48) ≥ 1 , assume [ p ( a (cid:48) )]
π ( a (cid:48) ) ≤ 1 k = ⇒ a (cid:48) is another smaller valid a , which violates the pre-condition; therefore, [ p ( a (cid:48) )]
π ( a (cid:48) )
> [ p ( a )] π ( a (cid:48) )
= 1 k , the a ∗ must lead to a p ∗ that violate the Ω k constrain, hence can’t be the optimal solution, contradiction.
• Suppose a (cid:48) >
a : i) if β ∗ ( a ) = 0 , then by pre-condition [ γ ( a )] ∗ π ( a ) [Equation]
[ γ [Equation]
0 , consequently β ∗ ( a (cid:48) )
= 0 , which deviates from the optimality of the objective function:
[Equation]
(notice that [ γ ( a )] ∗ π ( i ) =
[ γ ( a (cid:48) )] ∗ π ( i ) > 0 , ∀ i < a in order to hold the Ω( k ) constrain).
ii) if β ∗ ( a ) > 0 , then [ γ ( a )] ∗ π ( a ) =
and [ γ ( a (cid:48) )] ∗ π ( a ) > 0 =
⇒ β ∗ ( a (cid:48) )
< β ∗ ( a )
= ⇒ ∀
i ≥ a (cid:48) , [ p ( a (cid:48)
π ( i ) >
[ p ( a )] π ( i ) .
On the other hand, ∀ j < a (cid:48) , [ p ( a (cid:48) )]
π ( j ) = 1 k ≥ [ p ( a )]
π ( j ) =
⇒ (cid:80) Ki
= a (cid:48)
[ p ( a (cid:48) )]
π ( i ) ≤ 1 − (cid:80) a (cid:48) − 1 i =
[ p ( a (cid:48) )] π ( i ) ≤ 1 − (cid:80) a (cid:48) − 1 i =
[ p ( a )] π ( i ) = (cid:80) Ki = a (cid:48)
[ p ( a )] π ( i ) , which is contradictory with ∀ i ≥ a (cid:48) , [ p ( a (cid:48) )]
π ( i ) >
[ p ( a )] π ( i ) .
E Experiments Details E.1 Synthetic Data
[Figure]
, WW loss is deﬁned in a way that all other unlabeled classes are penalized equally; therefore, the model is spun up to infer the potential unlabeled true class by utilizing all the labeled training data.
LDR loss takes a trade off between CS and WW loss from the potential unlabeled true classes discovering perspective because the DW regularization controls whether LDR loss is closer to[-[Figure] True Data Dis. -10 -5 0 5 10 15 20 - -      (b) w/ CE loss -20 -
-]
[-) w/ CS loss -20 -
-]
CS loss or WW loss.
CE loss is a speciﬁc form of LDR loss.
With DW regularization, LDR loss is closer to CE loss with adjustable temperature parameter.
DW regularization also could be treated as a uncertainty capturing module for potential unlabeled true labels.
We justify the effectiveness of the proposed DW regularization for LDR loss on a synthetic data shown in Figure 1.
We ﬁrst deﬁne 3 bi-variate normal distributions, whose centers are (10 , 0) (cid:62) , ( − 10 , 0) (cid:62) , (0 , 10) (cid:62) and co-variances are (cid:20) 8 0 0 8 (cid:21) .
They are used to mimic the underlying distributions fordifferent classes.
For training, we ﬁrst sample 100 data samples for one class and 5 data samples for the other two classes, which is used to simulate imbalanced classes scenario.
Next, we manually impose 4 outliers, which are either denoted as black ‘ × ’ and ‘ ◦ ’ as an analogy to similar images mislabeled as different classes for image classiﬁcation task.
They are either sitting on the different classes comparing to their underlying, or on the middle of two classes.
For the purpose of justiﬁcation for DW regularization, we only compare CE, CS, WW and LDR in this subsection and leave out their top-k variants.
The DW regularization is tuned in the range {1e-1,1,1e1}, and (cid:96) 2 regularization is tuned in {1e-4,1e-3,1e-2,0} for all the methods.
Margin is ﬁxed as 1 for all the methods.
The ﬁnal model are selected by 5-fold cross-validation.
The classiﬁcation boundaries are drawn in Figure 1.
As we can observe from the results, DW regularization improves robustness for LDR loss.
E.2 Benchmark Datasets
In this section, we conduct experiments on some benchmark datasets for multi-class classiﬁcation.
The statistics for the datasets are summarized in Table 2.
The evaluation metrics include top- k accuracy ( k = { 1 , ..., 5 } ).
Structured Data: We ﬁrst compare different losses on the three structured datastesm, namely aloi, news20 and letter for learning a linear model [3, 6, 4] 1 .
We split 10% data samples for testing, the [Table] [Table] remaining for training and validation.
It is worth noting that we utilize PCA for news20 dataset to keep 128 principle components, to avoid the high dimensionality issue
[Table]
We grid-tune the hyper-parameters for all the baseline models by 5-fold cross validation.
The (cid:96) 2 − regularization hyper-parameters are tuned in the range {1e-4,1e-3,1e-2,0} for all the loss functions.
The margin hyper-parameters c ( y, k ) are tuned in the range {1e-1,1,1e1} for all the margin-based loss functions.
To avoid heavily tuning cost, we ﬁrst grid-tune LDR-KL and LDR- k -KL with c ( y, k ) and distributional weight (DW) in the range {1e-1,1,1e1} with ﬁxed (cid:96) 2 − regularization hyper-parameter as 1e-4.
After that, tune them with (cid:96) 2 − regularization hyper-parameters in the same range as other baselines.
It is worth noting that the top- k losses for k = 1 (SVM- k , CE- k and LDR- k -KL) would be reduced to their classical top-1 version losses (CS, CE and LDR-KL).
Hence, they have the same top-1 precision values as their top-1 version losses.
We run each method on each dataset independently 5 times and report the averaged top- k ( k =1,2,3,4,5) accuracy together with their standard deviations in the
Table 3.
As we can see from the results, LDR-KL and LDR- k -KL performs better comparing to other existing loss functions.
For aloi dataset with a large number of classes, LDR-KL and LDR- k -KL are consistently better than other methods, which means the proposed LDR loss could beneﬁt from the data with a large class number.
It is consistent with our assumption because it is more possible that such data could have more potential unlabeled true class.
For news20 and letter datasets, the proposed
LDR-KL loss is generally better than CE loss for top-1 precision; and the proposed LDR- k -KL loss is generally better than CE- k loss.
The non-smooth SVM losses are better for some cases thanks to the large margin classiﬁer effect.
Image Datasets: To eliminate the effect of different losses on feature learning (wich is not the focus of this paper), we conduct experiments by using a pre-trained network for extracting the feature vector of an image.
We ﬁrst pre-train deep neural networks on the benchmark datasets (Resnet-32 on cifar and Resnet-50 on imagenet by optimizing the CE loss) for extracting a feature representation [26]; then based on the pre-trained hidden features we learn linear models by optimizing different loss functions.
For the pre-training process, we use momentum SGD with stage-wised decreasing step size to optimize 50 epochs for cifar-10, 150 epochs for cifar-100 and 90 epochs for Imagenet.
The hidden feature dimension for CIFAR-(10 and 100) is 64; and the hidden feature dimension for Imagenet is 2048.
For learning the linear model based on the extract feature representations, we also utilize momentum SGD with stage-wised decreasing step size (10000 iterations as one stage; initial learning rate is 0.1 and decreasing factor is 0.1) for optimizing different loss functions, with a total ofand 50000 iterations for CIFAR-(10 and 100) and Imagenet, respectively.
The batch size is 128 for all the experiments in this work.
We keep the same protocol for model validation and evaluation as for the structured data, except that we keep the default testing splitting.
The results are summarized in the Tables 3.
From the results, LDR-KL and LDR- k -KL losses generally outperform the other loss functions.
The advantages are more obvious for cifar100 and Imagenet, who have a large number of classes.
For cifar10, the classes should be more clearly separated and easier for both human and machine learning model to recognize.