C URRICULUM L OSS : R OBUST L EARNING AND G ENERALIZATION AGAINST L ABEL C ORRUPTION
A BSTRACT
Deep neural networks (DNNs ) have great expressive power, which can even mem orize samples with wrong labels.
It is vitally important to reiterate robustness and generalization in DNNs against label corruption.
To this end , this paper stud - ies the 0-1 loss, which has a monotonic relationship with empirical adversary (reweighted) risk
(Hu et al
Although the 0-1 loss has some robust properties, it is difﬁcult to optimize.
To efﬁciently optimize the 0-1 loss while keeping its robust properties, we propose a very simple and efﬁcient loss, i e. curriculum loss (CL).
Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses.
Moreover, CL can adaptively select samples for model training.
As a result, our loss can be deemed as a novel perspective of curriculum sample selection strategy, which bridges a connection between curriculum learning and robust learning.
Experimental results on benchmark datasets validate the robustness of the proposed loss
I NTRODUCTION
Noise corruption is a common phenomenon in our daily life.
For instance, noisy corrupted (wrong) labels may be resulted from annotating for similar objects (Su et al., 2012; Yuan et al., 2019), crawling images and labels from websites (Hu et al., 2017; Tanaka et al., 2018) and creating training sets by program (Ratner et al., 2016; Khetan et al., 2018).
Learning with noisy labels is thus an promising area.
Deep neural networks (DNNs) have great expressive power (model complexity) to learn challenging tasks.
However, DNNs also undertake a higher risk of overﬁtting to the data.
Although many regularization techniques, such as adding regularization terms, data augmentation, weight decay, dropout and batch normalization, have been proposed, generalization is still vitally important for deep learning to fully exploit the super-expressive power.
Zhang et al.
(2017) show that DNNs can even fully memorize samples with incorrectly corrupted labels.
Such label corruption signiﬁcantly degenerates the generalization performance of deep models.
This calls a lot of attention on robustness in deep learning with noisy labels.
Robustness of 0-1 loss: The problem resulted from data corruption or label corruption is that test distribution is different from training distribution.
Hu et al.
(2018) analyzed the adversarial risk that the test distribution density is adversarially changed within a limited f -divergence (e.g. KLdivergence) from the training distribution density.
They show that there is a monotonic relationship between the (empirical) risk and the (empirical) adversarial risk when the 0-1 loss function is used.
This suggests that minimizing the empirical risk with the 0-1 loss function is equivalent to minimize the empirical adversarial risk (worst-case risk).
When we train a model based on the corrupted training distribution, we want our model to perform well on the clean distribution.
Since we do not know the clean distribution, we want our model to perform well for the worst case estimate of the clean distribution in some constrained set.
It is thus natural to employ the worst-case classiﬁcation risk of the estimated clean distribution as the objective.
Note that the worst-case classiﬁcation risk is an upper bound of the classiﬁcation risk of the true clean distribution, minimizing the worst-case risk can usually decrease the true risk.
When we employ the 0-1 loss, because of the equivalence between the classiﬁcation risk and the worst-case classiﬁcation risk, we can directly minimize the classiﬁcation risk under the corrupted training distribution instead of minimizing the worst-case classiﬁcation risk.
From the learning perspective, the 0-1 loss is more robust to outliers compared with an unbounded (convex) loss (e.g. hinge loss) (Masnadi-Shirazi & Vasconcelos, 2009).
This is due to unbounded convex losses putting much weight on the outliers (with a large loss value) when minimizing the losses (Masnadi-Shirazi & Vasconcelos, 2009).
If the unbounded (convex) loss is employed in deep network models, this becomes more prominent.
Since training loss of deep networks can often be minimized to zero, outlier with a large loss has a large impact on the model.
On the other hand, the 0-1 loss treats each training sample equally.
Thus, each sample does not have too much inﬂuence on the model.
Therefore, the model is tolerant of a small number of outliers.
Although the 0-1 loss has many robust properties, its non-differentiability and zero gradients make it difﬁcult to optimize.
One possible way to alleviate this problem is to seek an upper bound of the 0-1 loss that is still efﬁcient to optimize but tighter than conventional (convex) losses.
Such a tighter upper bound of the 0-1 loss can reduce the inﬂuence of the noisy outliers compared with conventional (convex) losses.
At the same time, it is easier to optimize compared with the 0-1 loss.
When minimizing the upper bound surrogate, we expect that the 0-1 loss objective is also minimized.
Learnability under large noise rate: The 0-1 loss cannot deal with large noise rate.
When the noise rate becomes large, the systematic error (due to label corruption) grows up and becomes not negligible.
As a result, the model’s generalization performance will degenerate due to this systematic error.
To reduce the systematic error produced by training with noisy labels, several methods have been proposed.
They can be categorized into three kinds: transition matrix based method (Sukhbaatar et al., 2014; Patrini et al., 2017; Goldberger & Ben-Reuven, 2017), regularization based method (Miyato et al., 2016) and sample selection based method (Jiang et al., 2018;
Han et al., 2018b).
Among them, sample selection based method is one promising direction that selects samples to reduce noisy ratio for training.
These methods are based on the idea of curriculum learning (Bengio et al., 2009) which is one successful method that trains the model gradually with samples ordered in a meaningful sequence.
Although they achieve success to some extents, most of these methods are heuristic based.
To efﬁciently minimize the 0-1 loss while keeping the robust properties, we propose a novel loss that is a tighter upper bound of the 0-1 loss compared with conventional surrogate losses.
Speciﬁcally, giving any base loss function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R , our loss Q ( u ) satisﬁes (cid:80) ni =1 1 (cid:0) u
i < 0 (cid:1) ≤ Q ( u ) ≤ (cid:80) ni
=1 l ( u i ) , where u
= [ u 1 , · · · , u n ] with u i being the classiﬁcation margin of i th sample, and 1 ( · ) is an indicator function.
We name it as Curriculum Loss (CL) because our loss automatically and adaptively selects samples for training, which can be deemed as a curriculum learning paradigm.
Our contributions are listed as follows:
• We propose a novel loss (i
e. curriculum loss) for robust learning against label corruption.
We prove that our CL is a tighter upper bound of 0-1 loss compared with conventional summation based surrogate loss.
Moreover, CL can adaptively select samples for stagewise training, which bridges a connection between curriculum learning and robust learning.
• We prove that CL can be performed by a simple and fast selection algorithm with O ( n log n ) time complexity.
Moreover, our CL supports mini-batch update, which is convenient to be used as a plug-in in many deep models.
• We further propose a Noise Pruned Curriculum Loss (NPCL) to address label corruption problem by extending CL to a more general form.
Our NPCL automatically prune the estimated noisy samples during training.
Moreover, NPCL is also very simple and efﬁcient, which can be used as a plug-in in deep models as well.
C URRICULUM L OSS
In this section, we present the framework of our proposed Curriculum Loss (CL).
We begin with discussion about robustness of the 0-1 loss in Section 2.1.
We then show that our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses in Section 2.2.
A tighter bound of the 0-1 loss means that it is less sensitive to the noisy outliers, and it better preserves the robustness of the 0-1 loss with a small rate of label corruption.
For a large rate of label corruption, we extend our CL to a Noise Pruned Curriculum Loss (NPCL) to address this issue in Section 2.3.
A simple multi-class extension and a novel soft multi-hinge loss are included in the Appendix.
All the detailed proofs can be found in the Appendix as well.
2. R OBUSTNESS OF 0-1 LOSS AGAINST LABEL CORRUPTION
We rephrase Theorem 1 in (Hu et al., 2018) from a different perspective, which motivates us to employ the 0-1 loss for training against label corruption.
Theorem 1.
(Monotonic Relationship)
(Hu et al.
(Hu et al., 2018)) Let p ( x, y ) and q ( x, y ) be the training and test density,respectively.
Deﬁne r ( x, y )
=
q ( x, y ) /p
( x, y ) and r i = r
( x i , y i ) .
Let l ( (cid:98) y, y )
= 1 (cid:0) sign ( (cid:98) y )
(cid:54) =
y (cid:1) and l ( (cid:98) y, y )
= 1 (cid:0) argmax k
( (cid:98) y k ) (cid:54) =
y (cid:1) be 0-1 loss for binary classiﬁcation and multi-class classiﬁcation, respectively.
Let f ( · ) be convex with f (1) = 0 .
Deﬁne risk R ( θ ) , empirical risk (cid:98) R ( θ ) , adversarial risk R adv ( θ ) and empirical adversarial risk (cid:98) R adv ( θ ) as
[Equation]
where U f = (cid:8) r ( x, y ) (cid:12)(cid:12) E p ( x,y )
[ f ( r ( x, y ))] ≤ δ, E p ( x,y )
[ r ( x, y )] = 1 , r ( x, y ) ≥ 0 , ∀
( x, y ) ∈
X × Y (cid:9) and (cid:98) U f = (cid:8) r (cid:12)(cid:12) 1 n (cid:80) ni =1 f ( r i ) ≤ δ, 1 n (cid:80) ni
=1 r i = 1 , r ≥ 0 (cid:9) .
Then we have that
[Equation]
If R adv ( θ 1 ) = 1 , then R ( θ 1 )
[Equation]
[Equation]
The same monotonic relationship holds between their empirical approximation: (cid:98) R ( θ ) and (cid:98) R adv .
Theorem 1 (Hu et al., 2018) shows that the monotonic relationship between the (empirical) risk and the (empirical) adversarial risk (worst-case risk) when 0-1 loss function is used.
It means that minimizing (empirical) risk is equivalent to minimize the (empirical) adversarial risk (worst-case risk) for 0-1 loss.
When we train a model based on the corrupted training distribution p ( x, y ) , we want our model to perform well on the clean distribution q ( x, y ) .
Since we do not know the clean distribution q , we want our model to perform well for the worst-case estimate of the clean distribution, with the assumption that the f -divergence between the corrupted distribution p and the clean distribution q is bounded by δ .
Note that the underlying clean distribution is ﬁxed but unknown, given the corrupted training distribution, the smallest δ that bounds the divergence between the corrupted distribution and clean distribution measures the intrinsic difﬁculty of the corruption, and it is also ﬁxed and unknown.
The corresponding worst-case distribution w.r.t the smallest δ is an estimate of the true clean distribution, and this worst-case risk upper bounds the risk of the true clean distribution.
In addition, this bound is tighter than the other worst-case risks w.r.t larger δ .
It is natural to use this upper bound as the objective for robust learning.
When we use 0-1 loss (that is commonly employed for evaluation), because of the equivalence of the risk and the worst-case risk, we can directly minimize risk under training distribution p instead of directly minimizing the worst-case risk (i.e., the upper bound).
Moreover, this enables us to minimize the upper bound without knowing the true δ beforehand.
When the true δ is small, i.e., the corruption of the training data is not heavy, the upper bound is not too pessimistic.
Usually, minimizing the upper bound can decrease the true risk under clean distribution.
Particularly, when the clean distribution coincides with the worst-case estimate w.r.t the smallest δ , minimizing the risk under the corrupted training distribution leads to the same minimizer as minimizing the risk under the clean distribution.
2. T IGHTER UPPER BOUNDS OF THE 0-1 L OSS
Unlike commonly used loss functions in machine learning, the non-differentiability and zero gradients of the 0-1 loss make it difﬁcult to optimize.
We thus propose a tighter upper bound surrogate loss.
We use the classiﬁcation margin to deﬁne the 0-1 loss.
For binary classiﬁcation, classiﬁcation margin is u = (cid:98) yy , where (cid:98) y and y ∈ { +1 , − 1 } denotes the prediction and ground truth, respectively.
(A simple multi-class extension is discussed in the Appendix.)
Let u i ∈
R be the classiﬁcation margin of the i th sample for i ∈ { 1 , ..., n } .
Denote u =
[ u 1 , ..., u n ] .
The 0-1 loss objective can be deﬁned as follows:
[Equation]
Given a base upper bound function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R , the conventional surrogate of the 0loss can be deﬁned as
[Equation]
Our curriculum loss Q ( u ) can be deﬁned as Eq.(9).
Q ( u ) is a tighter upper bound of 0-1 loss J ( u ) compared with the conventional surrogate loss (cid:98) J ( u ) , which is summarized in Theorem 2:
Theorem 2.
(Tighter Bound)
Suppose that base loss function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R is an upper bound of the 0-1 loss function.
Let u i ∈
R be the classiﬁcation margin of the i th sample for i ∈ { 1 , ..., n } .
Denote max( · , · ) as the maximum between two inputs.
Let u =
[ u 1 , ..., u n ] .
Deﬁne
Q ( u ) as follows:
[Equation]
Then J ( u ) ≤ Q ( u ) ≤ (cid:98) J ( u ) holds true.
Remark:
For any ﬁxed u , we can obtain an optimum solution v ∗ of the partial optimization.
The index indicator v ∗ can naturally select samples as a curriculum paradigm for training models.
The partial optimization w.r.t index indicator v can be solved by a very simple and efﬁcient algorithm
(Algorithm 1) in O ( n log n ) .
Thus, the loss is very efﬁcient to compute.
Moreover, since Q ( u ) is tighter than conventional surrogate loss (cid:98) J ( u ) , it is less sensitive to outliers compared with (cid:98) J ( u ) .
Furthermore, it better preserves the robust property of the 0-1 loss against label corruption.
The difﬁculty of optimizing the 0-1 loss is that the 0-1 loss has zero gradients in almost everywhere (except at the breaking point).
This issue prevents us from using ﬁrst-order methods to optimize the 0-1 loss.
Eq.(9) provides a surrogate of the 0-1 loss with non-zero subgradient for optimization, while preserving robust properties of the 0-1 loss.
Note that our goal is to construct a tight upper bound of the 0-1 loss while maintaining informative (sub)gradients.
Eq.(9) balances the 0-1 loss and conventional surrogate by selecting (the trust) samples (index) for training progressively.
Updating with all the samples at once is not efﬁcient for deep models, while training with mini-batch is more efﬁcient and well supported for many deep learning tools.
We thus propose a batch based curriculum loss (cid:98) Q ( u ) given as Eq.(10).
We show that (cid:98) Q ( u ) is also a tighter upper bound of 0-1 loss objective J ( u ) compared with conventional loss (cid:98) J ( u ) .
This property is summarized in Corollary 1.
Corollary 1.
(Mini-batch Update) Suppose that base loss function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R is an upper bound of the 0-1 loss function.
Let b , m be the number of batches and batch size, respectively.
Let u ij ∈ R be the classiﬁcation margin of the i th sample in batch j for i ∈ { 1 , ..., m } and j ∈ { 1 , ..., b } .
Denote u =
[ u 11 , ..., u mb ] .
Let n = mb .
Deﬁne (cid:98) Q ( u ) as follows:
[Equation]
Then J ( u ) ≤ Q ( u ) ≤ (cid:98) Q ( u ) ≤ (cid:98) J ( u ) holds true.
Remark: Corollary 1 shows that a batch-based curriculum loss is also a tighter upper bound of 0-1 loss J ( u ) compared with the conventional surrogate loss (cid:98) J ( u ) .
This enables us to train deep models with mini-batch update.
Note that random shufﬂe in different epoch results in a different batch-based curriculum loss.
Nevertheless, we at least know that all the induced losses are upper bounds of 0-1 loss objective and are tighter than (cid:98) J ( u ) .
Moreover, all these losses are induced by the same base loss function l ( · ) .
Note that, our goal is to minimize the 0-1 loss.
Random shufﬂe leads to a multiple surrogate training scheme.
In addition, training deep models without shufﬂe does not have this issue.
We now present another curriculum loss E ( u ) which is tighter than Q ( u ) .
E ( u ) is an (scaled) upper bound of 0-1 loss.
This property is summarized as Theorem 3.
Input: u
} , the selection threshold C ;
Output: Index v
Compute the losses l i =
( u i ) for n ;
Sort samples (index) w.r.t.
the losses { l i } ni =1 in a non-decreasing order; // Get l 1 ≤ · · · ≤ l n
Initialize L 0 = 0 ; for i = 1 to n do
L i =
L i − 1 + l i ; if L i ≤ ( C + 1 − i ) then
Set v i = 1; else
Set v i = 0; end if end for
Theorem 3.
(Scaled Bound)
Suppose that base loss function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R is an upper bound of the 0-1 loss function.
Let u i ∈
R be the classiﬁcation margin of the i th sample for i ∈ { 1 , ..., n } .
Denote u =
[ u 1 , ..., u n ] .
Deﬁne E ( u ) as follows:
[Equation]
Then J ( u ) ≤ 2 E ( u ) ≤ 2 (cid:98) J ( u ) holds true.
Remark: E ( u ) has similar properties to Q ( u ) discussed above.
Moreover, it is tighter than Q ( u ) , i.e. E ( u ) ≤ Q ( u ) .
Thus, it is less sensitive to outliers compared with Q ( u ) .
However, Q ( u ) can construct more adaptive curriculum by taking 0-1 loss into consideration during the training process.
Directly optimizing E ( u ) is not as efﬁcient as that optimizing Q ( u ) .
We now present a batch loss objective (cid:98) E ( u ) given as Eq.(12).
(cid:98) E ( u ) is also a tighter upper bound of 0-1 loss objective J ( u ) compared with conventional surrogate loss (cid:98) J ( u ) .
Corollary 2.
(Mini-batch Update for Scaled Bound) Suppose that base loss function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R is an upper bound of the 0-1 loss function.
Let b , m be the number of batches and batch size, respectively.
Let u ij ∈ R be the classiﬁcation margin of the i th sample in batch j for i ∈ { 1 , ..., m } and j ∈ { 1 , ..., b } .
Denote u =
[ u 11 , ..., u mb ] .
Let n = mb .
Deﬁne (cid:98) E ( u ) as follows:
[Equation]
Then J ( u ) ≤ 2 E ( u ) ≤ 2 (cid:98) E ( u ) ≤ 2 (cid:98) J ( u ) holds true.
All the curriculum losses deﬁned above rely on minimizing a partial optimization problem (Eq.(13)) to ﬁnd the selection index set v ∗ .
We now show that the optimization of v with given classiﬁcation margin u i ∈ R , i ∈ { 1 , ..., n } can be done in O ( n log n ) .
Theorem 4.
(Partial Optimization) Suppose that base loss function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R is an upper bound of the 0-1 loss function.
For ﬁxed u i ∈ R , i ∈ { 1 , ..., n } , an minimum solution v ∗ of the minimization problem in Eq.
(13) can be achieved by Algorithm 1:
[Equation]
where C is the threshold parameter such that 0 ≤ C ≤ 2 n .
Remark: The time complexity of Algorithm 1 is O ( n log n ) .
Moreover, it does not involve complex operations, and is very simple and efﬁcient to compute.
Algorithm 1 can adaptively select samples for training.
It has some useful properties to help us better understand the objective after partial minimization, we present them in Proposition 1.
Proposition 1.
(Optimum of Partial Optimization) Suppose that base loss function l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) , u ∈ R is an upper bound of the 0-1 loss function.
Let u i ∈ R for i ∈ { 1 , ..., n } be ﬁxed values.
Without loss of generality, assume l ( u 1 ) ≤ l ( u 2 ) · · · ≤ l ( u n ) .
Let v ∗ be an optimum solution of the partial optimization problem in Eq.(13).
Let T ∗ =
(cid:80) ni
=1 v ∗
i and L T ∗ =
(cid:80) T ∗ i =1 l ( u i ) .
Then we have
[Equation] [Equation]
Remark: When C ≤ n + (cid:80) ni
=1 1 (cid:0) u
i < 0 (cid:1) , Eq.(17) is tighter than the conventional loss (cid:98) J ( u ) .
When C ≥ n , Eq.
(17) is a scaled upper bound of 0-1 loss J ( u ) .
From Eq.(17) , we know the optimum of the partial optimization problem (13) (i.e. our objective) is max( L T ∗ , C − T ∗ ) .
When
L T ∗ ≥ C − T ∗
, we can directly optimize L T ∗ with the selected samples for training.
When L T ∗ <
C − T ∗ , note that L T ∗ +1 > max( L T ∗ , C − T ∗ ) from Eq.(16), we can optimize L T ∗ +1 for training.
Note that when T ∗ < n , we have that L T ∗ +1 ≤ L n = (cid:80) ni =1 l ( u i ) , which is still tighter than the conventional loss (cid:98) J ( u ) .
When T ∗ = n , for the parameter C ≤ n + (cid:80) ni
=1 1 (cid:0) u
i < 0 (cid:1) , we have that
L T ∗ = (cid:98) J ( u ) ≥ J ( u ) ≥ C − n = C − T ∗ .
Thus we can optimize max( L T ∗ , C − T ∗ ) =
(cid:98) J ( u ) .
In practice, when training with random mini-batch, we ﬁnd that optimizing L T ∗ in both cases instead of L T ∗ +1 does not make much inﬂuence.
2. N OISE P RUNED C URRICULUM L OSS
The curriculum loss in Eq.(9) and Eq.(11) expect to minimize the upper bound of the 0-1 loss for all the training samples.
When model capability (complexity) is high, (deep network) model will still attain small (zero) training loss and overﬁt to the noisy samples.
The ideal model is that it correctly classiﬁes the clean training samples and misclassiﬁes the noisy samples with wrong labels.
Suppose that the rate of noisy samples (by label corruption) is (cid:15) ∈ [0 , 1] .
The ideal model is to correctly classify the (1 − (cid:15) )
n clean training samples, and misclassify the (cid:15)n noisy training samples.
This is because the label is corrupted.
Correctly classify the training samples with corrupted (wrong) label means that the model has already overﬁtted to noisy samples.
This will harm the generalization to the unseen data.
Considering all the above reasons, we thus propose the Noise Pruned Curriculum Loss (NPCL) as
[Equation]
(18) where C
=
(1 − (cid:15) ) n or C =
[Equation]
When we know there are (cid:15)n noisy samples in the training set, we can leverage this as our prior.
(The impact of misspeciﬁcation of the prior is included in the supplement.)
When C =
(1 − (cid:15) )
n (assume C, (cid:15)n are integers for simplicity), from the selection procedure in Algorithm 1, we know (cid:15)n 1 samples with largest losses l ( u ) will be pruned.
This is because C − (cid:80) ni
=1 v i + 1 ≤ 0 when (cid:80) ni
=1 v i ≥ (1 − (cid:15) ) n + 1 .
Without loss of generality, assume l ( u 1 ) ≤ l ( u 2 ) · · · ≤ l ( u n ) .
After pruning, we have v (1 − (cid:15) )
n +1 = · · · = v n = 0 , the pruned loss becomes
[Equation]
It is the basic CL for (1 − (cid:15) ) n samples and it is the upper bound of (cid:80) (1 − (cid:15) )
n i = 1 (cid:0) u i < 0 (cid:1) .
If we prune more noisy samples than clean samples, it will reduce the noise ratio.
Then the basic CL can handle.
Fortunately, this assumption is supported by the "memorization" effect in deep networks (Arpit et al., 2017), i.e. deep networks tend to learn clean and easy pattern ﬁrst.
Thus, the loss of noisy or hard
Input: noise
Output:
Initialize model parameter w .
for k = 1 to N do
Shufﬂe training set D ; while Not fetch all the data from D do
Fetch a mini-batch (cid:98) D from D ;
Compute losses { l i } mi =1 for data in (cid:98) D ;
Compute the selection threshold C according to Eq.(21).
Compute selection index v ∗ by Algorithm 1;
Update w = w − α ∇ l (cid:16) (cid:98) D v ∗ (cid:17) w.r.t the subset (cid:98) D v ∗ of (cid:98) D selected by v ∗ ; end while end for data tend to remain high for a period (before being overﬁtted).
Therefore, the pruned samples with largest loss are more likely to be the noisy samples.
After the rough pruning, the problem becomes optimizing basic CL for the remaining samples as in Eq.(19).
Note that our CL is a tight upper bound approximation to the 0-1 loss, it preserves the robust property to some extent.
Thus, it can handle case with small noise rate.
Speciﬁcally, our CL(Eq.19) further select samples from the remaining samples for training adaptively according to the state of training process.
This generally will further reduce the noise ratio.
Thus, we may expect our NPCL to be robust to noisy samples.
Note that, all the above can be done by the simple and efﬁcient Algorithm 1 without explicit pruning samples in a separated step.
Namely, our loss can do all these automatically under a uniﬁed objective form in
Eq.(18).
When C =
(1 − (cid:15) )
n , the NPCL in Eq.(18) reduces to basic CL E ( u ) in Eq.(11) with (cid:15) = 0 .
When
C = (1 − (cid:15) ) 2 n +(1 − (cid:15) ) (cid:80) ni
=1 1 (cid:0) u
i < 0 (cid:1) , for an ideal target model (that misclassiﬁes noisy samples only), we know that E
[ C ] = (1 − (cid:15) )
2 n + (1 − (cid:15) )
E [ (cid:80) ni
=1 1 (cid:0) u
i < 0 (cid:1) ]
= (1 − (cid:15) )
2 n + (1 − (cid:15) )
(cid:15)n =
(1 − (cid:15) ) n .
It has similar properties as choosing C =
(1 − (cid:15) ) n .
Moreover, it is more adaptive by considering 0-1 loss during training at different stages.
In this case, the NPCL in Eq.(18) reduces to the CL Q ( u ) in Eq.(9) when (cid:15) = 0 .
Note that C is a prior, users can deﬁned it based on their domain knowledge.
To leverage the beneﬁt of deep learning, we present the batched NPCL as
[Equation]
(20) where (cid:98) C
j = (1 − (cid:15) ) m or as in Eq.
[Equation] [Equation]
Similar to Corollary 1, we know that L ( u ) ≤ (cid:98) L ( u ) .
Thus, optimizing the batched NPCL is indeed minimizing the upper bound of NPCL.
This enables us to train the model with mini-batch update, which is very efﬁcient for modern deep learning tools.
The training procedure is summarized in
Algorithm 2.
It uses Algorithm 1 to select a subset of samples from every mini-batch.
Then, it uses the selected samples to perform gradient update.
E MPIRICAL S TUDY 3. E VALUATION OF R OBUSTNESS AGAINST L ABEL C ORRUPTION
We evaluate our NPCL by comparing Generalized Cross-Entropy (GCE) loss (Zhang & Sabuncu, 2018), Co-teaching (Han et al., 2018b), Co-teaching+ (Yu et al., 2019), MentorNet (Jiang et al., 2018) and standard network training on MNIST, CIFAR10 and CIFAR100 dataset as in (Han et al., 2018b; Patrini et al., 2017; Goldberger & Ben-Reuven, 2017).
Two types of random label corruption, i.e. Symmetry ﬂipping (Van Rooyen et al., 2015) and Pair ﬂipping (Han et al., 2018a), are
[Figure]
considered in this work.
Symmetry ﬂipping is that the corrupted label is uniformly assign to one of K − 1 incorrect classes.
Pair ﬂipping is that the corrupted label is assign to one speciﬁc class similar to the ground truth.
The noise rate (cid:15) of label ﬂipping is chosen from { 20% , 50% , 35% } as a representative.
As a robust loss function, we further compare NPCL with GCE loss in detail with noise rate in { 0% , 10% , 20% , 30% , 40% , 50% } .
We employ same network architecture and network hyperparameters as in Co-teaching (Han et al., 2018b) for all the methods in comparison.
Specifically, the batch size and the number of epochs is set to m = 128 and N = 200 , respectively.
The
Adam optimizer with the same parameter as (Han et al., 2018b) is employed.
The architecture of neural network is presented in Appendix L. For NPCL, we employ hinge loss as the base upper bound function of 0-1 loss.
In the ﬁrst few epochs, we train model using full batch with soft hinge loss (in the supplement) as a burn-in period suggested in (Jiang et al., 2018).
Speciﬁcally, we start
NPCL at 5 th epoch on MNIST and 10 th epoch on CIFAR10 and CIFAR100, respectively.
For Coteaching (Han et al., 2018b) and MentorNet in (Jiang et al., 2018), we employ the open sourced code of Co-teaching (Han et al., 2018b).
For Co-teaching+ (Yu et al., 2019), we employ the code provided by the authors.
We implement NPCL by Pytorch.
For NPCL, Co-teaching and Co-teaching+, we employ the true noise rate as parameter.
Experiments are performed ﬁve independent runs.
The error bar for STD
For performance measurements, we employ both test accuracy and label precision as in (Han et al., 2018b).
Label precision is deﬁned as : number of clean samples / number of selected samples , which measures the selection accuracy for sample selection based methods.
A higher label precision in the mini-batch after sample selection can lead to a update with less noisy samples, which means that model suffers less inﬂuence of noisy samples and thus preforms more robustly to label corruption.
The pictures of test accuracy and label precision vs. number of epochs on MNIST are presented in
Figure 1.
The results on CIFAR10 and CIFAR100 are shown in Figure 5 and Figure 6 in Appendix, respectively.
It shows that NCPL achieves superior performance compared with GCE loss in terms of test accuracy.
Particularly, NPCL obtains signiﬁcant better performance compared with GCE loss in hard cases: Symmetry-50% and Pair-ﬂip-35%, which shows that NPCL is more robust to label corruption compared with GCE loss.
Moreover, NPCL obtains better performance on MNIST, and competitive performance on CIFAR10 and CIFAR100 compared with Co-teaching.
Furthermore,
NPCL achieves better performance than Co-teaching+ on CIFAR10 and two cases on MNIST.
In addition, we ﬁnd that Co-teaching+ is not stable on CIFAR100 with 50% symmetric noise.
Note that NPCL is a simple plug-in for a single network, while Co-teaching/Co-teaching+ employs two networks to train the model concurrently.
Thus, both the space complexity and time complexity of
Co-teaching/Co-teaching+ is doubled compared with our NPCL.
Both our NPCL and Generalized Cross Entropy (GCE) loss are robust loss functions as plug-in for single network.
Thus, we provide a more detailed comparison between our NPCL and GCE loss with noise rate in { 0% , 10% , 20% , 30% , 40% , 50% } .
The experimental results on CIFAR10 are presented in Figure 3.
The experimental results on CIFAR100 and MNIST are provided in Figureand Figure 7 in Appendix.
From Figure 3, Figure 8 and Figure 7, we can observe that NPCL obtains similar and higher test accuracy in all the cases.
Moreover, from Figure 3 and Figure 7, we can see that NPCL achieves similar test accuracy compared with the GCE loss when the noise rate is small.
The improvement increases with the increase of the noise rate.
Particularly, NPCL obtains remarkable improvement compared with the GCE loss on CIFAR10 with noise rate 50%.
It shows that NPCL is more robust compared with GCE loss against label corruption.
GCE loss employs all samples for training, while NPCL prunes the noisy samples adaptively.
As a result, GCE loss still employs samples with wrong labels for training, which misleads the model.
Thus, NPCL obtains better performance when the noise rate becomes large.
3. M ORE EXPERIMENTS WITH DIFFERENT NETWORK ARCHITECTURES
We follow the experiments setup in (Lee et al., 2019).
We use the online code of (Lee et al., 2019) , and only change the loss for comparison.
We cite the numbers of Softmax, RoG and D2L (Ma et al., 2018) in (Lee et al., 2019) for comparison.
The test accuracy results on uniform noise, semantic noise and open-set noise are shown in Table 1,
Table 2 and Table 3, respectively.
From Table 1, we can observe that both NPCL and CL outperforms
Softmax (cross-entropy) and RoG (cross-entropy) on ﬁve cases for uniform noise.
Note that RoG is an ensemble method, while CL/NPCL is a single loss for network training, one can combine them to boost the performance.
From Table 2, we can see that CL obtains consistently better performance than cross-entropy and D2L
(Ma et al., 2018) for the semantic noise.
Table 3 shows that NPCL achieves competitive performance compared with RoG for open-set noise.
[Table]
[Table] [Table]
We further evaluate the performance of CL/NPCL on the Tiny-ImageNet dataset.
We use the
ResNet18 network as the test-bed.
For GCE loss, we employ the default hyper-parameter q = 0 .
7 in all cases.
All the methods are performed ﬁve runs with seeds { 1 , 2 , 3 , 4 , 5 } .
The curve of mean test accuracy (shaded in std) are provided in Figure 2.
We can see that NPCL and CL obtain higher test accuracy than generalized cross-entropy loss and stand cross-entropy loss on both cases.
Note that CL does not have parameters, it is much convenient to use.
[Figure] [Figure]
C ONCLUSION AND F URTHER W ORK
In this work, we proposed a curriculum loss (CL) for robust learning.
Theoretically, we analyzed the properties of CL and proved that it is tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses.
We extended our CL to a more general form (NPCL) to handle large rate of label corruption.
Empirically, experimental results on benchmark datasets show the robustness of the proposed loss.
As a further work, we may improve our CL to handle imbalanced distribution by considering diversity for each class.
Moreover, it is interesting to investigate the inﬂuence of different base loss functions in CL and NPCL.
A CKNOWLEDGEMENT
We sincerely thank the reviewers for their insightful comments and suggestions.
This paper was supported by Australian Research Council grants DP180100106 and DP200101328.
A E XPLANATION OF T HEOREM 1 FOR ROBUST LEARNING
Theorem.
(Monotonic Relationship)
((Hu et al., 2018)
Let p ( x, y ) and q ( x, y ) be the training and test density,respectively.
Deﬁne r ( x, y )
=
q ( x, y ) /p
( x, y ) and r i = r
( x i , y i ) .
Let l ( (cid:98) y, y )
= 1 (cid:0) sign ( (cid:98) y )
(cid:54) =
y (cid:1) and l ( (cid:98) y, y )
= 1 (cid:0) argmax k
( (cid:98) y k ) (cid:54) =
y (cid:1) be 0-1 loss for binary classiﬁcation and multi-class classiﬁcation, respectively.
Let f ( · ) be convex with f (1) = 0 .
Deﬁne risk
R ( θ ) , empirical risk (cid:98) R ( θ ) , adversarial risk R adv ( θ ) and empirical adversarial risk (cid:98) R
adv ( θ ) as
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
where U f = (cid:8) r ( x, y ) (cid:12) (cid:12) E p ( x,y )
[ f ( r ( x, y ))] ≤ δ, E p ( x,y )
[ r ( x, y )] = 1 , r ( x, y ) ≥ 0 , ∀
( x, y ) ∈
X × Y (cid:9) and (cid:98) U f = (cid:8) r (cid:12) (cid:12) 1 n (cid:80) ni =1 f ( r i ) ≤ δ, 1 n (cid:80) ni
=1 r i = 1 , r ≥ 0 (cid:9) .
Then we have that
[Equation]
θ 1 ) < 1 , then R (
[Equation]
If R
[Equation]
, then R
[Equation] [Equation]
The same monotonic relationship holds between their empirical approximation: (cid:98) R ( θ ) and (cid:98) R adv .
Hu et al.
(2018) show that minimizing (empirical) risk is equivalent to minimize the (empirical) adversarial risk (worst-case risk) for 0-1 loss.
Thus, we can directly optimize the risk instead of the worst-case risk.
Speciﬁcally, suppose we have an observable training distribution p ( x, y ) .
The observable distribution p ( x, y ) may be corrupted from an underlying clean distribution q ( x, y ) .
We train a model based on the training distribution p ( x, y ) , and we want our model to perform well on the clean distribution q ( x, y ) .
Since we do not know the clean distribution q ( x, y ) , we want our model to perform well for the worst-case estimate of the clean distribution, with the assumption that the f -divergence between the corrupted distribution p and the clean distribution q is bounded by δ .
Note that the underlying clean distribution is ﬁxed but unknown, given the corrupted training distribution, the smallest δ that bounds the divergence between the corrupted distribution and clean distribution measures the intrinsic difﬁculty of the corruption, and it is also ﬁxed and unknown.
The corresponding worst-case distribution w.r.t the smallest δ is an estimate of the true clean distribution, and this worst-case risk upper bounds the risk of the true clean distribution.
In addition, this bound is tighter than the other worst-case risks w.r.t larger δ .
Formally, the upper bound w.r.t the smallest δ is given as
[Equation]
where (cid:101) U f is an equivalent constrainted set w.r.t U f for q ( x, y ) .
Then, we have
[Equation]
When l ( · ) is 0-1 loss, from Theorem 1, we know that minimize G ( θ ) is equivalent to minimize (cid:101) G ( θ ) .
Thus, we can minimize (cid:101) G ( θ ) instead of G ( θ ) .
[Equation]
Minimize the Eq.(30) enables us to minimize the Eq.(28) without knowing the true divergence parameter δ beforehand.
Usually, minimizing the upper bound can decrease the true risk under clean distribution.
Particularly, when the clean distribution coincides with the worst-case estimate w.r.t the smallest δ , minimizing the risk under the corrupted training distribution leads to the same minimizer as minimizing the risk under the clean distribution.
Relationship between label corruption and general corruption
Label corruption is a special case of general corruption.
Label corruption restricts the corruption in the space Y instead of the space X × Y .
That is to say, the training distribution p ( x ) is same as the clean distribution q ( x ) over X .
Then, we have the robust risk for label corruption as
[Equation]
) where H := { q ( x, y )
| q ( x ) =
p ( x ) , ∀ ( x, y ) ∈
X × Y } .
The supremum in G y ( θ ) is taken over (cid:101)
U
f ∩ H , while the supremum in G ( θ ) is taken over (cid:101) U f .
Due to the additional constrain q ( x ) = p ( x ) , ∀ ( x, y ) ∈
X × Y , we thus know that the robust risk G y ( θ ) is bounded by G ( θ ) , i.e., G y ( θ ) ≤
G ( θ ) .
Moreover, it is more piratical and important to be robust for both label corruption and feature corruption.
B P ROOF OF T HEOREM
Proof.
Because 1 (cid:0) u < 0 (cid:1) ≤ l ( u ) , we have (cid:80) n i =1 l ( u i ) ≥ (cid:80) n i =1 1 (cid:0) u
i < 0 (cid:1) .
Then
[Equation] [Equation]
=
[Equation]
( u i )
[Equation]
(cid:88) n i =
[Equation]
(cid:88) n i =
[Equation]
(cid:0) u
[Equation] [Equation]
[Equation]
Since loss (cid:98) J ( u ) = (cid:80) ni =1 l ( u i ) , we obtain Q ( u ) ≤ (cid:98) J ( u ) .
On the other hand, we have that
[Equation]
[Equation] [Equation]
u i < 0 (cid:1) (37)
Since J ( u ) = (cid:80) n i =1 1 (cid:0) u i < 0 (cid:1) , we obtain Q ( u ) ≥ J ( u )
C P ROOF OF C OROLLARY
Proof.
Since n = mb , similar to the proof of Q ( u ) ≤ (cid:98) J ( u ) , we have (cid:98) Q ( u ) =
(cid:88)
b j = min
[Equation] [Equation]
[Equation]
ij l
[Equation]
u ij )
[Equation]
− (cid:88) m i =1 v ij + (cid:88) m i =1 1 (cid:0) u ij < 0 (cid:1)(cid:1) ≤ (cid:88)
b
[Equation]
On the other hand, since the group (batch) separable sum structure
[Equation]
we have that
[Equation]
D P ROOF OF P ARTIAL O PTIMIZATION T HEOREM (T HEOREM 4)
Proof.
For simplicity
[Equation]
let l i
= l ( u i )
[Equation]
i ∈ { 1 , ..., n } .
Without loss of generality, assume l 1 ≤ l 2 · · · ≤ l n .
Let v ∗ be the solution obtained by Algorithm 1.
Assume there exits a v such that
[Equation]
Let T = n (cid:80) i
=1 v i and T ∗ = n (cid:80) i =1 v ∗ i .
Case 1: If T = T ∗ , then there exists an v k = 1 and v ∗ k = 0 .
From Algorithm 1, we know k > T ∗ ( v ∗ k =
⇒ k > T ∗ ) and l k ≥ l j , j ∈ { 1 , ..., T ∗ } .
Then we know n (cid:80) i
=1 v ∗ i l i ≤ n (cid:80) i =
1 v i l i .
Thus, we can achieve that
[Equation] [Equation]
This contradicts the assumption in Eq.(44)
Case 2: If T > T ∗ , then there exists an v k = 1 and v ∗ k = 0 .
Let L T ∗ =
T ∗ (cid:80) i =1 l i .
Since l k ≥ 0 , we have L T ∗ + l k ≥ L T ∗ .
From Algorithm 1, we know that L T ∗ + l k > C − T ∗ .
Thus we obtain that
[Equation] [Equation]
This contradicts the assumption in Eq.(44)
Case 3: If T < T ∗ , we obtain C − T ≥ C − T ∗ + 1 .
Then we can achieve that
[Equation]
This contradicts the assumption in Eq.(44).
Finally, we conclude that v ∗ obtained by Algorithm 1 is the minimum of the optimization problem given in (13).
E P ROOF OF P ROPOSITION
Proof.
Note that T ∗ = n (cid:80) i
=1 v ∗ i , from the condition of v ∗ i = 1 in Algorithm 1, we know that L T ∗ ≤
C +1 − T ∗ .
From the condition of v ∗ k = 0 in Algorithm 1, we know that L T ∗ +1 > C − T ∗ .
Because l ( u i ) ≥ 1 (cid:0) u i < 0 (cid:1) ≥ 0 for i ∈ { 1 , ..., n } , we have L T ∗ +
=
L T ∗ + l ( u T ∗ +1 ) ≥ L T ∗ .
Thus, we obtain L T ∗ +1 > max( L T ∗ , C − T ∗ ) .
By substitute the optimum v ∗ into the optimization function, we obtain that
[Equation]
F P ROOF OF T HEOREM
Proof.
We ﬁrst prove that objective (11) is tighter than the loss objective (cid:98) J ( u ) in Eq.(8).
After this, we prove that objective (11) is an upper bound of the 0/1 loss deﬁned in equation (7).
For simplicity, let l i
=
l ( u i ) , we obtain that
[Equation] [Equation]
[Equation]
Note that (cid:98) J ( u ) =
n (cid:80) i =
1 l ( u i ) , thus, we have E ( u ) ≤ (cid:98) J ( u ) .
Without loss of generality, assume l 1 ≤ l 2 · · · ≤ l n .
Let L i
=
i (cid:80) j
=1 l j , T = n (cid:80) i =1 v ∗
i , where v ∗ =[ v ∗ 1 , v ∗ 2 · · · v ∗ n ] T is the optimum of v for ﬁxed u .
Let k = n (cid:80) i =1 1 ( u i ≥ 0) .
Then we achieve that the 0/1 loss J ( u ) is as follows:
[Equation]
From Algorithm 1 with C = n , we achieve that L T ≤ n − T + 1 and L T+1 > n − T .
Case 1: If k ≥ T , we can achieve that
[Equation]
Case 2: If k < T, n − T ≥ L T , we can obtain that
[Equation]
Since k < T , if follows that
[Equation] [Equation]
[Equation]
Together with n − T ≥ L T , we can obtain that
[Equation]
Thus, we can achieve that
[Equation]
Case 3: If k < T, n − T <
L T , we can obtain that
[Equation] [Equation]
( n −
[Equation]
n
[Equation]
k
[Equation]
From (67), we have L T ≥ T − k .
Together with (72), it follows that
[Equation]
Finally, we can achieve that J ( u ) ≤ 2 E ( u ) ≤ 2 (cid:98) J ( u ) .
G P ROOF OF C OROLLARY
Proof.
Since n = mb , similar to the proof of (cid:98) Q ( u ) ≤ (cid:98) J ( u ) , we have
[Equation]
On the other hand, since the group (batch) separable sum structure, we have that
[Equation]
Together with Theorem 3, we obtain that J ( u ) ≤ 2 E ( u ) ≤ 2 (cid:98) E ( u ) ≤ 2 (cid:98) J ( u )
[Figure]
H M ULTI -C LASS E XTENSION
For multi-class classiﬁcation, denote the groudtruth label as y ∈ { 1 , ..., K } .
Denote the classiﬁcation prediction (the last layer output of networks before loss function) as t i , i ∈ { 1 , ..., K } .
Then, the classiﬁcation margin for multi-class classiﬁcation can be deﬁned as follows
[Equation]
We can see that 1 (cid:0) u < 0 (cid:1) = 1 (cid:0) t y − max i (cid:54) =
y t i < 0 (cid:1) is indeed the 0-1 loss for multi-class classiﬁcation.
With the classiﬁcation margin u , we can compute the base loss l ( u ) ≥ 1 (cid:0) u < 0 (cid:1) .
In this work, we employ the hinge loss.
As we need the upper bound of 0-1 loss, the multi-class hard hinge loss function Moore & DeNero (2011) can be deﬁned as
[Equation]
The multi-class hard hinge loss in Eq.(81) is not easy to optimize for deep networks.
We propose a novel soft multi-class hinge loss function as follows:
[Equation]
The soft hinge loss employs the LogSumExp function to approximate the max function when the classiﬁcation margin is less than zero, i.e., misclassiﬁcation case.
Intuitively, when the sample is misclassiﬁed, it is far away from being correctly separate by a positive margin (e.g. margin u ≥ 1 ).
In this situation, a smooth loss function can help speed up gradient update.
Because
LogSumExp ( t ) > max i ∈{ 1 , ··· K } t i we know that the soft hinge loss is an upper bound of the hard hinge loss, i.e., S ( t , y ) ≥ H ( t , y ) .
Moreover, we can obtain a new weighted loss F ( t , y ; β ) =
βS ( t , y )
+ (1 − β ) H
( t , y ) , β ∈ [0 , 1] that is also an upper bound of 0-1 loss.
I E VALUATION OF E FFICIENCY OF THE P ROPOSED S OFT - HINGE L OSS
We compare our soft multi-class hinge loss with hard multi-class hinge loss Moore & DeNero (2011) on CIFAR100 dataset training with Adam and SGD optimizer, respectively.
We keep both the network architecture and hyperparameters same.
We employ the default learning rate and momentums of Adam optimizer in PyTorch toolbox, i.e. lr = 10 − 3 , β 1 = 0 .
9 , β 2 = 0 .
For SGD optimizer, the learning rate ( lr ) and momentum ( ρ ) are set to lr = 10 − 2 and ρ = 0 .
respectively.
The pictures of training/test accuracy v.s number of epochs are presented in Figure 4.
We can observe that both the training accuracy and the test accuracy of our soft hinge loss increase greatly fast as the number of epochs increase.
In contrast, the training and test accuracy of hard hinge loss grow very slowly.
The training accuracy of soft hinge loss can arrive 100% trained with both optimizers.
Both training and test accuracy of soft hinge loss are consistently better than hard hinge loss.
In addition, training accuracy of hard hinge loss can also reach 100% when SGD optimizer is used.
However, its test accuracy is lowever than that of soft hinge loss
. J I MPACT OF M ISSPECIFIED E STIMATION OF N OISE R ATE (cid:15)
We empirically analyze the impact of misspeciﬁed prior for the noise rate (cid:15) .
The average test accuracy over last ten epochs on MNIST for different priors are reported in Table 4.
We can observe that
NPCL is robust to misspeciﬁed prior for small noise cases (Symmetry-20%).
Moreover, it becomes a bit more sensitive on large noise case (Symmetry-50%) and on the pair ﬂipping case (Pair-35%)
[Table]
[Table] [Table]
[Table]
K R ELATED L ITERATURE
Curriculum Learning:
Curriculum learning is a general learning methodology that achieves success in many area.
The very beginning work of curriculum learning (Bengio et al., 2009) trains a model gradually with samples ordered in a meaningful sequence, which has improved performance on many problems.
Since the curriculum in (Bengio et al., 2009) is predetermined by prior knowledge and remained ﬁxed later, which ignores the feedback of learners, Kumar et al.
(Kumar et al., 2010) further propose Self-paced learning that selects samples by alternative minimization of an augmented objective.
Jiang et al.
(Jiang et al., 2014) propose a self-paced learning method to select samples with diversity.
After that, Jiang et al.
(Jiang et al., 2015) propose a self-paced curriculum strategy that takes different priors into consideration.
Although these methods achieve success, the relation between the augmented objective of self-paced learning and the original objective (e.g. cross entropy loss for classiﬁcation) is not clear.
In addition, as stated in (Jiang et al., 2018), the alternative update in self-paced learning is not efﬁcient for training deep networks.
Learning with Noisy Labels: The most related works are the sample selection based methods for robust learning.
This kind of works are inspired by curriculum learning (Bengio et al., 2009).
Among them, Jiang et al.
(Jiang et al., 2018) propose to learn the curriculum from data by a mentor net.
They use the mentor net to select samples for training with noisy labels.
Co-teaching (Han et al., 2018b) employs two networks to select samples to train each other and achieve good generalization performance against large rate of label corruption.
Co-teaching+ (Yu et al., 2019) extends Coteaching by selecting samples with disagreement of prediction of two networks.
Compared with
Co-teaching/Co-teaching+, our CL is a simple plugin for a single network.
Thus both space and time complexity of CL are half of Co-teaching’s.
Recently, Zhang & Sabuncu (2018) propose a generalized Cross-entropy loss for robust learning.
Construction of tighter bounds of 0-1 loss: Along the line of construction of tighter bounds of the 0-1 loss, many methods have been proposed.
To name a few, Masnadi-Shirazi et al.
(MasnadiShirazi & Vasconcelos, 2009) propose Savage loss, which is a non-convex upper bound of the 0-
[Table]
[Figure]
loss function.
Bartlett et al.
(Bartlett et al., 2006) analyze the properties of the truncated loss for conventional convex loss.
Wu et al.
(Wu & Liu, 2007) study the truncated hinge loss for SVM.
Although the results are fruitful, these works are mainly focus on loss function at individual data point, they do not have sample selection property.
In contrast, our curriculum loss can automatically select samples for training.
Moreover, it can be constructed in a tighter way than these individual losses by employing them as the base loss function.
L A RCHITECTURE OF N EURAL N ETWORKS
CNN on MNIST
CNN on CIFARCNN on CIFAR28 ×
Gray Image 32 × 32 RGB Image
× 32 RGB Image 3 × 3 conv, 128 LReLU 3 × 3 conv, 128 LReLU 3 × 3 conv, 128 LReLU 2 × 2 max-pool, stridedropout, p = 0.
[Table]
conv, 256 LReLU 3 × 3 conv, 256 LReLU 3 × 3 conv, 256 LReLU
[Table]
-pool, stridedropout, p
[Table] [Table]
conv, 512 LReLU
[Table]
conv
[Table]
256 LReLU
[Table]
LReLU
[Table]
[Figure]
[Figure] [Figure]