A Curriculum Perspective of Robust Loss Functions
Anonymous Author(s)
AffiliationAddress email
Abstract
Learning with noisy labels is a fundamental problem in machine learning.
A largebody of work aims to design loss functions robust against label noise.
However,it remains open questions why robust loss functions can underfit and why lossfunctions deviating from theoretical robustness conditions can appear robust.
Totackle these questions, we show that each loss function implicitly induces a sampleweighting curriculum.
We then adopt the resulting curriculum perspective toanalyze how robust losses interact with various training dynamics, which helpselucidate the above questions.
Based on our findings, we propose simple fixesto make robust losses that severely underfit competitive to state-of-the-art losses.
Notably, our curriculum perspective connects robust losses to the seemingly distinctapproaches based on curriculum learning.
Introduction
Labeling errors are non-negligible [1] in datasets from expert annotations [2, 3], crowd-sourcing[4] and automatic annotations [5, 6].
The resulting noisy labels can hamper generalization, as overparameterized neural networks can memorize all training samples [7].
To combat the impact of noisylabels, a large body of research aims to design loss functions robust against label noise [8–13], whichis the main focus of this work.
Theoretical results show that losses satisfying robustness conditions[9, 11] will lead to the same optimum with clean or noisy labels.
Existing robustness conditions are agnostic to training dynamics.
Although theoretically appealing,they may fail to fully characterize the performance of robust losses with noisy labels.
In particular,it has been shown that (1) robust losses can underfit difficult tasks [1, 10, 12, 13], while (2) lossesfailing to satisfy theoretical robustness conditions [12–14] can exhibit robustness.
The reasons behindthese observations remain open questions.
For (1), existing explanations [10, 15] can be limited,which we discuss in §2.3.
For (2), to our knowledge, there is no work directly addressing it.
To tackle the above questions, we consider training dynamics in our analysis, which complementsexisting robustness conditions.
By rewriting loss functions into a standard form, we show thateach loss function implicitly induces a sample-weighting curriculum (§3). Such a perspectiveconnects robust losses to the seemingly distinct [1] approaches with curriculum learning [16–20] fornoise-robust training.
The original definition [21] of curriculum learning aims to present trainingsamples with gradually increasing difficulty and diversity to ease learning.
We adopt a generalizeddefinition of curriculum [22], i.e., a curriculum specifies a sequence of re-weighting of trainingsample distributions, which can manifest as sample weighting [16–18] or sample selection [19, 20].
The curriculum perspective helps elucidate underfitting and noise robustness from the interactionbetween the sample-weighting curriculums and various training dynamics.
We first attribute theunderfitting problem to the marginal average sample weights from the implicit curriculums (§4.1)
We then show that an increased number of classes leads to low initial sample weights with fixedcurriculums (§4.2).
By adapting the curriculums accordingly, we make robust losses that severely underfit perform competitively to state-of-the-art losses (§4.2).
Finally, we attribute the noiserobustness of losses to higher average sample weights for clean samples compared to noisy ones(§4.3).
We hypothesize that with sample-weighting curriculums that magnify the learning speeddifferences and neglect unlearnt samples, clean samples will receive more weights, explaining theevolution of sample performance during training (§4.3).
Such an explanation leads to two unexpectedresults when viewed from existing robustness conditions: by simply changing the learning rateschedule, robust losses can be vulnerable to label noise and cross entropy can appear robust (§4.3).
Background
We review classification, label noise, sufficient conditions for noise robustness and several typicalrobust loss functions before summarizing the open questions that we aim to address.
2.1 Classification with Label Noise and Noise Robustness
The k -ary classification problem with input x ∈ R d and ground truth label y ∈ { 1 , . . . , k } can besolved with ˆ y = arg max j s j , where s j corresponds to the j -th class in the class scoring functions θ : R d → R k with parameter θ .
Given a loss function L ( s θ ( x ) , y ) , we can estimate the optimalparameter with θ ∗ = arg min θ E x ,y L ( s θ ( x ) , y ) .
The un-normalized class scores s θ ( x ) can betransformed to class probabilities with the softmax function, where the i -th element of the output p isp i =
e s i / ( (cid:80)
j e s j ) .
For notation simplicity, we omit the dependence on θ and x if possible.
The annotation process may introduce errors, resulting in a potentially corrupted label ˜
y following˜ y = (cid:26) y, with probability P (˜ y
=
y
| x , y ) j, j ̸ = y with probability P (˜ y
= j | x , y )
Noise is symmetric (or uniform) if P (˜ y
= j | x , y )
=
η/k , with η = P (˜
y ̸
= y ) the noise rate constant.
Noise is asymmetric (or class-conditional) if P (˜ y
= j | x , y ) =
P (˜ y =
j | y ) .
A loss function L isrobust against label noise if the optimal parameter ˜ θ ∗ =
arg min θ E x , ˜ y L ( s θ ( x ) , ˜ y ) estimated usingdataset with noisy labels ˜
y equals to θ ∗ .
We review two existing sufficient conditions for noise robustness to demonstrate how they are agnosticto models and training dynamics .
Loss L is called symmetric [9] if
[Equation]
where C is a constant.
With η < ( k − 1) /k , L is robust against symmetric (uniform) label noise.
Such stringent condition is later extended by Zhou et al.
Suppose a loss can be written as L ( s θ ( x ) , j ) =
l ( p j ) .
As an equivalent rephrase of the sufficient condition, L is called asymmetric if
[Equation]
With ˜ r < 1 , i.e., the dataset is dominated by clean labels, L is robust under generic label noise.
2.2 Review of Selected Losses
In addition to cross entropy (CE) that is vulnerable to label noise [9, 11], we review several typicalrobust losses for our analysis.
We ignore the difference in constant scaling factors and constantadditive bias in the loss functions .
They are either equivalent to learning rate scaling in SGD orirrelevant in gradient computation.
See Table 1 for the exact formulas and Appendix A for derivations.
Symmetric Losses Mean absolute error (MAE) [9] and the subsequent reverse cross entropy (RCE)[13] are essentially equivalent, both satisfying Eq.
Ma et al.
[10] normalize generic losses satisfying L ( s , j ) > 0 , ∀ j ∈ { 1 , . .
, K } into symmetric losses with L N ( s , y )
=
L ( s , y ) /
( (cid:80) j L ( s , j )) .
We include normalized cross entropy (NCE) as an example.
Asymmetric Losses We include two asymmetric losses [11]: asymmetric generalized cross entropy(AGCE) and asymmetric unhinged loss (AUL).
Notably, with q ≥ 1 for AGCE and q ≤ 1 in AUL,both losses are complete asymmetric [11], i.e., Eq.
(2) always holds when ˜ r < 1 . [Table]
Combined Losses Losses can be combined to enjoy both robustness and sufficient learning.
Forexample, generalized cross entropy (GCE) [12] can be viewed as a smooth interpolation between CEand MAE.
Alternatively, symmetric cross entropy (SCE) [13] uses a weighted average of CE and RCE (MAE).
Finally, Ma et al.
[10] propose the active-passive-loss framework and argue that robustand sufficient learning requires a balanced combination of active and passive losses.
Suppose loss L can be rewritten into L ( s , y )
=
(cid:80) j l ( s , j ) .
An active loss requires ∀ j ̸ = y, l ( s , j ) = 0 , while apassive one satisfies ∃ j ̸ = y, l ( s , j ) ̸ = 0 .
Accordingly, CE and NCE are active losses while MAE(RCE) is a passive loss.
Note that passive losses can be viewed as a combination of active and passivelosses.
We include NCE+MAE as an example.
2.3 Open Questions Addressed in this Work
Why do robust losses underfit?
Ma et al.
[10] attribute underfitting to failure in balancing activepassive components.
However, different specifications of L ( s , y )
= (cid:80) j l ( s , j ) can lead to ambiguities in the active-passive dichotomy.
For example, with L MAE ( s , y )
∝ (cid:80) j | I ( j = y ) − p y | where I ( · )is the indicator function, MAE is a passive loss; yet the equivalent L MAE ( s , y ) ∝ (cid:80) [-ki =
-]j I ( j = y )(1 − p y )makes MAE an active loss.
Wang et al.
[15] view ∥∇ s L ( s , y )
∥ 1 as weights for sample gradientsand attribute underfitting to their low variance, making clean and noisy samples less distinguishable.
However, as we show in §4.1, MAE also underfits on clean datasets.
Therefore, why robust lossesunderfit remains an open question.
What affects the robustness of a loss?
Although combined losses such as GCE and SCE fail tosatisfy existing robustness conditions in Eq.
(1) and (2), it is unclear why they also exhibit robustnessagainst label noise [12, 13].
Furthermore, it is unclear how training dynamics, which are irrelevant inexisting robustness conditions, affect the noise robustness of a loss.
Implicit Curriculums of Robust Losses
We derive the standard form of losses and show that each implicitly induces a sample-weightingcurriculum, which helps examine how robust losses interact with various training [-dynamics.3.
-]dynamics.
3.1 The Implicit Sample-Weighting Curriculum
Losses in Table 1 generally follow the form L ( s , y )
=
l ( p y ) , where p y is the softmax output of classy .
Note that p y can be rewritten as
[Equation]
where
[Equation]
is the soft margin between s y and the maximum of other scores, a smooth approximation of thehard margin ∆ ∗ y .
With classifier ˆ y = arg max j s j , ∆ y indicates how well a sampled is learnt, as∆ y ≥ 0 leads to ∆ ∗ y ≥ 0 , which ensures successful classification using scores s .
Since ∇ s l ( p y )
=
l ′ ( p y ) · p ′ y (∆ y ) · ∇ s ∆ y , losses can be rewritten into a standard form with equivalent gradients :
[Equation]
(5) where stop _ grad( · ) avoids backpropagating through w (∆ y )
=
l ′ ( p y ) · p ′ y (∆ y ) .
Each loss in the formof Eq.
(5) thus implicitly induces a sample-weighting curriculum with w (∆ y ) as the sample weightand ∆ y as the implicit loss .
Such a view is only valid with first-order derivatives.
By examiningthe interaction between w (∆ y ) and various training dynamics, we can elucidate the reasons behindunderfitting and noise robustness.
Table 1 shows the derived sample weights for losses in §2.2.
By analyzing ∇ s L ( s , y ) , Wang et al.
[14, 15] extract ∥∇ s L ( s , y ) ∥ 1 as weights for sample gradients,which share similar forms in Table 1.
Instead of directly weighting sample gradients, our derivationidentifies the implicit loss ∆ y , making the resulting sample-weighting scheme compatible with thestandard definition of curriculum learning [22].
In addition, the extracted ∆ y and ∆ ∗ y can serve asdirect metrics for sample performance in curriculum learning, compared to loss [23, 24] and gradientmagnitude [25] that are affected by preference specified with sample weight w (∆ y ) of a loss function.
Finally, the ∆ y distribution is essential in analyzing the interaction between losses and trainingdynamics as shown in §4.
3.2 The Additional Entropy-Reducing Curriculum of NCE
As shown in Table 1, due to normalization, L NCE ( s , y ) additionally depends on p j where j ̸ = y ,which cannot be trivially rewritten into Eq.
A derivation of the gradient gives
[Equation]
Thus NCE can be rewritten as
[Equation]
The first term results in a similar sample-weighting curriculum as CE with an additional weightγ NCE = 1 / ( (cid:80) j − log p j ) ≤ 1 / ( k log k ) which has marginal magnitude when k is large.
The secondterm is a regularizer
[Equation]
which reduces the entropy of the softmax output p .
The regularizer has per-sample weights ϵ NCE
=k ( − log p y ) /
( (cid:80) j − log p j ) .
It can thus be interpreted as a regularization curriculum.
Notably, thetwo curriculums work synergically in reducing the entropy of the softmax output.
As ∆ y inducesgradients with constant L1 {+norm ∥∇ s ∆ y ∥ 
 = 2 as shown in Appendix B,+} we approximate theeffective sample-weighting curriculum with w NCE for analysis of underfitting (§4.1):
[Equation]
with derivations in Appendix B. Note that ∇ s L NCE ( s , y ) and ∇ s ∆ y may have different directions.
Understanding Robust Losses with Their Implicit Curriculums
We empirically investigate the interaction between sample-weighting curriculum w (∆ y ) and varioustraining dynamics for questions in §2.3, following the standard settings [10, 11].
Experiments arebased on the standard MNIST [26] and CIFAR10/100 [27] benchmarks with synthetic symmetric andasymmetric label noises.
We use a 4-layer CNN for MNIST, an 8-layer CNN for CIFAR10 and a ResNet-34 [28] for CIFAR100, all with batch normalization [29].
Without further specifications, allmodels are trained with a fixed number of epochs using SGD with momentum, weight decay andcosine learning rate annealing .
Different from the standard settings, as hyperparameters of lossescan change the maximum of w (∆ y ) , which essentially adjust the learning rate of SGD, we normalizethe maximum of w (∆ y ) to 1 to avoid complications. We leave hyperparameters, data augmentationand noise generation to Appendix C. We state explicitly when deviating from the standard settings. [Table]
− − −
[Figure]
4.1 Underfitting of Robust Losses from a Sample-Weighting Curriculum Perspective
Robust losses do underfit.
We confirm that on difficult tasks like CIFAR100 [10, 12, 13], underfitingresults from robust losses themselves rather than inferior hyperparameters.
We tune hyperparametersof each loss on CIFAR100 and report results on CIFAR100 and CIFAR10 without label noise in
Table 2. As shown in Table 2, the performance of NCE, AGCE and AUL lag behind CE by a nontrivialmargin on CIFAR100.
Notably, MAE performs much worse compared to CE, similar to AGCE and AUL with inferior hyperparameters.
In contrast, there is no underfitting in CIFAR10.
See Table 5 in Appendix D for similar results with a smaller learning rate.
Marginal effective learning rate α ∗ t explains underfitting.
We attribute underfitting of a loss tothe diminishing effective learning rate α ∗ t = ¯ w t · α t , where ¯ w t is the average sample weights of thebatch and α t the learning rate at step t .
Let ¯ α ∗ t = (cid:80) ti =1 α ∗ i /t be the average effective learning rateup to step t .
In Table 2 we report ¯ α ∗ t at the final step. For losses that heavily underfit on CIFAR100,their final ¯ α ∗ t is marginal compare to CE.
Marginal α ∗ t due to marginal initial sample weights.
To examine why robust losses result inmarginal α ∗ t , in Fig.
1 we compare their sample-weighting functions w (∆ y ) to the ∆ y distribution of CIFAR10 and CIFAR100 at initialization.
For robust losses that severely underfit (Fig. 1a), the ∆ ydistribution of CIFAR100 concentrates at regions with marginal sample weights, resulting in smalleffective learning rate α ∗ t .
It can be hard for these samples to escape the region with marginal weightsbefore the learning rate attenuates.
In contrast, losses with non-trivial initial sample weights (Fig. 1b)result in moderate or no underfitting in Table 2.
As a corroboration, we plot the average effectivelearning rate ¯ α ∗ t of AUL with different hyperparameters in Fig.
2a.
With superior hyperparameters( a = 1 . 1 , q = 5 . 0 ), ¯ α ∗ t quickly increase to a non-negligible value before annealing.
In contrast, ¯ α ∗ tstays marginal with inferior hyperparameters ( a = 3 . 0 , q = 0 . 7 ). [Figure]
Marginal α ∗ t due to fast diminishing sample weights.
In Fig.
2b, different from other robustlosses but similar to CE, α ∗ t of NCE peaks at initialization.
However, it decreases substantially fastercompared to CE, which can be attributed to the synergy between the two implicit curriculums of NCEin reducing w NCE .
As ∆ y improves, γ NCE , ϵ NCE and w CE all decreases.
In addition, the regularizer R CE ( s ) further decreases the entropy of softmax output and thus γ NCE .
As a result, w NCE decreasesmuch faster compared to w CE , leading to faster attenuating α ∗ t .
Loss combination mitigates marginal initial sample weights.
As w CE and w NCE peak at initialization, they compensate the marginal initial sample weights when combined with other robustlosses, helping initial learning and thus avoiding underfitting.
In Table 2, the effective learning rateon CIFAR100 is substantially increased when combining MAE with CE and NCE.
Interestingly, CEand NCE are both “active” as their sample weights peak at initialization, while other robust losses are“passive” due to their marginal initial sample weights.
Such dichotomy based on sample-weightingcurriculums complements the active-passive dichotomy of losses [10] from a distinct perspective.
4.2 Addressing Underfitting by Adapting the Sample-Weighting Curriculums
Although robust losses can underfit CIFAR100, the problem is absent on [-CIFAR
 but CIFAR10.-]CIFAR10 in Table 2.
Suchdifference has been vaguely attributed to the increased difficulty of CIFAR100 [1, 12].
We furthershow that the issue lies specifically in the increased number of classes k when combined with fixedsample-weighting curriculums.
By adapting the curriculums accordingly, robust losses that severelyunderfit can become competitive with the state-of-the-art losses.
Given findings in §4.1, we use MAEas a typical example.
Intuitively, the larger number of classes, the more subtile differences to be distinguished among theclasses, thus the harder the task is.
In addition, the number of classes k determines the ∆ y distributionat initialization.
Assuming that class scores s j at initialization are i.i.d.
variables following thenormal distribution s j ∼ N ( µ, σ ) .
In particular, µ = 0 and σ = 1 for most neural networks withstandard initializations [30] and normalization layers [29, 31]. ∆ y = s y − log (cid:80) j ̸ =
y e s j is then acombination of a normal random variable and the logarithm of sum of log-normal random variables. We simulate the distribution of ∆ y with different number of classes k in Fig. 3a. The simulation fitsthe real distribution in geneal as shown in Fig. 6 of Appendix E. With more output class, the ∆ ydistribution has similar variance but shifts further left, corresponding to diminishing sample weightswith the fixed MAE curriculum.
The expected ∆ y can be approximated with
[Equation]
See Appendix E for detailed derivations.
Subtracting E (∆ y ) from ∆ y centers the distribution at 0where w MAE (∆ y ) peaks.
As shown in Fig.
3b, the centered distributions virtually overlaps.
Shifting or rescaling ∆ y mitigates underfitting from increased number of classes.
To assignnontrivial sample weights at initialization, the sample-weighting curriculum of robust losses shouldbe adapted according to the number of classes k .
A simple strategy is to make the expected initialsample weights agnostic to k .
Given a sample-weighting function w (∆ y ) , we can either shift
[Equation]
− − −  5 0.
k= k= k= k= (a) Simulated distributions − − −   (b) Shifted by − E [∆ y ] − − −   0.
1.0 MAE
MAE-shift MAE-rescale (c) Expected initial w (∆ y )
Clean
Symmetric Noise (Noise Rate η ) Loss η =η = 0 .η = 0 .η = 0 .
[Table]
η =
[Table]
[Table]
or rescale
[Equation]
it, where a is a hyperparameter.
As shown in Table 3, we find that a = 2 . 6 makes MAE comparableto the previous best (NCE+AUL) [11] on CIFAR100 with symmetric label noise, which substantiallyoutperforms the vanilla MAE that underfits.
Notably, since E [∆ y ] ≈ 2 . 6 when k = 10 , µ = 0 andσ = 1 , we essentially match the expected initial sample weights of CIFAR100 to that of CIFAR10, asillustrated in Fig. 3c.
For similar results with asymmetric label noise, see Table 6 in Appendix E.
4.3 Noise Robustness from a Sample-Weighting Curriculum Perspective
Intuitively, losses exhibiting noise robustness should weight clean samples more than noisy ones.
Weprovide an explanation based on the interaction between w (∆ y ) and two different training dynamics.
As NCE additionally involves a regularization curriculum, we leave its discussions for future work.
Robust losses assign larger weights to clean samples.
The average weight assigned to noisy samplesduring training can be characterized with ¯ w noise =
(cid:80)
i,t I (˜ y
i,t = y i,t ) w i,t / ( (cid:80) i,t I (˜ y
i,t ̸ =
y i,t ) α t ) ,where we report the drop in test accuracy and snr on CIFAR10 under symmetric label noise in Table 4.
Losses with less performance drop under label noise has higher snr in general.
To explain what leads to a high snr, we first examine how ∆ y distributions of noisy and clean samplesevolve during training on CIFAR10 in Fig.
When trained with robust losses (Fig. 4b and 4c), thenoisy and clean distributions of ∆ y gets better separated and more spread.
In addition, ∆ y ’s of somenoisy samples gets decreased, suggesting that noisy samples can be unlearnt .
In contrast, with CE(Fig. 4a), the noisy and clean distributions of ∆ y are less separated and more compact.
We now give a possible explanation for Fig.
4 with the following two training dynamics: (D1) cleansamples are learnt faster than noisy samples ; (D2) noisy samples can be unlearnt when trainedon clean samples.
D1 is identified in [7, 32] and later manifests in curriculum-based robust training Clean η = 0 .η = 0 . [Table]
η = 0 . [Table] η = [Table]
Loss
Hyperparams
[Table]
AGCE
[Table]
[Figure]
It can result from the dominance of clean samples ( ˜ r < 1 ) in the expected gradient.
Inaddition, gradients of clean samples are more correlated than those of noisy samples [35].
Thusperformance on clean samples can be improved when training on one another, leading to D1.
D
onlybecome apparent when examining Fig.
4b and 4c, which can result from generalization with cleansamples.
Suppose in MNIST, a sample of 0 is erroneously labeled as 9.
Then a model well-trainedwith clean samples of class 9 and 0 can result in a low ∆ y for this noisy sample.
D1 and D2 can actin synergy to separate the clean and noisy distributions of ∆ y , as shown in Fig. 4.
We hypothesis that robust losses enhance the synergy of D1 and D2.
In Table 1, w (∆ y ) of lossescan be decomposed into f (∆ y ) · g (∆ y ) , where f (∆ y ) is a monotonically increasing function andg (∆ y )
a decreasing one.
For example, f CE (∆ y ) degenerates to constant 1 and g CE (∆ y )
= 1 − p y ,while f MAE (∆ y ) =
p y and g MAE (∆ y )
= 1 − p y .
Notably, g (∆ y ) shared by all losses converge to0 as ∆ y increases, preventing ∆ y from growing infinitely large.
In addition, a non-degeneratedf (∆ y ) can enhance the synergy between D1 and D2 .
Since the initial ∆ y distribution generallylies on the monotonically increasing part of w (∆ y ) determined by f (∆ y ) , faster learning of samplesresults in their larger weights during training.
Thus robust losses magnify the difference in learningspeed between clean and noisy samples, which can also account for the substantially spread ∆ ydistributions in Fig.
4b and 4c.
As w (∆ y ) can assign negligible sample weights with low ∆ y dueto the monotonically increasing f (∆ y ) , unlearnt noisy samples are neglected with diminishingweights, which can account for the decrease of ∆ y ’s for noisy samples in Fig.
4b and 4c.
In contrast,as w CE (∆ y ) assign high sample weights for small ∆ y ’s, it compensates the synergy of D1 and
D2, thus results in compact ∆ y distribution, larger ∆ y ’s for noisy samples, and less separated ∆ ydistributions in Fig.
4a.
With sufficient training, clean samples will eventually have high ∆ y ’s with diminishing sampleweights thanks to g (∆ y ) .
Noisy samples then dominate the expected gradient and can lead tooverfitting, leading to two unexpected results when viewed from robustness conditions [9, 11]:
Robust losses are vulnerable to label noise with extended training.
In Fig.
5a we show the learningcurve of CE and MAE using constant learning rate under different symmetric noises on MNIST. [Figure] a
[Figure] Although enjoying theoretically guaranteed noise robustness [9, 11], similar to CE, MAE eventuallyoverfits to noisy samples, becoming vulnerable to label noise as weights of clean samples diminish.
Losses can become robust by adjusting the learning rate schedule.
Interestingly, in Fig.
4a, despitethe compensation of w CE (∆ y ) , the synergy between D1 and D2 still results in partially-separated∆ y distributions of noisy and clean samples.
We can thus improve the noise robustness of CE bypreventing the weights of clean samples from diminishing due to g (∆ y ) , which can be achieve byslowing down the convergence or early stopping [36].
In Fig.
5b we show the learning curve of CEusing fixed learning rates under symmetric noise on MNIST.
By simply increasing or decreasing thelearning rate, which strengthens the implicit regularization of SGD [37] or directly slows down theconvergence, the noise robustness of CE can be substantially improved.
Related Work
Our work is closely related to robust loss functions [8–13] for robust training with noisy labels [1].
Theoretical results [9, 11] derive sufficient conditions for robustness against label noise without considering the training dynamics.
We complement these results by considering the interaction betweenrobust losses and various training dynamics.
The underfitting of robust losses has been heuristicallymitigated with loss combination [10, 12, 13].
We further elucidate the cause of underfitting from acurriculum perspective, based on which we provide an effective solution.
Curriculum-based approaches combat label noise with either sample selection [19, 20] or sampleweighting [16–18].
In particular, sample weights are designed [14–16] or predicted by a modeltrained on a separated dataset [17, 18].
In contrast, the sample-weighting curriculums consideredin this work are implicitly induced by robust loss functions.
Most related to our work, Wang et al.[14] identifies gradient norms as weights for sample gradients of each robust loss.
In contrast, asdiscussed in §3.1, we explicitly extract the implicit loss, which helps draw the connection to standardcurriculum learning [22] and facilitates analysis of training dynamics.
Our work is also related to the ongoing debate [22, 38] on strategies for selecting or weightingsamples in curriculum learning: whether easier first [21, 23] or harder first [24, 39] is better.
Theimplicit curriculums of robust losses in this work differ in two important ways.
First, the implicitloss identified in §3.1 more directly measures sample difficulty than loss value [23, 24] and gradientmagnitude [25].
Second, the implicit sample-weighting curriculums can be viewed as a combinationof both weighting strategies by emphasizing moderately difficult samples, as discussed in §4.3.
Conclusion
We identify the implicit sample-weighting curriculums of selected loss functions.
The resultingimplicit loss as a direct sample performance metric and sample weight specifying the implicitpreference of each loss function can help analyze and design robust loss functions and curriculums.
Following such a curriculum perspective, we elucidate the reasons behind underfitting and robustnessagainst label noise for existing robust loss functions, and design a simple approach to address theunderfitting issue. Notably, our curriculum perspective connects robust loss functions to the seeminglydistinct approaches based on curriculum learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In §3.1 we state that the curriculum
view is valid when considering the first order derivatives.
We also analyze the exception
with NCE in §3.2.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] In §4.2 we
explicitly state the assumed distributions of s j when deriving E [∆ y ] .
(b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix B
we
include the detailed derivations of ∥∇ s ∆ y ∥ 1 and w NCE ; in Appendix E we include the
detailed derivation of E [∆ y ] .
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] All default settings are in Appendix C, specific hyperparameters
deviation from default settings are stated near each result.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Tables 2 to 4, 5 and 6.
We omit error bars for figures to
improve readability.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Appendix C
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] At the beginning of§4, we cite MNIST, CIFAR10/100.
(b) Did you mention the license of the assets?
[N/A] MNIST and CIFAR10/100 are classic
benchmarks
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Detailed Review of Selected Losses in §2.2
The standard Cross Entropy (CE)
[Equation]
(12) is neither symmetric [9] or asymmetric [11], thus can be vulnerable to label noise.
A.1 Symmetric Losses
The classic Mean Absolute Error (MAE) proposed by Ghosh et al. [9] is
[Equation]
where I ( j = y ) is the indicator function.
Reverse Cross Entropy (RCE) [13] is
[Equation]
where log 0 is truncated to a constant A to avoid numerical overflow. Thus implementation-wise, RCE is equivalent to MAE.
Ma et al.
[10] argued that any generic losses with L ( s , j ) > 0 , ∀ j ∈ { 1 , . . .
, K } can becomesymmetric by simply normalizing them.
As an example, the Normalized Cross Entropy (NCE) is
[Equation]
A.2 Asymmetric Losses
Asymmetric Generalized Cross Entropy (AGCE) is
[Equation]
q where a > 0 and q > 0 .
It is asymmetric
when I ( q ≤ 1)( a +1 a ) 1 − q + I ( q > 1) ≤ 1 / ˜ r .
Asymmetric Unhinged Loss (AUL) is
[Equation]
q where a > 1 and q > 0 .
It is asymmetric
when I ( q ≤ 1)( aa − 1 ) q − 1 + I ( q ≤ 1) ≤ 1 / ˜ r .
[-where q > 
-]
A.3 Combined Losses
Losses can be combined in different ways.
For example, Generalized Cross Entropy (GCE)
[Equation]
14) can be viewed as a smooth interpolation between CE and MAE, where 0 < q ≤ 1 .
CE or MAE canbe recovered by setting q → 0 or q = 1 .
Alternatively, Symmetric Cross Entropy (SCE) [13] is a weighted average of CE and RCE (MAE):
[Equation]
(15) where a > 0 , b > 0 , and 0 < q < 1 .
Finally, Ma et al.
[10] propose to combine active and passive robust losses.
Here we review NCE+MAE as an example:
[Equation]
where a > 0 , b > 0 , and 0 < q < 1 .
B Additional Derivations for §3.2
Constant Norm of ∥∇ s ∆ y ∥ 1 : Since
[Equation]
then
[Equation]
Approximating w NCE in Eq.
[Equation]
The derivation is based on the inequality | x ± y | ≤ | x | + | y | , following the intuition [14, 15] that∥∇ s L NCE ( s , y ) ∥ 1 can be regarded as sample weights.
CIFAR
{+CIFAR
+}
Underfitting Loss [Table] [Figure]
C Detailed Experimental Settings
Our settings are mainly identical to [10, 11], with differences explicitly stated in the main text.
Herewe describe the default settings. All experiments are trained on NVIDIA 2080ti gpus with FP32.
Noise generation
The noisy labels are generated following [10, 11, 40].
For symmetric labelnoise, the training labels are randomly flipped to a different class with with probabilities η ∈{ 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .
Asymmetric label noise are generated by a class-dependent flipping pattern.
On CIFAR-100, the 100 classes are grouped into 20 super-classes each having 5 sub-classes.
Each classare flipped within the same super-class into the next in a circular fashion.
The flip probabilities areη ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } .
Models and Training
We use a 4-layer CNN for MNIST, an 8-layer CNN for CIFAR10 and a ResNet34 [28] for CIFAR100, all with batch normalization [29].
Data augmentation including randomwidth/height shift and horizontal flip are applied to CIFAR10/100.
Without further specifications,all models are trained using SGD with momentum 0.9 for 50, 120 and 200 epochs on MNIST, CIFAR10 and CIFAR100, respectively.
Learning rates with cosine annealing are 0.01 on MNIST and CIFAR10, and 0.1 on CIFAR100.
Weight decays are 10 − 3 on MNIST, 10 − 4 on CIFAR10 and 10 − 5on CIFAR100.
Notably, all losses are normalized to have unit maximum in sample weights, which isdifferent from [10].
D Additional Results for §4.1
Underfitting of robust losses with smaller learning rate.
We report similar results as Table 2with smaller learning rates in Table 5.
Although settings that severe underfits slightly improve, theperformance gap compared to CE is still substantial.
Such results further confirms that underfittingresults from robust losses themselves.
[Table]
E Additional Results and Derivation for §4.2
Simulated initial ∆ y approximate real dataset well.
As shown in Fig.
6 simulated distributionsapproximate CIFAR10 and CIFAR100 well, which supports the analysis with Fig.
Deriving E (∆ y ) in Eq.
[Equation]
where ≈ 1 follows E
[log X ] ≈ log E [ X ] − V
[ X ] 2 E [ X ] 2 [41], = 2 utilize properties of sum of log-normalvariables [42], and = 3 substitutes the expression of E [ e s y ] and V [ e s y ] for log-normal distributions.
Results of shifted and rescaled MAE with asymmetric label noise.
In addition to Table 3, we alsoreport results with asymmetric label noise in Table 6.
Shift and rescale ∆ y also mitigate underfittingof MAE with asymmetric label noise, resulting in performance similar to the state-of-the-arts.