U NDERSTANDING
G ENERALIZED L ABEL S MOOTHING WHEN L EARNING WITH N OISY L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels.
It was shown that LS serves as a regularizer for training data with hard labels and therefore improves the generalization of the model.
Later it was reported LS even helps with improving robustness when learning with noisy labels.
However, we observe that the advantage of LS vanishes when we operate in a high label noise regime.
Puzzled by the observation, we proceeded to discover that several proposed learning-with-noisy-labels solutions in the literature instead relate more closely to negative label smoothing (NLS), which deﬁnes as using a negative weight to combine the hard and soft labels!
We show that NLS differs substantially from
LS in their achieved model conﬁdence.
To differentiate the two cases, we will call LS the positive label smoothing (PLS), and this paper uniﬁes PLS and NLS into generalized label smoothing (GLS).
We provide understandings for the properties of GLS when learning with noisy labels.
Among other established properties, we theoretically show NLS is considered more beneﬁcial when the label noise rates are high.
We provide extensive experimental results on multiple benchmarks to support our ﬁndings too.
I NTRODUCTION
Label smoothing (LS) (Szegedy et al., 2016) is an arising learning paradigm that uses positively weighted average of both the hard training labels and uniformly distributed soft label:
[Equation]
where we denote the one-hot vector form of hard label and an all one vector as y , 1 respectively.
K is the number of label classes, and r is the smooth rate in the range of [0 , 1] .
It was shown that LS serves as a regularizer for the hard training data and therefore improves generalization of the model.
The regularizer role of LS prevents the model from ﬁtting overly on the target class.
Empirical studies have demonstrated the effectiveness of LS in improving the model performance across various benchmarks (Pereyra et al., 2017) (such as image classiﬁcation (Szegedy et al., 2016), machine translation (Vaswani et al., 2017), language modelling (Chorowski & Jaitly, 2017)) and model calibration (Müller et al., 2019).
Later it was reported LS even helps with improving robustness when learning with noisy labels (Lukasik et al., 2020).
However, we observe that the advantage of LS vanishes when we operate in a high label noise regime.
In Figure 1, we present a set of experiments on UCI datasets (Dua & Graff, 2017).
We highlight best two smooth rates (possible to have tied smooth rates) under each label noise rate.
Indeed, non-negative smooth rates (circles colored in red) outperform negative ones when the label noise rates are low.
Nonetheless, with the increasing of noise rates, negative smooth rates r < 0 (Eqn.
(1), diamonds colored in green) appear to be more competitive when learning with noisy labels.
Puzzled by the observation, we proceeded to discover that several proposed learning-with-noisy-labels solutions in the literature, including Loss Correction (Patrini et al., 2017), NLNL (Kim et al., 2019) and Peer Loss (Liu & Guo, 2020), instead relate more closely to negative label smoothing (NLS), which deﬁnes as using a negative weight to combine the hard and soft labels!
[Figure]
This paper uniﬁes label smoothing with either positive or negative smooth rate into a generalized label smoothing (GLS) framework.
Our paper is motivated by the above inconsistent observations.
And we aim to provide a more thorough understanding of GLS under the setting of learning with noisy labels, rather than proposing a new method to compete for state-of-the-art performances.
We ﬁrstly show that negative label smoothing differs substantially from positive label smoothing in their achieved model conﬁdence.
With the presence of label noise, we then proceed to show that there exists a phase transition behavior when ﬁnding the optimal label smoothing rate for GLS.
Particularly, when label noise is low, positive label smoothing is able to uncover the optimal model while negative label smoothing is considered more beneﬁcial in a high label noise regime.
We provide extensive experimental evidences to support our ﬁndings.
For instance, on multiple benchmark datasets, we present the clear transition of the optimal smoothing rate going from positive to negative when we keep increasing noise rates.
On CIFAR-10 test data, we show a negative smoothing rate elicits higher model conﬁdence on correct predictions and lower conﬁdence
on wrong predictions compared with the behavior of a positive one.
Our contributions summarize as follows:
• We provide understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels, where the label smooth rate can go negative.
(An understanding paper rather than a method paper) •
We show that several robust loss functions in the noise learning literature correspond to learning with GLS, under certain noise rate models.
(Section 3) • We show that negative label smoothing improves the conﬁdence of model prediction.
With the presence of label noise, we demonstrate learning with a negative smooth rate can be more robust to label noise compared with a positive rate when label noise rates are high.
(Section 4, 5) • Empirical experiments on multiple datasets and methods demonstrate that with the presence of label noise, NLS becomes competitively robust to label noise.
We also empirically show how GLS results in trade-offs in model conﬁdence, bias and variance of the generalization error.
(Section 6, Appendix D and E) • Extensive experiment results validate our main theoretical conclusions.
Besides, we also discuss practical considerations and multi-class extensions of GLS to mitigate the impact of label noise.
(Appendix B, C)
We defer all proofs to Appendix F. Our work primarily contributes to the literature of learning with noisy labels (Scott et al., 2013; Natarajan et al., 2013; Liu & Tao, 2015; Patrini et al., 2017; Liu & Guo, 2020).
Our core results are contingent on recent works of understanding the effect of label smoothing when training deep neural network models, i e., label smoothing improves model calibration (Müller et al., 2019), more complexed forms of label smoothing (Li et al., 2020; Yuan et al., 2020) and in particular when label noise presents (Lukasik et al., 2020; Liu, 2021).
We generalize a concept called generalized label smoothing and provide new understandings for when, instead of setting a positive smoothing rate as the literature would normally do, a negative smoothing rate is considered a better option.
Due to the space limit, we defer a more detailed discussion of related works to Appendix A.
P RELIMINARIES 2. L EARNING WITH SMOOTHED LABELS
For a K -class classiﬁcation task, we denote by X ∈ X a high-dimensional feature and Y ∈ Y :
= { 1 , 2 , ..., K }
the corresponding label.
Suppose ( X, Y ) ∈
X × Y are drawn from a joint distribution D .
Let y i be the one-hot encoded vector form of y i which generates according to Y .
The random variable of smoothed label
Y LS ,r with smooth rate r ∈ [0 , 1] generates y LS ,r i as (Szegedy et al., 2016): [Equation]
[Equation]
For example, when r = 0 . 3 , the smoothed label of y i =
[0 , 1 , 0] becomes y LS ,r =0 .
3 i = [0 .
[Equation] [Equation]
We consider a broader setting where the smoothed label might be negatively related to the corresponding feature.
As a supplementary to existed works on label smoothing, we explore the beneﬁts of learning with generalized label smoothing (GLS), i.e., r ∈ ( −∞ , 1] instead of a non-negative r .
[Equation]
where y GLS ,r i is given by the random variable of generalized smooth label Y GLS ,r .
We name the scenario r < 0 as negative label smoothing (NLS).
To clarify, we don’t assume a strict lower bound for r .
If r → −∞ , normalizing y GLS ,r i by 1 − r returns y GLS ,r i =
y
i − 1 K .
We will show when imposing a negative smoothing parameter will be considered beneﬁcial as compared to a positive one.
In the main paper, we focus on the binary classiﬁcation task where y i ∈ { 0 , 1 } and K = 2 .
And we defer multi-class extensions to Appendix C. Denote f as a deep neural network, f ( x i ) is the model prediction of x i ∈ X with element f ( x i )
y i :=
P ( Y = y i | X = x i , f ) .
Given the sample x ∈ X and a hard label y ∈ Y , binary CE loss is deﬁned as (cid:96) CE
( f ( x ) , y ) :
= − log( f ( x ) y ) .
Throughout this paper, we shorthand (cid:96) as (cid:96) CE for a clean presentation.
2. L EARNING WITH NOISY LABELS
The noisy label literature considers the setting where we only have access to samples with noisy labels from ( X, (cid:101) Y ) .
Suppose random variables ( X, (cid:101) Y ) ∈
X × (cid:101) Y are drawn from a noisy joint distribution (cid:101) D .
Statistically, the random variable of noisy labels (cid:101)
Y can be characterized by a noise transition matrix T , where each element T i,j represents the probability of ﬂipping the clean label Y
= i to the noisy label (cid:101)
Y =
j , i.e,
T ij = P ( (cid:101) Y =
j | Y = i ) .
In this paper, we are interested in the widely studied class-dependent label noise.
We assume the label noise is conditionally independent of features, i.e.,
[Equation]
For the binary classiﬁcation setting, deﬁne e 0 :
= P ( (cid:101) Y = 1 | Y = 0) , e 1 := P ( (cid:101) Y =
| Y = 1) .
Without loss of generality, we assume e 1 − e
=
e ∆ ≥ 0 .
We denote the binary noise transition matrix in the noisy label setting as: T = (cid:18) 1 − ee 0 e1 − e(cid:19) .
2. M ODEL CONFIDENCE
We deﬁne a key quantity, model conﬁdence, that plays an important role in later sections.
Deﬁnition 1.
Model conﬁdence of model f for sample ( x, y ) .
Given a model f , a sample x with its target label y ∈ { 0 , 1 } , the model conﬁdence of f w.r.t.
sample x is deﬁned as MC ( f ; x, y ) =
f ( x ) y − f ( x ) 1 − y .
MC ( f ; x, y ) in deﬁnition 1 characterizes the difference of the predicted probability between target class and the other class.
MC ( f ; x, y ) = 0 simply means f has no conﬁdent on its predictions since the model can not identify the target class of x . MC ( f ; x, y ) returns a negative value when f gives a wrong prediction and is not conﬁdent to predict the label of x as the target label y .
To dig into how GLS inﬂuences the model conﬁdence on correct and wrong predictions in following sections, we separate the distribution D into
[Equation]
C ONNECTION TO OTHER ROBUST METHODS
In this section, we aim to theoretically explore the connection between GLS and popular methods such as backward/forward loss correction (Natarajan et al., 2013; Patrini et al., 2017), NLNL (Kim et al., 2019) and peer loss (Liu & Guo, 2020).
We defer the corresponding empirical validations to Appendix B.
For r ≤ 1 , let ˜y be the vector form of noisy label ˜
y obtained from (cid:101) Y , we deﬁne the r smoothed label of ˜ y
as ˜y GLS ,r , where ˜y GLS ,r :=
(1 − r ) · ˜y + ( r/K ) · 1 and is generated by the random variable (cid:101) Y GLS ,r .
Risk minimization of the Generalized Label Smoothing (GLS) w.r.t.
noisy labels becomes:
Risk Minimization Using GLS:
[Equation]
The GLS framework covers three special methods: PLS ( r ∈ (0 , 1] ), Vanilla (CE) Loss ( r = 0 ) and NLS ( r < 0 ).
Besides, we observe that NLS connects to a special case of label smoothing regularization.
We highlight this in Theorem 1.
Theorem 1.
∀ r ∈ [0 , 1] , NLS with smooth rate − r is a special form of label smoothing regularization:
[Equation]
3. L OSS CORRECTION
Loss correction (Patrini et al., 2017) studies two robust loss designs which are based on the knowledge of nonsingular noise transition matrix T .
The backward correction (cid:96) ← ( f ( X ) , (cid:101) Y ) re-weights the loss (cid:96) ( f ( X ) , (cid:101) Y ) by
T − 1 ˆ Y , (cid:101) Y with ˆ Y being the model predicted label, while the proposed forward correction (cid:96) → ( f ( X ) , (cid:101) Y ) multiplies the model predictions by T .
Proposition 1.
For r LC := 2 e 0 2 e 0 − 1 < 0 , λ LC :=
e ∆ · 1 1 − 2 e 0 , risk minimization of both backward and forward correction (with the knowledge of noise rates) are equivalent to the combination of NLS and an extra
[Table]
The incurred Bias-LC controls the model conﬁdence on ( X, Y = 1) ∼ D f .
Note that when the noise rate is not substantially high, i.e, e 0 ∈ [0 , 12 ) , λ LC > 0 .
Then, compared with loss correction, NLS with smooth rate r LC makes the model f to be less conﬁdent on ( X, Y = 1) ∼ D + f and more conﬁdent on ( X, Y = 1) ∼ D − f
(wrong predictions).
However, the impact of term Bias-LC is diminishing when either e ∆ → 0 (symmetric noise rates) or e 0 → 0 (low noise rates) as speciﬁed in Theorem 2.
Theorem 2.
Assume the noise transition matrix is symmetric, i.e,
e ∆ = 0 , backward and forward loss correction are a special form of NLS with smooth rate r LC .
3. L EARNING FROM COMPLEMENTARY LABELS
Complementary label (Ishida et al., 2017) was ﬁrstly introduced to mitigate the cost of collecting data.
Rather than encouraging the model to ﬁt directly on the target, learning from complementary labels trains the model to not ﬁt on the complementary label which differs from the target.
Later, an indirect training method “Negative Learnin” (NL) (Kim et al., 2019) was proposed to reduce the risk of providing incorrect information with the presence of noisy labels and is robust to label noise in multi-class classiﬁcation tasks.
A more generic unbiased risk estimator of learning with complementary labels was proposed (Ishida et al., 2019) and is deﬁned as: (cid:96) CL ( f ( X ) , (cid:101) Y ) :
= (cid:96) ( f ( X ) , (cid:101) Y ) − (cid:96) ( f ( X ) , 1 − (cid:101) Y ) .
Theorem 3.
Learning from complementary labels with (cid:96) CL is equivalent to NLS with smooth rate r CL → −∞ : min E
[Equation] D (cid:104) (cid:96) CL ( f ( X ) , (cid:101) Y ) (cid:105) [Equation]
X [Equation] (cid:101) Y
[Equation]
(cid:101) [Equation] (cid:96) [Equation]
[Table]
[Equation] (cid:101) [Equation] [Table]
[Equation]
r CL
[Table]
[Equation]
3. P EER LOSS FUNCTIONS
Peer loss functions (Liu & Guo, 2020) propose a family of robust loss measures which do not require the knowledge of noise rates.
The mathematical representation of peer loss functions is (cid:96) PL ( f ( X ) , (cid:101) Y ) :
= (cid:96) ( f ( X ) , (cid:101) Y ) − (cid:96) ( f ( X 1 ) , ˜ Y 2 ) , where ( X i , (cid:101) Y i ) ∼ (cid:101) D .
The second term of peer loss evaluates on randomly paired data samples and labels to punish f from overly ﬁtting on noisy labels.
Proposition 2.
For r PL := 2 · P ( (cid:101) Y = 1) , λ PL := 1 − r PL , risk minimization of peer loss is equivalent to negative label smoothing regularization with an extra term Bias-PL, i.e.,
[Equation]
The incurred term Bias-PL controls the model conﬁdence on ( X, (cid:101) Y = 1) ∼ (cid:101) D and has a diminishing effect as P ( (cid:101) Y = 1) → 1 / 2 .
Generally, peer loss relates to GLS as the negatively weighted GLS term appears to be a regularizer.
Note that we have access to the P ( (cid:101) Y = 1) , we can bridge the gap between GLS and peer loss by adding an estimable term Bias-PL.
With some derivations, we further show in Theorem 4, when noisy priors are equal, peer loss has an exact GLS form.
Theorem 4.
When the noisy labels have equal prior, i.e, P ( (cid:101) Y = 0)
=
P ( (cid:101) Y = 1) , peer loss is a special form of NLS regularization with smooth rate r PL .
Besides,
[Equation]
GLS AND MODEL CONFIDENCE
Now we show that NLS differs substantially from PLS in their achieved model conﬁdence.
This discussion sets the foundation for our discussion when learning with noisy labels in next section. 4. GLS ON CLEAN DATA We ﬁrstly show that NLS differs substantially from PLS in their achieved model conﬁdence.
This discussion sets the foundation for our discussion when learning with noisy labels in proceeding subsections.
When the label is clean, i.e, e 0 = e 1 = 0 , Eqn. (3) reduces to:
[Equation]
To clarify, we are not restricting D to have inﬁnite samples, i.e., for the discrete distribution D
= { x i , y i } Ni =1 ,
Eqn.
(4) becomes: min (cid:104) 1 N (cid:80) i ∈ [ N ] (cid:96) ( f ( x i ) , y [Equation] [Equation]
f [Equation] x i ) , y i ) (cid:1)(cid:105) . Note:
[Equation] Both log (cid:0) x/ (1 − x ) (cid:1) and 2 x − 1 are monotonically increasing for x ∈ (0 , 1) , model f with a high
MC (cid:96) ( f ; X, Y ) has high MC ( f ; X, Y ) . The difference between PLS and NLS lie in the weight of Term
MC (cid:96) ( f ; X, Y ) : NLS encourages high MC (cid:96) ( f ; X, Y ) and MC ( f ; X, Y ) while PLS has an opposite effect.
We adopt the generation of 2D (binary) synthetic dataset from (Amid et al., 2019) by randomly sampling two circularly distributed classes. The inner annulus indicates one class (blue), while the outer annulus denotes the other class (red).
We hold 20% data samples for performance comparison.
Side-effects of over-conﬁdent In Figure 2, the colored bands depict the different levels of prediction probabilities: light blue + orange bands indicate samples that satisfy MC < 0 .
4 (low model conﬁdence).
When learning with clean data, GLS with non-positive smooth rate may yield over-conﬁdence on the model prediction and a relatively low test accuracy.
Label noise reduces model conﬁdence
Recent works (Liu, 2021; Cheng et al., 2020) have demonstrated that with the presence of label noise, learning with noisy labels directly will eventually result in unconﬁdent model predictions.
Continuing the synthetic 2D dataset, we ﬂip the clean labels according to a symmetric noise transition matrix with noise rate e i for both classes.
With the presence of label noise in Figure 3, GLS generally becomes less conﬁdent on its predictions.
Besides, when the smooth rate increases from negative to positive, more samples are of uncertain predictions. Thus, a smaller/negative smooth rate is beneﬁcial when the noise rate increases by encouraging more conﬁdent predictions.
GLS WITH LABEL NOISE
In this section, we target at the optimal candidates of r in GLS when the label noise presents.
In Section 4, we have shown that NLS and PLS have the opposite functionality on the model conﬁdence when training on clean data. Given the unseen test data, learning with non-negative smooth rates may not always return the best outcome (Figure 3).
Based on this observation, we delve into details to show when NLS is more favorable than PLS and Vanilla Loss.
We start with stating Assumption 1:
Assumption 1.
We assume learning with clean data distribution D with smooth rate r ∗ ≤ 1 in GLS returns the best performance on the unseen clean test data distribution D test .
Assumption 1 simply offers an “anchor” point to initiate our analysis for the noisy label setting.
To clarify, we don’t rule out the possibility that other methods outperform GLS with optimal smooth rate r ∗ .
Later in Section 6.1 and Appendix D, we will empirically test what r ∗ usually is on various benchmarks.
We deﬁne optimal classiﬁer on Y ∗ which follows r ∗ smooth label distribution as: f
[Equation]
With the introduction of r ∗ and f ∗D , our goal is then to recover the classiﬁer f using the noisy training labels.
To bridge learning with noisy labels and clean labels for GLS, we deﬁne λ 1 , λ 2 and offer Theorem 5.
[Equation]
Theorem 5.
The risk minimization of GLS (Eqn. (3)) in the noisy setting relates to the risk deﬁned on the clean data with two additional bias terms: [Equation] [Table] [Figure]
[Equation]
The True Risk is the risk minimization w.r.t.
clean optimal label distribution Y ∗ .
Training GLS on noisy labels results in two extra bias terms which affect the model conﬁdence.
We defer an empirical validation of
Theorem 5 to Appendix B.
Now we proceed to answer “what parameters are preferred in the noisy setting”.
5. S YMMETRIC ERROR RATES WITH e ∆ =
Symmetric error rates e := e 0 = e 1 indicates the probability of ﬂipping to the other class is equal for both classes.
In this case, λ 2 = 0 and Term M-Inc2 is cancelled and Eqn.
(5) reduces to min
[Equation]
[Figure]
t e s
PLS, r [0,1] NLS, r <
Noisy labels impairs model conﬁdence on Vanilla Loss
In the GLS framework, deﬁne the optimal r that will cancel the impact of Term M-Inc1 as: [Equation]
The threshold r opt in Eqn.
7 implies:
Theorem 6.
With Assumption 1, GLS with smooth rate r = r opt yields f ∗D .
• When error rate e < r ∗ / 2 , r = r opt > 0 (PLS); • When error rate e = r ∗ / 2 , r =
(Vanilla Loss); • When error rate e > r ∗ / 2 , r = r opt < 0 (NLS).
In Theorem 6, adopting NLS when noise rate e < r ∗ / 2 induces λ 1 < 0 , Term M-Inc1 makes f overlyconﬁdent on its predictions compared with Y ∗ .
In Figure 4, with the decreasing of r ∗ , PLS is less tolerant of labels with high noise.
Similarly, if e ≥ r ∗ 2 , with the decreasing of r ∗ , NLS is more robust in the high noise regime while PLS makes the model f become less-conﬁdent on its predictions.
Clearly, NLS outperforms PLS especially when noise rates are large and r ∗ is small.
5. A SYMMETRIC ERROR RATES WITH e ∆ (cid:54) =
In this case, adopting r = r ∗ − 2 e 0 1 − 2 e 0 removes the Term M-Inc1.
However, when r < 1 , Term M-Inc2 is not negligible due to assymetric noise transition matrix.
As a result, Term M-Inc2 becomes:
[Equation]
, with e ∆ [Equation] Term M-Inc2 in the minimization increases the model conﬁdence on ( X, Y = 0) ∼ D + f .
The model will then become overly-conﬁdent with the class that has a low noise rate e 0 .
Meanwhile, Term M-Inc2 decreases the model conﬁdence on ( X, Y = 1) ∼ D + f (less-conﬁdent to the class with a high noise rate e 1 ).
E XPERIMENT RESULTS
In this section, we present our empirical observations regarding the role of GLS under clean and noisy labels by using UCI datasets, CIFAR-10 and CIFAR-100.
6. W HAT IS THE PRACTICAL DISTRIBUTION OF r ∗ AND r OPT
r ∗ and r opt on UCI datasets (Dua & Graff, 2017)
As for UCI datasets, we pick Twonorm and Splice for illustration in the main paper.
The noisy labels are generated by a symmetric noise transition matrix with noise rate e i =
As highlighted in Table 1, r opt appears with positive values when the data is clean (same as r ∗ ) or of a low noise rate.
With the increasing of noise rates, the performance of PLS results in a much larger degradation compared with NLS.
We color-code different noise regimes where either PLS (red-ish) or NLS (green-ish) outperforms the other.
Clearly there is a separation of the favored smoothing rate for different noise scenarios (upper left & low noise for PLS, bottom right & high noise for NLS).
[Table]
0.800 r = 0 .0.
0.806 r = − 0 .0.
0.817 r = − 0 .0.
0.819 r = − 1 .0. 0.819 r = − 2 .0.
0.830 r = − 4 .0. 0.830 r = − 8 .0.
0.845 r opt = [ 0.0, 0.8 ] [ 0.4, 0.8 ] [ -1.0, -0.4 ]
r ∗ and r opt on CIFAR datasets (Krizhevsky et al., 2009)
When learning with a larger scale and more complex dataset, like CIFAR-10 and CIFAR-100, models are prone to converge on a local optimal solution rather than the global optimum.
This phenomenon occurs frequently in NLS which ends up with performance degradation.
Thus, in Table 2, when learning with noisy labels, we report the better performance of GLS between direct training and loading the same warm-up model.
And we observe that the performance of NLS is more competitive than PLS when learning with clean data.
Clearly, NLS outperforms PLS in CIFARand CIFAR-100 under various synthetic noise settings.
The gap is larger when the noise rates are high.
[Table]
[Table]
C OMPARISONS BETWEEN GLS AND MORE ROBUST METHODS
In Table
[Table]
we compare GLS with several robust methods in synthetic noisy CIFAR datasets.
We emphasize that we are not proposing a new method to compete with state-of-the-art methods. Instead, we hope to help readers understand how the generalized label smoothing fare when label noise presents.
Clearly, GLS (especially NLS) can deﬁnitely be viewed as a competitive and efﬁcient robust loss function which outperforms
Cross Entropy, Bootsrap (Reed et al., 2014), SCE (Wang et al., 2019), APL (Ma et al., 2020) and Forward correction (Patrini et al., 2017) in most settings.
[Table]
Additional results More extensive empirical results are included in Appendix: empirical validation of main theorems (Appendix B); practical considerations of GLS (Appendix C); experiment results on additional datasets (Appendix D); bias and variance of the generalization error on the clean data (Appendix E).
Conclusion In this paper, we provide understandings for a generalized notion of label smoothing where the label smoothing rate can go negative.
We show that learning with negatively smoothed labels explicitly improves the conﬁdence of model prediction.
This key property acts as a signiﬁcant role when the conﬁdence of model prediction drops.
We make connections between negative label smoothing and existing learning with noisy label solutions.
In contrast to existing works that promote the use of positive label smoothing, we show both theoretically and empirically the advantage of a negative smooth rate when the label noise rate increases.
Our observations provide new understanding for the effects of label smoothing, especially when the training labels are imperfect.
A PPENDIX
The Appendix is organized as follows.
• Section A presents the full version of related works.
• Section B includes empirical validations of theoretical conclusions in Section 3 and Theorem 5.
• Section C discusses practical considerations and multi-class extensions of GLS.
• Section D shows additional experiments on synthetic dataset and UCI datasets.
• Section E illustrates the bias and variance trade-off when learning with GLS from clean data.
• Section F includes omitted proofs for theoretical conclusions in the main paper.
A F ULL VERSION OF RELATED WORKS
Our work supplements to two lines of related works.
Learning with noisy labels Annotated labels from human labelers usually consists of an non-negligible amount of mis-labeled data samples.
Making deep neural nets perform robust training on “noisily" labeled datasets remains a challenge.
Classical approaches of learning with noisy labels assume the noisy labels are independent to features.
They ﬁrstly estimate the noise transition matrix (Liu & Tao, 2015; Menon et al., 2015; Harish et al., 2016; Patrini et al., 2017), then proceed with a loss correction (Natarajan et al., 2013;
Patrini et al., 2017; Liu & Tao, 2015) to mitigate label noise.
Recent works propose robust loss functions (Xu et al., 2019; Kim et al., 2019; Liu & Guo, 2020; Wei & Liu, 2020) to train deep neural nets directly without the knowledge of noise rates, or design a pipeline which dynamically select and train on “clean" samples with small loss (Jiang et al., 2018; Han et al., 2018;
Yu et al., 2019; Yao et al., 2020a).
More recently, several approaches target at addressing more challenging noise settings, such as instance-dependent label noise (Cheng et al., 2020; Berthon et al., 2021).
Understanding the effect of label smoothing Learning with one-hot labels is prone to over-ﬁtting, soft label learning then naturally draws attentions of machine learning researchers. Successful applications of soft label learning include the label distribution learning (Geng, 2016) which provides an instance with description degrees of all the labels. Label smoothing (LS) (Szegedy et al., 2016) is another arising learning paradigm that uses positively weighted average of both the hard training labels and uniformly distributed soft labels.
Empirical studies have demonstrated the effectiveness of LS in improving the model performance (Pereyra et al., 2017; Szegedy et al., 2016; Vaswani et al., 2017; Chorowski & Jaitly, 2017) and model calibration (Müller et al., 2019).
However, knowledge distilling a teacher network (trained on smoothed labels) into a student network is much less effective (Müller et al., 2019).
Later, generalization effects of more advanced forms of label smoothing was studied, such as structural label smoothing (Li et al., 2020) and non-uniform label smoothing (Chen et al., 2020).
More recently, it was shown that an appropriate label smoothing regularizer with reduced label variance boosts the convergence (Xu et al., 2020).
When label noise presents, (Liu, 2021) gives theoretical justiﬁcations for the memorizing effects of label smoothing.
And the effectiveness of label smoothing in mitigating label noise is investigated in (Lukasik et al., 2020).
B E MPIRICAL VALIDATIONS OF MAIN THEOREMS
In this section, we empirically validate our main theoretical conclusions, i.e, the connection between GLS and popular methods, and the comparison between clean empirical risks and noisy ones.
B. C ONNECTION TO OTHER ROBUST METHODS
We ﬁrstly compare GLS with backward correction (Natarajan et al., 2013), forward correction (Patrini et al., 2017) and peer loss (Liu & Guo, 2020) on CIFAR-10 dataset.
To approximate the performance of backward/forward Loss Correction, we adopt GLS with smooth rate (cid:15) ( (cid:15) − 1) .
As for the approximation of peer loss, we choose (cid:96) ( f ( X ) , (cid:101) Y ) − (cid:96) ( f ( X ) , (cid:101) Y GLS ,r =0 . 5 ) which is equivalent to GLS when r → −∞ .
Experiment results in Table 4 on CIFAR-10 under symmetric noise settings demonstrate that the equivalent forms of GLS are robust to label noise.
[Table]
Explanation of the performance gap In practice, we adopt the same hyper-parameter setting as used for all other smooth rates for GLS form.
Loss corrections will ﬁrstly warm-up with the cross-entropy loss, estimate the noise transition matrix with this pre-trained model, and then proceed to train with the backward/forward corrected loss.
Peer loss functions adopt a dynamical adjustment for learning rate.
The warming up, estimation error of noise transition matrix as well as the special hyper-parameter settings explain performance gaps.
B. C LEAN EMPIRICAL RISK V . S . NOISY EMPIRICAL RISK
Now we empirically verify Theorem 8, which relates the risk of GLS in the noisy setting to the clean ones.
Assume the the noise label is generated through the symmetric noise transition matrix.
We restate the connection between two risks as below:
[Equation]
In practice, obtaining the exact value of r ∗ is not trivial.
Finding an accurate r for GLS to make λ 1 be exactly 0 is pretty hard.
Note that when M-Inc1 is equipped with the Cross-entropy loss and NLS encourages conﬁdent predictions, one mistake could make the value of M-Inc1 has a huge difference.
To be more speciﬁc, suppose the label of an image is a “cat”, if the NLS-trained model makes a mistake on this image, it is very likely to predict this image “cat” with a probability that p → 0 since NLS may push the model become overly-conﬁdent.
Then log( p ) can be extremely small.
Thus, the empirical risk (evaluated on the clean data) for NLS will change much more signiﬁcantly than PLS ones.
We use a UCI dataset (Waveform) for illustration.
The value of r ∗ is approximately 0, and the estimated (clean) true risk in above is 0.1798.
When the noise rates are 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , the optimal smooth rate should be − 0 .
25 , − 0 .
67 , − 1 . 5 , − 4 according to Eqn.
The estimated noisy risk of GLS on these noise settings can be summarized in Table 1.
Clearly, when e = 0 .
1 , r = − 0 .
25 is closest to the estimated clean true risk (also returns the best test accuracy among these smooth rates).
Similarly observations reach to all other e .
Although the risks evaluated on noisy settings with the corresponding r opt can not be exactly the same as the true risk (evaluates on the clean data with r = 0 . 0 ), r opt already reaches the smallest one among these smooth rates and the non-zero Gap of Risk is best explained by the precision of r ∗ as well as the estimation error of risks.
[Table]
C P RACTICAL CONSIDERATION OF GLS
In the main paper
[Table]
we theoretically show when we should adopt NLS and PLS.
In this section, we discuss more practical considerations of GLS, including the optimal smoothing parameter, how to reduce the impacts of bias terms [Table] and multi-class extensions.
C. T HE OPTIMAL SMOOTHING PARAMETER
In practice, we don’t have access to noise rates e i .
Our work does not intend to particularly focus on the noise rate estimation.
For readers interested in the noise rate estimation, please refer to (Liu & Tao, 2015;
Menon et al., 2015; Harish et al., 2016; Patrini et al., 2017; Yao et al., 2020b; Zhu et al., 2021).
To estimate r opt = r ∗ − 2 e 1 − 2 e , one can simply assume r ∗ → 0 .
And the noise rate e is estimable by a large family of noise estimation methods mentioned above.
Our practical observations show that NLS with a CE warm-up is not sensitive to the negative smooth rate, for example, on CIFAR-10 and CIFAR-100 synthetic noisy datasets, r < − 1 . 0 frequently achieves best results (see Table 2 in the main paper).
Our current contribution focuses on understanding the generalized label smoothing, and we prefer leaving the task of identifying the optimal smooth rate to future works.
C. M AKING GLS MORE ROBUST TO LABEL NOISE
There is a line of related works targeting at distinguishing clean labels from the noisy labels.
Current literature in selecting clean samples from noisily labeled dataset is based on the empirical evidence that samples with noisy/wrong labels have a larger loss than clean ones.
For interested readers, please refer to (Han et al., 2018;
Jiang et al., 2018; Yu et al., 2019; Yao et al., 2020a; Wei et al., 2020; Northcutt et al., 2021).
Compared with the risk minimization over the clean data distribution ( X, Y ) ∼ D , learning directly with GLS on the noisy distribution ( X, (cid:101) Y ) ∼ (cid:101) D will result in an extra term ( e 1 − e 0 ) · (1 − r ) · E ( X,Y =1) ∼D [ (cid:96) ( f ( X ) , 0) − (cid:96) ( f ( X ) , 1)] compared to the clean scenario.
Empirically, we can estimate the bias term, perform a bias correction by subtracting the estimated bias term from the objective function in Eqn.
Suppose we have access to a clean distribution D clean which consists of selected clean samples.
Denote the estimated noise rates as ˆ e i , when e ∆ (cid:54) = 0 , in order to make GLS be more robust to label noise and ﬁt on the optimal distribution Y ∗ , we improve GLS by performing a model conﬁdence correction on the dominating class through:
[Table]
C. M ULTI - CLASS EXTENSION
As an extension to the binary classiﬁcation task, we next show how GLS extends to the multi-class setting under two broad families of noise transition model.
When learning with multi-class classiﬁcation tasks, we assume Assumption 1 holds in the multi-class setting.
And for Y, (cid:101) Y ∈
[ K ] , we extend the deﬁnition of model conﬁdence to multi-class classiﬁcation tasks as:
Deﬁnition 2.
Model conﬁdence of sample x (K-class classiﬁcation).
Given a model f , a sample x with its target label y ∈ [ K ] , the model conﬁdence score of f w.r.t.
sample x is deﬁned as
[Equation]
Sparse noise transition matrix Sparse noise model (Wei & Liu, 2020) assumes K is an even number.
For c ∈ [ K/ 2] , i c < j c , sparse noise model speciﬁes K/ 2 disjoint pairs of classes ( i c , j c ) to simulate the scenario where particular pairs of classes are ambiguity and misleading for human annotators.
The off-diagonal element of T reads T i c ,j c = e 0 , T j c ,i c = e 1 .
Suppose e
+ e 1 < 1 , the diagonal entries become T
i c ,i c
= 1 − e 1 , T j c ,j c = 1 − e 0 .
Clearly, our conclusions in Section 5 extends directly to the sparse noise transition matrix by simply splitting the K -class classiﬁcation task into K 2 disjoint binary ones.
Symmetric noise transition matrix Symmetric noise model (Kim et al., 2019) is a widely accepted synthetic noise model in the literature of learning with noisy labels.
The symmetric noise model generates the noisy labels by randomly ﬂipping the clean label to the other possible classes with probability (cid:15) .
∀
i (cid:54) =
j ,
T i,j = (cid:15)/ ( K − 1) , and the diagonal entry T i,i = 1 − (cid:15) .
Deﬁne the optimal r for GLS in the multi-class setting as r opt :=
( K − 1) · r ∗ − K · (cid:15) ( K − 1) − K · (cid:15) , Theorem 6 can be extended to the multi-class setting as:
Theorem 7.
Under Assumption 1, suppose the symmetric noise rate is not too large, i.e, (cid:15) <
K − 1 K , GLS with smooth rate r = r opt
yields f ∗D .
• When error rate (cid:15) < ( K − 1) · r ∗ K , r = r opt > 0 (PLS); • When error rate (cid:15) =
( K − 1) · r ∗ K , r = 0 (Vanilla Loss); • When error rate (cid:15) > ( K − 1) · r ∗ K , r = r opt < 0 (NLS).
D A DDITIONAL E XPERIMENT R ESULTS AND D ETAILS
In this section, we include more experiment results, observations and details for learning with GLS.
D. E XPERIMENT DETAILS ON CIFAR-10, CIFAR-
We ﬁrstly introduce experiment details on CIFAR-10 dataset adopted in our experiment designs.
Training settings of clean CIFAR-10 dataset (Krizhevsky et al., 2009)
We adopted ResNet34 (He et al., 2016), trained for 200 epochs with batch-size 128, SGD (Robbins & Monro, 1951) optimizer with Nesterov momentum of 0.9 and weight decay 1e-4.
The learning rate of ﬁrst 100 epochs is 0.1.
Then it multiples with 0.1 for every 50 epochs.
Generating noise labels on CIFAR datasets We adopt symmetric noise model which generates noisy labels by randomly ﬂipping the clean label to the other possible classes with probability (cid:15) .
And we set (cid:15) = 0 .
6 for CIFAR-10, (cid:15) = 0 .
6 for CIFAR-100.
We also make use of asymmetric noise model.
The asymmetric noise is generated by ﬂipping the true label to the next class with probability (cid:15) .
We set (cid:15) = 0 .
3 for CIFAR-10.
Training settings of synthetic noisy CIFAR datasets
The generation of symmetric noisy dataset is adopted from (Cheng et al., 2020).
The symmetric noise rates are [0 . 2 , 0 . 4 , 0 . 6] .
We choose two methods to
train GLS
• Direct training: this setting is the same as training on clean CIFAR-10 dataset.
• Warm-up: in this case, we ﬁrstly train a ResNet34 model with Cross-Entropy loss for 120 epochs.
For this warm-up, the only difference in hyper-parameter setting is the learning rate, where the initial learning rate is 0.1 and it multiplies 0.1 for every 40 epochs.
After the warm-up, GLS loads the same pre-trained model and trains for 100 epochs with learning rate 1e-6.
D. W HY NLS IS OVERLOOKED ?
When learning from a relative large scale dataset, NLS tends to push the model become overly conﬁdent early in the training.
The poor performances of NLS (direct-train) in Table 6 explain why NLS is neglected.
When there is no warm-up, training NLS directly without warming up will reach a 88% − 92% test accuracy on the clean data.
The performance will degrade much more signiﬁcantly than PLS when the noise level is high or | r | is large.
In Table 6, we provide the comparisons between direct-train and warm-up in several settings.
The improvement bring by a warm-up procedure becomes much more signiﬁcantly in the high noise regime.
NLS makes the classiﬁer be overly conﬁdent at the early training which results in converging to a bad local optimum (without CE warm-up, NLS frequently results in a worse performance in CIFAR-10 and
CIFAR-100).
Since the model will usually ﬁt on the clean data ﬁrst, then over-ﬁts on the noisy ones (Liu et al., 2020), a large number of approaches (such as Loss corrections (Patrini et al., 2017), Peer Loss (Liu &
Guo, 2020), etc) adopt a CE warm-up ﬁrstly.
Note that there is no difference in the computing costs between
NLS (with CE warmup) and CE loss, proceeding with NLS to enhance the model conﬁdence makes NLS much more competitive in the high noise regime, also gives practical insights on how to make NLS work better when learning with clean data.
[Table]
D. E XPERIMENT DETAILS ON SYNTHETIC DATASETS AND UCI
We introduce experiment details on synthetic datasets and UCI datasets adopted in our experiment designs.
Generation of synthetic dataset In the synthetic (Type 1) dataset, we generate 500 points for both classes.
Class +1 distributes inside the circle with radius 0.25.
Class -1 generates by randomly sampling 500 data points in the annulus with inner radius 0.28 and outer radius 0.45.
As for synthetic (Type 2) dataset, we uniformly assign labels for 50% samples in the annulus (with inner radius 0.22, outer radius 0.31) based on
Type 1 dataset.
Generating noise labels on synthetic datasets and UCI datasets Note that these datasets are all binary classiﬁcation datasets, each label in the training and validation set is ﬂipped to the other class with probability e , and we set e = 0 .
4 for synthetic Type 1 dataset, e = 0 .
3 for synthetic Type 2 dataset.
Training settings of synthetic datasets For both types of synthetic datasets, we adopted a three-layer
ReLU Multi-Layer Perceptron (MLP), trained for 200 epochs with batch-size 128 and Adam (Kingma & Ba, 2014) optimizer.
The initial learning rate is 0.1, and it multiplies 0.1 for every 40 epochs.
Training settings of UCI datasets Dua & Graff (2017)
We adopted (Liu & Guo, 2020) a two-layer ReLU
Multi-Layer Perceptron (MLP) for classiﬁcation tasks on multiple UCI datasets, trained for 1000 episodes with batch-size 64 and Adam (Kingma & Ba, 2014) optimizer.
We report the best performance for each smooth rate under a set of learning rate settings, [0 . 0007 , 0 .
D. A DDITIONAL EXPERIMENT ON r ∗ AND r OPT
r ∗ and r opt on synthetic dataset We generate 2D (binary) synthetic dataset by randomly sampling two circularly distributed classes.
The inner annulus indicates one class (blue), while the outer annulus denotes the other class (red).
Clearly, the generated synthetic dataset is well-separable (Type 1) and we hold 20% data samples for performance comparison.
The noise transition matrix takes a symmetric form with noise rate e i for both classes.
To simulate the scenario where the clean data may not be perfectly separated due to a non-negligible amount of uncertainty samples clustering at the decision boundary, we ﬂip the label of 50% samples near the intersection of two annulus to the other class (Type 2).
As speciﬁed in Table 7, r ∗ =
[0 . 1 , 0 . 4] for Type 1 data and r ∗ =
2] for Type 2 data.
With the presence of label noise, the distribution of r opt shifts from non-negative ones to negative values.
Even though NLS fails to outperform
PLS on clean data, we observe that NLS is less sensitive to noisy labels.
Data with high level noise rates clearly favor NLS with a low smooth rate!
[Table]
r ∗ and r opt on more UCI datasets We further test the performance of generalized label smoothing onmore UCI datasets (Heart, Breast 1, Breast 2, Diabetes, German, Image and Waveform).
Our observation remains unchanged: there exists a general trend that with the increasing of noise rates, NLS becomes much more competitive than PLS.
Here, we attach the results of 4 additional UCI datasets for illustration.
The noisy labels are generated by a symmetric noise transition matrix with noise rate e i =
As highlighted in Table 8, r opt appears with positive values when the data is clean (same as r ∗ ) or of a low
[Table]
noise rate.
With the increasing of noise rates, NLS becomes more competitive than PLS.
We color-code different noise regimes where either PLS (red-ish) or NLS (green-ish) outperforms the other.
Clearly, there is a separation of the favored smoothing rate for different noise scenarios (upper left & low noise for PLS, bottom right & high noise for NLS).
D. A DDITIONAL EXPERIMENT RESULTS ON MODEL CONFIDENCE
NLS improves model conﬁdence on Synthetic Type 2 dataset
In this case, the clean data that are close to decision boundary distributes randomly.
In Figure 5-6, the colored bands depict the different levels of prediction probabilities.
When the smooth rate increases from negative to positive, more samples fall in the orange and light blue band which indicates uncertain predictions.
When the smooth rate increases from negative to positive, learning with GLS will result in more uncertain predictions.
With the increasing of noise rates ( e i =
→ 0 . 4 ), GLS with a ﬁxed smooth rate becomes less conﬁdent on its predictions.
Thus, a smaller smooth rate is required when the noise rate increases.
Model conﬁdence of GLS on CIFAR-10 test dataset When GLS trained on symmetric 0.2 noisy CIFAR10 training dataset (see Figure 7), with the decreasing of smooth rates (from right to left), the model conﬁdence on correct predictions gradually approach to its maximum, while for wrong predictions, the model conﬁdence converges to its minimum value.
NLS makes the model prediction become over-conﬁdent on correct predictions and in-conﬁdent on wrong predictions.
[Figure]
D. E FFECT OF GLS ON PRE - LOGITS
We visualise the pre-logits of a ResNet-34 for three classes on CIFAR-10.
We adopt the method from (Müller et al., 2019) which illustrates how representations differ between penultimate layers of networks trained with different smooth rates in GLS.
In Figure 8, NLS makes the model f be conﬁdent on her predictions and the distances between three clusters are clearly larger than those appeared in Vanilla Loss and PLS.
PLS; trained with symmetric 0.2 noisy CIFAR-10 training dataset).
E B IAS AND VARIANCE TRADE - OFF OF GLS
Denote ˆ f H , ˆ f S as pre-trained models on the training dataset D w.r.t.
hard labels and soft labels, respectively.
The vector form of the prediction w.r.t.
sample x given by ˆ f H and ˆ f S are ˆf H ( x ; D ) and ˆf S ( x ; D ) .
For the ease of presentation, we relate notations with subscript H/S to hard/soft labels without further explanation.
Given the sample x and the one-hot label y , we denote the averaged model prediction by:
[Equation]
where Z H , Z S are normalization constants.
The bias of model prediction is deﬁned as the KL divergence D KL between target distribution (one-hot encoded vector form) y and the averaged model prediction.
[Equation]
While the variance of model prediction measures the expectation of KL divergence between the averaged model prediction and model prediction over D : Var H := E D
[Equation]
Empirical observation from (Zhou et al., 2021) shows that the variance brought by learning with positive soft labels given by a teacher’s model (Hinton et al., 2015) is less than the direct training w.r.t hard labels.
As an extension, we are interested in how GLS interferes with the bias and variance of model prediction.
Bias and variance of GLS on clean dataset We introduce our empirical observation regarding the role of
GLS in bias and variance trade-off in Figure 9.
We select nine smooth rates of GLS for illustration.
Each smooth rate setting of GLS trains on the CIFAR-10 dataset for 5 times with different data augmentations.
To estimate the variance and bias of pre-trained models, we adopt the implementation in (Yang et al., 2020).
Empirical results show that learning directly with a larger positive smooth rate typically results in lower variance and higher bias.
In Figure 9, we can observe almost constant bias values and very low variance for
NLS.
This is best explained by the warm-up of pre-trained models and the fact that NLS pushes the classiﬁer to give conﬁdent predictions.
As for PLS, with the increase of smooth rate, the overall bias has an increasing tendency while the variance has the decreasing pattern.
Especially when the smooth rate approaches to 1, i.e., r = 0 .
9 , the variance is close to 0.
[Figure]
F O MITTED PROOFS F. P ROOF OF T HEOREM
Before we prove Theorem 1, we ﬁrst introduce Lemma 1.
Lemma 1.
∀ ( x, y GLS ,r ) , (cid:96) (cid:0) f ( x ) , y GLS ,r (cid:1) =
(cid:0) 1 − r 2 (cid:1) · (cid:96) (cid:0) f ( x ) , y (cid:1) + r 2 · (cid:96) (cid:0) f ( x ) , 1 − y (cid:1) .
Proof of Lemma
Proof.
For CE loss, due to its linear property w.r.t.
the label, we directly have:
[Equation]
Proof of Theorem
Proof.
Based on Lemma 1, with a bit of math, for NLS, we have:
[Equation]
[Table]
Proof.
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
F.
P ROOF OF P ROPOSITION
Proof.
The risk minimization of backward correction is equivalent to:
[Equation]
Theorem 1 in
[Equation]
Patrini et al
[Equation]
The risk minimization of forward correction is equivalent to:
[Equation]
(cid:96)
[Equation]
f
[Equation]
X
[Equation] Y ) [Equation] (By Theorem 2 in (Patrini et al., 2017))
Theorem 1 and 2 in (Patrini et al., 2017) demonstrate that forward and backward corrected losses equal the original loss (cid:96) computed on the clean data in expectation.
Thus, for r LC = 2 e 0 2 e 0 − 1 , by Theorem 5 (adopt r ∗ = 0 ), we have:
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] Thus,
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
F. P ROOF OF T HEOREM
Proof.
Based on Proposition 1, when e ∆ = 0 , λ LC = [Equation] we directly have:
[Equation]
F. P ROOF OF T HEOREM
Proof.
Note that
[Equation]
[Table]
We have:
[Equation]
X
[Equation]
When r CL → −∞ , we have r CL 2 − r CL → − 1 .
Thus,
[Equation]
F. P ROOF OF P ROPOSITION
Proof.
Note that:
[Equation]
And we have:
[Equation]
= (cid:104)
[Equation]
[Equation]
[Equation]
0 (cid:1)
+
[Equation]
[Equation]
[Equation]
[Equation] 1 (cid:1)(cid:105) = [Equation] (cid:104)
[Equation]
0) · (cid:96)
[Equation]
[Equation] (cid:0) 1 − [Equation] 0) (cid:1) · (cid:96)
[Equation]
[Equation]
+
[Equation]
P
[Equation] 0) (cid:1) · (cid:96) (cid:0) [Equation] [Equation]
[Equation]
[Equation]
[Equation]
+ [Equation] (cid:1) [Equation]
(cid:96)
[Equation]
f
[Equation]
[Equation]
0 (cid:1)(cid:105)
Thus,
[Table]
Thus
[Table]
for r PL = 2 · P ( (cid:101) Y = 1)
[Table]
λ PL = 1 − r PL
[Table]
we have:
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
And we can conclude that:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
F. P ROOF OF T HEOREM
Proof.
When P ( (cid:101) Y = 0) = P ( (cid:101) Y = 1) , according to Proposition 2, we have λ PL =
and:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] When r PL → −∞ , we further have:
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] CL [Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
Thus, Theorem 4 is proved
[Equation]
F. P ROOF OF T HEOREM
Proof.
Note that the optimal r that will cancel the impact of Term M-Inc1 is:
[Equation]
• When e < r ∗ 2 , r opt > 0 .
In this case, learning PLS with smooth rate r opt results in:
[Equation]
which yields f ∗D
; • When e = r ∗ 2 , r opt = 0 .
Learning with the Vanilla Loss yields f ∗D since:
[Equation]
X
[Equation]
• Similarly, when e > r ∗ 2 , learning NLS with r = r opt < 0 yields f ∗D .
F. P ROOF OF T HEOREM
Proof.
Denote p i =
P ( Y = i ) as the clean label distribution, ˜ p
i = P ( (cid:101)
Y = i ) as the clean label distribution.
Let (cid:15) (cid:48)
=
K · (cid:15) K − 1 , we have:
[Equation]
Adopting r opt = r ∗ − (cid:15)
[Equation] 1 − (cid:15) (cid:48) , with a bit of math [Equation] the weight of Term M-Inc1 becomes 0 and
[Equation]