R OBUST L EARNING VIA G OLDEN S YMMETRIC L OSS OF ( UN )T RUSTED L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning robust deep models against noisy labels becomes ever critical when today’s data is commonly collected from open platforms and subject to adversarial corruption.
The information on the label corruption process i.e., corruption matrix can greatly enhance the robustness of deep models but still fall behind in combating hard classes
In this paper , we propose to construct a golden symmetric loss (GSL) based on the estimated confusion matrix as to avoid overﬁtting to noisy labels and learn effectively from hard classes.
GSL is the weighted sum of the corrected regular cross entropy and reverse cross entropy.
By leveraging a small fraction of trusted clean data, we estimate the corruption matrix and use it to correct the loss as well as to determine the weights of GSL.
We theoretically prove the robustness of the proposed loss function in the presence of dirty labels.
We provide a heuristics to adaptively tune the loss weights of GSL according to the noise rate and diversity measured from the dataset.
We evaluate our proposed golden symmetric loss on both vision and natural language deep models subject to different types of label noise patterns.
Empirical results show that GSL can signiﬁcantly outperform the existing robust training methods on different noise patterns, showing accuracy improvement up to 18% on CIFAR-100 and 1% on real world noisy dataset of
Clothing1M.
I NTRODUCTION
Diverse datasets collected from the public domain which power up deep learning models present new challenges – highly noisy labels.
It is not only time consuming to collect labels but also difﬁcult to ensure a consistent label quality due to various annotation errors (Patrini et al., 2017) and adversarial attacks (Goodfellow et al., 2015).
The large capacity of deep learning models enables effective learning from complex datasets but also suffers from overﬁtting to the noise structure in the dataset.
The curse of memorization effect (Jiang et al., 2018) can degrade the accuracy of deep learning models in the presence of highly noisy labels.
For example, in (Zhang et al., 2017) the accuracy of AlexNet to classify CIFAR10 images drops from 77% to 10%, when there are randomly ﬂipped labels.
Designing learning models that can robustly train on noisy labels is thus imperative.
To distill the impact of noisy labels, the related work either ﬁlters out suspiciously noisy data, derives robust loss functions or tries to proactively correct labels.
Symmetric Cross entropy Loss (SCL) is shown effective in combating label noise especially for hard classes by combing the regular with the reverse cross entropy.
The former avoids overﬁtting and the latter is resilient to label noise.
Given its promising results, there is yet to have a clear principle on how to weight the regular and reverse cross entropy terms, e.g., at different noise rates and patterns.
In contrast, Distilling (Li et al., 2017) and Golden Loss Correction (GLC) (Hendrycks et al., 2018) advocate to use a small clean data to improve the estimated corruption matrix.
Speciﬁcally, GLC trains the deep model on both a clean and noisy set, whose loss is corrected through the corruption matrix.
While the clean set is evenly chosen from all classes, the corrupted labels may appear unevenly across classes depending on the noise pattern (Xiao et al., 2015).
As the corrected loss of GLC does not differentiate the difﬁculty of classes, it may not learn those hard classes effectively.
We propose GSL constructing the golden symmetric loss that dynamically weights regular/reverse cross entropy and corrects the label prediction based on the estimated corruption matrix.
Similar to
Noise 0 1 2 3 4 5 6 7 8 9 Given label 0 13 46
T r u e l a b e l 0.
[Figure]
Corruption matrix GLC SCL GSL 0 1 2 3 4 5 6 7 8 9 Given label 0 13 46
T r u e l a b e l 0.
0 1 2 3 4 5 6 7 8 9 Given label 0 13 46
T r u e l a b e l 0.
0 1 2 3 4 5 6 7 8 9 Given label 0 13 46
T r u e l a b e l 0.
[Figure] matrix GLC, GSL leverages clean data to estimate the corruption matrix which is used to correct labels and decide the weights of the golden symmetric loss.
As such, GSL can effectively differentiate the difﬁculty level of classes by adjusting the weights and mitigate the impact of noise overﬁtting via the golden symmetric cross entropy.
Speciﬁcally, we use the noise rate and noise diversity to adaptively tune the weights of modiﬁed cross entropy and reverse cross entropy.
We prove that modiﬁed cross entropy by using confusion matrix is noise tolerant same as the reverse cross entropy.
Motivation example.
We demonstrate the advantages and disadvantages of GLC and SCL, and the their combination (the proposed GSL) through the example of learning convolution networks on
CIFAR-10 injected with 60% symmetric noise.
The experimental setup is detailed in § 6.
Figureshows the corruption matrix of the injected noise and the confusion matrices from the predictions of SCL, GLC, and GSL.
Even if the injected noise is symmetric across all classes (see Figure 1a), prediction errors are distributed asymmetrically across the classes (see Figure 1b).
Though GLC can achieve a lower average error rate than SCL (reﬂected in darker diagonal elements on average), it performs worse in hard classes, e.g., class 4 (cat) and class 6 (dog) (difference in blue shades across the diagonal elements).
By setting up proper weights for two types of cross entropy, GSL is able to achieve both superior average and per class accuracy.
R ELATED W ORK
Enhancing the robustness of deep models against noisy labels is an active research area.
The massive datasets needed to train deep models are commonly found corrupted, (Wang et al., 2018), severely degrading the achievable accuracy, (Zhang et al., 2017).
The impact of label noise on deep neural networks is ﬁrst characterized by the theoretical testing accuracy over a limited set of noise patterns (Chen et al., 2019).
(Vahdat, 2017) suggest an undirected graph model for modeling label noise in deep neural networks and indicate symmetric noise to be more challenging than asymmetric.
Solutions of the prior art can be categorized into three directions: (i) ﬁltering out noisy labels: (Malach & Shalev-Shwartz, 2017; Han et al., 2018b; Yu et al., 2019; Wang et al., 2018); (ii) correcting noisy labels: (Patrini et al., 2017; Hendrycks et al., 2018; Li et al., 2017); and (iii) deriving noise resilient loss functions: (Ma et al., 2018; Konstantinov & Lampert, 2019).
Noise Resilient Loss Function .
The loss function is modiﬁed to enhance the robustness to label noise by introducing new loss functions, (Ghosh et al., 2017; Wang et al., 2019), or adjusting the weights of noisy data instances, (Ren et al., 2018b; Konstantinov & Lampert, 2019; Ma et al., 2018).
Mean
Absolute Error (MAE) (Ghosh et al., 2017; Zhang & Sabuncu, 2018) and General Cross Entropy loss (Zhang & Sabuncu, 2018) are proposed as a noise resilient alternative but at the cost of slow convergence.
To avoid overﬁtting to noise, D2L
(Ma et al., 2018) uses the subspace dimensionality to assign weights to each data point, whereas Konstantinov (Konstantinov & Lampert, 2019) determines the loss weights based on the trustworthiness level of data sources.
(Wang et al., 2019) propose symmetric cross-entropy loss that combines a new term of reverse cross entropy with traditional cross entropy via constant weights on both terms.
Meta-Weight-Net (Shu et al., 2019) re-weights samples during optimizing loss function in the training process by using a multi-layer perceptron to predict the weight of each sample.
With the same perspective, (Ren et al., 2018a) uses the similarity of samples to the clean instances in the validation set for re-weighting them in loss function.
Label correction .
To avoid the data reduction caused by ﬁltering, label correction methods adjust the predicted/given labels by using only noisy labels (Patrini et al., 2017; Tanaka et al., 2018) or jointly with a small fraction of trusted data (Veit et al., 2017; Han et al., 2018a; Li et al., 2017; Hendrycks et al., 2018).
Reed et al.
train the classiﬁer by the “new” labels combining the raw and predicted labels without access to label ground truth.
(Patrini et al., 2017) estimate the noise confusion matrix by ﬁrst training a classiﬁer on the noisy labels and then using the softmax probabilities.
(Veit et al., 2017) acquire human-veriﬁed labels to train a cleaning network for correcting noisy labels of multi-label classiﬁcation problems.
(Han et al., 2018a) estimate the noise transition probability by incorporating human assistance.
(Li et al., 2017) and (Hendrycks et al., 2018) leverage a small set of clean data to estimate noise corruption matrix from the clean and noisy sets, respectively.
DivideMix (Li et al., 2020) is a semi-supervised method, including two networks and Gaussian Mixture Model for sample selection.
The proposed GSL combines resilient loss function and label correction by curating a small fraction of trusted data.
We solicit a subset of informative data instances to estimate the confusion matrix and provide a minimum supervision on noisy labels.
We also provide a heuristic to adaptively tune the weights of golden symmetric loss according to the noise characteristics of the dataset.
G OLDEN S YMMETRIC L OSS
Consider the classiﬁcation problem having dataset ˜ D
= { ( x n , ˜ y n ) } Nn =1 where x n ∈ X ⊂ R d denotes the n th observed sample, and ˜ y n ∈ Y := { 1 , ..., K } the corresponding given label over K classes.
Hereon n is ignore for the simplicity.
˜
y is affected by label noise.
The label corruption process is characterised by a corruption matrix C ij = P
(˜ y =
j | y
= i ) for i
= 1 , . .
, K and j = 1 , . .
, K where y is the true label.
Synthetic noise patterns are expressed as a label corruption probability ε plus a noise label distribution.
Let g ( · , θ ) denote a neural network-based classiﬁer parameterized by θ .
For each data point x , f ( · , θ ) predicts the probability for each class label k : p ( k | x ) =
[Equation]
where z j are the logits.
Symmetric Cross Entropy .
Let q ( k | x ) denote the ground truth probability distribution over the K class labels where q ( k | x )
= 1 for k equal to the true class y and q ( k | x ) =
for all k (cid:54) =
y .
The cross entropy loss ( (cid:96) ce ) and reverse cross entropy loss 1 ( (cid:96) rce ) for sample x are:
[Equation]
(Wang et al., 2019) combine cross entropy and reverse cross entropy into the symmetric cross entropy:
[Equation]
where α and β are hyperparameters.
On the one hand cross entropy loss is not robust to noise (Ghosh et al., 2017) but achieves good convergence (Zhang & Sabuncu, 2018).
On the other hand reverse cross entropy is tolerant to noise (Wang et al., 2019).
Estimating Noise Corruption Matrix .
We estimate the noise corruption matrix as in (Hendrycks et al., 2018).
The method fosters training a ﬁrst classiﬁer g ( · , Θ) on noisy data to approximate the elements C ij of the noise corruption matrix via a small fraction of trusted data D with known true label y .
Practically given A i the subset of trusted data with label of class i { A i ⊂ D : y = i } , the elements of C can be approximated by:
[Equation]
where g (˜ y
=
j | x , Θ) denotes predicted probability of x having class label j .
That is ˆ C ij is computed as the mean predicted probability of class j for all trusted data points having true label of class i .
Training with Corrected Labels .
Let ˆ C be the estimated noise corruption matrix.
Using the method in (Patrini et al., 2017), we increase the noise resilience by correcting the predictions of the [Figure] classiﬁer using ˆ C .
Let ˆ p be the corrected predicted probabilities ˆ p
= ˆ C T p , i.e. for data point x : ˆ p ( k | x ) =
(cid:80) K i =
1 ˆ C ik p ( i | x ) for k = 1 , . . .
, K .
We enhance the regular cross entropy term.
Applying the prediction correction to both terms holds lower beneﬁts.
We evaluate this empirically with extensive experiments on datasets of text, i.e. Twitter in Figure.
2a, and images, i.e. CIFAR10 and
CIFAR100 in Appendix A.Experiment details can be found in § 6.
We consider different datasets, noise rates, noise types and fractions of trusted data.
We see that in all cases, except one with a difference < 0 .
3 %, correcting only the cross entropy ( ce-only ) holds better results than correcting only the reverse cross entropy ( rce-only ) or correcting both .
Focusing on Figure.
2a, ce-only improves accuracy by up to 5 % and 8 % for bimodal and symmetric noise, respectively.
In case of CIFAR-10 and
CIFAR-100 datasets the improvements are more pronounced with up to 11% and 50% respectively.
Golden Symmetric Loss .
Towards a more effective and robust learning we propose to leverage the estimated noise corruption matrix ˆ C to tune the two loss terms based on the observed noise pattern.
α and β can signiﬁcantly impact the ﬁnal model accuracy.
Tuning these parameters is no mean feat as different datasets affected by different noise patterns beneﬁt from different optimal values (Wang et al., 2019).
Again we show this behavior by training a 2-layer FC neural network on the Twitter dataset under eleven different ( α , β ) combinations and two noise patterns with 80% noise.
Figure.
2b reports for each noise pattern the evolution over the training epochs of the test accuracy for the best and worst ( α , β )-pair.
For bimodal noise even with a small number of trials, the impact of ( α , β ) ranges from an accuracy close to 60 % all the way down to almost 0 %.
Moreover only few (two out of eleven) ( α , β )-pairs hold accuracy close to 60 %.
For symmetric noise the tuning impact is lower (limited between 70 % and 80 %) but the best and worst ( α , β )-pair differ from the bimodal noise case.
This underlines both the importance and difﬁculty of tuning ( α , β ).
Motivated by the high impact of α and β , we propose to dynamically weight the regular and reverse cross entropy terms.
Let A () and
B () be weighting functions mapping ˆ C −→ R we deﬁne a new loss function:
[Equation]
We call this new loss function golden symmetric loss.
A () and B () should capture not only the intensity of the noise pattern, but also the diversity of the noise pattern (see Figure. 2b).
Determining Weights of Golden Symmetric Loss ( A () and B () ) .
In general the more intense and asymmetric the noise pattern, the lower the weight values should be.
Since the ﬁnal loss function learns from both dirty and clean data (see next paragraph), lower values of α and β reduce the inﬂuence of dirty data over the one of clean data.
Hence, we design A () and B () to capture both noise intensity and diversity.
The intensity is given by the noise rate ε ∈ [0 , . . .
, 1] , i.e. one minus the average of the diagonal elements of ˆ C .
The diversity is measured via Jain’s fairness index
J ( x 1 , x 2 , . . . , x n ) (cid:44) ( (cid:80) ni = n x i ) 2 / n (cid:80) ni = n x 2 i .
We choose J because it bounds the diversity on a similar scale as ε between 1 (all equal, full symmetry) down to 1 / n (highest asymmetry).
We apply J on all the K ( K − 1) noise, i.e. off the diagonal, elements of ˆ C :
[Equation]
[Equation]
[Table]
[Table]
/AFpe5Na</latexit>
For symmetric noise J = 1 , the more asymmetric the smaller J .
Final weights proportional to J, ε .
Putting It All Together .
As a ﬁnal step, to maximize the utility of the trusted data, we foster D as additional trusted training data for f () .
Since D contains the true labels y no prediction correction is applied.
Hence, the overall loss function for data points from both D and ˜ D is: [Equation] x
[Equation]
D .
Figure.
3 summarises visually the training process divided into two main steps: (i) estimating noise corruption matrix through the ﬁrst network g trained on untrusted dataset ˜ D and (ii) training classiﬁer f on both untrusted ˜ D and trusted D through the golden symmetric loss.
T HEORETICAL A NALYSIS
We prove that the cross entropy loss with label correction is noise tolerant under the deﬁnition put forth by (Ghosh et al., 2017; Manwani & Sastry, 2013) and extending prior art results.
Let the risk of classiﬁer f and loss function (cid:96) ce under clean labels be R ( f ) =
E x ,y (cid:96) ce
( f ( x ) , y ) and the risk under noise rate ε be R ε ( f ) =
E x , ˜ y (cid:96) ce
( f ( x ) , ˜ y ) .
E indicates the expectation taken over the random variables indicated as its subscripts.
With prediction correction via C , the risk becomes R ε ( f, C ) =
E x , ˜ y (cid:96) ce
( C T f ( x ) , ˜ y ) .
Let f ∗ and f ∗ ε be the global minimizers of R ( f ) and R ε ( f ) , respectively, and C ∗ = p
(˜ y
| y )
and ˆ C be the true and estimated noise confusion matrices, respectively.
Theorem 4.1.
In a multi-class classiﬁcation problem, (cid:96) ce with prediction correction is noise tolerant under symmetric label noise if the noise rate ε < K − 1 K − ∆
A ∆ R , where ∆ A = (cid:80) Kk =1 (cid:96) ce ( C ∗ T f ( x ) , k ) − (cid:80) Kk =1 (cid:96) ce
( ˆ C T f ( x ) , k ) , and ∆ R is the difference of risk minimization between optimal classiﬁer and f .
And (cid:96) ce with prediction correction is also noise tolerant under ﬂip noise when noise rate ε yk ≤ (1 + ∆ W y ∆ W k ) − ε y
(1 + ∆ W y ∆ W k ) where ε k and ε yk are the correct and ﬂipped class probabilities, respectively.
The proof is based on the risk minimization framework aiming to show under which condition
R ε ( f ∗ , C ∗ ) − R ε ( f, ˆ C ) ≤ 0 , i.e. the loss function is robust to noise.
The detailed steps of the proof can be found in Appendix C.
The condition ε < K − 1 K − ∆ A ∆ R is a generalization of the previous bound ε <
K − 1 K by (Ghosh et al., 2017).
Without label correction ∆
A = 0 which corresponds to the previous result.
Label correction improves the robustness by allowing higher noise rates, i.e. with label correction ∆
A ∆ R ≥ 0 and hence K − 1 K ≤ K − 1 K − ∆ A ∆ R .
Similar observations hold for ﬂip noise bound.
E XPERIMENTAL S ETUP
Dataset, Architecture and Parameters .
We consider two types of datasets: vision and text analysis.
For vision, we use convolution neural networks (CNN) to classify CIFAR-10 and CIFAR-100 with injected label noise and Clothing1M as real world noisy dataset.
For text, we use fully connected neural networks to classify noisy Twitter and Stanford Sentiment Treebank (SST).
In principle, we use the same network architecture on all comparative approaches across different noise resilience techniques.
In addition, we test the original network from the respective papers too and report the best results among the two.
• CIFAR-10 (Krizhevsky et al., 2009).
It contains 60K images classiﬁed into 10 classes: 50K as a training set and 10K as validation set.
We use the architecture of Wide-ResNet by (Zagoruyko & Komodakis, 2016) with depth 28 and a widening factor 10 and train it with SGD with Nesterov momentum and a cosine learning rate schedule (Loshchilov & Hutter).
For GSL, we ﬁrst train f for 75 epochs to obtain the noise corruption matrix.
Then we train g for 120 epochs.
• CIFAR-100 (Krizhevsky et al., 2009).
It contains 60K images classiﬁed into 100 classes: 50K as training set and 10K as the validation set.
We use the same Wide-ResNet architecture used for CIFAR-10.
For GSL, we train the f and g networks for 75 and 200 epochs, respectively.
Clothing1M (Xiao et al., 2015).
This is a real world dataset with label noise.
It includes images scrapped from the Internet classiﬁed into 14 categories.
We resize and crop each image to 224 × 224 pixels.
This dataset contains 47K and 10K images for training and testing, respectively.
These two sets have both given (scrapped) and true (human-checked) labels.
We use ResNetpretrained with ImageNet and further train for 10 epochs with batch size 32 , SGD optimizer, momentum 0 .
9 , weight decay 10 − 3 , and learning rate 10 − 3 which is divided by 10 after 5 epochs.
• Twitter (Gimpel et al., 2011).
The Twitter dataset includes 1,827 tweets annotated with 25 POS tags split in 1000 tweets as training set, 327 tweets as development set and 500 tweets as test set.
We add development set to training set, and consider it as a training set.
We use a 2-layer fully connected network with 256 hidden neurons each and GELU nonlinearity as activation function.
We train f with Adam for 15 epochs with batch size 64 and learning rate of 0.001.
We train g for 25 epochs.
To regularize all linear output layer, we use (cid:96) 2 weight decay with λ = 5 × 10 − 5 .
• Stanford Sentiment Treebank (Socher et al., 2013).
The SST dataset includes single sentence movie reviews.
We use the 2-class version, including 6911 reviews in the training set, a development set with 872 reviews, and 1821 reviews in the test set.
We augment the training set by using development set.
We learn 100-dimensional word vectors from scratch for a vocab size of 10000.
We train a word-averaging model with an afﬁne output layer using Adam optimizer for 5 epochs for network f and 10 epochs for network g .
The batch size and learning rate are 50 and 0.001, respectively.
To regularize all linear output layer, we use (cid:96) 2 weight decay with λ = 1 × 10 − 4 .
Noise Corruption .
We consider symmetric noise and two different asymmetric noises, namely ﬂip and bimodal.
Symmetric noise corrupts the true label into a random other labels with equal probability based on the noise rate.
The ﬂip noise is generated by ﬂipping the original label to a paired other class with a speciﬁc probability.
The bimodal noise imitates targeted adversarial attacks (Goodfellow et al., 2015).
Speciﬁcally, the true labels are corrupted into two neighborhoods centered on two targeted classes, each of which follows truncated normal distribution, N T ( µ, σ, a, b ) .
µ speciﬁes the target and σ controls the spread.
a and b simply deﬁne the class label boundaries.
For CIFAR-10 we target class 3 and 7, for CIFAR-100 class 30 and 70, for Twitter class 6 and 18, and for SST class 0 and 1.
Instead, Clothing1M is already affected by real world label noise and left untouched.
E VALUATION
In this section, we empirically compare GSL against state of the art noise resilient networks on noisy vision and text data.
We aim to show the effectiveness of GSL via testing accuracy on diverse and challenging noise patterns.
Our target evaluation metric is the accuracy achieved on the clean testing set, i.e. not affected by noise. [Table]
6. V ISION A NALYSIS
We compare GSL against four noise resilient networks from the state of the art: GLC (Hendrycks et al., 2018), SCL (Wang et al., 2019), F ORWARD (Patrini et al., 2017), B OOTSTRAP (Reed et al.) and
C O - TEACHING + (Yu et al., 2019).
In addition, we extend F ORWARD by adding reverse cross-entropy based on SCL and loss correction through the confusion matrix same as GLC, called SG F ORWARD .
For training GSL, C O - TEACHING +, SG F ORWARD and GLC, we use PyTorch v1.4.0.
For all other methods, we use Keras v2.2.4 and Tensorﬂow v1.13.0.We assume 10% of trusted data is available for
GSL, GLC and SG F ORWARD .
Table 1 summarizes the testing accuracy for all combinations of noise patterns and comparative approaches.
For CIFAR-10, GSL achieves the highest accuracy among all resilient networks except for ﬂip noise with 30% noise rate.
SG F ORWARD is the closest rival to GSL because both use the same mechanism in the loss function.
Besides, GSL has 2 to 8% higher accuracy than GLC, demonstrating the beneﬁt of introducing symmetric cross-entropy, especially in high noise rates.
In terms of comparison between GSL and SCL, the accuracy difference is even more visible, implying the beneﬁt of using corruption matrix to assign weights on two terms in symmetric cross-entropy.
We note that SCL uses an 8-layer CNN with 6 convolutional layers followed by 2 fully connected layers instead of a
Wide ResNet because of the superior results.
SCL performs particularly worse in 60% bimodal noise because this is a more challenging pattern and has no access to the corruption matrix.
Moreover, our method can still obtain 11 to 30% higher test accuracy than C O - TEACHING + that uses two deep networks concurrently.
CIFAR-100 is more challenging than CIFAR-10 due to the larger number of classes.
GSL achieves the highest accuracy except for ﬂip noise with 30% rate, and SG F ORWARD is the second best result among other competitors.
Although for ﬂip noise with 30% rate SG F ORWARD performs better than
GSL, the improvement of GSL is more signiﬁcant than SG F ORWARD compared to the CIFARdataset.
The largest difference (more than 2%) in accuracy between the GSL and SG F ORWARD methods is with bimodal noise.
In case of 60% symmetric noise, GSL achieves the accuracy of 68%, whereas GLC and SCL trail far behind.
Moreover, given the difﬁculty of training a robust classiﬁer for CIFAR-100 with 60% label noise, it is worth mentioning that SCL can achieve similar performance as GLC that is given 10% of trusted data in case of 30% symmetric noise.
This also indicates the effectiveness of symmetric cross entropy in learning hard classes even without trusted data.
However, when facing extremely noisy labels and patterns, the small amount of trusted data can greatly improve the robustness of the classiﬁer but not necessarily the symmetric cross entropy.
Seen from the high accuracy compared to GLC and SCL, GSL effectively uses the trusted data to correct symmetric cross entropy loss and improve the learning on the hard classes.
GSL performs slightly better with symmetric noise than with bimodal and ﬂip noise that is more challenging for
CIFAR-10.
In the CIFAR-100, GSL works better on the asymmetric noise rather than symmetric.
For Clothing1M dataset, as shown in Table [Table] GSL obtains the highest test accuracy compared to other methods.
Same as CIFAR-10 and CIFAR-100, SG F ORWARD achieves a relatively good performance.
The difference between GSL and SCL comes from the effectiveness of corruption matrix that makes the regular cross entropy robust. [Table]
T EXT A NALYSIS
We evaluate GSL on text datasets of Twitter and SST, against resilient networks that leverage corruption matrix, namely GLC and F ORWARD .
Both GSL and GLC use the trusted data for estimating the corruption matrix, wheres the original F ORWARD (Patrini et al., 2017) relies solely on the noisy data.
As the proposed loss of golden symmetric cross entropy is general and can be combined with different resilient networks, we hence use following four variations of loss correction and symmetric cross entropy on the existing work:
• Forward gold ( G F ORWARD ): we replace the estimation of the corruption matrix by the identity matrix on trusted samples and apply loss correction through the confusion matrix.
• True corruption matrix (TM ATRIX ): we directly use the true corruption matrix and apply loss correction through it.
• Forward gold with symmetric cross entropy ( SG F ORWARD ): we extend the corrected loss of G F ORWARD to the corrected symmetric cross entropy as in the GSL.
• True corruption with symmetric cross entropy ( S TM ATRIX ): we apply golden symmetric cross entropy and the true corruption matrix instead of the estimated matrix.
We extensively evaluate GSL, GLC G F ORWARD , TM ATRIX , SG F ORWARD , and S TM ATRIX on
Twitter and SST, with label corruption ranging from 0% to 100%.
We also vary the percentage of trusted data among 1% and 5%.
We summarize the average accuracy across 11 noise rates in Table 2.
Twitter .
GSL consistently achieves the highest average accuracy in most cases.
Compared to GLC,
GSL has signiﬁcant higher accuracy for Twitter corrupted with symmetric and bimodal noises, but the difference diminishes with increasing amounts of trusted data.
When the percent of trusted data is low, say, 1%, GLC is unable to estimate the corruption matrix accurately nor to correct the loss, seen by the difference between GLC and TM ATRIX .
SST .
Here, the classiﬁcation involves only two classes and turns out to be less challenging than the
Twitter case.
The difference among the different comparative approaches considered is smaller than for Twitter.
For instance, though GSL consistently achieves the best average accuracy in almost all cases, the difference between GSL and GLC is around 1-3%.
Again, we see that GSL visibly outperforms GLC on low amounts of trusted data because of using cross entropy and the difference among them becomes limited.
We note that TM ATRIX and G F ORWARD collapse under Flip noise.
C ONCLUSION
To enhance the robustness of deep models against by label noise we propose GSL that features on correcting the symmetric cross entropy loss by the noise corruption matrix.
GSL uses a small fraction of trusted data to accurately estimate the corruption matrix, and further determine the weights applied on regular and reverse cross entropy.
GSL learns deep networks from trusted samples through regular cross entropy and from untrusted noisy samples through golden symmetric cross entropy.
We prove that the cross entropy corrected by the corruption matrix is noise robust.
To adapt to noise patterns of dataset, we heuristically set the weights of golden symmetric loss based on the corruption matrix.
We extensively evaluate GSL on vision and text analysis under diversiﬁed noise rates and patterns.
Evaluation results show that GSL can achieve a remarkable accuracy improvement, i e., from 2 to 18% on CIFAR benchmarks and real world noisy data, compared to methods that either correct loss or leverage symmetric cross entropy.
A PPENDIX A T RAINING WITH C ORRECTED L ABELS : V ISION R ESULTS
Here we present the extensive results of our empirical evaluation on training with corrected labels for the vision datasets.
This complements the results presented in § 3.
We compare the impact of correcting labels only on the cross entropy term ( ce only ), only on the reverse cross entropy term ( rce only ), or both .
Table 3 and Table 4 show the achieved accuracy for CIFAR-10 and CIFAR-100, respectively, under two noise rates, 30% and 60%, three different noise types, symmetric, bimodal and ﬂip, and three fractions of trusted data, 5%, 10% and 15%.
For each noise scenario the best case is highlighted in bold.
ce only achieves the highest accuracy in all cases except one.
Under 60% bimodal noise on CIFAR-100 with 10% trusted data rce only is slightly better by 0 .
28 percent points.
More in general, rce only typically performs second best and both achieves the worst accuracy.
Focusing on ce only over the other two, the gain tends to increase with the difﬁculty of the noise scenarios, i.e. with higher number of classes, higher noise rates and less trusted data.
ce only outperforms the other two by up to 11.74 percent points for CIFAR-10 and up to 51.03 percent points for CIFAR-100.
B T EXT A NALYSIS ON T WITTER AND SST D ATASETS WITH V ARYING N OISE R ATES
Figure 4 shows how the accuracy changes with respect to different noise rates on the Twitter and SST datasets. GSL and GLC are provided with one percent trusted data.
In contrast, GSL can effectively use the symmetric cross entropy to overcome the limitation of low trusted data.
This also explains why S TM ATRIX always trails closely behind GSL by using the true confusion matrix and symmetric entropy loss.
One may further improve S TM ATRIX by using the optimal weights of A and B according to the true corruption matrix, instead of estimated corruption matrix of GSL. The
Twitter dataset highlights well the differences (see Figure 4a).
The SST dataset is an easier problem with only two classes and all methods are able to perform equally well (see Figure 4b).
[Figure]
C P ROOF OF R OBUSTNESS OF C ROSS E NTROPY L OSS WITH L ABEL C ORRECTION T HEOREM 4.
Theorem 4.1.
In a multi-class classiﬁcation problem, (cid:96) ce with prediction correction is noise tolerant under symmetric label noise if the noise rate ε < K − 1 K − ∆
A ∆ R , where ∆ A = (cid:80) Kk =1 (cid:96) ce ( C ∗ T f ( x ) , k ) − (cid:80) Kk =1 (cid:96) ce
( ˆ C T f ( x ) , k ) , and ∆ R is the difference of risk minimization between optimal classiﬁer and f .
And (cid:96) ce with prediction correction is also noise tolerant under ﬂip noise when noise rate ε yk ≤ (1 + ∆ W y ∆ W k ) − ε
y
(1 + ∆ W y ∆ W k ) where ε k and ε yk are the correct and ﬂipped class probabilities, respectively.
Proof.
For symmetric noise:
[Equation]
,y (cid:96) ce
( ˆ C T f ( x ) , y ) =
E x ,y [(1 − ε ) (cid:96) ce
( ˆ C T f ( x ) , y )
+ ε
K − 1( (cid:88) k
(cid:54) =
y (cid:96) ce
( ˆ C T f ( x ) , k ))]
=
E x ,y [(1 − ε ) (cid:96) ce
( ˆ C T f ( x ) , y )
+ ε
K − 1(
K (cid:88) k
= (cid:96) ce
( ˆ C T f ( x ) , k ) − (cid:96) ce
( ˆ C T f ( x ) , y ))]
= (1 − ε )
R
( f ( x , ˆ C ) + ε K − 1(
K (cid:88) k
= (cid:96) ce
( ˆ C T f ( x ) , k ) − R ( f, ˆ C ))
=
R ( f, ˆ C )(1 − εK K − 1) + ε
K − 1(
K (cid:88) k
= (cid:96) ce
( ˆ C T f ( x ) , k ))
Let A ( ˆ C T f ( x ) , y )
=
(cid:80)
Kk =1 (cid:96) ce
( ˆ C T f ( x ) , k ) .
Then we can rewrite (7) as R ε ( f, C ) =
(1 − εKK − 1 ) R ( f, C ) + εK − 1 A ( C T f ( x ) , y ) , thus:
R ε ( f ∗ , C ∗ ) − R ε ( f, ˆ C ) = (1 − εK K − 1) ( R ( f ∗ , C ∗ ) − R ( f, ˆ C )) (cid:124) (cid:123)(cid:122) (cid:125) ∆ R + ε
K − 1 ( A ( C ∗ T f ∗ ( x ) , y ) − A ( ˆ C T f ( x ) , y ))
(cid:124) (cid:123)(cid:122)
[Equation]
where ∆ R ≤ 0 , because f ∗ is the global minimizer for R and C ∗ the optimal noise confusion matrix.
Similarly, ∆ A ≤ 0 because for the optimal case we can say A ( C ∗ T f ∗ ( x ) , y )
≈ 0 .
(cid:96) ce with label correction is robust to noise when R ε ( f ∗ , C ∗ ) − R ε ( f, ˆ C ) ≤ 0 .
This is true when:
[Equation]
With no label correction, C is missing in ∆ A and the two terms become equal, i.e. ∆ A = 0 .
In this condition the bound becomes ε < K − 1 K as found by (Ghosh et al., 2017) for cross-entropy without label correction.
Since ∆ A ∆ R ≥ 0 , ε < K − 1 K ≤ K − 1 K − ∆ A ∆ R the new bound can be seen as generalization of the previous bound.
∆ A ∆ R should also be less than one to ensure a meaningful bound on ε , avoiding scenarios of noise rate greater than 1.
For asymmetric ﬂip noise, 1 − ε y
is the probability of a label being correct (i.e., k = y ), and the noise condition ε yk < 1 − ε y generally states that a sample x has a higher probability ( 1 − ε y ) of being classiﬁed correctly as class y , rather than the probability ( ε yk ) of being classiﬁed incorrectly as class k (cid:54) = y .
[Equation]
Similar to the symmetric case we require that R ε ( f ∗ , C ∗ ) − R ε ( f, ˆ C ) ≤ 0 for the loss to be robust to noise: [Equation]
[Equation]
where ∆ W y ≤ 0 and ∆ W k ≤ 0 because C ∗ is the optimal noise confusion matrix.
Rewriting (11):
[Equation]
According to (12), the bound is ε yk ≤ (1 + ∆ W y ∆ W k ) − ε y
(1 + ∆ W y ∆ W k ) .
With no label correction ∆ W y =
and the bound becomes ε yk < 1 − ε y as found by prior art.