Investigating Why Contrastive Learning Benefits Robustness against Label Noise
Abstract
Self-supervised contrastive learning has recently been shown to be very effective in preventing deep networks from overfitting noisy labels.
Despite its empirical success, the theoretical understanding of the effect of contrastive learning on boosting robustness is very limited.
In this work, we rigorously prove that learned the representation matrix has certain desirable properties in terms its SVD that benefit robustness against label noise.
We further show that the low-rank structure of the
Jacobian of deep networks pre-trained with contrastive learning allows them to achieve a superior performance initially, when fine-tuned on noisy labels.
Finally, we demonstrate that the initial robustness provided by contrastive learning enables robust training methods to achieve state-of-the-art performance under extreme noise levels.
1. Introduction
Very recently, self-supervised contrastive learning has shown a lot of promise in boosting robustness of deep networks against noisy labels (Zheltonozhskii et al., 2022;
Hendrycks et al., 2019; Ghosh & Lan, 2021).
Despite its empirical success, the theoretical understanding of the effect of contrastive learning on improving robustness of deep networks against noisy labels is very limited.
In this work we theoretically characterize the beneficial properties of representations obtained by contrastive learning for enhancing robustness against noisy labels.
We prove that contrastive learning produces a representation matrix that has: (i) a prominent singular value corresponding to each sub-class in the data, and significantly smaller remaining singular values; and (ii) a large alignment between the prominent singular vectors and the ground-truth labels.
Then we analyze the case where a linear model is trained on the obtained representations with labels that are either perturbed with
Gaussian noise or flipped at random to other classes
We show that noise has minimal effect on learning the clean labels and the model can hardly memorize the wrong labels.
We further show that deep networks pre-trained with contrastive learning can be fine-tuned on noisy labels to achieve a superior performance initially, before overfitting the noise.
We attribute this to the initial low-rank structure of the Jacobian with a few large singular values associated to its prominent directions and insignificant singular values otherwise.
Finally, we conduct extensive experiments on noisy CIFAR10 and CIFAR-100 (Krizhevsky & Hinton, 2009) and the mini Webvision datasets (Li et al., 2017) to demonstrate that the initial robustness provided by contrastive learning can be further leveraged by robust methods to achieve state-ofthe-art performance under extreme levels of noise.
2. Problem Formulation and Background
Suppose we have a dataset D
= { ( xxx i ,yyy i ) } ni =1 , where ( xxx i ,yyy i ) denotes the i -th sample with input xxx i ∈ R d and its clean one-hot encoded label yyy i ∈
R K corresponding to one of the K classes.
For example, for a data point xxx i from class j ∈
[ K ] , we have yyy i = eee j where e j denotes the vector with a 1 in the j th coordinate and 0’s elsewhere.
We further assume that there are ¯ K ≥ K sub-classes in the data.
Sub-classes of a class share the same label, but are distinguishable from each other.
We assume that for every data point xxx i , we only observe a noisy version of its label ˆ yyy i .
We denote by YYY , ˆ YYY ∈ R n × K the matrices of all the one-hot encoded clean and noisy labels of the training data points.
2.1. Self-supervised Contrastive Learning
Self-supervised contrastive learning learns representations of different data points by maximizing agreement between differently augmented views of the same example via a contrastive loss in the latent space, as we discuss below.
Augmentation graph.
The augmentations can be used to construct the population augmentation graph (HaoChen et al., 2021), whose vertices are all the augmented data in the population distribution, and two vertices are connected with an edge if they are augmentations of the same natural example.
Hence, ground-truth classes naturally form connected sub-graphs.
Formally, let P be the distribution of all natural data points (raw inputs without augmentation).
For a natural data point xxx ∗ ∼ P , let A ( ·| xxx ∗ ) be the distribution of xxx ∗ ’s augmentations.
For instance, when xxx ∗ represents an image,
A ( .
| xxx ∗ ) can be the distribution of common augmentations (Chen et al., 2020) including Gaussian blur, color distortion and random cropping.
Then, for an augmented data point xxx , A ( xxx | xxx ∗ ) is the probability of generating xxx from xxx ∗ .
The edge weights w x i x j = E xxx ∗ ∼ P
[ A ( xxx i | xxx ∗ )
A ( xxx j | xxx ∗ )] can be interpreted as the marginal probability of generating xxx i and xxx j from a random natural data point.
Contrastive loss.
The embeddings produced by contrastive learning can be viewed as a low-rank approximation of the normalized augmentation graph.
Effectively, minimizing a loss that performs spectral decomposition on the population augmentation graph can be succinctly written as a contrastive learning objective C ( f ) on neural network representations (HaoChen et al., 2021):
[Equation]
where f ( xxx ) ∈ R p is the neural network representation for an input xxx , and xxx,xxx + are drawn from the augmentations of the same natural data point, and xxx,xxx − are two augmentations generated independently either from the same data point or two different data points.
The above loss function is similar to many standard contrastive loss functions (Oord et al., 2018; Sohn, 2016; Wu et al., 2018), including SimCLR (Chen et al., 2020) that we will use in our experiments.
Minimizing this objective leads to representations with provable accuracy guarantees under linear probe evaluation.
We use f min to denote the minimizer, i.e., f min = arg min f C ( f ) .
2.2. Training the Linear Head with Label Noise
After obtaining the representations of dimension p by minimizing the contrastive loss, a linear layer parameterized by WWW ∈ R p × K is trained on the representations.
Given a matrix FFF ∈ R n × p where each row FFF i = f min ( xxx i )
⊤ is the learned representation of a data point
xxx
i , we consider the downstream task of training a linear model, parameterized by WWW ∈ R p × K , to minimize the MSE loss with lregularization with parameter β
[Equation]
Let ˆ WWW ∗ denote the solution that has the following closedform expression
[Equation]
While we use MSE in our analysis, we empirically show that our results hold for other losses, such as cross-entropy.
3. Contrastive learning Boosts Robustness 3.1. Provable Robustness of the Linear Head
To theoretically understand the robustness provided by contrastive learning, we assume certain properties of the augmentation graph and analyze the low-rank structure of the the resulting representation matrix.
In particular, we utilize the following natural assumptions that formalize the following two properties on the data augmentation: (1) the augmented examples of one sub-class are similar to each other; and (2) the augmented examples of one sub-class are different from the augmented examples of other sub-classes.
Assumption 3.1 ( Compact sub-class structure ) .
For a triple of augmented examples x x x j , x x x s and x x x t from the same sub-class, the marginal probability of xxx s , xxx j being generated from a natural data point is close to that of xxx t , xxx j .
Formally, we have w xxx s xxx
j /w xxx
t xxx j ∈
[ 11+ δ , 1 + δ ] , for small δ ∈ [0 , 1) .
Assumption 3.2 ( Distinguishable sub-class structure ) .
For two pairs of augmentated examples ( xxx i ,xxx j ) and ( xxx s ,xxx t ) where xxx i , xxx j are from different sub-classes and xxx s , xxx t are from the same sub-class, the marginal probability of xxx i , xxx j being generated from a natural data is much smaller than that of xxx s , xxx t .
Formally, we have w xxx
i xxx
j /w
xxx
s xxx t ≤ ξ , for small ξ ∈ [0 , 1) .
The above assumptions result in an augmentation graph where augmented data points from different subclasses form nearly disconnected subgraphs with similar edge weights.
In particular for ξ = 0 , we get diconnected subgraph structure.
Desirable Properties of Contrastive Representations
The key to our analysis is that, based on compact and distinguishable sub-class structure assumptions 3.1, 3.2, contrastive learning produces a low-rank representation matrix
FFF that captures the sub-class structure.
More formally, the representation matrix has ¯ K singular vectors that align well with the ground-truth labels, and the corresponding ¯ K singular values are significant larger than the other singular values.
The following theorem is a summary of Lemmas
A.2 A.3 B.2 B.3 and Corollary A.6 in the Appendix which details the desirable properties of the representation matrix.
Theorem 3.3.
Having ¯ K compact and distinguishable subclasses in the data, the representation matrix FFF learned by contrastive learning has ¯ K prominent singular values of magnitude O (1) .
At the same time, the sum of the remaining singular values is significantly smaller, i.e., O ( √ δ + ξ ) .
Furthermore, the most prominent ¯ K singular vectors and the ground-truth labels has a O (1) alignment, measured by the normalized projection of the clean labels YYY onto the
[Figure]
span of the singular vectors.
Intuitively, the above three properties of the representation matrix affect the downstream training in the following sense: (1) the magnitude of largest singular values determines the speed at which the model evolves as well as the extent to which the model can fit the training data; (2) the alignment between prominent singular vectors and clean labels indicates whether the model evolves in the right direction; and (3) the magnitude of smaller singular values dictates the amount of overfitting.
As a result, Theorem 3.3 implies that the model trained on such representation learns mainly the correct information from the training data, which we formally show in Theorems 3.4 and 3.5.
Training performance w.r.t. Ground-truth Labels Reflects Robustness
To simplify the theoretical analysis, instead of studying the generalization performance (usually measured by the expected loss over the data distribution), we will examine the loss and accuracy on the training data w.r.t.
groundtruth labels.
This strongly correlates with the test accuracy, especially under large noise.
We empirically confirm this correlation in Figure 1, where the dashed lines show training loss and training accuracy w.r.t.
ground-truth labels, and solid lines show test loss and test accuracy.
We clearly see the high correlation between training and performance, in particular under significant levels of label noise.
Gaussian Label Noise
We first consider the case where label noise is generated from a Gaussian distribution.
Formally, ˆ YYY =
YYY + ∆ YYY , where YYY is the clean label matrix containing all the one-hot encoded labels, and ∆ YYY is the label noise matrix, where each column drawn independently from N (0 , σ 2 I n /K ) .
We consider this setting first, as it provides the most convenient way to analyze robustness.
Here, our analysis mainly aims at breaking down the effect of label perturbations on training dynamics, in terms of bias and variance.
This could provide theoretical insights into the benefits of contrastive learning for boosting robustness.
The following theorem bounds the expected error on training data w.r.t.
ground-truth labels, and shows how contrastive learning exploits the augmented sub-class structure to improve robustness.
Theorem 3.4.
For a dataset of size n with K classes, ¯ K balanced compact and distinguishable sub-classes (c.f. assumptions 3.2, 3.1) and labels corrupted with Gaussian noise N (0 , σ 2 I n /K ) , a linear model trained by minimizing the objective in Eq.
(2) with the representations obtained by minimizing contrastive loss in Eq.
(1) has the following expected error on the training set w.r.t.
the ground-truth labels YYY :
[Equation]
We note that the above results can be easily extended to imbalanced sub-class structure.
The proof can be found in the Appendix.
The proof follows the intuition discussed in Section 3.1 that the desirable properties of the learned representation benefit the downstream training.
In a nutshell, we derive the bound by writing the error in terms of singular values and vectors of FFF and then applying Theorem 3.3.
In Eq.
(4), the error is decomposed into bias and variance.
The bias captures the mismatch between the average prediction of the model and the ground-truth labels.
It depends on the magnitude of the prominent singular values as well as the alignment of the corresponding singular vectors with the ground-truth labels.
Contrastive learning reduces the bias by aligning the first ¯ K singular vectors with ground-truth labels (Theorem 3.3), thus producing a small second term in the bias.
The variance quantifies the sensitivity to label noise, and is controlled by the magnitude of the non-prominent singular values, which is guaranteed to be small by Theorem 3.3.
The regularization parameter β appears in both terms and can be tuned as a trade-off between underfitting and overfitting.
With small enough δ and ξ , one can select a small β to not explicitly penalize the variance much.
This results in a small bias, and subsequently a small total error.
For example, when there exists a β such that √ δ + ξ ≪ β ≪ 1 , the error ≈
σ 2 ¯ Kn ( 1 β +1 ) 2 , which is the inevitable cost of achieving a small bias, when the representation matrix FFF has ¯ K prominent singular values.
Random Label Flipping
Next, we study the case where the label noise ∆ YYY =
ˆ YYY − YYY is generated by flipping a fraction of the clean labels at random.
Formally, for an example xxx i belongs to class j with y y y
i
=
e e e j , if its label is flipped to class k , we have ∆ yyy i =
eee k − eee j .
We introduce the following notations to analyze the case of asymmetric label noise which mimics the real-world noise, where wrong labels are generated in a (sub)class-dependent way.
Let m ¯
k
be the number of mislabeled examples in subclass ¯ k , m ¯ k,k be the number of mislabeled examples in subclass ¯ k that are labeled as class k , and Z k be the set of sub-classes in class k .
We define c k | ¯ k := m ¯ k,k m ¯ k for all ¯ k , k such that
¯ k / ∈
Z k , to be the fraction of mislabeled examples in sub-class ¯ k that are mislabeled as k .
We have that (cid:80) k :
¯ k/ ∈
Z k c k | ¯ k = 1 , ∀ ¯ k ∈ [ ¯ K ] .
When c k | ¯ k = 1 K − 1 ∀ k ∈ [ K ] , ¯ k ∈
[ ¯ K ] , the noise is symmetric.
For simplicity we assume ξ = 0 .
The general case of ξ ≥ 0 requires more involved analysis which we discuss in the Appendix.
The following theorem shows that for a dataset with compact and distinguishable sub-class structure the linear classifier trained on the representations obtained by contrastive learning can recover the clean label for all training data.
Theorem 3.5 (Asymmetric Noise) .
For a dataset with K classes and ¯ K compact and distinguishable sub-class structure (c.f. assumptions 3.2 3.1) with ξ = 0 , let n min , n max be the size of the smallest and largest sub-class, and α be the fraction of mislabeled examples in the training set.
Let c max :=
max k ∈ [ K ] , ¯ k ∈
[ ¯ K ] c k | ¯ k ∈
[ 1 K − 1 , 1] be the maximum fraction of wrong labels in a subclass that are flipped to another class.
Then as long as
[Equation]
a linear model trained by minimizing the objective in Eq.
(2) with the representations obtained by minimizing contrastive loss in Eq.
(1) can predict the ground-truth labels for all training examples, i.e.,
[Equation]
In other words contrastive learning can prevent the linear model from memorizing any wrong label even under large noise.
Theorem 3.5 also shows that the model can tolerate more noise when the sub-class structure is more compact, i.e., δ is smaller, or the noise is more symmetric, or the sub-classes are more balanced.
The following corollary for symmetric noise is simply obtained by setting c max
= 1 K − 1 in Theorem 3.5.
Corollary 3.6 (Symmetric Noise) .
For symmetric noise, under the same assumption as in Theorem 3.5, as long as
[Equation]
a linear model trained by minimizing the objective in Eq.
(2) with the representations obtained by minimizing contrastive loss in Eq.
(1) can predict the ground-truth labels for all training examples.
If we further let δ = 0 and n max /n min = 1 , we get ( K − 1) /K
noise tolerance.
We note that this, however, does not imply that a dataset with more classes necessarily has a higher noise tolerance.
In Appendix B.1, we show that less distinguishable sub-class structure, i.e. ξ > 0 , introduces a O ( ¯ K 5 / 2 ξ ) perturbation to the singular values and a O ( ¯ K 5 / 2 ξ ) rotation in the direction of singular vectors of the representation matrix.
Datasets with more classes usually contains more sub-classes, which greatly reduces the noise tolerance.
This is also reflected by our empirical results (Figure 1) where the performance of the linear model is worse on CIFAR-100 compared to CIFAR-10 under the same noise level.
3.2. Contrastive Learning Slows down Overfitting for Fine-tuning
As shown in Fig.
2c, finetuning achieves a higher test accuracy than the randomly initialized one.
Recall that the theoretical guarantee for linear model (theorems 3.4 and 3.5) is obtained by examining singular values and singular vectors of FFF .
Here, we use a similar idea to understand benefits of contrastive learning on robustness when all the layers are trained.
Intuitively, during the early stage of training, it is natural to assume that the gradient does not considerably change, and therefore the model is nearly linear.
In this case, the initial Jacobian matrix plays the same role as the representation matrix, FFF , to the linear model.
This is supported by the recent studies suggesting the following properties of training neural networks: the early learning dynamics can by mimicked by training a linear model (Hu et al., 2020), SGD
[Figure] [Figure]
C o un t s
Pretrained
[Figure]
initialized (a) 100 200
[Figure]
Singular values
I nn e r p r o d u c t b t w s i n g u l a r v e c t
o r a n d l a b e l
Pretrained Randomly initialized (b)     50 Epoch
T e s t
A cc u r a c y
Randomly intialized Pretrained (c)     50 Epoch 1.
L o ss o n
M i s l a b e l e d E x a m p l e s (d)
[Table]
on neural networks learns a linear model first (Kalimeris et al., 2019), and a network that provides a better alignment between prominent directions of the Jacobian matrix and the label vector is more likely to generalize well (Oymak et al., 2019).
We examine the SVD of the Jacobian of a ResNet pretrained with contrastive learning and compare it to that of a randomly initialized network.
Fig.
2a, 2b present the distribution of singular values and the alignment of singular vectors with clean labels.
The Jacobian is computed on a random sample of 1000 data points from CIFAR10.
Interestingly, Fig.
2b shows that while pre-training does not considerably improve (in Appendix C we show a slight improvement) the alignment between singular vectors of the Jacobian and the clean label vector, it greatly shrinks the smaller singular values of the Jacobian, as is illustrated by Fig.
2a.
As a result, it takes substantially longer for the pre-trained network to overfit the noisy labels.
As Fig.
2d shows, while a randomly initialized network experience a sharp drop in loss of noisy labeled data points during the first few epochs of training, it takes much longer for a pre-trained network to overfit the noise.
Details of the experiment can be found in Appendix D.3.
Contrastive learning boosts robust methods.
As discussed, pre-training the network with contrastive learning effectively shrinks the smaller singular values of the
Jacobian and slows down overfitting the noisy labels.
Our experiments ( c.f.
Table 1 and Appendix D) show that the initial level of robustness provided by contrastive learning can be leveraged by existing robust training methods to achieve a superior performance under extreme noise levels.
In particular, we consider three methods that prevent overfitting through regularization, namely, Mixup (Zhang et al., 2017) and ELR (Liu et al., 2020), or identifying clean examples, namely, CRUST (Mirzasoleiman et al., 2020).
4. Conclusion
We showed that representations learned by contrastive learning provably boosts robustness against noisy labels.
In particular, contrastive learning provides a representation matrix that has: (i) a significant gap between the prominent singular values and the remaining ones; (ii) a large alignment between the prominent singular vectors and the clean labels.
The above properties allow a linear layer trained on the representations to effectively learn the clean labels well while barely overfitting the noise.
Then we explained why fine-tuning all layers of a network pre-trained with contrastive learning can also achieve a good performance in the early phase.
Crucially, contrastive learning greatly reduces the magnitude of nonprominant singular values of the initial Jacobian matrix, which slows down the overfitting.
Finally, we demonstrated that the initial robustness provided by contrastive learning can boost robust methods and achieve state-of-the-art performance under extreme noise levels.
Our results confirm benefits of contrastive pretraining for robust machine learning.
Acknowledgements
This research was supported in part by Cisco Systems and
UCLA-Amazon Science Hub for Humanity and Artificial
Intelligence.
A. Analysis for Disconnected Subclasses
In this section we consider the case where ξ = 0 in assumption 3.2, which implies that the probability of two augmentation data from different subclasses being generated from the same random natural datum is exactly zero.
And in section B we extend the results to any ξ ∈ [0 , 1) via eigenvalue and eigenvector perturbation.
We use ∥ · ∥ 1 , ∥ · ∥ 2 and ∥∥ F to denote the 1 -norm, operator norm and Frobenius norm, respectively.
A.1. Spectral Decomposition of Augmentation Graph
An important technical idea we use to formalize the representations obtained by contrastive learning is augmentation graph (HaoChen et al., 2021), which is an undirected graph with all augmentation data { xxx 1 ,xxx 2 , . .
xxx n } as its vertices and w xxx
i xxx j as the weight for edge ( xxx i ,xxx j ) .
Let AAA denote the adjacency matrix of the augmentation graph, that is, each entry a ij =
w xxx i xxx j .
And the normalized adjacency matrix AAA is defined as
[Equation]
where DDD = diag ( w xxx 1 , w xxx 2 , . . . , w xxx n ) with w xxx i = (cid:80) nj =1 w xxx
i xxx j .
For simplicity we index the augmentation data in the following way: the first n 1 data are from subclass 1 , the next n 2 data are from subclass 2 , ..., the last n ¯ K data are from subclass ¯ K .
Lemma A.1 states an important property of AAA .
Lemma A.1.
Assumption 3.2 with ξ = 0 implies that AAA is a block diagonal matrix diag ( AAA 1 ,AAA 2 , . .
. ,AAA ¯ K ) where AAA ¯ k ∈ R n ¯ k × n ¯ k .
This combined with 3.1 gives us: for each block AAA ¯ k , the ratio between two entries in the same column is bounded as follows
[Equation]
where ¯ a k,i,j are the entries of AAA ¯ k and
δ ′ =
(1 + δ ) 3 / 2 − 1 =
δ + O ( δ 2 ) .
Let f min =
arg min f C ( f ) and FFF min =
[ f min ( xxx 1 ) f min ( xxx 2 ) . . .
f min ( xxx n )] ⊤ , according the theorem in (HaoChen et al., 2021), FFF min is also the minimizer of the following matrix factorization problem
[Equation]
and therefore can be further decomposed as
[Equation]
by Eckart–Young–Mirsky theorem (Eckart & Young, 1936), where FFF ∗ ∈ R n × p
=
[ vvv 1 ,vvv 2 , . . .
,vvv p ] ∈ R n × p , ΣΣΣ = diag ( √ λ 1 , √ λ 2 , . . .
, (cid:112) λ p ) , RRR ∈ R
p × p is some orthogonal matrix, λ 1 , λ 2 , . . .
, λ p are the p largest eigenvalues of AAA and vvv 1 ,vvv 2 , . . .
,vvv p are the corresponding unit-norm eigenvectors.
W.l.o.g.
, we assume λ 1 ≥ λ 2 ≥ · · · ≥ λ p .
Our following proofs are based on this decomposition.
To avoid cluttered notation we drop the subscript of FFF min , i.e., we use FFF for the learned representation.
A.2. Properties of Singular Values/Vectors of the Representation Matrix
From the above section we know that the singular values/vectors of FFF are the first p eigenvalues/vectors of AAA .
For each block
AAA ¯ k , let λ ¯ k, 1 , λ ¯ k, 2 , . .
, λ ¯ k,n ¯ k denote the eigenvalues (in descending order) and vvv ¯ k, 1 ,vvv ¯ k, 2 , . .
,vvv ¯ k,n ¯ k denote the corresponding eigenvectors.
The eigenvalues of AAA are the list of the eigenvalues of all blocks.
The corresponding eigenvectors are the block vectors ( ⃗ 0 1 ,⃗ 0 2 , . . .
,⃗
¯ k − 1 ,vvv ¯ k,i ,⃗ 0 ¯ k +1 , . . .
,⃗ 0 ¯ K ) :
= ˙
vvv ¯ k,i with each ⃗
j being a zero vector of length n j .
Since AAA is a normalized adjacency matrix, each block AAA ¯ k is also normalized.
Then the largest eigenvalue for each block is exactly 1 , i.e., λ ¯ k, 1 = 1 .
It immediately follow Lemma A.2.
Lemma A.2.
The ¯ K largest eigenvalues of A A A are all 1 , i.e., λ
= λ 2 = · · · = λ ¯ K
= 1 .
As long as p ≥ ¯ K , all λ ¯ k, 1 and vvv ¯ k, 1 appear in the decomposition of FFF .
Let p ¯ k ≥ 1 be the number of eigenvalues/eigenvectors of block AAA ¯ k that also appear in the decomposition of FFF .
The following Lemmas and Corollaries states other important properties of eigenvalues/eigenvectors of AAA .
Lemma A.3.
By assumption 3.1, the 1 -norm of vvv ¯ k, 1 has the following lower bound.
[Equation]
Proof.
Write vvv ¯ k, 1 as [ e 1 , e 2 , . .
, e n ¯ k ] T .
By Perron-Frobenius theorem, all the elements here are positive since AAA ¯ k is a positive matrix.
Then the quotient of any two elements in vvv ¯ k, 1 can be bounded in terms of the entries of AAA ¯ k (Minc, 1970) and therefore 1 + δ ′ by lemma A.1:
[Equation]
Let e min =
min i e i , we have
[Equation]
and
[Equation]
Combining the preceding two equations yields
[Equation]
Lemma A.4.
The sum of squared eigenvalues of each block AAA ¯ k can be bounded.
[Equation]
Proof.
First we have
[Equation] [Equation]
where ccc ¯ k,j = [¯ a ¯ k, 1 ,j , ¯ a ¯ k, 2 ,j , . .
, ¯ a ¯ k,n ¯ k ,j ] T denotes the j -th column in AAA ¯ k .
Analogous to lemma A.3, here we can bound ∥
ccc ¯ k,j ∥ 22 in terms of ∥ ccc ¯ k,j ∥ 21 by lemma A.
[Equation]
We also have
[Equation]
where the second inequality holds because of assumption 3.1 and the third inequality holds because of Perron-Frobenius theorem.
Combining equations 9, 10 and 11 completes the proof.
Corollary A.5.
The eigenvalues except the ¯ K largest ones are each upper bounded by
[Equation]
Proof.
By Lemmas A.2 and A.
[Equation]
Then
[Equation]
Corollary A.6.
For each block A A A ¯ k , the sum of the eigenvalues from the second to the p ¯ k -th is bounded by
[Equation]
And the sum of eigenvalues of AAA from the ¯ K + 1 -th to the p -th is bounded by
[Equation]
Proof.
Applying Cauchy–Schwarz inequality to equation 12 yields the bound for (cid:80) p ¯ k i
=
λ ¯ k,i .
Summing both sides of equation 12 over ¯ k ∈ [ ¯ K ] and then applying Cauchy–Schwarz inequality give the bound for (cid:80) pi =
¯
K +1 λ i .
A.3.
Error under Gaussian Noise when ξ =
With the decomposition in Equation 8, the learned parameter of the linear model in Equation 3 can be rewritten as
[Equation]
The output on the training set FFF ˆ WWW ∗ is
[Equation]
where BBB = diag ( b 1 , b 2 , . . .
, b p ) with b i =
λ
i λ i + β .
And the error on training set w
[Equation]
r
[Equation]
t. ground-truth labels can be therefore written in terms of the eigenvalues and eigenvectors of AAA
[Equation]
where yyy j ∈ R n is the j -th column of YYY .
We first calculate the bias 2 term.
We have b 2 i − 2 b i ≤ 0 since b i ∈ (0 , 1] .
Then we drop items with i ≥ ¯ K + 1 in the summation and apply Lemma A.2 to get an upper bound
[Equation]
By Perron-Frobenious theorem all elements in vvv ¯ k, 1 are positive, thus the sum of elements of vvv ¯ k, 1 can be written as ∥ vvv ¯ k, 1 ∥ 1 .
With the observation that ˙ vvv T ¯ k, 1 yyy j = ∥
vvv ¯ k, 1 ∥ 1 when ¯ k is a subclass within class j and otherwise ˙ vvv T ¯ k, 1 yyy j = 0 , the above equation can be rewritten as
[Equation]
Then by Lemma A.3,
[Equation]
Now we bound the variance term.
By Lemma A.
[Equation] [Equation]
Apply Cauchy–Schwarz inequality and Corollary A.6 to bound the summation in the last term
[Equation]
Plugging the preceding into Equation 16 yields
[Equation]
A.4. Accuracy under Label Flipping (Proof for Theorem 3.5)
We study the accuracy by looking at the entries of the output FFF ˆ WWW ∗ .
[Equation]
For convenience we define the notations C k and S ¯ k as the sets of indices of examples from class k and subclass ¯ k , respectively
[Equation]
to class k
[Equation]
i belongs to subclass ¯ k }
= { i :
[Equation]
Let the notation µµµ ( j ) denote the j -th element of vector µµµ .
Then zzz ( j ) k can be written as (cid:80) pi =1 b i vvv ( j )
i vvv ⊤ i ( yyy k + ∆ yyy k ) .
Let ¯ k j denote the subclass that xxx j belongs to, i.e., j ∈ S ¯ k and define e min ,s :=
min
j vvv ( s ) ¯ k j , 1 and e max ,s :=
max s vvv ( s ) ¯ k j , 1 .
We have the following two lemmas.
Lemma A.7.
zzz ( j ) k can be bounded
[Equation]
Proof.
Let b ¯ k,i denote λ ¯ k,i β
+ λ ¯ k,i .
Recalling that one property of the block vector ˙
vvv ¯ k,i is that ˙ vvv ( j ) ¯ k,i = 0 when j / ∈ S ¯ k , we have
[Equation]
For the nonzero elements in ˙ vvv ¯ k j , 1 , if j ∈ C k , there are at least n min (1 − α ) elements being 1 at corresponding positions in yyy k + ∆ yyy k ; if j / ∈ C k , there are at most n max αc max elements being 1 at corresponding positions in yyy k + ∆ yyy k .Then
the inner product in the first term in equation 17 can be bounded by
[Equation]
For the inner product in the second term in equation 17, if j ∈ C k , then ˙ vvv ⊤ ¯ k j ,i ( yyy k + ∆ yyy k ) is the sum of at most n max elements in ˙
vvv ¯ k j ,i .
Since ∥ ˙ vvv ¯ k j ,i ∥ 22 = 1 , the sum is bounded by [ −√ n max , √ n max ] .
Similarly we can get the bound [ −√ n max c max , √ n max c max ] for j / ∈ C k .
We also know that | ˙ vvv ( j ) ¯ k j ,i | ≤ 1 .
Now it remains to bound (cid:80) p ¯ kj i =2 b ¯ k j ,i by applying
Cauchy–Schwarz inequality and Corollary A.
[Equation]
Lemma A.8.
e min ,j , e max ,j and e max,j /e min ,j can be bounded.
e min ,j ≥
[Equation]
Proof.
The proof is analogous to that for lemma A.3.
Now we are ready to calculate the maximum noise level that allows correct prediction on all training examples, i.e., zzz jk : j ∈ C k > zzz jk ′ :
j/ ∈ C k ′ .
Let the preceding hold, then by lemma A.7 we get
[Equation]
Plugging lemma A.8 into equation 18 with some algebraic manipulation yields
[Equation]
B. Considering Off-Diagonal Entries in the Adjacency Matrix (Connected Subclasses)
For here on we assume n ¯ k = n ¯ K , ∀ ¯ k ∈ ¯ K for simplicity, despite that our results can easily extend to unbalanced dataset.
Lemma B.1.
Under assumption 3.2, the off-diagonal entries in A is no longer zero.
Let (cid:101) AAA denote the new normalized matrix, which also contains non-zero off-diagonal entries.
With a bit abuse of notation, in the following we use AAA to denote the matrix obtained by normalizing A with off-diagonal elements ignored.
Then all the properties of eigenvectors and eigenvalues of AAA stated before including those lemma A.1, A.3, A.4 still hold.
And (cid:101) AAA can be written as a perturbation of AAA (cid:101) AAA = AAA + EEE, with ∥ EEE ∥ F = O (cid:0) ¯ K 5 / 2 ξ (cid:1) .
Proof.
Let ˚ AAA be a matrix in the same shape of AAA containing all elements of (cid:101) AAA in the diagonal blocks.
Let HHH be a matrix that collects the remaining off-diagonal elements.
Therefore (cid:101) AAA =
˚ AAA + HHH , which can be rewritten as
[Equation]
For all off-diagonal elements h i,j in HHH
[Equation]
i
[Equation]
Since there are n 2 − (cid:80) ¯ K ¯ k =1 n 2¯ k entries outside of the diagonal blocks, the norm of HHH can be bounded by
[Equation]
Each element in the diagonal blocks of AAA − ˚ AAA is non-negative.
Also, supposing xxx i and xxx j are from subclass ¯ k , we have
[Equation]
(cid:113)(cid:80)
[Equation]
(cid:113)(cid:80)
[Equation]
: xxx
[Equation]
∈ C
[Equation]
k
[Equation]
xxx t xxx
[Equation]
− w xxx
i xxx
[Equation]
xxx
[Equation]
xxx
[Equation]
xxx
[Equation]
by which the norm of AAA − ˚ AAA is bounded
[Equation]
Combining equations 20, 21 and 22 completes the proof.
AAA has the following eigendecomposition
[Equation]
, where ΣΣΣ I collects the ¯ K largest eigenvalues λ 1 , . .
. , λ ¯ K on the diagonal and ΣΣΣ N collects the remaining λ ¯ K +1 , . .
, λ ¯ n .
VVV I and VVV N collects the corresponding ¯ K and n − ¯ K eigenvectors, respectively.
Let (cid:101) AAA has analogous decomposition
[Equation]
with eigenvalues (cid:101) λ 1 , . . .
, (cid:101)
[Table]
B.1. Perturbation in Eigenvalues and Eigenvectors
The following two lemmas bound the changes in eigenvalues, eigenvectos and the alignment between labels and eigenvectors caused by ξ .
Lemma B.2.
We have the following bound for eigenvalues of (cid:101) AAA :
[Equation]
Proof.
From Lemma A.2 and Corollary A.6 we know that
[Equation]
By Weyl’s inequality on perturbation, we have
[Equation]
The right-hand-side is ≤ ∥ EEE ∥ F and therefore O ( ¯ K 5 / 2 ξ ) by lemma B.1.
Combining the preceding with equationcompletes the proof.
Lemma B.3.
The norm of the projection of YYY onto the range of (cid:101) VVV
I is bounded from below, i.e.,
[Equation]
Proof.
By Lemma A.2 and Corollary A.5 we have
[Equation]
By Wedin’s Theorem (Wedin, 1972; Stewart, 1990), we have the following bound on the principle angle between the range of (cid:101) VVV I and the range of VVV I as long as 1 ≥ O ( √ δ + ¯ K 5 / 2 ξ )
[Equation]
TI ( (cid:101) VVV I (cid:101) VVV T
I − I )
[Equation]
F
[Equation]
(cid:18)
∥ EEE ∥ F λ ¯ K − λ ¯ K + (cid:19) ≤ O (cid:18)
O
( ¯ K 5 / 2 ξ ) 1 − O ( √ δ ) (cid:19)
=
O ( ¯ K 5 / 2 ξ ) .
[Equation]
B.2. Error under Gaussian Noise (Proof for Theorem 3.4)
Considering ξ , rewrite bias 2 as
[Equation]
where (cid:101) b i =
[Equation]
λ
i
[Equation]
λ i + β .
Also, Lemma A.3 gives us the lower bound for ∥ VVV I VVV TI YYY ∥
[Equation]
Combining lemma B.2, lemma B.3, equation 24 and equation 25 yields the bound for the bias
[Equation]
We bound the variance in the same manner as in Section A.3 by applying Cauchy–Schwarz inequality, Corollary A.6 and
Lemma B. variance = σn
[Equation]
C. Contrastive Learning Slightly Improves the Alignment Between Jacobian Matrix and Ground-truth Labels
We compare the alignments between the clean label vector and the initial Jacobian matrix of (1) network pretrained using
SimCLR for 1000 epochs, (2) network pretrained using SimCLR for 100 epochs and (3) randomly initialized network.
yyy ∈ R nK is the vector obtained by flattening the label matrix YYY , i.e., concatenating the n rows of YYY .
Let zzz ( xxx i ,WWW )
∈
RRR K be the output of the network given example xxx i and parameters
WWW ∈ R d (we see the parameters of the network as a vector).
Then the Jacobian JJJ is defined as
[Equation]
Note that ∂zzz ( xxx i ,WWW )
WWW ∈ R d × K , therefore JJJ
( WWW ) ∈ R nK × d .
In table ??
Π I ( yyy ) is the projection of yyy onto the span of thesingular vectors of JJJ ( WWW 0 ) with larges singular values and Π N ( yyy ) is the projection of yyy onto the span of the remaining singular vectors.
Interestingly, pretraining for more epochs leads to larger Π
I ( yyy ) and smaller Π N ( yyy ) and therefore larger ∥ JJJJJJ T yyy ∥
F / ∥ JJJJJJ T ∥ F .
How much this slight improvement in the alignment contributes to the robustness deserves further investigation.
D. Experiments
We evaluate the effectiveness of contrastive learning in boosting the robustness of deep networks under various levels of label noise.
We first consider fine-tuning all layers of a network pre-trained with contrastive learning on noisy labels, and show that it can achieve a comparable generalization performance to the state-of-the-art robust methods.
Then, we show that the structure of the representation matrix obtained by contrastive learning can be leveraged by robust methods to achieve a superior generalization performance under extreme noise levels.
For our evaluation, we use artificially corrupted versions of CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009), as well as a subset of the real-world dataset Webvision (Li et al., 2017), which naturally contains noisy labels.
Our method was developed using PyTorch (Paszke et al., 2017).
We use 1 Nvidia A40 for all experiments.
Baselines .
We compare our results with many commonly used baselines for robust training against label noise: (1) Fcorrection (Patrini et al., 2017) is a two step process, where a neural network is first trained on noisily-labelled data, then retrained using a corrected loss function based on an estimation of the noise transition matrix.
(2) Decoupling (Malach & Shalev-Shwartz, 2017) is a meta-algorithm that trains two networks concurrently, only training on examples where the two networks disagree.
(3) Co-teaching (Han et al., 2018) also trains two networks simultaneously.
Each network selects subsets of clean data with high probability for the other network to train on.
(4) MentorNet (Jiang et al., 2018) uses two neural networks, a student and a mentor.
The mentor dynamically creates a curriculum based on the student, while the student trains on the curriculum provided by the mentor.
(5) D2L
(Ma et al., 2018) learns the training data distribution, then dynamically adapts the loss function based on the changes in dimensionality of subspaces during training.
(6) INCV (Chen et al., 2019) identifies random subsets of the training data with fewer noisy labels, then applies Co-teaching to iteratively train on subsets found with the most clean labels.
(7) T-Revision (Xia et al., 2019) learns the transition matrix efficiently using an algorithm that does not rely on known points with clean labels.
(8) L DMI (Xu et al., 2019) uses a novel information-theoretic loss function based on determinant based mutual information.
(9) ELR (Liu et al., 2020) uses semi-supervised learning techniques to regularize based on the early-learning phase of training, to ensure the noisy labels are not overfit.
(10) CRUST (Mirzasoleiman et al., 2020) dynamically selects subsets of clean data points by clustering in the gradient space.
(11) Mixup (Zhang et al., 2017) smooths the decision boundary by adding linear interpolations of feature vectors and their labels to the dataset.
D.1. Empirical results on artificially corrupted CIFAR
We first evaluate our method on CIFAR-10 and CIFAR-100, which each contain 50 , 000 training images, and 10 , 000 test images of size 32 × 32 × 3 .
CIFAR-10 and CIFAR-100 have 10 and 100 classes respectively.
We use the same testing protocol as (Xu et al., 2019; Liu et al., 2020; Xia et al., 2019), by evaluating our method on symmetric and asymmetric label noise.
For both CIFAR-10 and CIFAR-100, we use symmetric noise ratios of 0 . 2 , 0 . 5 , 0 . 8 , and an asymmetric noise ratio of 0 . 4 .
Effects of Contrastive Learning In our experiments, we first pre-train ResNet-32 (He et al., 2016) using SimCLR (Chen et al., 2020; SimCLR) for 1000 epochs using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 3 × 10 − 4 , a weight decay of 1 × 10 − 6 and a batch size of 128.
When pre-training, the last linear layer of ResNet-32 is replaced with a 2-layer projection head with an output dimensionality of 64.
When pretraining is finished, we replace the projection head with a new, randomly initialized classification layer, and begin training normally.
We also report the results when ELR,
Mixup, and Crust are applied to fine-tune the pre-trained network.
For ELR, we use β = 0 .
7 for the temporal ensembling
[Table]
parameter, and λ = 3 for the regularization strength.
For mixup, we use a mixup strength of α = 1 .
For Crust, we choose a coreset ratio of 0 . 5 .
The results are shown in Table 2.
We note that SimCLR pretraining leads to an across the board improvement for Crust,
ELR, and Mixup.
For lower noise ratios, the improvement is marginal.
However, for extreme noise ratios, the improvement is more dramatic.
In particular, pre-training boosts the performance of Crust by up to 44.1%, ELR by up to 8.2%, and Mixup by up to 34.1% under 80% noise.
We also note that under 80% noise, SimCLR pretraining alone outperforms all methods without SimCLR pretraining on CIFAR-10 and CIFAR-100.
D.2. Empirical Study on WebVision
WebVision is large scale image dataset with noisy labels (Li et al., 2017).
It contains 2.4 million images crawled from
Google Images search and Flickr that share the same 1000 classes as the ImageNet dataset.
The noise ratio in classes varies from 0.5% to 88%, and the number of images per class varies from 300 to more than 10,000 (Fig. 4 in (Li et al., 2017) shows the noise distribution).
We follow the setting in (Jiang et al., 2018) and create a mini WebVision dataset that consists of the top 50 classes in the Google subset with 66,000 images.
We use both WebVision and ImageNet test sets for testing the performance of the model.
We train InceptionResNet-v2 (Szegedy et al., 2017) for 120 epochs with a starting learning rate of 0 .
02 , which we anneal by a factor of 0 .
01 at epochs 40 and 80.
We use the SGD optimizer with a weight decay of 1 × 10 − 3 , and a minibatch size of 32.
For Mixup and ELR, we use the same hyperparameters as CIFAR.
Table 1 shows the Top-1 and Top-5 accuracy of different methods evaluated on WebVision and ImageNet.
We see that for both ELR and Mixup, SimCLR pretraining leads to average improvements of 4 .
11% and 3 .
20% for mini Webvision and ImageNet respectively.
Furthermore, we note that SimCLR pretraining on its own outperforms every method without
SimCLR pretraining, except ELR and Crust.
D.3. Training Only the Linear Layer v.s. Training All Layers
Figure 3 compares the performance of fine-tuning only the last linear layer (i.e., with the encoder frozen) and fine-tuning all layers (i.e., with the encoder unfrozen).
For both CIFAR-10 and CIFAR-100 we first pretrain Res-Net 32 using SimCLR for 1000 epochs using the Adam optimizer with a learning rate of 3 × 10 − 4 , a weight decay of 1 × 10 − 6 and a batch size of 128.
Then, for both frozen and unfrozen fine-tunings, we use the SGD optimizer with a learning rate of 5 × 10 − 3 , a weight decay of 1 × 10 − 3 , a batch size of 64.
Interestingly, in most cases training all layers achieves a higher test accuracy, which, however, does not mean it is necessarily better under some other hyperparameter settings.
Also, we note that training all
[Figure]
layers is more likely to overfit, especially under large noise level (column 3 in figure 3)
[Figure]