C HAOS IS A L ADDER : A N EW T HEORETICAL U NDERSTANDING OF C ONTRASTIVE L EARNING VIA A UGMENTATION O VERLAP
A BSTRACT
Recently, contrastive learning has risen to be a promising approach for large-scale self-supervised learning.
However, theoretical understanding of how it works is still unclear.
In this paper, we propose a new guarantee on the downstream performance without resorting to the conditional independence assumption that is widely adopted in previous work but hardly holds in practice.
Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations, thus simply aligning the positive samples (augmented views of the same sample) could make contrastive learning cluster intra-class samples together.
Based on this augmentation over- lap perspective, theoretically, we obtain asymptotically closed bounds for downstream performance under weaker assumptions, and empirically, we propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
Our theory suggests an alternative understanding of contrastive learning: the role of aligning positive samples is more like a surrogate task than an ultimate goal, and the overlapped augmented views (i.e., the chaos) create a ladder for contrastive learning to gradually learn class-separated representations.
The code for computing ARC is available at https://github.com/zhangq327/ARC .
I NTRODUCTION
Contrastive Learning (CL) emerges to be a promising paradigm for learning data representations without labeled data (Oord et al., 2018; Hjelm et al., 2019).
Recently, it has achieved impressive results and gradually closed the gap between supervised and unsupervised learning, hopefully leading to a new era that resolves the hunger for labeled data in the deep learning ﬁeld (He et al., 2020; Chen et al., 2020b; Wang et al., 2021).
However, despite its intriguing empirical success, a theoretical understanding of how contrastive learning actually works in practice is still under-explored.
The general methodology of contrastive learning is quite simple, that is to maximize the similarity between augmented views of the same image ( a.k.a.
positive samples), and minimize the similarity between that of two random images ( a.k.a.
negative samples).
Intuitively, it is an instance discrim- ination task (differing each image from others) instead of a classiﬁcation task (clustering images from the same class together and differing with other classes).
Nevertheless, as shown in Figure 1(a), CL representations are also class-separated.
Therefore, understanding how the pretraining task (CL) and the downstream task (classiﬁcation) interact plays a central role in both theoretical understandings and practical designings of contrastive methods.
Previously, Saunshi et al.
and Lee et al.
(2020) have tried to establish guarantees on the classiﬁcation performance for self-supervised representations.
However, their analysis relies heavily on the assumption that the two positive samples, as augmented views of the same image, are (nearly) conditionally independent on the class y .
However, this is hardly practical as the augmented views are still strongly input-dependent (see Figure 1(b)).
In fact, if the conditional independence is satisﬁed, the unsupervised task will become as informative as the supervised task, making this discussion
[Figure]
almost unnecessary.
This motivates us to ﬁnd more practical and weaker assumptions to understand how contrastive learning actually works (even without conditional independence).
To achieve this, we need to re-examine the contrastive learning process.
Previously, Wang & Isola (2020) show that CL objective involves two goals: alignment (for positive samples) and uniformity (for negative samples).
Nevertheless, we show that there exist bad cases where the features could still have poor performance even with perfect alignment and uniformity.
Thus, contradictory to the common belief of contrastive learning as learning invariance, we note that invariance alone is inadequate to learning useful representations for downstream tasks.
In this paper, we provide a novel understanding of contrastive learning that requires only practical and minimal assumptions, while also guarantee class-separated representations.
Our core insight hinges on the observation that contrastive learning usually adopts much more aggressive data augmentations than that in supervised learning (He et al., 2020; Chen et al., 2020a).
As shown in Figure 1(b), we notice that aggressive random cropping of two images can generate views that are very much alike that we could even hardly tell them apart, e.g., the wheels of two different cars.
In other words, there will be support overlap between different intra-class images through aggressively augmented views of them, a phenomenon we call augmentation overlap .
Thus, the alignment of positive samples will also cluster all the intra-class samples together, and lead to class-separated representations.
From our perspective, the role of data augmentation is to create a certain degree of “chaos” between intra-class samples, and the role of contrastive loss is to “climb the ladder of chaos”, i.e., the process that we gradually cluster intra-class samples by aligning positive samples.
Following this intuition, we develop a new theory for understanding the effectiveness of contrastive learning from the perspective of augmentation overlap.
Speciﬁcally, we derive the upper and lower bounds for its downstream performance and show how the two bounds will asymptotically converge with our assumptions on augmentation overlap.
Driven by this analysis, we further discuss how varying augmentation will affect the performance of contrastive learning from both synthetic and real-world datasets, and show that the results align well with our theory.
In summary,
• We characterize the failure of the previous analysis of contrastive learning, and develop a new understanding through the augmentation overlap effect.
Compared to existing theories on contrastive learning, ours can provide guidance to the practical designing of contrastive methods and evaluation metrics.
• We establish general guarantees (both upper and lower bounds) for the downstream performance without assumptions on conditional independence.
And we further show how the two bounds could asymptotically converge under our less restrictive assumptions.
• We provide a quantitative discussion on the effect of augmentation strength, which veriﬁes our theory from both theoretical and empirical aspects.
Motivated by our theory, we further propose a new unsupervised evaluation metric for contrastive learning named ARC and show that it aligns well with downstream performance on real-world datasets.
R ELATED W ORK
Contrastive Learning in Practice.
Contrastive self-supervised learning originates from a mutual information perspective of representation learning (Oord et al., 2018;
Hjelm et al., 2019), and soon becomes a general learning paradigm that contrasts between positive and negative pairs (He et al., 2020; Chen et al., 2020a).
It is rapidly closing the performance gap between unsupervised and supervised learning on large-scale dataset like ImageNet (Chen et al., 2021), and outperforms supervised learning when combined with a few ( e.g., 10%) labels (Chen et al., 2020b).
Several recent works show that similar performance could be achieved without negative samples by adopting certain training techniques (Grill et al., 2020;
Chen & He, 2020).
Understanding Contrastive Learning Objectives.
Both the original InfoNCE loss (Oord et al., 2018) and its InfoMax variants (Hjelm et al., 2019; Poole et al., 2019) are designed as variational estimates of the mutual information between inputs and representations, but these estimators are shown to have poor bias-variance trade-offs (Song & Ermon, 2020).
Instead, Wang & Isola (2020) simply understand contrastive learning through the two terms in the InfoNCE loss: alignment of positive samples and uniformity of negative samples.
However, as we show later, this perspective is also insufﬁcient to explain the effectiveness of contrastive learning, and we should take the interplay between augmentation and alignment into consideration.
Understanding Downstream Generalization.
Saunshi et al.
(2019) propose the ﬁrst theoretical guarantees by bridging the contrastive and classiﬁcation objectives.
Lee et al.
(2020) further link the reconstruction-based objective to the downstream objective.
However, both Saunshi et al.
and Lee et al.
(2020) rely on the unrealistic assumption that the positive samples are (nearly) conditionally independent.
Huang et al.
(2021) establish bounds by assuming a very small intra-class support diameter, which is also not practical.
Besides, some also explore the information-theoretical perspectives for analyzing contrastive learning (Tian et al., 2020; Tsai et al., 2021; Tosh et al., 2020; 2021), though their mutual information assumptions are hard to verify.
Recently, similar to our analysis, HaoChen et al.
(2021) also study the augmentation graph and establish guarantees in terms of graph connectivity.
Our work differs to theirs mainly in three aspects: 1) our analysis is applicable for the widely adopted InfoNCE and CE losses, while theirs is developed for their own spectral loss; 2) ours starts from the alignment and uniformity perspective while theirs starts from the matrix decomposition perspective; 3) our theory is empirically veriﬁed and inspires a useful evaluation metric for data augmentation, while their analysis focusing on minimizing the decomposition error is farther from the practical designing of positive and negative samples.
In a nutshell, compared to previous discussions, our theory has a closer connection to the actual contrastive learning process, and we verify the feasibility of each assumption with empirical evidence.
L IMITATIONS OF P REVIOUS U NDERSTANDINGS
We begin by introducing the basic notations and common practice of contrastive learning in the image classiﬁcation task.
In general, it has two stages, unsupervised pretraining, and supervised ﬁnetuning.
In the ﬁrst stage, with N unlabeled samples D u
= { x i } Ni =1 , we pretrain an encoder mapping from the d -dimensonal input space to a unit hypersphere f ∈ F : R d → S m − 1 in the mdimensional space.
In the second stage, we evaluate the learned representations z with the labeled data D l = { ( x i , y i ) } where labels y i ∈ { 1 , . . .
, K } .
Speciﬁcally, we ﬁx the encoder and learn a linear classiﬁcation head g : R m → R K on top from ˜ D
l = { ( z, y ) |
z = f ( x ) ∈ R m } .
Contrastive Pretraining .
Taking a training example x ∈ D u , we draw its positive sample x + = t ( x ) by applying a random data augmentation t ∼ T , and draw M randomly augmented samples { x − i } Mi =1 from D u as its negative samples.
Then, we can learn the encoder f
with the widely used InfoNCE loss (Oord et al., 2018)
[Equation]
Let p ( x ) be the data distribution, p ( x, x + ) be the joint distribution of positive pairs, and we simply assume p ( x, x + )
=
p ( x + , x ) and p ( x ) = (cid:82) p ( x, x + ) dx + , ∀ x ∈ R d following Wang & Isola (2020).
Linear Evaluation .
To evaluate the learned representations by contrastive learning, we usually adopt the Cross Entropy (CE) loss (Chen et al., 2020a) for a labeled pair ( x
, y ) ∈
D l
[Equation]
with a linear classiﬁer g ( z ) =
Wz where W =
[ w 1 , w 2 , . .
. , w K ] .
3. E XISTING T HEORETICAL A SSUMPTIONS AND T HEIR L IMITATIONS
As discussed above, there are some previous understandings on how contrastive learning yields good performance, and they mainly differ by their theoretical assumptions.
[Figure]
First, Wang & Isola (2020) interpret the ﬁrst and second terms of the InfoNCE loss (Eq. 1) as they are aiming at the following two properties: 1) alignment (the nominator): positive samples x, x + has similar features, i.e., f ( x ) ≈ f
( x + ) ; 2) uniformity (the denominator): features are roughly uniformly distributed in the unit hypersphere S m − 1 .
In particular, they show that
InfoNCE can be minimized with 1) perfect alignment and 2) perfect uniformity.
However, as we illustrate in Figure 2, the features could still have very poor downstream performance in the ﬁnite sample scenario.
This issue can be described rigorously by the following proposition.
Proposition 3.1 (Class-uniform Features Also Minimize the
InfoNCE Loss) .
For N training examples of K classes, consider the case when features { f ( x i ) } N i =1 are randomly distributed in S m − 1 with maximal uniformity (i.e., , minimizing the 2nd term of Eq. 1) while also satisfying ∀ x i , x + i ∼ p ( x, x + ) , f ( x i ) =
f ( x + i ) .
Because we have these two properties, the InfoNCE loss achieves its minimum.
However, the downstream classiﬁcation accuracy is at most 1 /K + ε and ε is nearly zero when N is large enough.
Proofs can be found in Appendix A.
This proposition indicates that the instance discrimination task (alignment + uniformity) alone cannot guarantee the learning of class-discriminative features as desired in the downsteam classiﬁcation.
Instead, Saunshi et al.
and Lee et al.
both establish the relationship between pretraining and classiﬁcation objectives and provide guarantees for the downstream performance.
In fact, the two works both assume the conditional independence of the two positive samples, i.e., p ( x, x + | y )
=
p ( x | y ) p ( x + | y ) .
However, this assumption is too strong as it is hardly practical.
As shown in Figure 1(b), augmented views from the same class are not actually independent as views from the same sample are more alike than that from other samples.
N EW A UGMENTATION O VERLAP T HEORY FOR C ONTRASTIVE L EARNING
The analysis above motivates us to ﬁnd a minimal and practical assumption: 1) it is enough to guarantee good performance on downstream tasks; 2) it is less restrictive than the i.i.d.
assumptions as in Saunshi et al.
and Lee et al.
4. G AP B ETWEEN C ONTRASTIVE L EARNING AND D OWNSTREAM C LASSIFICATION
We start with an assumption on the label consistency between positive samples, that is, any pair of positive samples ( x, x + ) should belong to the same class.
Assumption 4.1 (Label Consistency) .
∀ x, x + ∼ p ( x, x + ) , we assume the labels are deterministic (one-hot) and consistent: p
( y | x ) =
p ( y | x + ) .
This is a natural and minimal assumption that is likely to hold in practice.
As shown in Figure 1(b), the widely adopted augmentations in contrastive learning (Chen et al., 2020a) like images cropping, color distortion, and horizontal ﬂipping will hardly alter the belonging image classes.
With this minimal assumption, we can characterize the generalization gap between unsupervised and supervised learning risks.
We ﬁrst introduce the mean CE loss, L µ CE ( f ) =
E p ( x
[Equation]
y )
[Equation]
µ i ) (cid:21) , where we use the classwise mean representation µ k =
E p ( x | y = k ) [ f ( x )] as the weight w k of the classiﬁer g .
It is easy to see that the mean CE loss upper bounds the CE loss, i.e., L µ CE ( f ) ≥ min g L CE ( f, g ) and Saunshi et al.
(2019) showed that the mean classiﬁer could achieve comparable performance to learned weights.
Then, we have the following upper and lower bounds on the downstream risk (measured by mean CE loss).
[Figure]
Theorem 4.2 (Guarantees for General Encoders) .
If Assumption 4.1 holds, then, for any f ∈ F , its downstream classiﬁcation risk L µ CE ( f ) can be bounded by the contrastive learning risk L NCE ( f )
[Equation]
[Equation]
where log( M/K ) is a constant * , Var( f ( x ) | y )
=
E
[Equation] [Equation]
f ( x ) − E
[Equation]
x | y
[Equation]
f ( x ) (cid:107) 2 (cid:3) denotes the conditional (intra-class) feature variance, and O (cid:0) M − 1 / 2 (cid:1) denotes the order of the approximation error by using M negative samples.
Notably, our generalization bounds above improve over previous ones in the following aspects:
1) we do not require the conditional independence assumption as in Saunshi et al.
(2019); 2) we directly analyze the widely adopted InfoNCE loss (for contrastive learning) and CE loss (for supervised ﬁnetuning), while Saunshi et al.
(2019) are restricted to hinge and logistic objectives that have worse performance in practice (Chen et al., 2020a); 3) the class collision error terms introduced in Saunshi et al.
(2019) (due to the existence of same-class samples in the negative samples) now disappear in our bounds by adopting the InfoNCE loss, which also helps understand why InfoNCE performs better in practice; and 4) the bounds in Saunshi et al.
(2019) will become looser with more negative samples, which is contradictory to the common practice (Chen et al., 2020a).
While in our bounds, a larger M indeed has a lower approximation error and helps close the generalization gap.
In fact, several recent works have also been devoted to resolve the last “large- M ” problem (Ash et al., 2021; Merad et al., 2020).
Nevertheless, their analysis also requires the conditional independence assumption as in Saunshi et al.
(2019), while we show this problem can be resolved even without conditional independence.
Nozawa & Sato (2021) also establish bounds for the InfoNCE loss, but their bounds have incompressible class collision terms while ours do not.
Nevertheless, an important message of the theorem above is that Assumption 4.1 alone is still insufﬁcient to guarantee good downstream performance.
As there are intra-class variance terms in the upper and lower bounds, when they are large enough, contrastive learning might still have inferior performance as shown in Proposition 3.1.
Although the variance terms can be easily eliminated with the canonical conditional independence assumption, discussions in Section 1 have already demonstrated its impracticality.
In the next part, we will present a new understanding of how contrastive learning could control this variance term in practice.
4. C LOSING THE G AP WITH I NTRA - CLASS C ONNECTIVITY
The theorem above motivates us to study how contrastive learning could effectively control its intraclass variance and learn class-separated features.
Here, we propose a new understanding of this clustering ability through a dissection of the augmented views.
In particular, we notice that although samples are different from each other, applying aggressive augmentations like that in SimCLR (Chen * log( M/K ) could be absorbed in the loss et al., 2020a) can largely make them more alike.
For example, in Figure 1(b), two different cars become very similar when they are both cropped to the wheels.
Then, with contrastive learning, the two cars will have closer representations as they share a common view of the wheels.
In other words, two different intra-class samples could be aligned together if they have overlapped augmented views.
If all intra-class samples could be bridged by data augmentations, we can successfully cluster the whole class together.
Below, we formalize the intuition above with the language of graphs.
Notations.
A graph G is represented by a tuple G =
( V , E )
where
V =
( v 1 , v 2 , . .
. , v N ) is a set of vertices and E
⊆
V × V is a set of edges.
A path is a sequence of edges that joins a sequence of vertices, e.g., v i 1 − v i 2 − · · · − v i k .
We say that two vertices v and u are connected if G contains a path from v to u .
A graph is said to be connected if every pair of vertices in the graph is connected.
Two graphs are said to be disjoint if any pair of inter-graph vertices are not connected.
To begin with, we deﬁne the concept of T -connectivity of sample pairs, which describes whether two samples could be connected via the augmentation overlap of their augmented views.
Deﬁnition 4.3 ( T -connectivity) .
Given a collection of augmentations T = { t | t : R d → R d } , we say that two different images x i , x j ∈ R d are T -connected if they have overlapped views: supp( p ( x + i | x i ))
(cid:84) supp( p ( x + j | x j ))
(cid:54) = ∅ , or equivalently, ∃ t i , t j ∈ T such that t i ( x i ) =
t j ( x j ) .
Then, we can deﬁne an augmentation graph of all training samples in terms of their T -connectivity.
Deﬁnition 4.4 (Augmentation Graph) .
Given a set of N samples D = { x i } Ni =1 and an augmentation set T = { t | t : R d → R d } , we can deﬁne an augmentation graph G ( D , T ) =
( V , E ) as
• we take the N natural samples as the vertices of the graph, i.e., V = { x i } Ni =1 ; • there exists an edge e ij between two vertices x i and x j if they are T -connected.
Based on these concepts, we introduce the following assumption that with a proper choice of data augmentations, all intra-class samples could form a connected graph, as depicted in Figure 3(a).
Assumption 4.5 (Intra-class Connectivity) .
Given a training set D u , there exists an appropriate augmentation set T such that the augmentation graph G ( D u , T ) is class-wise connected, i.e., ∀ k ∈ { 1 , . .
, K } , the subgraph G k (graph G restricted to vertices in class k ) is connected.
Comparing to Saunshi et al.
and Lee et al.
(2020) that require (nearly) conditional independence p ( x, x + | y )
= p ( x | y ) p ( x + | y ) , ours only requires the connectivity of intra-class samples as in
Figure 1(b), and does not need them to be conditionally independent.
To make this analysis technically simpler, we make another assumption that we can align positive samples perfectly by minimizing the InfoNCE loss.
In practice, the alignment loss can typically be minimized up to a small error ε , and we have appended a more involved discussion of this weak alignment scenario in Appendix B.
For now, we focus on the simpliﬁed perfect alignment scenario.
Assumption 4.6 (Perfect Alignment) .
At the minimizer f (cid:63) of the InfoNCE loss, we can achieve perfect alignment, i.e., ∀ x, x + ∼ p
( x, x + ) , f (cid:63) ( x ) =
f (cid:63) ( x + ) .
Proposition 4.7.
Under Assumptions 4.5 & 4.6, by minimizing the InfoNCE loss we can conclude that the conditional variance terms vanish at the minimizer
[Equation]
[Equation]
Intuitively, for samples in each class k , if the corresponding subgraph G k is connected, there exists a path connecting every intra-class pairs ( x i , x j ) , as shown in Figure 3(a).
Consequently, aligning the positive pairs will also align all samples on the path, and eventually align x i and x j .
In this way, all intra-class samples can be clustered together and the intra-class variance shrinks to zero (under
Assumption 4.6).
Besides, because proper data augmentation will not cause inter-class augmentation overlap (Assumption 4.1), inter-class samples can be well separated with the uniformity term.
As a result, we can attain alignment of intra-class samples while maximizing the uniformity of inter-class samples.
According to Theorem 4.2, we will have an asymptotically closed generalization gap (with more negative samples M → ∞ ) for the encoder that minimizes the contrastive loss.
Theorem 4.8 (Guarantees for the Optimal Encoder) .
If Assumption 4.1, 4.5 & 4.6 hold and f is
L -smooth, then, for the minimizer f (cid:63) =
arg min
L NCE ( f ) , its classiﬁcation risk can be upper and lower bounded by its contrastive risk as
[Equation]
[Figure]
acc
[Figure]
acc =
[Figure]
acc
= 1.
acc
= 0.
We note that different to previous bounds that hold for any f ∈ F as in Theorem 4.2, our results here only stand for the minimizer of the contrastive loss f (cid:63) .
This indicates that the InfoNCE loss alone cannot simply guarantee good downstream performance, and the learning dynamics matters for the contrastive learning to learn useful features.
4. R ETHINKING THE R OLE OF D ATA A UGMENTATIONS
Our analysis above suggests a new understanding of the role of data augmentations in contrastive learning.
Conventionally, the success of contrastive learning is usually attributed to learning invariance w.r.t.
various data augmentations by matching positive examples.
However, as shown in
Proposition 3.1, matching positive pairs alone is theoretically inadequate to learn useful features.
Indeed, assuming that an ideal encoder that possesses invariance a priori does exist, like invariance to translation (CNNs), rotation (Cheng et al., 2016), and scaling (Xu et al., 2014), do we obtain class-discriminative features simply by random initialization?
Still NO, since these low-level properties are independent of high-level class information that we want to learn.
Thus, the reason why contrastive learning works cannot simply be attributed to the invariance learning principle.
We instead believe that the role of data augmentation is to create a certain degree of “chaos” between different intra-class samples (Figure 1(b)) such that they become more alike (or formally, T -connected).
In this way, the chaos serves as a “ladder” for bridging intra-class samples together when labels are absent, and the mission of the contrastive loss is to “climb this ladder”, that is, aligning intra-class samples by aligning the overlapped positive samples, as shown in Figure 3(a).
Therefore, from our perspective, instance discrimination by contrastive learning is actually a surrogate for the classiﬁcation task, and the surrogate can complete its misson when the ladder of chaos is complete (or formally, when intra-class connectivity holds).
Q UANTIFYING THE I NFLUENCE OF A UGMENTATION S TRENGTH
We have shown that with appropriate augmentations, we can derive guarantees on downstream performance.
However, in practice, as illustrated in Figure 3(b), there could be cases where augmentations are either too weak (intra-class features cannot be clustered together as in Figure 2) or too strong (inter-class features will also collapse to the same point) and lead to sub-optimal results.
In this section, we further provide a quantitative analysis of how different strength of data augmentation will affect the ﬁnal performance, both theoretically and empirically.
5. C HARACTERIZATION ON R ANDOM A UGMENTATION G RAPH
In practice, there are various data augmentation types that are hard to be described precisely.
For the ease of analysis, we consider a simple case where for each class k , there are N samples uniformly distributed around the cluster center c k on a hypersphere S d .
We then augment each sample x i with random samples in a hyper-disk of radius r on the hypersphere.
In Appendix D, we provide theoretical analysis on how different augmentation strength (measured by r ) will affect the connectivity of the augmentation as a function of the number of samples N , the position of the cluster centers c k and input dimensions d .
In particular, the minimal r for the graph to be connected decreases as N increases, so large-scale datasets can bring better connectivity.
Meanwhile, the required r also increases as d increases, so we need more samples or stronger augmentations for large-size inputs.
Here, we show our simulation results by applying contrastive
[Figure]
  
$XJ6WUHQJWK     7 H V W $ FF X U D F\     $ & $FF$&
[Figure]
s
[Figure]

  (SRFK  
   7 H V W $ FF X U D
F\  
  $ &5  $FF$&
[Figure]
   (SRFK 
[Figure]
H V W $
[Figure]
learning to the problem above.
From Figure 4, we can see that when r =
(no augmentation), the features are mixed together and hardly (linearly) separable, which corresponds to the under-overlap case in Figure 3(b).
As we increase r from 0 to 0 .
1 , the features become more and more discriminative.
And when r is too large ( r = 1 . 5 ), the inter-class features become mixed and inseparable again (over-overlap).
In Appendix C.2, we provide visualization results of the augmentation graphs, which also align well with our analysis.
Overall, our theoretical and empirical discussions verify our theory that intra-class augmentation overlap with a proper amount of data augmentation is crucial for contrastive learning to work well.
5. N EW S URROGATE M ETRICS FOR A UGMENTATION O VERLAP
From our theory and analysis above, we see that the augmentation overlap between intra-class samples indeed matters from contrastive learning to generalize better.
Inspired by this, we propose the
Confusion Ratio metric as a measure of the degree of augmentation overlap.
Speciﬁcally, for an unlabeled dataset D u with N samples, we randomly augment each raw sample x i ∈ D u for C times, and get an augmented set (cid:101)
D u
= { x ij , i ∈
[ N ] , j ∈ [ C ] } .
Then, for each x ip ∈ (cid:101) D u that is an augmented view of x i ∈ D u , denoting its k -nearest neighbors in (cid:101) D u in the feature space of f as N k ( x ip , f ) and other augmented views from the same image as C ( x ip ) =
{ x ij , j (cid:54) = p } , we can deﬁne its Confusion Ratio (CR) as the ratio of augmented views from different raw samples in its k -nearest neighbors,
[Equation]
We also deﬁne its average as Average Confusion Ratio (ACR):
[Equation]
When augmentation overlap happens, the nearest neighbors could be augmented views from a different sample, leading to a higher ACR.
Thus, ACR measures the degree of augmentation overlap, and a higher ACR indicates a higher degree of augmentation overlap.
Here we take k = 1 by default.
Here, to measure the augmentation strength in real-world datasets, following the common practice (Chen et al., 2020a), we adopt the RandomResizedCrop operator with scale range [ a, b ] for data augmentation, and we deﬁne its strength of augmentation as r = (1 − b ) + (1 − a ) (a comparison with other kinds of augmentations, e.g., color jittering, can be found in Appendix C.1).
As shown in
Figure 5(a), ACR (augmentation overlap) indeed increases with the strength of data augmentations, and only a moderate ACR achieves the best accuracy, which is consistent with our theory discussed above.
Besides, we also plot the change of ACR along the training process in Figure 5(b) & 5(c).
We can notice that for weak augmentations, the initial ACR is low, and it rapidly decreases to zero and seldom changes while training, which leads to poor test accuracy.
Instead, with proper augmentations, the initial ACR is higher, and it gradually decreases to zero and obtains good accuracy.
This is also consistent with our theory that we need a certain amount of augmentation overlap for contrastive learning to work well.
At the beginning, this will lead to a higher ACR, but as training continues, better alignment (lower ACR) will help bring up the test accuracy.
Average Relative Confusion (ARC).
In the discussion above, we notice that ACR itself does not indicate the test accuracy, but the relative change of ACR before and after training can be used as such an indicator.
A large change of ACR means a large change of augmentation overlap, which indicates that the contrastive loss can actually cluster intra-class samples together through overlapped
[Figure] [Figure]
views.
Based on this observation, we propose Average Relative Confusion (ARC) as
[Equation]
a ratio calculated with the initial ACR of the initialized model f init and the ﬁnal ACR of the pre- trained model f ﬁnal .
A higher ARC indicates that the contrastive learning process faces a hard task (augmentation overlap) at the beginning (high initial ACR), while successfully clustering intra-class samples with good alignment of positive samples at the end (lo ﬁnal ACR).
Therefore, a higher ARC score should correspond to higher downstream accuracy.
As shown in Figure 6 & 7, as augmentations become stronger, ARC scores indeed align well with the change of downstream accuracy across 1) different datasets, 2) different contrastive methods, and 3) different choices of k .
This justiﬁes our understanding of contrastive learning through augmentation overlap.
Meanwhile, as the calculation of ARC only involves unsupervised data, it could serve as a good surrogate metric for evaluating contrastive learning without using labeled data.
Compared to previous evaluation methods like linear classiﬁcation (Eq. 2), our ARC metric is more preferable as 1) it is theoretically motivated; 2) it does not need labeled data; 3) it does not need to learn additional modules like linear classiﬁers or rotation tasks (Reed et al., 2021).
More experimental details can be found in Appendix E.
C ONCLUSION
In this paper, we have proposed a new understanding of contrastive learning through a revisiting of the role of data augmentations.
In particular, we notice the aggressive data augmentation applied in contrastive learning can signiﬁcantly increase the augmentation overlap between intra-class samples, and as a result, by aligning positive samples, we can also cluster inter-class samples together.
Based on this insight, we develop a new augmentation overlap theory that could guarantee good downstream performance without relying on conditional independence and obtain asymptotically closed gaps.
With this perspective, we also characterize how different augmentation strength affects downstream performance with both random graphs and real-world datasets.
Last but not least, we also develop a new surrogate metric for evaluating contrastive learning without labels and show that it aligns well with downstream performance.
Overall, we believe that we pave a new way for understanding contrastive learning with insights on the designing of contrastive methods and evaluation metrics.
A CKNOWLEDGEMENT
Yisen Wang is partially supported by the National Natural Science Foundation of China under
Grant 62006153, Project 2020BD006 supported by PKU-Baidu Fund, and Huawei Technologies
Inc.
Jiansheng Yang is supported by the National Science Foundation of China under Grant No.
Zhouchen Lin is supported by the NSF China (No. 61731018), NSFC Tianyuan
Fund for Mathematics (No. 12026606), Project 2020BD006 supported by PKU-Baidu Fund and
Qualcomm.
A O MITTED P ROOFS A. P ROOF OF P ROPOSITION 3.
Proposition A.1 (Class-uniform features also minimize the InfoNCE loss) .
For N training examples of K classes, consider the case when features { f ( x i ) } Ni =1 are randomly distributed in S m − 1 with maximal uniformity while also satisfying ∀ x i , x + i ∼ p ( x, x + ) , f ( x i ) =
f ( x + i ) .
Because we have perfect alignment and perfect uniformity, the InfoNCE loss achieves its minimum.
However, the downstream classiﬁcation accuracy is at most 1 /K + ε and ε is nearly zero when N is large enough.
Proof.
We only need to give a counterexample that satisfy the desired classiﬁcation accuracy.
We consider the case when there is no T -connectivity between any pair of samples from { x i } Ni =1 , which is easily achieved if we adopt a small enough data augmentation.
In this scenario, the perfect alignment of positive samples ( x i , x + i ) could have no effect on the other samples.
Therefore, when the features { f ( x i ) } Ni =1 are uniformly distributed in S m − 1 , according to the law of large number, for any measurable set U ∈ S m − 1 , when N is large enough, there will be almost equal size of features from each class in U .
Consequently, any classiﬁer g that classiﬁes U to class k will only have 1 /K accuracy asymptotically.
A. P ROOF OF T HEOREM 4.
We will prove the upper and lower bounds separately as follows.
A.
[Equation]
T
[Equation]
U PPER B OUND
We ﬁrst provide the upper bound of the approximation error of the following Monte Carlo estimate.
Lemma A.2.
For LSE := log E p ( z ) exp( f ( x ) (cid:62) g ( z )) , we denote its (biased) Monte Carlo estimate with M random samples
z i ∼ p
( z ) , i = 1 , . . .
, M as (cid:100) LSE M = log 1 M (cid:80) Mi =
[Equation]
Then the approximation error A ( M ) can be upper bounded in expectation as
[Equation]
We can see that the approximation error converges to zero in the order of 1 /M − 1 / 2 .
Proof.
First, we have
[Equation]
where the ﬁrst inequality follows the Intermediate Value Theorem and e (the natural number) is the upper bound of the absolute derivative of log between two points when | f ( x ) (cid:62) g ( z i ) | ≤ 1 .
And the second inequality follows the Berry-Esseen Theorem given the bounded support of exp( f ( x ) (cid:62) g ( z i )) as following: for i.i.d random variables
Y i with bounded support supp( Y ) ⊂
[ − α, α ] , zero mean and bounded variance σ 2 Y < α 2 , we have:
[Equation]
where the constant C α only depends on α .
Here, we set Y i = exp( f ( x ) (cid:62) g
( z i )) −
E p ( z i ) exp( f ( x ) (cid:62) g
( z i )) .
As | f ( x ) (cid:62) g ( z i ) | ≤ 1 , | Y i | ≤ 2 e .
Y i has zero mean and bounded variance (2 e ) 2 .
Theorem A.3.
For each f ∈ F , the mean CE loss can be upper bounded by the InfoNCE loss:
[Equation]
where Var( f ( x ) | y )
=
E p ( y ) (cid:2) E p ( x | y ) (cid:107) f
( x ) − E p ( x | y ) f ( x ) (cid:107) 2 (cid:3) denotes the conditional variance.
Proof.
Denote p ( x, x + , y ) as the joint distribution of the positive pairs x, x + and the label y .
Denote the M independently negative smaples as { x − i } Mi =1 .
According to Assumption 4.1, x + and x here has the same label y .
Denote µ y as the center of features of class y, y = 1 , . .
, K .
Then we have the following lower bounds of the InfoNCE loss,
[Equation]
A ( M )
[Equation]
log M = −
[Equation]
x
[Equation]
) f
[Equation] [Equation]
x
[Equation] [Equation]
E p
[Equation]
x ) log
[Equation]
p
[Equation] [Equation]
[Equation]
f ( x − ))
− A ( M )
[Equation]
log M = −
[Equation]
x
[Equation]
x
[Equation] [Equation]
[Equation] [Equation]
x ) log
[Equation]
p
[Equation]
y − )
[Equation]
p
[Equation]
−
[Equation]
− )
[Equation]
f
[Equation] [Equation]
f
[Equation]
x − ))
− A ( M ) + log M
[Equation]
) ≥ − E
[Equation]
x
[Equation]
x
[Equation] [Equation]
[Equation] [Equation]
f ( x
[Equation] [Equation]
E p (
[Equation]
) log E p ( y − )
[Equation] [Equation]
p
[Equation]
y
[Equation]
) (cid:2)
[Equation] [Equation]
[Equation]
f ( x − ) (cid:3) )
− A ( M )
[Equation]
log M =
[Equation] [Equation]
log
[Equation] [Equation]
) − A ( M )
[Equation]
log M = −
[Equation]
y
[Equation] [Equation]
y )]
[Equation]
E
[Equation]
) log E
[Equation]
y − )
[Equation] [Equation]
µ y − ) − A ( M )
[Equation]
log M
[Equation]
) ≥ − E
[Equation] [Equation]
[Equation]
µ
[Equation]
f ( x
[Equation]
) − µ y )
[Equation]
(cid:3)
[Equation]
E p ( x ) log E p ( y − ) exp
[Equation]
f ( x )
[Equation]
µ y − )
− A ( M ) + log M (
[Equation]
) ≥ − E p ( x,y )
f ( x ) (cid:62) µ y − (cid:113) E p ( x,y ) (cid:107) f
( x ) − µ y (cid:107) 2 + E p ( x ) log E p ( y − ) exp( f ( x ) (cid:62) µ y − )
− A ( M ) + log M = − E p ( x,y ) f ( x ) (cid:62) µ y − (cid:112) Var( f ( x ) | y )
[Equation]
E p ( x ) log 1 K
K
[Equation]
k = exp( f ( x ) (cid:62) µ k ) − A ( M )
[Equation]
log M = E p ( x,
[Equation]
) (cid:2)
[Equation]
f ( x ) (cid:62) µ y + log
K
[Equation]
exp
[Equation]
K )
=
L µ
[Equation]
f )
[Equation]
( f ( x )
[Equation]
y )
[Equation]
A ( M )
[Equation]
log( M/K ) , which is equivalent to our desired results.
In the proof above, (1) follows Lemma A.2; (2) follows the Jensen’s inequality for the convex function exp( · ) ; (3) follows from the fact that because f ( x ) ∈ S m −, we have
[Equation]
and (4) follows the Cauchy–Schwarz inequality and the fact that because p ( x, x + ) = p ( x + , x ) holds, x, x + have the same marginal distribution.
A.2. T HE L OWER B OUND
In this part, we further show a lower bound on the downstream performance.
Lemma A.4 (Budimir et al.
Corollary 3.5 (restated)) .
Let g : R m → R be a differentiable convex mapping and z ∈ R n .
Suppose that g is L - smooth with the constant L > 0 , i.e., ∀ x, y ∈ R m , (cid:107)∇ g
[Equation]
x )
[Equation]
g
[Equation]
y )
[Equation]
≤ L
[Equation]
x − y (cid:107) .
Then we have
[Equation]
where x ( j ) denotes the j -th dimension of x .
With the lemma above, we can derive the lower bound of the downstream performance.
Theorem A.5.
For any f ∈ F , we have
[Equation]
where Var( u ( x ) | y )
=
E p ( y ) (cid:2) E p ( x | y ) (cid:107) u ( x ) − E p ( x | y ) u ( x ) (cid:107) 2 (cid:3) denotes the conditional variance
[Equation]
Proof.
Similar to the proof of Theorem A.
[Equation]
we have
[Equation]
i
[Equation]
i )
[Equation] [Equation]
log
[Equation]
−
[Equation] [Equation]
[Equation]
i
[Equation]
i
[Equation]
y )
[Equation] [Equation]
[Equation] [Equation]
log K =
[Equation] [Equation]
M
[Equation]
which is our desired result.
In the proof, (1) we adopt a Monte Carlo estimate with M samples from p ( y ) and bound the approximation error with Lemma A.2; (2) follows the same deduction in Theorem A.3; (3) the ﬁrst term is derived following the Cauchy–Schwarz inequality for the alignment term.
As for the second term, we ﬁrst show that the convex function logsumexp is L -smooth as a function of f ( x − j ) in our scenario.
Because (cid:107) f
( X ) (cid:107) ≤ 1 , we have ∀ f ( x j 1 ) , f ( x j 2 ) ∈ R m , the following bound on the difference of their gradients holds
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
So here the logsumexp is L -smooth for L = 12 .
Then
[Equation]
we can apply the reversed Jensen’s inequality in Lemma A.4; (4) holds because
[Equation]
A. P ROOF OF P ROPOSITION 4.
Proposition A.6.
Under Assumptions 4.5, & 4.6, by minimizing the InfoNCE loss, we can conclude that the conditional variance term vanishes, i.e.,
Var( f ( x ) | y )
= 0 .
Proof.
Consider any T -connected sample x i , x j .
Accoding to the deﬁnition of T -connectivity, there exist t i , t j ∈ T such that t i ( x ) =
t j ( x ) .
When perfect alignment holds as in Assumption 4.6, we will have f ( x i ) =
f ( t i ( x i )) and f ( x j ) =
f
( t j ( x j )) .
Combining with t i ( x i ) =
t j ( x j ) , we have f ( x i ) =
f
( x j ) .
That is, any T -connected pair has the same representation.
Then, in the augmentation subgraph G k that is connected according to Assumption 4.5, there exists a path for any pair of samples ˆ x i , ˆ x j ∈ G k where any two adjacent samples are T -connected.
As a result, ˆ x i and ˆ x j will also have the same representation by applying T -connectivity recursively.
At last, all samples in G k will have the same representation and the intra-class variance vanishes.
A. P ROOF OF T HEOREM 4.
Theorem A.7 (Guarantees for the optimal encoder) .
If Assumption 4.1, 4.5 & 4.6 hold and f is
L -smooth, then, for the minimizer f (cid:63) =
[Equation]
) , its classiﬁcation risk can be upper and lower bounded by its contrastive risk as
[Equation]
Proof.
A direct combination of Theorem 4.2 and 4.7 will give us the above two-sided bounds.
B G ENERALIZED G UARANTEES UNDER W EAK A LIGNMENT
In Section 4.2, we have shown that with perfect alignment (Assumption 4.6), the variance terms in the bounds of Theorem 4.2 can be minimized to zero, and consequently, the upper and lower bounds can be asymptotically closed.
Nevertheless, in practice, due to the constraint of hypothesis class F and optimization algorithms, we typically cannot achieve the exact minimizer, i.e., a perfect degree of alignment.
This motivates us to consider a less restrictive setting, namely the ε -weak alignment assumption, where the alignment error could be as large as ε .
Deﬁnition B.1 (Weak Alignment) .
A mapping f satisﬁes ε -weak alignment if ∀ x, x + ∼ p ( x, x + ) , (cid:107) f
( x ) − f ( x + ) (cid:107) ≤ ε .
For any ε -weak alignment f ∈ F , we have the following bounds on its downstream risk.
Theorem B.2 (Guarantees under weak alignment) .
If Assumption 4.1, 4.5 hold, then ∀ f ∈ F satisfying ε -weak alignment, its classiﬁcation risk can be upper and lower bounded by its contrastive risk as
[Equation]
(17) where D denotes the maximal diameter of the intra-class augmentation graphs {G k , k = 1 , . .
, K } and m denotes the output dimension of the encoder f .
In this way, we extend the guarantees developed for optimal encoders (Theorem 4.8) to even nonminimizers f ∈ F as long as it could align the positive samples within error ε .
[Figure]
function of different augmentation strength (a) and different number of samples (b) on the synthetic data in Section 5.1.
Empirical Veriﬁcation.
Besides the alignment error, we could notice this relaxation also introduces the dependence on an additional parameter D , the maximal diameter of the intra-class augmentation graphs.
As shown in Figure 8, when the augmentation is very weak, the intra-class graph is not connected and the diameter is ∞ .
Then, by applying stronger augmentations, D will become smaller and smaller, and ﬁnally converge to 1 (fully connected).
Besides, increasing the number of samples, ranging from 50 to 10 , 000 , does not have a large impact on D in practice.
Given these facts, we could reasonably assume that D is bounded and has a relatively small value with properly chosen augmentations.
As a result, with a bounded diameter D , a small alignment error ε will guarantee a small generalization gap between the upstream and downstream tasks.
This generalizes Theorem 4.8 by quantifying the generalization gap under weak alignment.
Proof.
Consider any pair of samples ( x, x (cid:48) ) from the same class y , and the positive sample of x as x + .
As intra-class connectivity holds, x and x (cid:48) are connected, and the maximal length of the path from x to x (cid:48) is D .
Therefore, under the ε -weak alignment that
[Equation]
[Equation]
we can bound the representation distance between x and x (cid:48) by the triangular inequality
[Equation]
With the inequality above, we can bound the variance terms in Theorem 4.2.
In particular, the conditional variance can be bounded as
[Equation]
where (1) follows Eq. 19.
At last
[Equation]
we can bound the variance items in Theorem 4.2 with Eq.
20, arrive at the desired bounds
[Equation]
which conclude our proof
[Equation]
C A DDITIONAL E MPIRICAL E VIDENCE C. F URTHER E VALUATION OF ARC M ETRIC
In the main text, we study the effect of different strength of RandomResizedCrop on the downstream accuracy as our proposed metrics (ACR and ARC), which help verify our theory.
Nevertheless, in practice, the augmentations adopted in contrastive learning is composed of a list of different kinds of augmentations.
Therefore, in this part, we further study the effect of other types of data augmentations, and we show that our ARC metric is also effective for evaluating not only other kinds of data augmentations, but also their composed ones.
Comparing different kinds of augmentations.
We begin by comparing the four kinds of data augmentations adopted in SimCLR (Chen et al., 2020a): RandomResizedCrop, ColorJitter, Grayscale, etc.
For a fair comparison, we apply each one alone for contrastive learning, and evaluate both the downstream accuracy and ARC.
From Figure 11, we can conclude that among the six kinds of augmentations, RandomResizedCrop is the most important augmentation, and ColorJitter is the second.
The rest of them are less powerful, as they cannot even learn useful features by themselves.
We can also see that our ARC metric aligns well with the downstream accuracy for different kinds of augmentations.
Comparing ColorJitter with different strength.
Based on the observation above, as we have discussed RandomResizedCrop in Section 5.2, we now choose ColorJitter, the second important augmentation, as another kind of augmentation for the study of different augmentation strength.
Speciﬁcally, we study the four parameters of brightness, contrast, saturation, and hue, where a large value corresponds a large degree of augmentation.
Note that we also adopt the default augmentations in SimCLR while only changing the parameters of ColorJitter (different to the setup in Figure 9).
As shown in Figure 10, there is also a reverse-U curve like that in RandomResizedCrop, and
the
[Figure]
sweet spot is usually achieved with 0.8, which corresponds to the default of choice in SimCLR (which is selected with exhausted hyperparameter search).
Meanwhile, our ARC metric still aligns well with the downstream accuracy for different strength of different kinds of color jittering, which demonstrates its wide applicability.
Comparing composed augmentations.
In the above discussion, we focus on the effect of a single kind of augmentations.
Here, we show that our ARC metric is still effective for evaluating the composition of different augmentations.
Notably, it is hard to deﬁne a metric of augmentation strength in this case, as the effect of different augmentations could be nested.
Nevertheless, we can still draw a “ACC - log(ARC)” plot to show the correlation between the downstream accuracy (ACC) and our ARC metric, where each point denotes a model trained with randomly selected parameters of RandomResizedCrop and ColorJitter.
As shown in Figure 11, we can see there is indeed a strong correlation between the two metrics, with a Pearson correlation coefﬁcient ρ = 0 .
Therefore, our metric can be used for selecting different kinds of augmentations as well as their compositions in an unsupervised fashion.
C. V ISUALIZATION OF A UGMENTATION G RAPH
For a more intuitive and practical understanding of our augmentation overlap theory developed in
Section 4, we visualize of the augmentation graphs on both synthetic data (Section 5.1) and realworld data (Section 5.2).
[Figure]
Synthetic data.
Following the setting of experiments in Section 5.1, we construct the adjacent matrix of different samples, calculate its connected components, and visualize it in Figure 12 with different colors.
It shows that when there is no augmentation, i.e., r = 0 , each sample is a connected component alone, and the number of connected components is the same as the number of samples N .
As we increase the augmentation strength, samples will be connected together through the augmented views.
In particular, when r = 0 .
1 , the whole intra-class samples are connected while inter-class samples are separated, which exactly satisfy our assumptions on intra-class connectivity and label consistency, respectively.
Therefore, this is the perfect overlap as desired, and indeed, as shown in Figure 5.1, contrastive learning on it obtains 100% test accuracy.
When we keep increasing the augmentation strength to be as large as 1 . 5 , inter-class samples also become connected and inseparable, leading to a random guess in test accuracy (50%).
This shows that the relationship between the augmentation graph and the downstream performance aligns well with our augmentation overlap theory.
[Figure]
Real-world data.
For the ease of analysis, our augmentation overlap theory adopts a simpliﬁed scenario by assuming label consistency (Assumption 4.1) and intra-class connectivity (Assumption 4.5), and we have veriﬁed their feasibility on the synthetic data.
In comparison, these assumptions cannot hold exactly on real-world data as the chosen augmentations could be sub-optimal.
Nevertheless, as shown in the augmentation graphs of CIFAR-10 (Figure 13), our assumptions could still approximately hold: with a properly chosen augmentation strength, the inter-class connections will be much less frequent than intra-class connections: 96.4% edges are intra-class edges.
Further considering the continuity property and extrapolation ability of deep neural networks, these approximate conditions could still achieve close performance to the optimal performance guaranteed under exact conditions.
Besides, we also have similar conclusions for the under-overlap and over-overlap scenarios: 1) the lack of enough augmentations produces only a few edges in the augmentation graph, as a result, even though all edges are intra-class edges, the downstream performance is still poor (25% test accuracy); 2) too strong augmentations instead produce too many inter-class edges (87.6%), which laso leads to poor downstream accuracy (29%).
This highlights that our assumptions on label consistency and intra-class connectivity are indeed effective guidelines for the designing of contrastive methods.
D T HEORETICAL C HARACTERIZATION OF A UGMENTATION S TRENGTH
Following the setting in Section 5.1, we can take the radius r as a notation of augmentation strength, and analyze its effect on the connectivity of the corresponding augmentation graph.
Theorem D.1.
For N random samples taken from a class, while gradually increasing the augmentation strength r , we have the following results.
(a) Under-overlap.
When 0 ≤ r ≤ r 1 =
[( d
[Equation]
where r 1 is the minimal distance between N samples, all samples (vertices) in the aug- mentation graph will be isolated.
As a result, the learned features could be totally random as in Proposition 3.1.
Instead, if r ≥ r 1 , there are at least two intra-class samples are T -connected and enjoy the same representation.
(b) Perfect overlap.
When r ≥ r 2 = [( d/ 2)!
[Equation]
where r 2 is the maximal distance between N samples, all samples in the augmentation graph will be T -connected.
As a result, the classwise connectivity in Assumption 4.5 will be guaranteed.
(c) Over-overlap.
When 0 ≤ r < r 3 = 12 min i,j (cid:107) c i − c j (cid:107) − 1 , where r 3 is the (asymptotic) minimal distance between samples from different classes, the label consistency is guaranteed.
Otherwise, when the augmentation is too large, e.g., r > r 3 , there will be inter-class augmentation overlap and Assumption 4.1 not longer holds.
In the theorem above, we show that the proper augmentation strength is a function of the number of samples N and the input dimensions d .
In particular, for each x , as N increases, there will be more natural examples and we only need a smaller r to obtain an overlap sample.
Instead, as d increases, due to the curse of dimensionality, there will be less samples within the same distance, thus it requires a larger r .
Nevertheless, we actually only need the augmentation sub-graph G k to be connected, instead of being fully connected as in Theorem D.1 (b).
While the connectivity is hard to analyze in the ﬁnite sample scenario ( N < ∞ ) , we have the following asymptotic property as N → ∞ .
Theorem D.2.
For N uniformly distributed samples deﬁned in R d as above, we denote the minimal augmentation strength needed for connectivity as a function of N : c N = inf { r > 0 :
G ( r ) k is connected } , and the minimal augmentation strength needed for avoiding isolated points as a function of N : d N = inf { r > 0 : every vertex at least has a neighbour } .
V u is the volume of unit hyperball.
Then we have the following asymptotic result:
[Equation]
From the theorem we can see that c dN decreases in the order of Θ
[Equation]
(cid:19) as N → ∞ .
First, this result is aligned with the empirical ﬁnding that self-supervised learning can beneﬁt more from large scale dataset (Chen et al., 2020b).
Second, it also indicate a curse of dimensionality that the required augmentation strength is exponentially large.
D. P ROOF OF T HEOREM D.
Proof.
From deﬁnition and notation in section 4.
We can construct an augmentation Graph G ( D , T ) given N random samples.
We deﬁne D k as the distance from a random point to its k-th nearest neighbour.
Percus & Martin (1998) discuss D k in random grpah and give the estimation of that:
[Equation]
where d is the dimension of hypersphere and N is the number of random points.
When r < D 1 there is no edge in the graph.
So the class is separated.
When r > D N − 1 , any pair of vertexes have an edge between them,so the graph is full connected.
D
[Equation]
P ROOF OF T HEOREM D.
Proof.
Denote
[Equation]
With Theorem 1.1 from Penrose (1999) and features are uniformly distributed in the surface of unit hypersphere, V u denotes to the volume of unit hypershpere
[Equation]
∃ N 0 when N > N 0 , and augmentation strength is larger than ( 2( d − 1) S log N 2 N
[Equation]
d )
[Equation]
+
[Equation]
1 , the graph is connected,i.e the class is overlapped.
Then we want specify the case the class will be depart begin with some concepts in graph theory.
The largest nearest-neighbor link: For a give edge distance x and for each i= 1,...,n,let
[Equation]
[Equation]
to be the degree of the vertex U j in the random graph G N ( V, E, r i ) , and let
[Equation]
26) be the minimum vertex degree.
Deﬁne the largest nearest-neighbor link, the smallest edge distance for which each vertex has at least one neighbor
[Equation]
With Theorem 1.2 from Penrose (1999),
[Equation]
28) ∃ N (cid:48)
when N > N (cid:48) 0 , and augmentation strength is less than ( 2( d − 1)
S log N 2 N 2 V u d ) 1 d − (cid:15) 2 , there will be at least 1 isolated point which is not connected to any other point,i.e the class is departed.
Thus ∃ N= max( N 0 , N (cid:48) 0 ) ,when N >
N 1 , if augmentation strength is larger than ( 2( d − 1) S log N 2 N 2 V u d ) 1 d + (cid:15) 1 , the graph is connected, if augmentation strength is less than ( 2( d − 1) S log N 2 N 2 V u d ) 1 d − (cid:15) 2 , there will be at least 1 isolated point.
E A DDITIONAL E XPERIMENTAL D ETAILS E. S IMULATION ON R ANDOM A UGMENTATION G RAPH
Following our setting in Section 5.1, we consider a binary classiﬁcation task with InfoNCE loss.
We generate data from two uniform distribution on a unit ball S 2 in the 3 -dimensional space.
One center is (0 , 0 , 1) and another is (0 , 0 , − 1) .
The area of both parts are 1.
We take 5000 samples as train set and 1000 samples as test set.
For the encoder class F , we use a single-hidden-layer neural network with softmax activation, and we use InfoNCE loss to optimize it.
E. E XPERIMENTS ON R EAL - WORLD D ATASETS
To better understand and verify our theorem, we conduct experiments on real-world datasets, including CIFAR-10, CIFAR-100 and STL-10.
We use SimCLR (Chen et al., 2020a) and BYOL
Grill et al.
(2020) as our training framework and use ResNet18 as our network.
For CIFAR-10 and
CIFAR-100, we adopt C = 10 augmentations for each image, and search neural neighbors in the entire augmented dataset.
For STL-10, we adopt C = 6 due to its relatively large size.