E XPLORING G ENERALIZATION OF N ONC ONTRASTIVE SELF - SUPERVISED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive learning have recently produced results comparable to the state-ofthe-art supervised models.
Non-contrastive methods do not use negative samples, but separate samples of different classes by explicitly or implicitly optimizing the representation space.
Although we have some understanding of the core of the non-contrastive learning method, theoretical analysis of its generalization performance is still missing.
Thus we present a theoretical analysis of generalizability of non-contrastive models.
We focus on the inter-class distance, show how noncontrastive methods increase the inter-class distance and how the distance affects the generalization performance of the model.
We find that the generalization of non-contrastive methods is affected by the output dimension and the number of latent classes.
Models with much fewer dimensions than the number of latent classes are not sufficient to generalize well.
We demonstrate our findings through experiments on the CIFAR dataset
I NTRODUCTION
Self-Supervised Learning (SSL) is gaining popularity as a result of its competitive performance comparing to supervised learning, while is free of costly labels.
Among SSL models, contrastive learning has attracted great attention due to its strong generalization performance and wide range of applications.
Contrastive learning generates multiple augmented views from samples, and treats the views generated from the same sample with the same label, whereas the views from different samples have different labels.
Contrastive models have been widely used in various of scenarios including but not limited to graph representation (You et al., 2020; Zhu et al., 2020), computer vision (Chuang et al., 2020; Tian et al., 2020a) and natural language processing (Saeed et al., 2021;
Iter et al., 2020; Chi et al., 2020).
Typical contrastive models like MoCo (He et al.,
2020)and SimCLR (Chen et al., 2020) use InfoNCE loss to attract positive pairs and push negative pairs.
However a negative sample of contrastive learning can be false negative (Huynh et al., 2022).
Views of two samples with the same latent will be treated as negative pairs.
This can hurt the performance of the model.
There are many works trying to avoid this disadvantage (Huynh et al., 2022; Kalantidis et al., 2020; Kim et al., 2021).
BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2020) uses an asymmetric network structure, cancel the use of negative samples, and surprisingly achieves comparable performance.
Tian et al.
; Jing et al.
(2021) analyse the dynamics of features and networks in BYOL and SimSiam and explore why they perform well without falling into trivial solutions.
However, studies that quantitatively investigate their generalization performance have yet to emerge.
In this paper, we explore the generalizability of non-contrastive learning.
Under appropriate assumptions, we prove that the cross-correlation matrix of output features constrains the upper bound of generalization error rate.
The closer the matrix is to the identity matrix, the smaller this upper bound becomes.
Moreover, we demonstrate that the cross-correlation matrix is optimized by non-contrastive methods during training.
The sections of this paper are arranged as follow:
In Section 2 we review contrastive learning methods and the theoretical analysis of contrastive learning.
Then in Section 3 we formulate the generalization problem and agree on some notation.
In Section 4 the error rate theorem is proposed.
It illustrates how the cross-correlation matrix and the dimensionality of the representation space can limit the upper bound on the error rate.
Then we shows that SimSiam (Chen & He, 2020) and Barlow
Twins (Zbontar et al., 2021) optimize the cross-correlation matrix during training.
At last in Section 5 we conduct experiments on CIFAR-10 and CIFAR-100 to verify the effect of feature dimension and number of classes on error rate in Section 4.
R ELATED W ORKS 2. C ONTRASTIVE S ELF -S UPERVISED L EARNING
Self-supervised learning methods strive to learn representations using well-designed pretext tasks that do not require the use of expensive labels.
As a kind of self-supervised learning method, contrastive learning uses data augmentation to generate multiple augmented samples (views) from an original sample, and treats the augmented samples generated by the same sample with the same label, whereas the augmented samples from different samples have different labels.
Recent methods such as MoCo (He et al.,
2020)and SimCLR (Chen et al., 2020) have produced results comparable to the state-of-the-art supervised method on ImageNet (Deng et al., 2009) dataset.
They use the InfoNCE loss (Van den Oord et al., 2018) to pull positive sample pair together while pushing away negative samples, i.e., those augmented views from different samples.
However views from different sample can have the same latent label.
Treating them as negative samples will hurts model performance.
To overcome this shortcoming, Hu et al.
(2021); Xiao et al.
(2020) carefully design the negative sampling process.
Some works like BYOL (Grill et al., 2020), SimSiam (Chen & He, 2020) and Barlow Twins (Zbontar et al., 2021) directly created models that do not use negative samples at all.
These contrastive models free of negative samples are also called non-contrastive models.
2. T HEORETICAL A NALYSIS FOR C ONTRASTIVE L EARNING
Despite the fact that contrastive learning has shown to be effective in learning usable representations and has outperformed supervised learning in key downstream transfer learning benchmarks (He et al., 2020), their underlying mechanism remain opaque and poorly understood.
Arora et al.
(2019) provides a theoretical analysis on the explanation why the learned features via contrastive learning are useful for downstream tasks.
Tian et al.
; Jing et al.
(2021) give theoretical analysis of dynamics of non-contrastive methods and investigate how these models avoid trivial solutions.
Huang et al.
(2021) studies the generalization property of SimCLR and Barlow Twins, however, methods like BYOL with special optimization procedures are not considered.
Tosh et al.
(2021) study the generalization performance in the aspect of the mutual information.
Tian et al.
(2020b) propose the optimal views for contrastive learning that reserve relevant task information to ensure the mutual information among all views are task-relevant.
Research above explored the learning mechanism of contrastive learning and proposed a number of methods to modify the generalization performance of the model using loss function, however, most of which are based on InfoNCE object model with both positive and negative samples.
Noncontrastive method with no negative samples can not be completely explained by theorems based on mutual information, and the analysis on its generalization performance also remains open.
Our study focus on this open problem, exploring the factors that impact on self-supervised learning generalization performance, revealing how these factors affect the generalization error rate, and give out the upper bound of the non-contrastive model error rate.
P ROBLEM F ORMULATION
In this section we formulate the loss of contrastive methods and the generalization error rate in preparation for the analysis in the next section.
Contrastive learning treats views of the same sample as having the same label.
Given an underlying distribution D =
C 1 ∪ C 2 ∪ · · · ∪ C K .
C i , i ∈
[ K ] are disjoint latent classes.
Each sample x is independently and identically distributed (i.i.d.)
according to D and belongs to a unique C i .
An augmentation A generate different views ˆ x of x .
We denote A ( x ) and A ( S ) as the set of distorted views of x and sample set S =
( x 1 , . . .
, x m ) ∼ D .
Typical models (Chen et al., 2020;
He et al., 2020) learn an effective encoder f through pulling augmented views of the same sample together
[Figure]
and push views of all others away.
They use the InfoNCE loss (Van den Oord et al., 2018):
[Equation]
where ˆ x − ∈ A ( S ) are views from negative samples excluding x .
The architecture of SimCLR (Chen et al., 2020) who use L InfoNCE is shown in Fig. 1(a).
However, since we do not know the latent class of samples during training, some false negative samples will inevitably appear (Huynh et al., 2022).
InfoNCE loss will push one view away from its false negative samples, that is, the negative samples that actually have the same latent label, which will have a negative impact on the model.
Non-contrastive learning models are contrastive methods without using negative samples.
They either explicitly forces the cross-correlation matrix between positive samples close to the identity matrix (Barlow Twins (Zbontar et al., 2021), as shown in Fig. 1(b)), or makes use of asymmetric networks (BYOL (Grill et al., 2020), SimSiam (Chen & He, 2020), as shown in Fig.
1(c))
in order to avoid trivial constant features.
The loss function of the former is:
[Equation]
where d is the dimension of the encoder outputs,
[Equation]
E
[Equation]
x
[Equation]
x )
[Equation]
f
[Equation]
ˆ x 1 ) f
(ˆ x 2 ) T ) is the crosscorrelation matrix computed between the outputs of the two identical networks along the batch dimension and λ is a postive trade off parameter.
The latter is represented by SimSiam, whose loss function is:
[Equation]
where sim ( x , y )
=
x T y ∥ x ∥∥ y ∥ is the cosine similarity between ( x , y ) , h is the predictor layer, often a multi-layer perceptron.
sg is the stop-gradient operator, which means the backpropagation does not go through this path.
As for the generalization, given an encoder f , we consider a nearest neighbor classifier G : D
→ [ K ] :
[Equation]
where µ k :=
E x ∼C k , ˆ x 1 ∈A ( x ) [ f (ˆ x 1 )] is the mean of output features for samples in latent class C k .
The classifier maps each x ∼ D to a class C i .
Here we denote C ( x ) as the real latent class of x ∼ D , the generalization error rate can be formulated as
[Equation]
−1.0 −0.
−1.
−0.
−1.0 −0.
(a
[Figure]
Distributions minimizing L cor −1.0 −0.
−1.
−0.
−1.0 −0.
[Figure]
The generalization properties of L InfoNCE have been well studied (Huang et al., 2021).
In the following section we focus on Barlow Twins and SimSiam.
Exploring the relationship between their optimization mechanism and generalization error rate.
G ENERALIZATION A NALYSIS 4. E RROR R ATE T HEOREM
In this section, we study the generalization property of non-contrastive methods, and how the models improves generalization by optimizing the statistical characteristics of the output features.
Let d be the dimension of output features.
For simplicity, we assume ∥ f ( x ) ∥ 2 = r
=
d .
Which means features f ( x ) are scattered on a sphere with radius r .
We assume that views of the same latent class are close to each other and never transfer to another class, formally:
[Equation]
[Equation]
Note the we suppose the encoder f is Lipschitz continuous so that ˆ x 1 and ˆ x 2 close to each other infers f (ˆ x 1 ) and f (ˆ x 2 ) are close.
In the next section, we will show that Barlow Twins and SimSiam optimize the following norm:
[Equation]
rather than the Gaussian potential based uniformity(Wang & Isola, 2020):
[Equation]
where G t ( f (ˆ x 1 ) , f (ˆ x 2 )) =
exp( − t ∥
f
(ˆ x 1 ) − f (ˆ x 2 ) ∥ 22 ) is the Gaussian potential kernel (Cohn &
Kumar, 2007).
These two kinds of optimization lead to different distribution patterns.
The distribution generated by L cor ( f ) does not guarantee that each class of samples is separated from other classes.
So the generalization of the model depends on the randomness in training and the dimension of the entire space and the number of latent classes.
In this section, we first investigate when the model is guaranteed not to misclassify samples, and then derive an upper bound on the classification error rate in relation to L cor , d , and K .
As the assumption that views belong to the same latent class are close to each other, it is easy to think that as long as the inter-class distance between two classes is large enough, they will not be misclassified from each other.
So we have the following theorem of a pair of latent classes.
(See appendix for the proof)
Theorem 1 If µ Ti µ j < r 2 (1 − 4 ε + ε 2 ) , S i ∼ C
i , S j ∼ C j , then ∀
[Equation]
This theorem states that as long as µ i and µ j are far enough apart, views belong to class C i and C j will not be misclassified into each other’s classes.
This inspires us to study the constraints on the inter-class distance between latent classes.
Let Λ =
E x ∼D , ˆ x 1 ∈A ( x ) [ f (ˆ x 1 ) f
(ˆ x 1 ) T ] .
In the next section, we will show that non-contrastive losses
L BT and L SimSiam minimize L cor =
∥ Λ − I d ∥ 2 explicitly and implicitly, respectively.
Before that, here we prove that ∥ Λ − I d ∥ 2 can constrains µ Ti µ j and thus affects the error rate.
(See appendix for the proof)
Theorem 2 Let p i =
E x ∼C k , ˆ x ∈A ( x ) [ 1 [ˆ x ∈A ( S i )] ] , then
[Equation]
This is a critical theorem of our study.
As we know from Theorem 1, a large distance between classes ensures that samples of different classes are not misclassified.
Theorem 2 gives a lower bound on the expectation of the inter-class distance, i.e. the upper bound on the cosine distance.
We can reduce the expectation in Theorem 2 by optimizing the upper bound, so that the probability of the inter-class distance is large enough increases, which can further improve the generalization error rate.
By Theorem 2, if we can minimize ∥ Λ − I d ∥ , the inter-class distances will be constrained.
If the expectation is constrained, there cannot be a large number of inter-class distances that exceed this limit.
Formally, we have the inequality for a non-negative random variable X :
[Equation] [Equation]
Combining the above two theorems, we can get the error rate theorem (See appendix for the proof) :
Theorem
The error rate satisfies:
[Equation]
Here as we know, ε is related to the strength of data augmentation.
The remaining part of the upper bound shows that, given the dataset and data augmentation, the upper bound of the error rate is related to the following formula:
[Equation]
Here we get that minimizing ∥ Λ − I d ∥ 2 = L cor can help reducing the error rate.
As for d , if it is too small, that error rate bound will be larger than 1 , thus the upper bound will become meaningless.
In the next section, we look into the loss function of contrastive learning models, and shows how they optimize L cor .
4. L OSS A NALYSIS
In this section we study the loss of Barlow Twins (Zbontar et al., 2021) and SimSiam (Chen & He, 2020), and explore how they guarantee their generalizability.
Specifically, we will demonstrate how they optimize
[Equation]
cor
[Equation]
For Barlow Twins, the loss is
[Equation]
Let λ = 1 , and by definition of C , it can be inferred that optimizing L BT leads to minimizing ∥ Λ − I d ∥ 2 (See appendix for the proof) :
[Equation]
Barlow Twins explicitly minimize the cross-correlation matrix loss, but SimSiam is more complicated:
[Equation]
Let z 1 , z 2 be the corresponding representations f
(ˆ x 1 ) , f (ˆ x 2 ) of augmented views ˆ x 1 , ˆ x 2 in Eqn.
We make the following assumption.
(1) A linear bias-free network serve as the projection layer.
That is p 1 =
h
( z 1 ) =
W p z 1 in Eqn.
(2) Representation z is updated via backpropagation.
Let g ( z 1 , z 2 ) =
[Equation]
Z
[Equation]
W p z 1 ∥ = sim ( p 1 , z 2 ) .
The loss function of SimSiam is
[Equation]
Because Cor ( z ) = zz T is symmetric, ∃ U ∈ R n × n , s.t.UU T
=
U T U
=
I d , U diagonalizes
Cor ( z ) .
That is
[Equation]
The column of U , i.e. u i , is an orthonormal basis of R n × n .
α z =
U T z is the coordinate of z ∈ S n −under U .
Then, we have the following:
[Equation]
[Equation]
Lemma
If A ∈ M n ( C ([ a, b ])) is Hermitian with the property that the eigenvalues of A ( t ) are distinct for all t ∈ [ a, b ] , then A is diagonalizable in M n ( C ([ a, b ]))
(Grove & Pedersen, 1984)
Here M n ( C ([ a, b ])) is the set of n × n matrices whose elements are continuous in C [ a, b ] .
A diagonalizable A in M n ( C ([ a, b ])) means that there is a unitary element U ∈ M n ( C [ a, b ]) , such that for each t ∈ [ a, b ] , U ( t ) ∗ A ( t ) U ( t ) is diagonal.
According to Lemma 1, the correlation matrix Cor ( z ) =
zz T is real-valued and symmetrical, and thus a Hermitian.
During training there exists continuous U ( t ) that diagonalizes Cor ( z ( t )) .
Moreover, because
Cor ( z ( t )) is a real-valued symmetric, U ( t ) must be real-valued too.
That is, we have a continuous basis { u i ( t ) } ni =1 fir the training step t
It also allow us to consider the local change of Λ by assuming that U is locally constant.
By denoting Λ p = U T W p U , we have the following:
Theorem
According to Lemma 1, the local dynamics of SimSiam is
[Equation]
where ˜Λ p =
[Equation]
p 1 · ˜ α p 1 , ϵ ≜ E [(˜ α z 2 − ˜ α z 1 )˜
α Tz 1 ] ≪ 1 , c ≜ z 2 T p 1 is close toand r 1 , r 2 are learning rate of W p , z 1 , respectively.
[Table]
Tian et al.
(2021) shows that during training, the eigenspaces of W p and Λ gradually aligns, that is, they can be diagonalized by the same matrix U .
Thus we can assume W p = diag ( λ 1 p , . . .
, λ dp ) are diagonal, and then derive the element-wise form :
[Equation]
[Equation]
There is a clear intuitive understanding of how uniformity is optimized with this local dynamics.
If λ i is close to each other, then the output representation distribution will be uniform.
We begin by examining the simplest version.
By definition, λ i ¿ 0 always holds.
Accordingly, there are three cases for Eqn.
18 and 19 ignoring ϵ i .
(1) When ˜ λ ip ≤ 0 , (1 − c ˜ λ ip ) λ
i ≥ 0 , dλ i ≤ 0 and dλ ip ≥ 0 .
(2) When 0 ≤ ˜ λ ip ≤ 1 c , (1 − c ˜ λ ip ) λ i ≥ 0 , dλ i ≥ 0 and dλ ip ≥ 0 .
(3) When ˜ λ ip ≥ 1 c , (1 − c ˜ λ ip ) λ i ≤ 0 , dλ i ≤ 0 and dλ ip ≤ 0 .
The equality holds only if ˜ λ ip = 0 , ˜ λ ip = 1 c or λ i = 0 .
In conclusion, if ˜ λ i p > 1 c or ˜ λ i p < 0 , λ i deceases and λ i p moves to boundary 0 or 1 c .
If ˜ λ i p ∈ (0 , 1 c ) or ˜ λ ip < 0 , λ i deceases and λ ip moves to boundary 0 or 1 c , resulting in λ i close to each other or λ i = 0 .
That is, some dimensions collapse and others tend to have equivalent λ i .
So in those dimensions that do not collapse, the model optimize ∥
Λ − I d ∥ .
Taking ϵ
i and weight decay into consideration, similar conclusion can be obtained.
(See Appendix for the proof)
E XPERIMENTS
In this section, we experimentally study the relationship between the upper bound of error rate and the actual generalization performance, indicating that the upper bound we obtained is a relatively tight bound, which has practical significance, and also explores the representation space dimension and the number of latent classes.
We conduct experiments on the CIFAR-10 and CIFAR-100 dataset (Krizhevsky et al., 2009) with classification as downstream tasks.
We choose two independent augmentations of the same image as positive pairs, following the standard practice.
We use the Barlow Twins model and simplified
0.8  bound
K NN A cc u r a c y   32128 256 512
[Figure]
0.4  bound
K NN A cc u r a c y 32
[Figure]
bound -KNN Acc fitting
[Figure]
[Figure] [Figure]
[Figure]
K NN A cc u r a c y 832 64
[Figure]
K NN A cc u r a c y  64
[Figure]
SimSiam model whose projector and predictor layer is set to be a linear network W p to conduct the experiment.
The simplified predictor layer maintains the optimization properties of the model while reducing the impact of additional factors such as batch normalization.
We use ResNet- (He et al., 2016) as the encoder, run Barlow Twins model for 800 epochs and Simsiam models for 600 epochs and record the k-nearest neighbors(KNN) accuracy and L bound in the minibatch.
The code is modified on the basis of https://github.com/facebookresearch/simsiam and https://github.com/IgorSusmelj/barlowtwins .
The results are recorded in Table 1.
Note that in Theorem 3, except the dataset property p
i ’s, the dimension d and number of classes K plays a key role.
If K is very large and d is relatively small, for example K = 100 and d = 4 , then K − d d 2 = 6 .
the upper bound
[Equation]
so that it could no longer give a meaningful bound of Er ( f ) ∈ [0 , 1] .
Let V ( d ) =
K − d d 2 achieves its minimum value at d = 2 K , lim d → 0 V ( d ) = ∞ and lim d →∞ V ( d ) = 0 .
It induces that while d > 2 K , the generalization performance does not hurt too much, but if d gets smaller, especially d < K , the model can no longer maintain its generalizability.
In Table 1,
CIFAR-10 and CIFAR-100 have 10 and 100 classes respectively, i.e. K = 10 , 100 respectively.
the minimum points are d = 20 , 200 .
When d > K for CIFAR-100, the KNN accuracy remains around 43% .
When d gets lower, the accuracy drops quickly.
As for CIFAR-10, a 32 − dim model performs as well as hundreds of dimensions.
The accuracy drops when d < 2 K = 20 , this matches our prediction.
In Fig.
3 and Fig.
4, we did scatter plots of knn accuracy and L bound on CIFAR-10 and CIFARdatasets.
The points in Fig.
3 correspond to the models in Table 1.
Although it is not strict, there is still a clear correlation between knn accuracy and L bound in the figure.
The smaller L bound is,
[Table] [Table]
[Table] [Table]
[Figure]
K NN
A cc u r a c y
[Figure]
the larger knn accuracy is.
Note that by Theorem 3, L bound upper bounds the generalization error rate.
If L bound becomes smaller, then the error rate will also become smaller, which means that the accuracy increases.
If this upper bound is too loose, its value will not be so significantly related to the knn accuracy.
So the theorem agrees with the experiment, the relationship between the upper bound and the knn accuracy is significant.
In Fig.
4, the linear relationship among the sample points is significant, and the closer the sample points are to the end of training (upper left), the closer they are to the fitted line.
This shows that
Theorem 3 can accurately estimate the generalization error, and the upper bound of the error rate we proposed can determine the actual error rate to some extent.
On the other hand, as training progresses proceeding, sample points determined by L bound and KNN accuracy move from right to left along the fitting line, L bound decreases and K-nearest neighbor accuracy increases (corresponding to a decrease in error rate).
The optimization process described in Section 4.2 for L bound is validated.
C ONCLUSION
In this paper we study the generalization of non-contrastive learning methods represented by Barlow Twins and SimSiam.
We give an upper bound on the generalization error rate to illustrate the effect of inter-class distance on generalization performance.
We demonstrate the relationship between the cross-correlation matrix of output features and inter-class distance.
Then we analyze how non-contrastive learning methods optimize correlation so that they are sufficient to produce good generalization.
Moreover, we find that the dimension of feature space and the number of latent classes also affect the generalizability.
We conduct an experiment on CIFAR-10 and CIFARdataset to verify our theoretical results
A PPENDIX A P ROOF OF T HEOREM
Proof 1 Without loss of generality, we only prove that if µ Ti µ j < r 2 (1 − 4 ε + ε 2 ) , S i ∼ C
i , ∀ ˆ x ∈
A ( S i ) , G (ˆ x ; f ) ̸
= j .
∀ k ∈ [ K ] , ∥ µ k ∥ =
∥
E ˆ x ∈A ( S k )
[ f ( x )]
∥ =
∥ f
( x 0 )
+ E ˆ x ∈A ( S k )
[ f ( x ) − f ( x 0 )]
∥ ≥ ∥
f ( x 0 ) ∥ − rε = r
(1 − ε ) (21) ∥
f (ˆ x ) − µ i ∥ 2 − ∥ f
(ˆ x ) − µ j ∥=2 f
(ˆ x ) T µ j − 2 f
(ˆ x ) T µ i + ∥ µ i ∥ 2 − ∥ µ j ∥=
( f (ˆ x ) T − µ Ti + µ Ti )( µ j − µ i ) + ∥ µ i ∥ 2 − ∥ µ j ∥=2( f (ˆ x ) T − µ Ti )( µ j − µ i ) + 2 µ Ti µ j − ∥ µ i ∥ 2 − ∥ µ j ∥≤ 2 rε ∥ µ j − µ i ∥ + 2 µ Ti µ j − 2 r 2 (1 − ε )≤ 2 µ Ti µ j + 4 r 2 ε − 2 r 2 (1 − ε )≤ 2 µ Ti µ j + 4 r 2 ε − 2 r 2 (1 − ε )<
That is, ∥ f
(ˆ x ) − µ i ∥ 2 < ∥ f
(ˆ x ) − µ j ∥ 2 , so we have for ˆ x ∈ A ( S i ) , G (ˆ x ; f ) ̸ =
j
B P ROOF OF T HEOREM
Proof
First we have:
E
[Equation] [Equation]
[Equation]
(ˆ x 1 ) ̸ = C
(ˆ x 2 )
[( µ T C (ˆ x 1 ) µ C (ˆ x 2 ) ) 2 ] =
[Equation]
As for ∥ Λ − I d ∥ 2 :
[Equation]
The last inequality is because for any A, B ,
[Equation]
Let A =
( √ p 1 µ 1 , . .
. , √ p K µ K ) , we have
[Equation]
Finally
[Equation]
it is obvious that:
[Equation]
C P ROOF OF T HEOREMProof
[Equation]
D P ROOF OF E QUATIONProof
[Equation]
E L OCAL DYNAMICS ANALYSIS OF S IMSIAM E. P ROVE OF LOCAL DYNAMICS ANALYSIS OF S IMSIAM
Proof
First we note three basic formula:
(1) Derivation of the unit vector
[Equation]
(2) Chain rule: ∀ vector x
[Equation]
Through the formulas above we have
[Equation]
Because p 1 = W p z
[Equation]
We have ∂g∂z 1 = W Tp ∂g∂p 1 and ∂g∂W p
=
∂g∂p i z T 1 , thus
[Equation] [Equation]
35) Λ p and α z i update via:
[Equation]
where ∆ = ˜
α z 2 − ˜ α
Tz 2 ˜ α p 1 · ˜ α p 1 and r 1 , r 2 are learning rate of W p , z 1 , respectively.
As for Λ , we need the dynamic of ˜ α
z 1 .
For any vector x , d ∥ x ∥ dx = x
∥ x ∥ , d x ∥ x ∥
[Equation]
=
∥ x ∥ 2 I − xx T
∥
x ∥. So
[Equation]
[Equation]
The last equality in Eqn.
40 is due to:
[Equation]
The function ∆˜ α Tz 1 =
(˜ α z 2 − ˜
α
Tz 2 ˜ α p 1 · ˜ α p 1 )˜
α Tz 1 is formally independent of ∥
z 1 ∥ and ∥ p 1 ∥ .
So we assume the distribution of ∆˜ α Tz 1 is independent of ∥
z 1 ∥ and ∥ p 1 ∥ .
So we have
[Equation]
For simplicity, we ignore E ∥ z 1 ∥ , ∥ p 1 ∥ [ · ] and denote ∥ z 1 ∥ =
∥
z ∥ , ∥ p 1 ∥ =
∥
p ∥ in the following.
It seems to be a strong assumption, but this can be regarded to analyze the dynamics of the part of points whose ∥ z ∥ and ∥ p ∥ are certain values.
Under previous assumption that ∆˜ α Tz 1 is independent of ∥ z 1 ∥ and ∥ p 1 ∥ , it is a natural simplification.
Eqn. 36 and 40 become:
[Equation] [Equation]
Both of them have the term E [∆˜ α Tz 1 ] , which is crucial for the dynamics.
We examine it closely:
[Equation] [Equation]
In the training phase, we pull ˜ z 2 and ˜ p 1 together.
˜
z
2 ˜ p 1 approaches 1 at the very beginning and remains close to 1 during the whole training phase.
So we assume that z T 2 p 1 =
c is close to 1.
For the term E [˜ α z 2 ˜ α Tz 1 ] , z 1 , z 2 are the representations of different views of the same image through the same model, We assume that ˜ α
z 2 = ˜
α
z 1 + ε and ∥ ε ∥ ≪ 1 is related to the intensity of the data augmentation.
Moreover, ˜ α p 1 = Λ p α z 1 ∥ p ∥ =
∥ z ∥ ∥ p ∥ Λ p ˜ α
z 1 .
Denoting ˜Λ p ≜
∥ z ∥
∥ p ∥ Λ p , ϵ ≜ E [ ε ˜ α
Tz 1 ] , we have
[Equation]
Substituting Eqn. 47 to Eqn. 19 and 18, we finally get the local dynamics of Λ p and Λ
E.
T AKING ϵ i AND WEIGHT DECAY INTO CONSIDERATION
In the preceding section, we present an intuitive description of how Λ and Λ p change, without addressing ϵ .
SimSiam use weight decay during training, which has an impact on the models’ performance.
In this section, we take them into consideration and come up with similar results.
We can obtain the similar local dynamics when there is weight decay η .
[Equation]
[Equation]
Proof 6 With weight decay η , similar to the previous section, we have
[Equation]
Note that
[Equation]
Then, by Eqn. 41, ˜ α z 1 ˜ α TzΛ Tp ∥
α z 1 ∥ 2 ∥
α p 1 ∥ ∆˜ α Tz 1 , so we have
[Equation]
Similar to the previous section
[Equation]
the local dynamics can be obtained:
[Equation]
Similarly assuming Λ p = diag ( λ 1 p , ..., λ np ) as in Sec 4.2.
Eqn. 48 and 49 can be obtained.
Note that ϵ
= E [ ε ˜ α
Tz 1 ] , ε = ˜
α
z 2 − ˜ α
z 1 is generated by the output of the model on the disturbed samples, and randomness operates on the samples rather than directly on ˜ α
z 1 .
However, the significance of each component varies depending on the model.
Thus, we may assume that ϵ i will be smaller than λ
i specifically ∃ e > 0 , | ϵ i λ i
| < e ≪ 1 .
New terms affect the boundaries of different phases, but the intuitive understanding remains unchanged.
We have
[Equation] [Equation]
[Equation]
Similarly, there are three cases for Eqn. 48 and 49:
(1) When ˜ λ ip ≤ 0 , Eqn. 56 holds and Eqn.
57 fails.
dλ i ≤ 0 and dλ ip ≥ 0 .
(2) When 0 ≤ ˜ λ ip ≤ 1 c , dλ ip > 0 ⇔ λ i > η ∥ p ∥ 2 ∥ z ∥ 2 / ( 1+ ϵiλi ˜ λ ip − c ) , which more likely holds when ˜ λ ip is near 0.
dλ i > 0 more likely holds when ˜ λ ip is not too close to the boundary 0 and 1+ ϵi λi c .
(3) When ˜ λ ip ≥ 1 c , both Eqn. 56 and 57 fail.
dλ i ≤ 0 and dλ ip ≤ 0 .
E. V ISUALIZATION OF THE CORRELATION MATRIX OF S IM S IAM REPRESENTATIONS
[Figure] [Figure]