H OW D OES S IM S IAM A VOID C OLLAPSE W ITHOUT N EGATIVE S AMPLES ? T OWARDS A U NIFIED U NDERSTANDING OF P ROGRESS IN SSL
Anonymous authors
Paper under double-blind review
A BSTRACT
Towards avoiding collapse in self-supervised learning (SSL), contrastive loss is widely used but often requires a large number of negative samples.
Without negative samples yet achieving competitive performance, a recent work (Chen & He, 2021) has attracted signiﬁcant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse.
However, the reason for its success remains not fully clear and our investigation starts by revisiting the explanatory claims in the SimSiam.
After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of l 2 normalized vector.
This yields a uniﬁed perspective on how negative samples and SimSiam predictor alleviate collapse.
Such a uniﬁed perspective comes timely for understanding the recent progress in SSL.
I NTRODUCTION
Beyond the success of NLP (Lan et al., 2020; Radford et al., 2019; Devlin et al., 2019;
Su et al., 2020; Nie et al., 2020), self-supervised learning (SSL) has also shown its potential in the ﬁeld of vision tasks (Li et al., 2021;
Chen et al., 2021; El-Nouby et al., 2021).
Without the ground-truth label, the core of most SSL methods lies in learning an encoder with augmentation invariant representation (Bachman et al., 2019;
He et al., 2020; Chen et al., 2020a; Caron et al., 2020; Grill et al., 2020).
Speciﬁcally, they often minimize the representation distance between two positive samples, i.e . two augmented views of the same image, based on a Siamese network architecture (Bromley et al., 1993).
It is widely known that for such Siamese networks there exists a degenerate solution, i.e . all outputs “collapsing” to an undesired constant (Chen et al., 2020a; Chen & He, 2021).
Early works have attributed the collapse to lacking a repulsive component in the optimization goal and adopted negative samples, i.e . views of different samples, to alleviate this problem.
Introducing momentum into the target encoder, BYOL shows that Siamese architectures can be trained with only positive pairs.
A technical report (Fetterman & Albrecht, 2020) has suggested the importance of batch normalization in BYOL for its success, however, a recent work (Richemond et al., 2020) refutes their claim by showing BYOL works without BN.
More recently, SimSiam (Chen & He, 2021) has caught great attention by further simplifying BYOL in terms of removing momentum encoder, which can be seen as a major milestone achievement in
SSL through providing a minimalist method for achieving competitive performance.
However, the reason for the success of Simsiam remains not fully clear with a core question summarized as:
How does SimSiam avoid collapse without negative samples?
Our work attempts to answer the above question and our investigation starts with revisiting the explanatory claims in the original SimSiam paper.
Notably, two components, i.e . stop gradient and predictor, are observed to be essential for the success of SimSiam (Chen & He, 2021).
The reason has been mainly attributed to the stop gradient (Chen & He, 2021) by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves like alternating between optimizing each set.
Chen & He argue that the predictor h is helpful in SimSiam because h ﬁlls the gap to approximate expectation over augmentations (EOA).
Unfortunately, the above explanatory claims are found to be ﬂawed due to reversing the two paths with and without gradient.
This motivates us to ﬁnd an alternative explanation, for which we in- troduce a simple yet intuitive framework for facilitating analyzing collapse in SSL.
Speciﬁcally, we propose to decompose a representation vector into center and residual components.
This decomposition facilitates understanding which gradient component is beneﬁcial for avoiding collapse.
Under this framework, we attempt to understand why a Naive Siamese architecture causes collapse, and provide a uniﬁed perspective on how negative samples help prevent such collapse as well as how predictor in SimSiam achieves it without negative samples.
For both InfoNCE with negative samples (He et al., 2019; Chen et al., 2020b;a; Tian et al., 2019; Khosla et al., 2020) and SimSiam with asymmetric predictor, their center gradient component behaves like de-centering for preventing collapse.
Interestingly, we also ﬁnd that for both, the residual vector component achieves de-correlation which somewhat surprisingly also prevents collapse.
Overall, our work demystiﬁes the success of InfoNCE and SimSiam, which bridges the gap between them and recent frameworks with explicit de-centering and de-correlation.
Our work contributes to a uniﬁed understanding of the recent progress of different frameworks in SSL.
Finally, our work points out an important detail on whether BN, which removes the mean center, is essential for preventing a collapse in BYOL or SimSiam.
Speciﬁcally, we reveal that BN alone prevents collapse only when BN is adopted immediately after the MSE loss.
Towards more explainable SimSiam our work investigates multiple new variants of predictors for preventing collapse.
We follow prior works to set the experiment, see Appendix A.1.
R EVISITING S IM S IAM AND ITS EXPLANATORY CLAIMS
l 2 normalized vector and optimization goal.
SSL trains an encoder f for learning discriminative representation and we denote such representation as a vector z , i.e . f
( x ) =
z where x is a certain input.
For the augmentation-invariant representation, a straightforward goal is to minimize the distance between the representations of two positive samples, i.e . augmented views of the same image, for which mean squared error (MSE) is a default choice.
To avoid scale ambiguity, the vectors are often l 2 normalized, i.e .
Z =
z / ||
z
||
(Chen & He, 2021), before calculating the MSE:
[Equation]
which shows the equivalence of normalized MSE loss to cosine loss (Grill et al., 2020).
Collapse in SSL and solution of SimSiam .
Based on a Siamese architecture, the loss in Eq.causes the collapse, i.e. f always outputs a constant regardless of the input.
We refer this Siamese architecture with loss Eq.
as Naive Siamese in the remainder of paper.
Towards alleviating this phenomenon, multiple works have investigated solutions, among them contrastive loss with negative samples is widely used (Chen et al., 2020a).
SimSiam solves the collapse problem via predictor and stop gradient, based on which the encoder is optimized with a symmetric loss:
[Equation]
where sg ( · ) is stop gradient and P is the output of predictor h , i.e .
p =
h ( z ) and P = p / || p
|| 2 .
2. R EVISING EXPLANATORY CLAIMS IN S IM S IAM
Interpreting stop gradient as AO.
Chen & He hypothesize that the stop gradient in Eq.
2 is an implementation of Alternating between the Optimization of two sub-problems, which is denoted as
AO.
Speciﬁcally, with the loss considered as L ( θ, η ) =
E x, T (cid:104)(cid:13)(cid:13)
F θ ( T ( x ))
− η x (cid:13)(cid:13) 2 2 (cid:105) , the optimization objective min θ,η L ( θ, η ) can be solved by η t ← arg min η L ( θ t , η ) and θ t +1 ← arg min θ L ( θ, η t ) alternatively.
It is acknowledged in (Chen & He, 2021) that their hypothesis does not fully explain why the collapse is prevented.
Nonetheless, they mainly attribute SimSiam success to the stop gradient with the interpretation that AO might make it difﬁcult to approach a constant for ∀ x .
Interpreting predictor as EOA.
Their AO problem (Chen & He, 2021) is formulated independent of predictor h , for which they believe that the usage of predictor h is related to approximating EOA for ﬁlling the gap of ignoring E T [ · ] in a sub-problem of AO.
The approximation of E T [ · ] is summarized in Appendix A.2.
Chen & He support their interpretation by proof-of-concept experiments.
Speciﬁcally, they show that updating η x with a moving-average η tx ← m ∗ η tx +(1 − m )
∗F θ t ( T (cid:48) ( x )) can help prevent collapse without predictor (see Fig 1 (b)).
Given that the training completely fails when the predictor and moving average are both removed, at ﬁrst sight, their reasoning seems valid.
2. D OES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA?
Reasoning ﬂaw.
Taking the stop gradient into account, we divide SimSiam into two sub-models with different paths and term them Gradient Path (GP) and Stop Gradient Path (SGP).
For SimSiam, only the sub-model with GP includes the predictor (see Fig.
1 (a)).
We point out that their reasoning ﬂaw of predictor analysis lies in the reverse of GP and SGP .
By default, the SGP is on the same side of moving-average sub-model, as shown in Fig.
1 (b).
Conceptually, Fig. 1 (b) is similar to Fig.(c) instead of Fig.
1 (a).
It is worth mentioning that the Mirror SimSiam in Fig.
1 (c) is what stop gradient in the original SimSiam avoids.
Therefore, it is problematic to perceive h as EOA.
[Figure]
New Figure
[Figure] [Figure]
Naive Siamese z a z b z a z b z a z b p a p b p a p b
SGP
[Table]
Explicit EOA does not prevent collapse.
(Chen & He, 2021) points out that “in practice, it would be unrealistic to actually compute the expectation E T [ · ] .
But it may be possible for a neural network (e.g., the preditor h ) to learn to predict the expectation, while the sampling of T is implicitly distributed across multiple epochs.”
If implicitly sampling across multiple epochs is beneﬁcial, explicitly sampling sufﬁcient large N augmentations in a batch with the latest model would be more beneﬁcial to approximate E T [ · ] .
However, Table 1 shows that the collapse still occurs and suggests the success of moving average should not be interpreted from the EOA perspective.
2. A SYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN S IM S IAM
Symmetric Predictor does not prevent collapse.
The difference between Naive Siamese and Simsiam lies in whether the gradient in backward propagation ﬂows through a predictor, however, we show that this propagation helps avoid collapse only when the predictor is not included in the SGP path.
With h being trained the same as Eq. 2, we optimize the encoder f through replacing the Z in Eq.
2 with P .
The results in Table.
2 show that it still leads to collapse.
Actually, this is well expected by perceiving h to be part of the new encoder F , i.e .
p =
F ( x ) =
h ( f ( x )) .
In other words, the symmetric architectures with and without predictor h both lead to collapse.
Predictor with stop gradient is asymmetric.
Clearly, how SimSiam avoids collapse lies in the asymmetric architecture, i.e . one path with predictor and the other without predictor.
Under this asymmetric architecture, the role of stop gradient is to only allow the path with predictor to be optimized with the encoder output as the target, not vice versa.
In other words, the SimSiam avoids collapse by excluding Mirror SimSiam (Fig. 1 (c)) which has a loss (mirror-like Eq 2) as L Mirror = − ( sg ( P a ) · Z b + sg ( P b ) · Z a ) , where the predictor h is still trained the same way as SimSiam.
[Table]
symmetric predictor (Fig. 2 (b)) leads to collapse, while SimSiam (Fig. 1 (a)) avoids collapse.
With the conceptual h − 1 , we interpret Fig.
1 (a) the same as Fig.
2 (c) which differs from Fig.
2 (b) via changing the optimization target from p b to z b , i.e . z
b = h − 1 ( p b ) .
This interpretation suggests that the collapse can be avoided by processing the optimization target with h − 1 .
By contrast, Fig. 1 (c) and Fig. 2 (a) both lead to collapse, suggesting that processing the optimization target with h is not beneﬁcial for preventing collapse.
Overall, asymmetry alone guarantee no collapse avoidance, and the optimization target needs to be processed by h − 1 not h .
[Figure]
Trainable inverse predictor and its implication on
EOA.
In the above, we propose a conceptual inverse predictor h − 1 in Fig.
2 (c), however, it remains yet unknown whether the model with h − 1 (Fig.
2 (c)) is experimentally trainable.
A detailed setup for this investigation is reported in Appendix A.5.
The results in Fig.
3 show that a learnable h − 1 leads to slightly inferior performance, which is expected because h −cannot make z ∗ b perfectly same as z b , where z ∗ b is the output of h − 1 .
Note that they would be perfectly equivalent if z ∗ b = z b .
Despite a slight performance drop, the results conﬁrm that h − 1 is trainable.
The fact h − 1 is trainable provides additional evidence that the role h plays in SimSiam is not EOA because theoretically h − 1 cannot restore a random augmentation T (cid:48) from an expectation p , where
[Equation] x )) (cid:105) .
V ECTOR DECOMPOSITION FRAMEWORK FOR UNDERSTANDING COLLAPSE
We introduce a gradient analysis framework based on vector decomposition to facilitate their investigation.
By default, SimCLR (Chen et al., 2020a) and SimSiam (Chen & He, 2021) both adopt lnormalization in their loss for avoiding scale ambiguity.
We treat the l 2 normalized vector, i.e . Z , as the encoder output, which signiﬁcantly simpliﬁes the gradient derivation and relevant analysis.
3. A NALYSIS FRAMEWORK FROM THE VECTOR DECOMPOSITION PERSPECTIVE
Vector decomposition.
For the purpose of analysis, we propose to decompose Z into two parts,
Z = o + r , where o , r denote center vector and residual vector respectively.
Speciﬁcally, the center vector o is deﬁned as an average of Z over the whole representation space o
z
=
E
[ Z ] , however, we approximate it with all vectors in current mini-batch, i.e . o z = 1 M (cid:80)
Mm =1 Z m , where
M is the mini-batch size.
We deﬁne the residual vector r as the residual part of Z , i.e .
r a =
Z a − o z .
Collapse from the vector perspective.
A Naive Siamese is well expected to collapse since the loss is designed to minimize the distance between positive samples, for which a constant constitutes an optimal solution to minimize such loss.
When such collapse occurs, ∀ i, Z i = 1 M (cid:80)
Mm =1 Z m =
o z , where i denotes a random sample index, which shows the constant vector is o z when collapse occurs.
This interpretation only suggests a possibility that a dominant o can be one of the viable solutions, while the optimization, such as SimSiam, might still lead to a non-collapse solution.
It merely describes o as the consequence of the collapse, and our work investigates the cause of such collapse through analyzing the inﬂuence of individual gradient components, i.e . o and z .
Competition between o and r .
Complementary to the Standard Deviation (Std) (Chen & He, 2021), we introduce the ratio of o in z , i.e . m o
=
||
o || /
||
z || , where || .
|| is the L 2 norm, for indicating collapse.
Similarly, the ratio of r in z is deﬁned as m r = ||
r || /
||
z
|| .
When collapse happens, i.e . all vectors Z are close to each other or o , m o approaches 1 and m r approaches 0, which is not desirable for SSL.
A desirable case would be a relatively small m o and a relatively large m r , suggesting a relatively small (large) contribution of o ( r ) in each Z .
We interpret the cause of collapse as a competition between o and r where o dominates over r , i.e .
m o (cid:29) m r .
For Eq 1, the derived negative gradient on Z a (ignoring Z b for simplicity due to symmetry) is shown as:
[Equation]
where the gradient component Z a is a dummy term because the loss − Z a · Z a = − 1 is a constant having zero gradient on the encoder f .
Conjecture1.
With Z a = o + r a , we conjecture that the gradient component of o is expected to update the encoder to boost the center vector thus increase m o , while the gradient component of r a is expected to behave in the opposite direction to increase m r thus decrease m o .
On the other hand, a random gradient component is expected to have a relatively small inﬂuence on the trend.
[Figure]
To verify the conjecture, we revisit the dummy gradient term Z a .
We design loss − Z a · sg ( o z ) and − Z a · sg ( Z a − o z ) to show the inﬂuence of gradient component o and r a , respectively.
The results in Fig.
4 (a) show that gradient component o z has the effect of increasing m o while decreasing m r .
On the contrary, r a helps increase m r while decreasing m o .
Overall, the results verify Conjecture1.
Revisit collapse in a symmetric architecture.
Based on Conjecture1, here, we provide an intuitive interpretation on why a symmetric
Siamese architecture, such as Fig.
2 (a) and (b), cannot be trained without collapse.
The gradient in Eq 3 can be interpreted as two equivalent forms, from which we choose Z b − Z a =
( o z + r b ) − ( o z + r a ) =
r b − r a .
Since r b comes from the same positive sample as r a , it is expected that r b also increases m r , however, this effect is expected to be smaller than that of r a , thus causing collapse.
Basic gradient and Extra gradient components.
E xtra G radient component, denoted as G e , needs to be introduced to break the symmetry for preventing collapse.
As the term suggests, G e is deﬁned as a gradient term relative to the basic gradient in a naive (symmetric) Siamese architecture.
For example, in Fig.
2 (a), the negative gradient on Z a is Z b (see Eq 3) and Z b can be seen as the basic gradient because Z a and Z b are represented by the same encoder.
The extra gradient introduced by negative samples can thus be perceived as G e .
Similarly, in Fig.
2 (b), the negative gradient on
P a is P b (see Eq 3) and P b can be seen as the basic gradient.
With the basic gradient P b , the extra gradient G e caused by the asymmetric predictor is then Z b − P b for Fig.
1 (a).
With G e = o e + r e , we analyze how its center vector o e and residual vector r e in various setups inﬂuence the collapse in the following.
3. A TOY EXAMPLE EXPERIMENT WITH NEGATIVE SAMPLE
Which repulsive component helps avoid collapse?
Existing works often attribute the collapse in
Naive Siamese to lacking a repulsive part during the optimization.
This explanation has motivated previous works to adopt contrastive learning, i.e . attracting the positive samples while repulsing the negative samples.
We experiment with a simple triplet loss 1 , L tri = − Z a · sg ( Z b − Z n ) , where
Z n indicates the representation of a N egative sample.
The derived negative gradient is Z b − Z n , where Z b is the basic gradient component in the symmetric architecture and thus G e = − Z n in this setup.
For a sample representation, what determines it as a positive sample for attracting or a negative sample for repulsing is the component, thus it might be tempting to interpret that r e is the key component of repulsive part that avoids the collapse .
However, the results in Table 3 show that the component beneﬁcial for preventing collapse inside G e is o e instead of r e .
Speciﬁcally, to explore the individual inﬂuence of o e and r e in the G e , we design two experiments by removing one component while keeping the other one.
In the ﬁrst experiment, we remove the r e in G e while keeping the o e .
By contrast, the o e is removed while keeping the r e in the second experiment.
In contrast to what existing explanations may expect, we ﬁnd that the gradient component o e prevents collapses.
With Conjecture1, a gradient component prevents collapse if it has negative center vector.
In this setup, o e = − o z , thus o e has the de-centering role for preventing collapse.
On the contrary, r e does not prevent collapse and keeping r e even decreases the performance ( 36 . 21% < 47 .
Since the negative sample is randomly chosen, r e just behaves like a random noise on the optimization. [Table] Removing r e [Table]
a · sg ( Z b + o e ) 0. × 47. Removing o e − Z a · sg ( Z b + r e )   (cid:88) 3. D ECOMPOSED GRADIENT ANALYSIS FOR S IM S IAM
It is worth mentioning that it is challenging to derive the gradient on z a in SimSiam due to a nonlinear MLP module in h .
The negative gradient on P a for L SimSiam in Eq.
2 can be derived as
[Equation]
Top
[Equation] [Table] [Equation] [Table]
×
[Table]
×
[Table]
[Equation]
[Table]
where G e indicates the aforementioned extra gradient component.
To investigate the inﬂuence of o e and r e on the collapse, similar to the analysis with the toy example experiment in Sec. 3.2, we design the experiment by removing one component while keeping the other.
The results are reported in Table 4.
As expected, the model collapses when both components in G e are removed and the best performance is achieved when both components are kept.
Interestingly, the model does not collapse when either o e or r e is kept.
How o e prevents collapse in SimSiam.
Here, o p is used to denote the center vector of P to differentiate from the above introduced o z for denoting that of Z . Since G e = Z b − P b , o e = o z − o p in this setup.
With Conjecture1, it is well expected that o e helps prevent collapse if o e contains negative o p since the analyzed vector is P a .
To determine the amount of component of o p existing in o e , we measure the cosine similarity between o e − η p o p and o p for a wide range of η p .
The results in Fig 5 (a) show that when η p is around − 0 .
5 for their cosine similarity being close to zero, suggesting o e has − 0 .
5 o p .
With Conjecture1, this negative η p explains why SimSiam avoids collapse from the perspective of de-centering.
How o e causes collapse in Mirror SimSiam. Notably, as mentioned above, Mirror SimSiam does not prevent collapse, which can also be explained by analyzing its o e . In this setup, o e =
o p − o z , for which we evaluate the amount of component o z existing in o e via reporting the similarity between o e − η
z
o z and o z . The results show that their cosine similarity is zero when η z is set to around 0.2.
This positive η z explains why Fig. 1(c) causes collapse from the perspective of de-centering.
Overall, we ﬁnd that processing the target vector Z b with h − 1 , as in Fig.
2 (c), alleviates collapse ( η p = − 0 . 5 ), while processing it with h , as in Fig. 1(c), actually strengthens the collapse ( η z = 0 . 2 ).
In other words, via the analysis of o e , our results help explain how SimSiam avoids collapse as well as how Mirror SimSiam causes collapse from the de-centering perspective.
[Figure]
Relation to prior works.
Motivated from preventing the collapse to a constant, multiple prior works, such as W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021), explicitly adopt de-centering to prevent collapse.
Despite various motivations, we ﬁnd that they all implicitly introduce an o e that contains a negative center vector.
The success of their approaches aligns well with our conjecture1 as well as our above empirical results.
Based on our ﬁndings, we argue that the effect of de-centering can be perceived as o e having the negative center vector.
With this interpretation, we are the ﬁrst to demonstrate that how SimSiam with predictor and stop gradient avoids collapse can be explained from the perspective of de-centering. Beyond de-centering for avoiding collapse. In the toy example experiment in Section 3.2, r e is found to be not beneﬁcial for preventing collapse and keeping r e even decreases the performance. Interestingly, as shown in Table 4, we ﬁnd that r e alone is sufﬁcient for preventing collapse and achieves comparable performance as G e .
This can be explained from the perspective of dimensional de-correlation, which will be discussed in Section 3.4.
3. D IMENSIONAL DE - CORRELATION HELPS PREVENT COLLAPSE
Conjecture 2 and motivation.
We conjecture that dimensional de-correlation increases m r for preventing collapse.
The motivation is straightforward as follows.
The dimensional correlation would be minimum if only a single dimension has a very high value for every individual class and the dimension changes for different classes.
In another extreme case, when all the dimensions have the same values, equivalent to having a single dimension, which already collapses by itself in the sense of losing representation capacity.
Conceptually, r e has no direct inﬂuence on the center vector, thus we interpret that r e prevents collapse through increasing m r .
To verify the above conjecture, we train SimSiam normally with the loss in Eq 2 and train with several epochs with the loss in Eq 1 for intentionally decreasing the m r to close to zero.
Then, we train the loss with only a correlation regularization term, which is detailed in Appendix A.6.
The results in Fig.
5 (b) show that this regularization term increases m r at a very fast rate.
Dimensional de-correlation in SimSiam.
Assuming h only has a single FC layer to exclude the inﬂuence of o e , the weights in FC are expected to learn the correlation between different dimensions for the encoder output.
This interpretation echos well with the ﬁnding that the eigenspace of h weight aligns well with that of correlation matrix (Tian et al., 2021).
In essence, the h is trained to minimize the cosine similarity between h ( z a ) and I ( z b ) , where I is identity mapping.
Thus, h is optimized close to I , which is conceptually equivalent to optimizing the de-correlation for Z .
[Figure]
As shown in Table 4, r e alone for SimSiam also prevents collapse.
Since r e has no de-centering effect, the reason that prevents is collapse is attributed to the de-correlation effect.
From Figure 6, we observe SimSiam decreases the covairance during the whole training except in the ﬁrst few epochs.
From Figure 6 also reports the results for InfoNCE which will be discussed in Sec. 3.5.
3. T OWARDS A UNIFIED UNDERSTANDING OF RECENT PROGRESS IN SSL
De-centering and de-correlation in InfoNCE. InfoNCE loss is a default choice in multiple seminal contrastive learning frameworks (Sohn, 2016; Wu et al., 2018;
Oord et al., 2018).
The derived negative gradient of InfoNCE on Z a is proportional to Z b + (cid:80) Ni =
− λ i
Z i , where λ i =
exp( Z a · Z i /τ )
(cid:80) Ni =
exp( Z a · Z i /τ ) , and Z 0 =
Z b for notation simplicity.
See Appendix A.7 for the detailed derivation.
The extra gradient component G e = (cid:80) Ni =0 − λ i
Z i = − o z − (cid:80) Ni
=
λ i r i , for which o e = − o z and r e = − (cid:80) Ni =
λ i r i .
Clearly, o e contains negative o z as de-centering for avoiding collapse, which is equivalent to the toy example in Section 3.2 when the r e is removed.
Regarding r e , the main difference between L tri in the toy example and InfoNCE is that the latter exploits a batch of negative samples instead of a random one.
λ i is proportional to exp( Z a · Z i ) , indicating that a large weight is put on the negative sample when it is more similar to the anchor Z a , for which, intuitively, its dimensional values tend to have a high correlation with Z a .
Thus, r e containing such negative representation with a high weight tends to decrease dimensional correlation.
To verify this intuition, we measure the cosine similarity between r e and the gradient on Z a induced by a correlation regularization loss.
The results in Fig.
5 (c) show that such gradient similarity is high for a wide range of temperature value τ , especially when τ is around 0.1 or 0.2.
[-=
-]Formally, we hypothesize that the de-correlation effect in InfoNCE arises from the biased weights ( λ i ) on negative samples, which is corroborated by the temperature analysis in Appendix A.7.
A high temperature leads to more balanced weights λ i on negative samples and alleviates the de-correlation effect.
Fig.
6 shows the results of InfoNCE, whose trend highly resembles that of SimSiam.
For example,
InfoNCE also decreases the covariance value during training except for the ﬁrst few epochs.
Moreover, we also report the result of InfoNCE where r e is removed for excluding the de-correlation effect.
We observe that removing r e leads to a signiﬁcant drop, which is attributed to the diminished de-correlation effect. In contrast to SimSiam or InfoNCE (with r e by default), removing r e leads to a high covariance value during the whole training.
Unifying InfoNCE and SimSiam.
At ﬁrst sight, there is no conceptual similarity between InfoNCE and SimSiam, and this is why the community is intrigued by the success of SimSiam without negative samples.
Through decomposing the G e into o e and r e , we ﬁnd that for both, their o e plays the role of de-centering and their r e behaves like de-correlation.
In this sense, we bring the two seemingly irrelevant frameworks into a uniﬁed perspective in the sense of de-centering and de-correlation.
Beyond InfoNCE and SimSiam.
A literature survey on the recent progress in SSL shows that there is a trend of performing explicit manipulation of de-centering and de-correlation, for which
W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021) are the three representative works. They often achieve performance comparable to those with InfoNCE or SimSiam.
Towards a uniﬁed understanding of recent progress in SSL, our work is most similar to a concurrent work (Bardes et al., 2021).
Their work is mainly inspired by Barlow-twins (Zbontar et al., 2021) but decomposes them into three components in a more explicit manner.
On the contrary, our work is motivated to answer the question of how SimSiam prevents collapse without negative samples.
The take-way is also different in the sense that we ﬁnd that de-correlation itself is sufﬁcient for preventing collapse, while their work claims that variance component (equivalent to de-centering) is an indispensable component for preventing collapse.
Overall, through demystifying
InfoNCE and SimSiam with the decomposed two effects, our work helps understand recent progress in SSL in a more uniﬁed manner.
T OWARDS SIMPLIFYING THE PREDICTOR IN S IM S IAM
Two important details: BN and l 2 -normalization.
Regarding the reason for SimSiam’s success without negative samples, to our knowledge, our work is the ﬁrst to revisit and refute the explanatory claims in (Chen & He, 2021).
Several works, however, have attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam.
The success has been ascribed to BN in (Fetterman & Albrecht, 2020), however, (Richemond et al., 2020) refutes their claim and (Chen & He, 2021) also claims that there is no evidence that BN itself is beneﬁcial for preventing collapse.
Here, we point out a critical implementation detail that determines whether BN helps prevent collapse.
Since the role of intermediate BNs is ascribed to stabilize training (Richemond et al., 2020; Chen & He, 2021), we only investigate the ﬁnal BN in the encoder.
Speciﬁcally, BN alone, i.e .
using the Naive Siamese as shown in Fig 2, prevents collapse only if it is applied immediately before the MSE loss.
For example, the cosine loss adopts l 2 -normalization before the MSE loss, for which the BN needs to be applied after the l 2 normalization.
The rationale is that, as our conjecture suggests, de-centering is the key to preventing collapse.
In other words, BN that removes the mean is beneﬁcial for preventing collapse.
However, after the l 2 normalization, a new mean is generated.
Moreover, we empirically ﬁnd that unnormalized MSE loss works poorly for SimSiam, thus our further investigation still adopts the default cosine loss.
Based on our understanding of how SimSiam prevents collapse, we demonstrate that simple components (instead of a non-linear MLP) in the predictor are sufﬁcient enough for preventing collapse.
For example, to achieve de-centering, ideally a single bias layer would be sufﬁcient because a bias vector can represent the center vector.
On the other hand, to achieve dimensional de-correlation, a single FC layer might be sufﬁcient because a single FC layer can realize the interaction among various dimensions.
With the original SimSiam encoder, we ﬁnd that the above goals cannot be achieved.
However, attaching a l 2 normalization layer at the end of the encoder, i.e .
the input of the predictor, facilitates the investigation.
[Table]
Pridictor with FC layers.
To learn the dimensional correlation, theoretically, an FC layer is sufﬁcient.
In practice, we ﬁnd that a single FC layer can be difﬁcult to train.
Inspired by (Bell-Kligler et al., 2019), we adopt two consecutive FC layers even though two
FC layers are mathematically equivalent to a single FC layer.
Multiple FC layers have the property to make the training more stable.
This implementation is equivalent to removing the BN and ReLU in the original predictor.
Moreover, to train a single FC layer, we apply a Tanh layer to constrain the FC weight values after every iteration to make the training more stable.
They achieve comparable performance as original SimSiam (see Table 5). [Table]
Predictor with a bias layer.
A predictor with a single bias layer can be trained, where the bias vector is found to a very high cosine similarity with the center vector (see Table 6).
A theoretical derivation justifying the 0.99 cosine similarity is reported in Appendix A.8. A bias in the MLP predictor also has high a cosine similarity of 0.89. A single bias layer is found to be sufﬁcient for preventing collapse (see Table 5).  C ONCLUSION
Our work has investigated how SimSiam prevents collapse without negative samples.
Our investigation starts with revisiting the explanatory claims in SimSiam and pointing out a hidden ﬂaw in their reasoning.
After refuting their claims, our work proposes to decompose the representation vector and analyze the decomposed gradients.
We ﬁnd that for both InfoNCE and SimSiam, the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect.
Both de-centering and de-correlation effects are found to be beneﬁcial for preventing collapse.
We also simplify the predictors and ﬁnd that a single bias layer is sufﬁcient for preventing collapse.
A A PPENDIX A. E XPERIMENTAL SETTINGS
Unsupervised pre-training: Below are settings for self-supervised pre-training stage.
All experiments setting are used as default from the solo-learn library (da Costa et al., 2021).
Data augmentation: We use the series of transformations as follows, notations are in Pytorch style: RandomResizedCrop with scale [0.2, 1.0], interpolation is bicubic.
Randomly apply with probability of 0.8 for ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4), hue (0.1)) with strength accordingly.
Random gray scale RandomGrayscale apply with p = 0 .
2 Random apply horizontal ﬂip with p = 0 .
5 Normalize image to mean and std as (0.4914, 0.4822, 0.4465) and (0.247, 0.243, 0.261), respectively.
Initialization: All layers of the network are initialized by default of Pytorch.
Network architecture:
Network backbone B is Resnet-18.
Projection MLP head contains three fully-connected (FC) layers follow by Batch Norm (BN) and ReLU.
We remove ReLU in the ﬁnal
FC layer: FC 1 + BN + ReLU + FC 2 + BN + ReLU + FC 3 + BN .
All FC layers haveneurons for both input, output and hidden dimensions.
Predictor MLP head includes two FC layers as follows:
FC 1 + BN + ReLU + FC 2 .
Input and output of predictor both have dimension of 2048, while hidden dimension is 512.
Optimizer: SGD optimizer is used for unsupervised pre-training.
Batch size is 256 and learning rate is linearly scaling by formula lr × Bsz/ 256 from the base learning rate of 0.5.
The schedule for learning rate is cosine decay as SimSiam.
Momentum 0.9 and weight decay 1 .
× 10 −are used for SGD.
We use one GPU for each pre-training experiment.
The learning rate of predictor is ﬁxed (not cosine decay) same as Simsiam trick.
We use a warmup training for ﬁrst 10-epochs.
If not speciﬁed, by default we train the model for 1000 epochs.
Online linear evaluation: The frozen features (2048 dimensions) from the training set are extracted (from the self-supervised pre-trained model) to feed into train a supervised linear classiﬁer (1 FC layer with input 2048 and output is 10 classes).
The test is done on the validation set.
Learning rate for linear classiﬁer is 0.1.
All online evaluation reports are done using the solo-learn library for self-supervised learning (da Costa et al., 2021).
A. T WO SUB - PROBLEMS IN AO OF S IM S IAM
In the sub-problem η t ← arg min η L ( θ t , η ) , η t indicating latent representation of images at step t is actually obtained through η tx ← E T (cid:104) F θ t ( T ( x )) (cid:105) , where they in practice ignore E T [ · ] and sample only one augmentation T (cid:48) , i.e .
η tx ← F θ t ( T (cid:48) ( x )) .
Conceptually, Chen & He equate the role of predictor to EOA.
A. E XPERIMENTAL DETAILS FOR E XPLICIT EOA IN T ABLE
In the Moving average experiment, we follow the setting in SimSiam (Chen & He, 2021) without predictor.
In contrast to (Chen & He, 2021) that uses the representations of the same image from prior epochs, we perform EOA in the same mini-batch.
In the Same batch experiment, multiple augmentations, 10 augmentations for instance, are applied on the same image.
With multi augmentations, we get the corresponding encoded representation, i.e .
z i , i ∈ [1 , 10] .
We minimize the cosine distance between the ﬁrst representation z 1 and the average of the remaining vectors, i.e .
¯ z = 1 9 (cid:80) 10 i =
z i .
Gradient detach is applied on the average vector.
A. E XPERIMENTAL SETUP AND RESULT TREND
FOR
ABLE 2.
Here we provide the pseudocode for mirror SimSiam, Symmetric Predictor.
In the Mirror SimSiam experiment which relates to Fig.
1 (c), we stop the gradient in the input of predictor.
Then we attract the predictor with non-stop gradient projector output.
Without taking symmetric loss into account, the pseudocode is shown in Algorithm 1.
Taking symmetric loss into account, the pseudocode is shown in Algorithm 2.
[Table] [Table]
To implement the training dynamic of Fig.
2 (b), the pseudocode in Algorithm 3.
Algorithm 3 Pytorch-like Pseudocode: Symmetric Predictor
[Table]
The results of SimSiam, Naive Siamese, Mirror SimSiam, Symmetric Predictor are shown in Fig.
We observe that all models collapse except for SimSiam.
[Figure]
A. E XPERIMENTAL DETAILS FOR INVERSE PREDICTOR .
In the inverse predictor experiment which relates to Fig.
2 (c), we introduce a new predictor which has the same structure as that of original predictor.
The training loss consists of 3 parts: predictor training loss, inverse predictor training and new encoder (old encoder+predictor) training.
The new encoder consists of old encoder + predictor.
Proper detach needs to be put in the implementation.
We provide the pseudocode in Algorithm 4.
# f: encoder (backbone + projector)
# h: predictor # h_inv: inverse predictor for x in loader: # load a minibatch x with n samples x_a,
x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b)
# projections p_a, p_b = h(z_a), h(z_b)
# predictions d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach())
# detached predictor output # predictor training loss
L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/2 # to train h inv_p_a, inv_p_b = h_inv(p_a.detach()), h_inv(p_b.detach())
# to train h_inv # inverse predictor training loss
L_inv_pred = D(inv_p_a, z_a)/2 + D(inv_p_b, z_
[Table]
A. R EGULARIZATION LOSS
Following the Zbontar et al.
(2021), we compute covariance regularization loss of encoder output along the mini-batch.
The pseudocode for de-correlation loss calculation is put in Algorithm 5.
[Equation]
# N
[Equation]
batch size # D:
[Equation]
number of dimension for representation vector
[Equation] A [Equation] G RADIENT DERIVATION AND TEMPERATURE ANALYSIS FOR I NFO NCE
Let S be the cosine similarity between Z a and Z b .
The InfoNCE loss function can be expressed as follows, when cosine similarity are used,
[Equation]
[Equation]
where N indicates the number of negative samples and Z
=
Z b for simplifying the notation.
By treating Z a · Z i as the logit in the normal CE loss, we have the corresponding probability for each logit as λ i = exp( Z a · Z i /τ )
{+(cid:80) Ni =
+}
exp( Z a · Z i /τ ) , where i = 0 , 1 , 2 , ..., N and we have (cid:80) Ni =
λ i = 1 .
The negative gradient of representation Z a with regard to InfoNCE loss is shown as follows:
[Equation]
where 1 τ can be adjusted through learning rate and is omitted for simple discussion.
With Z b as the basic symmetric gradient, G e = − o z − (cid:80) Ni =
λ i r i , for which o e = − o z and r e = − (cid:80) Ni =
λ i r i .
When the temperature is set to a large value, λ i =
[Equation]
Z a · Z i /τ ) , tends to be similar to each other for various i and InfoNCE will degenerate to a simple contrastive loss, i.e .
L simple = − Z a ·
Z b + 1 N (cid:80) Z a · Z n , which repulse every negative sample with an equal force.
In contrast, a relative smaller temperature will give more weight, i.e . larger λ , to samples that are more similar to the anchor.
Hypothesis: The biased repulsive weight on each negative sample in the gradient leads to a decorrelation effect.
The inﬂuence of the temperature on the covariance and accuracy is shown in Fig 8.
We observe that a higher temperature tends to decrease effect of de-correlation, indicated by higher covariance value, which also leads to a performance drop.
This veriﬁes the above hypothesis because a large temperature causes more balanced weights λ i , which is found to alleviate the effect of de-correlation.
The model is trained with 200 epochs with the default setting in Solo-learn for SimCLR framework.
[Figure]
A. T HEORETICAL DERIVATION FOR A SINGLE BIAS LAYER
The cosine similarity loss is deﬁned as Eq 7, and the derivative is presented as Eq 8:
[Equation]
The above equation is used a prior for our following derivations.
The single bias layer in the predictor is denoted as b .
p a =
Z a + b (note that Z a is l 2 -normalized as mentioned in the main manuscript) and P a = p a / || p a || .
The cosine similarity loss is shown as
[Equation]
The gradient on the bias vector is derived as
[Equation]
where m = cossim ( Z a , Z b ) (cid:107) p a (cid:107) .
Assuming that the training is stable and the bias vector converges to a certain value when − ∂cossim ( Z a , Z b ) ∂ b
=
Thus, the converged b satisﬁes the following constraint:
[Equation]
[Equation]
With a batch of samples, the average of ( r b − r a ) is expected to be close to 0.
Thus, the bias layer vector is expected to converge to:
[Equation]
In practice, with the training dynamics, b is expected to ﬂuctuate around 1 − mm o z .
Nonetheless, we empirically measure the cosine similarity between b and o z and ﬁnd they are around 0.99.
We further derive the gradient on Z a as follows:
[Equation]
Here, for the second part of the gradient, since the loss − Z a · Z a = − 1 is a constant, this second part is a dummy term.
Considering Eq 12 and the value of m , we have b = (
[Equation]
Then the above equation is equivalent to
[Equation]
With Z b as the basic symmetric gradient, the extra gradient component G e = − (1 − cossim ( Z a , Z b ) (cid:107) p a (cid:107) )
o z , p a =
Z a + b and (cid:107) Z a (cid:107) = 1 , thus (cid:107) p a (cid:107) <
1 only when Z a is negatively correlated with b .
In practice, however, Z a and b are often positively correlated to some extent due to their shared center vector component.
In other words, (cid:107) p a (cid:107) > 1 .
Moreover, cossim ( Z a , Z b is smaller than 1, thus − (1 − cossim ( Z a , Z b ) (cid:107) p a (cid:107) ) < 0 , suggesting G e consists of negative o z with the effect of de-centerization.
This derivation shows why a single bias layer can help avoid collapse.