H OW
D OES
S IM S IAM A VOID C OLLAPSE
W ITHOUT
N EGATIVE S AMPLES ? T OWARDS A U NIFIED U NDERSTANDING OF P ROGRESS IN SSL
Anonymous authors Paper under double-blind review A BSTRACT
To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples.
Without negative samples yet achieving competitive performance, a recent work (Chen & He, 2021) has attracted signiﬁcant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse.
However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam.
After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the l 2 -normalized representation vector.
This yields a uniﬁed perspective on how negative samples and SimSiam alleviate collapse.
Such a uniﬁed perspective comes timely for understanding the recent progress in SSL.
I NTRODUCTION
Beyond the success of NLP (Lan et al., 2020; Radford et al., 2019; Devlin et al., 2019;
Su et al., 2020; Nie et al., 2020), self-supervised learning (SSL) has also shown its potential in the ﬁeld of vision tasks (Li et al., 2021;
Chen et al., 2021; El-Nouby et al., 2021).
Without the ground-truth label, the core of most SSL methods lies in learning an encoder with augmentation-invariant representation (Bachman et al., 2019;
He et al., 2020; Chen et al., 2020a; Caron et al., 2020; Grill et al., 2020).
Speciﬁcally, they often minimize the representation distance between two positive samples, i.e . two augmented views of the same image, based on a Siamese network architecture (Bromley et al., 1993).
It is widely known that for such Siamese networks there exists a degenerate solution, i.e . all outputs “collapsing” to an undesired constant (Chen et al., 2020a; Chen & He, 2021).
Early works have attributed the collapse to lacking a repulsive component in the optimization goal and adopted negative samples, i.e . views of different samples, to alleviate this problem.
Introducing momentum into the target encoder, BYOL shows that Siamese architectures can be trained with only positive pairs.
More recently, SimSiam (Chen & He, 2021) has caught great attention by further simplifying BYOL by removing the momentum encoder, which has been seen as a major milestone achievement in SSL for providing a minimalist method for achieving competitive performance.
However, the reason for such success remains unclear with a core question summarized as:
How does SimSiam avoid collapse without negative samples?
Our investigation starts with revisiting the explanatory claims in the original SimSiam paper (Chen & He, 2021).
Notably, two components, i.e . stop gradient and predictor, are essential for the success of SimSiam (Chen & He, 2021).
The reason has been mainly attributed to the stop gradient (Chen & He, 2021) by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves like alternating between optimizing each set.
Chen & He argue that the predictor h is helpful in SimSiam because h ﬁlls the gap to approximate expectation over augmentations (EOA).
Unfortunately, the above explanatory claims are found to be ﬂawed due to reversing the two paths with and without gradient (see Sec. 2.2).
This motivates us to ﬁnd an alternative explanation, for which we introduce a simple yet intuitive framework for facilitating the analysis of collapse in SSL.
Speciﬁcally, we propose to decompose a representation vector into center and residual components.
This decomposition facilitates understanding which gradient component is beneﬁcial for avoiding collapse.
Under this framework, we show that a basic Siamese architecture cannot prevent collapse, for which an extra gradient component needs to be introduced.
With SimSiam interpreted as processing the optimization target with an inverse predictor, the analysis of its extra gradient shows that (a) its center vector helps prevent collapse via the de-centering effect; (b) its residual vector achieves dimensional de-correlation which also alleviates collapse. Moreover, under the same gradient decomposition, we ﬁnd that the extra gradient caused by negative samples in InfoNCE (He et al., 2019; Chen et al., 2020b;a; Tian et al., 2019; Khosla et al., 2020) also achieve de-centering and de-correlation in the same manner.
It bridges the gap between InfoNCE and SimSiam and contributes to a uniﬁed understanding of the recent progress of various frameworks in SSL.
Finally, towards simplifying the predictor for more explainable SimSiam, we show that a single bias layer is sufﬁcient enough for preventing collapse.
The basic experimental settings for our analysis are detailed in Appendix A.1 with a more speciﬁc setup discussed in the context.
Overall, our work is the ﬁrst attempt for performing a comprehensive study on how SimSiam avoids collapse without negative samples.
Several works, however, have attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam.
A technical report (Fetterman & Albrecht, 2020) has suggested the importance of batch normalization (BN) in BYOL for its success, however, a recent work (Richemond et al., 2020) refutes their claim by showing BYOL works without BN, which is discussed in Appendix B.
R EVISITING S IM S IAM AND ITS EXPLANATORY CLAIMS
l 2 normalized vector and optimization goal.
SSL trains an encoder f for learning discriminative representation and we denote such representation as a vector z , i.e . f
( x ) =
z where x is a certain input.
For the augmentation-invariant representation, a straightforward goal is to minimize the distance between the representations of two positive samples, i.e . augmented views of the same image, for which mean squared error (MSE) is a default choice.
To avoid scale ambiguity, the vectors are often l 2 normalized, i.e .
Z =
z / ||
z ||
(Chen & He, 2021), before calculating the MSE:
[Equation]
which shows the equivalence of a normalized MSE loss to the cosine loss (Grill et al., 2020).
Collapse in SSL and solution of SimSiam .
Based on a Siamese architecture, the loss in Eqcauses the collapse, i.e. f always outputs a constant regardless of the input variance.
We refer to this Siamese architecture with loss Eq 1 as Naive Siamese in the remainder of paper.
Contrastive loss with negative samples is a widely used solution (Chen et al., 2020a).
Without using negative samples, SimSiam solves the collapse problem via predictor and stop gradient, based on which the encoder is optimized with a symmetric loss:
[Equation]
where sg ( · ) is stop gradient and P is the output of predictor h , i.e .
p =
h ( z ) and P = p / || p
|| .
2. R EVISING EXPLANATORY CLAIMS IN S IM S IAM
Interpreting stop gradient as AO.
Chen & He hypothesize that the stop gradient in Eq 2 is an implementation of Alternating between the Optimization of two sub-problems, which is denoted as AO.
Speciﬁcally, with the loss considered as L ( θ, η ) =
E x, T (cid:104)(cid:13)(cid:13)
F θ ( T ( x ))
− η x (cid:13)(cid:13) 2 (cid:105) , the optimization objective min θ,η L ( θ, η ) can be solved by alternating η t ← arg min η L ( θ t , η ) and θ t +1 ← arg min θ L ( θ, η t ) .
It is acknowledged that this hypothesis does not fully explain why the collapse is prevented (Chen & He, 2021).
Nonetheless, they mainly attribute SimSiam success to the stop gradient with the interpretation that AO might make it difﬁcult to approach a constant ∀ x .
Interpreting predictor as EOA.
The AO problem (Chen & He, 2021) is formulated independent of predictor h , for which they believe that the usage of predictor h is related to approximating EOA for ﬁlling the gap of ignoring E T [ · ] in a sub-problem of AO.
The approximation of E T [ · ] is summarized in Appendix A.2.
Chen & He support their interpretation by proof-of-concept experiments.
Specifically, they show that updating η x with a moving-average η tx ← m ∗ η tx + (1 − m ) ∗ F θ t ( T (cid:48) ( x )) can help prevent collapse without predictor (see Fig 1 (b)).
Given that the training completely fails when the predictor and moving average are both removed, at ﬁrst sight, their reasoning seems valid.
2. D OES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA?
Reasoning ﬂaw.
Considering the stop gradient
[Figure]
we divide the framework into two sub-models with different paths and term them Gradient Path (GP) and Stop Gradient Path (SGP).
For SimSiam, only the sub-model with GP includes the predictor (see Fig.
1 (a)).
We point out that their reasoning ﬂaw of predictor analysis lies in the reverse of GP and SGP .
By default, the moving-average sub-model, as shown in Fig.
1 (b), is on the same side as SGP.
Note that Fig.
1 (b) is conceptually similar to Fig.
1 (c) instead of Fig.
1 (a).
It is worth mentioning that the Mirror SimSiam in Fig.
1 (c) is what stop gradient in the original SimSiam avoids.
Therefore, it is problematic to perceive h as EOA
[Figure]
New Figure
[Figure]
[Figure]
z a z b z a z b z a z b p a p b p a p b
[Figure]
Method [Table] aug
Collapse
[Table] [Table]
[Table]
[Table]
[Table]
Explicit EOA does not prevent collapse.
(Chen & He, 2021) points out that “in practice, it would be unrealistic to actually compute the expectation E T [ · ] .
But it may be possible for a neural network (e.g., the preditor h ) to learn to predict the expectation, while the sampling of T is implicitly distributed across multiple epochs.”
If implicitly sampling across multiple epochs is beneﬁcial, explicitly sampling sufﬁcient large N augmentations in a batch with the latest model would be more beneﬁcial to approximate E T [ · ] .
However, Table 1 shows that the collapse still occurs and suggests that the equivalence between predictor and EOA does not hold.
2. A SYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN S IM S IAM
Symmetric Predictor does not prevent collapse.
The difference between Naive Siamese and Simsiam lies in whether the gradient in backward propagation ﬂows through a predictor, however, we show that this propagation helps avoid collapse only when the predictor is not included in the SGP path.
With h being trained the same as Eq 2, we optimize the encoder f through replacing the Z in Eq 2 with P .
The results in Table.
2 show that it still leads to collapse.
Actually, this is well expected by perceiving h to be part of the new encoder F , i.e .
p =
F ( x ) =
h ( f ( x )) .
In other words, the symmetric architectures with and without predictor h both lead to collapse.
Predictor with stop gradient is asymmetric.
Clearly, how SimSiam avoids collapse lies in its asymmetric architecture, i.e . one path with predictor and the other without predictor.
Under this asymmetric architecture, the role of stop gradient is to only allow the path with predictor to be optimized with the encoder output as the target, not vice versa.
In other words, the SimSiam avoids collapse by excluding Mirror SimSiam (Fig. 1 (c)) which has a loss (mirror-like Eq 2) as L Mirror = − ( sg ( P a ) · Z b + sg ( P b ) · Z a ) , where the predictor h is still trained the same way as in SimSiam.
[Table]
Predictor vs .
inverse predictor.
We interpret h as a function mapping from z to p , and introduce a conceptual inverse mapping h − 1 , i.e . z = h − 1 ( p ) .
Here, as shown in Table 2, SimSiam with symmetric predictor (Fig. 2 (b)) leads to collapse, while SimSiam (Fig.(a)) avoids collapse.
With the conceptual h − 1 , we interpret Fig.
1 (a) the same as Fig.
2 (c) which differs from Fig.
2 (b) via changing the optimization target from p b to z b , i.e . z
b = h − 1 ( p b ) .
This interpretation suggests that the collapse can be avoided by processing the optimization target with h − 1 .
By contrast, Fig. 1 (c) and Fig. 2 (a) both lead to collapse, suggesting that processing the optimization target with h is not beneﬁcial for preventing collapse.
Overall, asymmetry alone does not guarantee collapse avoidance, which requires the optimization target to be processed by h − 1 not h .
[Figure]
Trainable inverse predictor and its implication on
EOA.
In the above, we propose a conceptual inverse predictor h − 1 in Fig.
2 (c), however, it remains yet unknown whether such an inverse predictor is experimentally trainable .
A detailed setup for this investigation is reported in Appendix A.5.
The results in Fig.show that a learnable h − 1 leads to slightly inferior performance, which is expected because h − 1 cannot make the trainable inverse predictor output z ∗ b completely the same as z b .
Note that it would be equivalent to SimSiam if z ∗ b = z b .
Despite a slight performance drop, the results conﬁrm that h − 1 is trainable.
The fact that h − 1 is trainable provides additional evidence that the role h plays in SimSiam is not EOA because theoretically h − 1 cannot restore a random augmentation T (cid:48) from an expectation p , where p = h ( z ) =
E T (cid:104) F θ t (
T ( x )) (cid:105)
V ECTOR DECOMPOSITION FOR UNDERSTANDING COLLAPSE
By default, InfoNCE (Chen et al., 2020a) and SimSiam (Chen & He, 2021) both adopt l 2 normalization in their loss for avoiding scale ambiguity.
We treat the l 2 normalized vector, i.e . Z , as the encoder output, which signiﬁcantly simpliﬁes gradient derivation and the following analysis.
Vector decomposition.
For the purpose of analysis, we propose to decompose Z into two parts,
Z = o + r , where o , r denote center vector and residual vector respectively.
Speciﬁcally, the center vector o is deﬁned as an average of Z over the whole representation space o
z
=
E
[ Z ] .
However, we approximate it with all vectors in current mini-batch, i.e . o z = 1 M (cid:80)
Mm =1 Z m , where
M is the mini-batch size.
We deﬁne the residual vector r as the residual part of Z , i.e . r
=
Z − o z .
3. C OLLAPSE FROM THE VECTOR PERSPECTIVE
Collapse: from result to cause.
A Naive Siamese is well expected to collapse since the loss is designed to minimize the distance between positive samples, for which a constant constitutes an optimal solution to minimize such loss.
When the collapse occurs, ∀ i, Z i = 1 M (cid:80)
Mm =1 Z m =
o z , where i denotes a random sample index, which shows the constant vector is o z in this case.
This interpretation only suggests a possibility that a dominant o can be one of the viable solutions, while the optimization, such as SimSiam, might still lead to a non-collapse solution.
This merely describes o as the consequence of the collapse, and our work investigates the cause of such collapse through analyzing the inﬂuence of individual gradient components, i.e . o and r during training.
Competition between o and r .
Complementary to the Standard Deviation (Std) (Chen & He, 2021), for indicating collapse, we introduce the ratio of o in z , i.e . m o
=
||
o || /
||
z || , where || ∗ || is the L 2 norm.
Similarly, the ratio of r in z is deﬁned as m r = ||
r || /
||
z
|| .
When collapse happens, i.e . all vectors Z are close to the center vector o , m o approaches 1 and m r approaches 0, which is not desirable for SSL.
A desirable case would be a relatively small m o and a relatively large m r , suggesting a relatively small (large) contribution of o ( r ) in each Z .
We interpret the cause of collapse as a competition between o and r where o dominates over r , i.e .
m o (cid:29) m r .
For Eq 1, the derived negative gradient on Z a (ignoring Z b for simplicity due to symmetry) is shown as:
[Equation]
where the gradient component Z a is a dummy term because the loss − Z a · Z a = − 1 is a constant having zero gradient on the encoder f .
[Figure]
Conjecture1.
With Z a =
o
z
+ r a , we conjecture that the gradient component of o z is expected to update the encoder to boost the center vector thus increase m o , while the gradient component of r a is expected to behave in the opposite direction to increase m r .
A random gradient component is expected to have a relatively small inﬂuence.
To verify the above conjecture, we revisit the dummy gradient term Z a .
We design loss − Z a · sg ( o z ) and − Z a · sg ( Z a − o z ) to show the inﬂuence of gradient component o and r a respectively.
The results in
Fig.
4 show that the gradient component o z has the effect of increasing m o while decreasing m r .
On the contrary, r a helps increase m r while decreasing m o .
Overall, the results verify Conjecture1.
3. E XTRA GRADIENT COMPONENT FOR ALLEVIATING COLLAPSE
Revisit collapse in a symmetric architecture.
Based on Conjecture1, here, we provide an intuitive interpretation on why a symmetric Siamese architecture, such as Fig.
2 (a) and (b), cannot be trained without collapse.
Take Fig.
2 (a) as example, the gradient in Eq 3 can be interpreted as two equivalent forms, from which we choose Z b − Z a =
( o z + r b ) − ( o z + r a ) =
r b − r a .
Since r b comes from the same positive sample as r a , it is expected that r b also increases m r , however, this effect is expected to be smaller than that of r a , thus causing collapse.
Basic gradient and Extra gradient components.
The negative gradient on Z a in Fig.
2 (a) is derived as Z b , while that on P a in Fig.
2 (b) is derived as P b .
We perceive Z b and P b in these basic Siamese architectures as the Basic Gradient .
Our above interpretation shows that such basic components cannot prevent collapse, for which an E xtra G radient component, denoted as G e , needs to be introduced to break the symmetry .
As the term suggests, G e is deﬁned as a gradient term that is relative to the basic gradient in a basic Siamese architecture.
For example, negative samples can be introduced to Naive Siamese (Fig.
2 (a))
for preventing collapse, where the extra gradient caused by negative samples can thus be perceived as G e with Z b as the basic gradient.
Similarly, we can also disentangle the negative gradient on P a in SimSiam (Fig. 1 (a)), i.e . Z b , into a basic gradient (which is P b ) and G e which is derived as Z b − P b (note that Z b
=
P b + G e ).
We analyze how G e prevents collapse via studying the independent roles of its center vector o e and residual vector r e .
3. A TOY EXAMPLE EXPERIMENT WITH NEGATIVE SAMPLE
Which repulsive component helps avoid collapse?
Existing works often attribute the collapse in
Naive Siamese to lacking a repulsive part during the optimization.
This explanation has motivated previous works to adopt contrastive learning, i.e . attracting the positive samples while repulsing the negative samples.
We experiment with a simple triplet loss 1 , L tri = − Z a · sg ( Z b − Z n ) , where Z n indicates the representation of a N egative sample.
The derived negative gradient on Z a is Z b − Z n , where Z b is the basic gradient component and thus G e = − Z n in this setup.
For a sample representation, what determines it as a positive sample for attracting or a negative sample for repulsing is the residual component, thus it might be tempting to interpret that r e is the key component of repulsive part that avoids the collapse .
However, the results in Table 3 show that the component beneﬁcial for preventing collapse inside G e is o e instead of r e .
Speciﬁcally, to explore the individual inﬂuence of o e and r e in the G e , we design two experiments by removing one component while keeping the other one.
In the ﬁrst experiment, we remove the r e in G e while keeping the o e .
By contrast, the o e is removed while keeping the r e in the second experiment.
In contrast to what existing explanations may expect, we ﬁnd that the residual component o e prevents collapses.
With Conjecture1, a gradient component alleviates collapse if it has negative center vector.
In this setup, o e = − o z , thus o e has the de-centering role for preventing collapse.
On the contrary, r e does not prevent collapse and keeping r e even decreases the performance ( 36 . 21% < 47 .
Since the negative sample is randomly chosen, r e just behaves like a random noise on the optimization to decrease performance.
[Table]
Removing r e − Z a · sg ( Z b + o e ) 0.
× 47.
Removing o e − Z a · sg ( Z b + r e )   (cid:88)
3. D ECOMPOSED GRADIENT ANALYSIS IN S IM S IAM
It is challenging to derive the gradient on the encoder output in SimSiam due to a nonlinear MLP module in h .
The negative gradient on P a for L SimSiam in Eq 2 can be derived as
[Equation]
[Table]
[Equation]
[Table]
where G e indicates the aforementioned extra gradient component.
To investigate the inﬂuence of o e and r e on the collapse, similar to the analysis with the toy example experiment in Sec. 3.3, we design the experiment by removing one component while keeping the other.
The results are reported in Table 4.
As expected, the model collapses when both components in G e are removed and the best performance is achieved when both components are kept.
Interestingly, the model does not collapse when either o e or r e is kept.
To start, we analyze how o e affects the collapse based on Conjecture1.
How o e alleviates collapse in SimSiam.
Here, o p is used to denote the center vector of P to differentiate from the above introduced o z for denoting that of Z .
In this setup G e = Z b − P b , thus the residual gradient component is derived to be o e =
o
z − o p .
With Conjecture1, it is well expected that o e helps prevent collapse if o e contains negative o p since the analyzed vector is P a .
To determine the amount of component of o p existing in o e , we measure the cosine similarity between o e − η p o p and o p for a wide range of η p .
The results in Fig 5 (a) show that their cosine similarity is zero when η p is around − 0 .
5 , suggesting o e has ≈ − 0 .
5 o p .
With Conjecture1, this negative η p explains why SimSiam avoids collapse from the perspective of de-centering.
How o e causes collapse in Mirror SimSiam.
As mentioned above, the collapse occurs in Mirror
SimSiam, which can also be explained by analyzing its o e .
Here, o e =
o p − o z , for which we evaluate the amount of component
o z existing in o e via reporting the similarity between o e − η
z
o z and o z .
The results in Fig 5 (a) show that their cosine similarity is zero when η z is set to around 0.2.
This positive η z explains why Fig.
1(c) causes collapse from the perspective of de-centering.
Overall, we ﬁnd that processing the optimization target with h − 1 , as in Fig.
2 (c), alleviates collapse ( η p ≈ − 0 . 5 ), while processing it with h , as in Fig. 1(c), actually strengthens the collapse ( η z ≈ 0 . 2 ).
In other words, via the analysis of o e , our results help explain how SimSiam avoids collapse as well as how Mirror SimSiam causes collapse from a straightforward de-centering perspective.
[Figure]
Relation to prior works.
Motivated from preventing the collapse to a constant, multiple prior works, such as W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021), explicitly adopt de-centering to prevent collapse.
Despite various motivations, we ﬁnd that they all implicitly introduce an o e that contains a negative center vector.
The success of their approaches aligns well with our Conjecture1 as well as our above empirical results.
Based on our ﬁndings, we argue that the effect of de-centering can be perceived as o e having a negative center vector.
With this interpretation, we are the ﬁrst to demonstrate that how SimSiam with predictor and stop gradient avoids collapse can be explained from the perspective of de-centering.
Beyond de-centering for avoiding collapse.
In the toy example experiment in Sec. 3.3, r e is found to be not beneﬁcial for preventing collapse and keeping r e even decreases the performance.
Interestingly, as shown in Table 4, we ﬁnd that r e alone is sufﬁcient for preventing collapse and achieves comparable performance as G e .
This can be explained from the perspective of dimensional de-correlation, which will be discussed in Sec. 3.5.
3. D IMENSIONAL DE - CORRELATION HELPS PREVENT COLLAPSE
Conjecture2 and motivation.
We conjecture that dimensional de-correlation increases m r for preventing collapse.
The motivation is straightforward as follows.
The dimensional correlation would be minimum if only a single dimension has a very high value for every individual class and the dimension changes for different classes.
In another extreme case, when all the dimensions have the same values, equivalent to having a single dimension, which already collapses by itself in the sense of losing representation capacity.
Conceptually, r e has no direct inﬂuence on the center vector, thus we interpret that r e prevents collapse through increasing m r .
To verify the above conjecture, we train SimSiam normally with the loss in Eq 2 and train for several epochs with the loss in Eq 1 for intentionally decreasing the m r to close to zero.
Then, we train the loss with only a correlation regularization term, which is detailed in Appendix A.6.
The results in
Fig.
5 (b) show that this regularization term increases m r at a very fast rate.
Dimensional de-correlation in SimSiam.
Assuming h only has a single FC layer to exclude the inﬂuence of o e , the weights in FC are expected to learn the correlation between different dimensions for the encoder output.
This interpretation echos well with the ﬁnding that the eigenspace of h weight aligns well with that of correlation matrix (Tian et al., 2021).
In essence, the h is trained to minimize the cosine similarity between h ( z a ) and I ( z b ) , where I is identity mapping.
Thus, h that learns the correlation is optimized close to I , which is conceptually equivalent to optimizing with the goal of de-correlation for Z .
As shown in Table 4, for SimSiam, r e alone also prevents collapse, which
[Figure]
is attributed to the de-correlation effect since r e has no de-centering effect.
We observe from Fig.
6 that except in the ﬁrst few epochs, SimSiam decreases the covariance during the whole training.
Fig 6 also reports the results for InfoNCE which will be discussed in Sec. 4.
T OWARDS A UNIFIED UNDERSTANDING OF RECENT PROGRESS IN SSL
De-centering and de-correlation in InfoNCE.
InfoNCE loss is a default choice in multiple seminal contrastive learning frameworks (Sohn, 2016;
Wu et al., 2018;
Oord et al., 2018; Wang & Liu, 2021).
The derived negative gradient of InfoNCE on Z a is proportional to Z b + (cid:80) Ni =
− λ i
Z i , where λ i =
[Equation]
=0 exp( Z a · Z i /τ ) , and Z 0 =
Z b for notation simplicity.
See Appendix A.7 for the detailed derivation.
The extra gradient component G e = (cid:80) Ni =0 − λ i
Z i = − o z − (cid:80) Ni
=
λ i r i , for which o e = − o z and r e = − (cid:80) Ni =
λ i r i .
Clearly, o e contains negative o z as de-centering for avoiding collapse, which is equivalent to the toy example in Sec. 3.3 when the r e is removed.
Regarding r e , the main difference between L tri in the toy example and InfoNCE is that the latter exploits a batch of negative samples instead of a random one.
λ i is proportional to exp( Z a · Z i ) , indicating that a large weight is put on the negative sample when it is more similar to the anchor Z a , for which, intuitively, its dimensional values tend to have a high correlation with Z a .
Thus, r e containing such negative representation with a high weight tends to decrease dimensional correlation.
To verify this intuition, we measure the cosine similarity between r e and the gradient on Z a induced by a correlation regularization loss.
The results in Fig.
5 (c) show that their gradient similarity is high for a wide range of temperature values, especially when τ is around 0.1 or 0.2, suggesting r e achieves similar role as an explicit regularization loss for performing de-correlation.
Replacing r e with o e leads to a low cosine similarity, which is expected because o e has no de-correlation effect.
The results of InfoNCE in Fig.
6 resembles that of SimSiam in terms of the overall trend.
For example, InfoNCE also decreases the covariance value during training.
Moreover, we also report the results of InfoNCE where r e is removed for excluding the de-correlation effect.
Removing r e from the InfoNCE loss leads to a high covariance value during the whole training.
Removing r e also leads to a signiﬁcant performance drop, which echos with the ﬁnding in (Bardes et al., 2021) that dimensional de-correlation is essential for competitive performance.
Regarding how r e in InfoNCE achieves de-correlation, formally, we hypothesize that the de-correlation effect in InfoNCE arises from the biased weights ( λ i ) on negative samples.
This hypothesis is corroborated by the temperature analysis in Fig.
We ﬁnd that a higher temperature makes the weight distribution of λ i more balanced indicated a higher entropy of λ i , which echos with the ﬁnding in (Wang & Liu, 2021).
Moreover, we observe that a higher temperature also tends to increase the covariance value.
Overall, with temperature as the control variable, we ﬁnd that more balanced weights among negative samples decrease the de-correlation effect, which constitutes an evidence for our hypothesis.
Unifying InfoNCE and SimSiam.
At ﬁrst sight, there is no conceptual similarity between InfoNCE and SimSiam, and this is why the community is intrigued by the success of SimSiam without negative samples.
Through decomposing the G e into o e and r e , we ﬁnd that for both, their o e plays the role of de-centering and their r e behaves like de-correlation.
In this sense, we bring two seemingly irrelevant frameworks into a uniﬁed perspective with disentangled de-centering and de-correlation.
Beyond InfoNCE and SimSiam.
A literature survey on the recent progress in SSL shows that there is a trend of performing explicit manipulation of de-centering and de-correlation, for which
W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021) are three representative works.
They often achieve performance comparable to those with InfoNCE or
SimSiam.
Towards a uniﬁed understanding of recent progress in SSL, our work is most similar to a
[Figure]
concurrent work (Bardes et al., 2021).
Their work is mainly inspired by Barlow-twins (Zbontar et al., 2021) but decomposes its loss into three explicit components.
By contrast, our work is motivated to answer the question of how SimSiam prevents collapse without negative samples.
Their work claims that variance component (equivalent to de-centering) is an indispensable component for preventing collapse, while we ﬁnd that de-correlation itself alleviates collapse.
Overall, demystifying InfoNCE and SimSiam helps understand various frameworks in SSL from a more uniﬁed perspective.
T OWARDS SIMPLIFYING THE PREDICTOR IN S IM S IAM
Towards simplifying the predictor.
Based on our understanding of how SimSiam prevents collapse, we demonstrate that simple components (instead of a non-linear MLP in SimSiam) in the predictor are sufﬁcient for preventing collapse.
For example, to achieve dimensional de-correlation, a single FC layer might be sufﬁcient because a single FC layer can realize the interaction among various dimensions.
On the other hand, to achieve de-centering, a single bias layer might be sufﬁcient because a bias vector can represent the center vector.
Attaching an l 2 normalization layer at the end of the encoder, i.e . before the predictor, is found to be critical for achieving the above goal.
[Table]
Pridictor with FC layers.
To learn the dimensional correlation, an FC layer is sufﬁcient theoretically but can be difﬁcult to train in practice.
Inspired by the property that Multiple FC layers make the training more stable even though they can be mathematically equivalent to a single FC layer (Bell-Kligler et al., 2019), we adopt two consecutive FC layers which are equivalent to removing the BN and ReLU in the original predictor.
The training can be made more stable if a Tanh layer is applied on the adopted single FC after every iteration.
Table 5 shows that they achieve performance comparable to that with a non-linear MLP.
[Table]
Predictor with a bias layer.
A predictor with a single bias layer can be utilized for preventing collapse (see
Table 5) and the trained bias vector is found to have a cosine similarity of 0.99 with the center vector (see
Table 6).
A bias in the MLP predictor also has a high cosine similarity of 0.89, suggesting that it is not a coincidence.
A theoretical derivation for justifying such a high similarity as well as how this single bias layer prevents collapse are discussed in Appendix A.8.
C ONCLUSION
We point out a hidden ﬂaw in prior works for explaining the success of SimSiam and propose to decompose the representation vector and analyze the decomposed components of extra gradient.
We ﬁnd that its center vector gradient helps prevent collapse via the de-centering effect and its residual gradient achieves de-correlation which also alleviates collapse. Our further analysis reveals that
InfoNCE achieve the two effects in a similar manner, which bridges the gap between InfoNCE and SimSiam and contributes to a uniﬁed understanding of recent progress in SSL.
Towards simplifying the predictor we have also found that a single bias layer is sufﬁcient for preventing collapse.
A A PPENDIX A. E XPERIMENTAL SETTINGS
Unsupervised pre-training: Below are the settings for self-supervised encoder training.
The settings are used as default from the solo-learn library (da Costa et al., 2021).
Data augmentation: We use a series of transformations as follows: RandomResizedCrop with scale [0.2, 1.0], bicubic interpolation.
ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4), hue (0.1)) are randomly applied with the probability of 0.8.
Random gray scale RandomGrayscale is applied with p = 0 .
2 Horizontal ﬂip is applied with p = 0 .
The images are normalized with the mean (0.4914, 0.4822, 0.4465) and Std (0.247, 0.243, 0.261).
Initialization: All layers of the network are by default initialized in Pytorch.
Network architecture: The backbone architecture is ResNet-18.
The projection head contains three fully-connected (FC) layers followed by Batch Norm (BN) and ReLU, for which ReLU in the ﬁnal
FC layer is removed, i.e . FC 1 + BN + ReLU + FC 2 + BN + ReLU + FC 3 + BN .
All projection
FC layers have 2048 neurons for input, output as well as the hidden dimensions.
The predictor head includes two FC layers as follows:
FC 1 + BN + ReLU + FC 2 .
Input and output of the predictor both have the dimension of 2048, while the hidden dimension is 512.
Optimizer: SGD optimizer is used for the encoder training.
The batch size is 256 and the learning rate is linearly scaled by formula lr × Bsz/ 256 from the base learning rate of 0.5.
The schedule for learning rate adopts cosine decay as SimSiam.
Momentum 0.9 and weight decay 1 .
× 10 − 5 are used for SGD.
We use one GPU for each pre-training experiment.
Following the practice of SimSiam, the learning rate of predictor is ﬁxed during the training.
We use warmup training for the ﬁrst 10 epochs.
If not speciﬁed, by default we train the model for 1000 epochs.
For the online linear revaluation, we follow the solo-learn library (da Costa et al., 2021). The frozen features (2048 dimensions) from the training set are extracted (from the self-supervised pre-trained model) to feed into train a supervised linear classiﬁer (1 FC layer with the input 2048 and output of 100).
The test is performed on the validation set.
The learning rate for the linear classiﬁer is 0.1.
All online evaluation reports are done using the.
A. T WO SUB - PROBLEMS IN AO OF S IM S IAM
In the sub-problem η t ← arg min η L ( θ t , η ) , η t indicating latent representation of images at step t is actually obtained through η tx ← E T (cid:104) F θ t ( T ( x )) (cid:105) , where they in practice ignore E T [ · ] and sample only one augmentation T (cid:48) , i.e .
η tx ← F θ t ( T (cid:48) ( x )) .
Conceptually, Chen & He equate the role of predictor to EOA.
A. E XPERIMENTAL DETAILS FOR E XPLICIT EOA IN T ABLE
In the Moving average experiment, we follow the setting in SimSiam (Chen & He, 2021) without predictor.
In the Same batch experiment, multiple augmentations, 10 augmentations for instance, are applied on the same image.
With multi augmentations, we get the corresponding encoded representation, i.e .
z i , i ∈ [1 , 10] .
We minimize the cosine distance between the ﬁrst representation zand the average of the remaining vectors, i.e . ¯ z = 19 (cid:80) 10 i =
z i .
Gradient detach is applied on the averaged vector.
We also experimented with letting the gradient backward through more augmentations, however, they consistently led to collapse.
A. E XPERIMENTAL SETUP AND RESULT TREND FOR T ABLE 2. Mirror SimSiam
Here we provide the pseudocode for Mirror SimSiam.
In the Mirror SimSiam experiment which relates to Fig.
1 (c).
Without taking symmetric loss into account, the pseudocode is shown in Algorithm 1.
Taking symmetric loss into account, the pseudocode is shown in Algorithm 2.
[Table]
[Table]
Symmetric Predictor.
To implement the SimSiam with Symmetric Predictor as in Fig.
2 (b), naively we can just perceive the predictor as part of the new encoder, for which the pseudocode is provided in
Algorithm 3.
Alternatively, we can additionally train the predictor similarly as that in SimSiam, for which the training involves two losses, one for training the predictor and another for training the new encoder and the corresponding pseudocode is provided in Algorithm 4.
Moreover, for the second implementation, we also experiment with another variant that ﬁxes the predictor while optimizing the new encoder and then train the predictor alternatingly.
All of them lead to collapse with similar trend as long as the symmetric predictor is used for training the encoder.
For avoiding redundancy, in Fig. 8 we only report one of them.
# f: encoder (backbone + projector)
# h: predictor for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x)
# augmentation z_a, z_b = f(x_a), f(x_b)
# projections p_a, p_b = h(z_a), h(z_b)
# predictions
L = D(p_a, p_b)/2 + D(p_b, p_a)/2 # loss
L.backward() # back-propagate update(f, h) # SGD update def D(p, z): # negative cosine similarity z = z.detach # stop gradient p = p,
1) # l2- z = (z, dim
1) # l2- return p*z sum mean()
# f: encoder (backbone + projector)
# h: predictor for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x)
# augmentation z_a, z_b = f(x_a), f(x_b)
# projections p_a, p_b = h(z_a), h(z_b)
# predictions d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach())
# detached predictor output # predictor training loss L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/ # encoder training loss L_enc =
D(p_a, d_p_b)/2 + D(p_b, d_p_a)/
L = L_pred + L_enc
L.backward() # back-propagate update(f, h) # SGD update def D(p, z): # negative cosine similarity with detach on z z = z.detach() # stop gradient p = normalize(p, dim=1)
# l2-normalize z = normalize(z, dim=1)
# l2-normalize return -(p*z).sum(dim=1).mean()
The results of SimSiam, Naive Siamese, Mirror SimSiam, Symmetric Predictor are shown in Fig.
We observe that all models collapse except for SimSiam.
[Figure]
A. E XPERIMENTAL DETAILS FOR INVERSE PREDICTOR .
In the inverse predictor experiment which relates to Fig.
2 (c), we introduce a new predictor which has the same structure as that of the original predictor.
The training loss consists of 3 parts: predictor training loss, inverse predictor training and new encoder (old encoder+predictor) training.
The new encoder consists of the old encoder + predictor.
Proper gradient detach needs to be put in the implementation.
We provide the pseudocode in Algorithm 5.
A. R EGULARIZATION LOSS
Following the Zbontar et al.
(2021), we compute covariance regularization loss of encoder output along the mini-batch.
The pseudocode for de-correlation loss calculation is put in Algorithm 6.
# f: encoder (backbone + projector)
# h: predictor # h_inv: inverse predictor for x in loader: # load a minibatch x with n samples x_a,
x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b)
# projections p_a, p_b = h(z_a), h(z_b)
# predictions d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach())
# detached predictor output # predictor training loss
L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/2 # to train h inv_p_a, inv_p_b = h_inv(p_a.detach()), h_inv(p_b.detach())
# to train h_inv # inverse predictor training loss
L_inv_pred = D(inv_p_a, z_a)/2 + D(inv_p_b, z_b)/ # encoder training loss L_enc =
D(p_a, h_inv(p_b))/2 + D(p_b, h_inv(p_a))
L = L_pred + L_inv_pred + L_enc
L.backward() # back-propagate update(f, h, h_inv) # SGD update def D(p, z): # negative cosine similarity with detach on z z = z.detach() # stop gradient p = normalize(p, dim=1)
# l2-normalize z = normalize(z, dim=1)
# l2-normalize return -(p*z).sum(dim=1).mean() like Pseudocode: De-correlation loss
[Table]
A. G RADIENT DERIVATION AND TEMPERATURE ANALYSIS FOR I NFO NCE
Let S be the cosine similarity between Z a and Z b .
The InfoNCE loss function can be expressed as follows, when cosine similarity are used,
[Equation]
where N indicates the number of negative samples and Z
=
Z b for simplifying the notation.
By treating Z a · Z i as the logit in the normal CE loss, we have the corresponding probability for each logit as λ i =
[Equation]
) , where i = 0 , 1 , 2 , ..., N and we have (cid:80) Ni =
λ i = 1 .
The negative gradient of representation Z a with regard to InfoNCE loss is shown as follows:
[Equation]
where 1 τ can be adjusted through learning rate and is omitted for simple discussion.
With Z b as the basic symmetric gradient, G e = − o z − (cid:80) Ni =
λ i r i , for which o e = − o z and r e = − (cid:80) Ni =
λ i r i .
When the temperature is set to a large value, λ i =
[Equation]
=
[Equation]
exp
[Equation]
Z a · Z i /τ ) , tends to be similar to each other for various i and InfoNCE will degenerate to a simple contrastive loss, i.e .
L simple = − Z a ·
Z b + 1 N (cid:80) Z a · Z n , which repulse every negative sample with an equal force.
In contrast, a relative smaller temperature will give more weight, i.e . larger λ , to samples that are more similar to the anchor.
The inﬂuence of the temperature on the covariance and accuracy is shown in Fig 7.
We observe that a higher temperature tends to decrease the effect of de-correlation, indicated by a higher covariance value, which also leads to a performance drop.
This veriﬁes the above hypothesis because a large temperature causes more balanced weights λ i , which is found to alleviate the effect of decorrelation.
The model is trained with 200 epochs with the default setting in Solo-learn for the
SimCLR framework.
A. T HEORETICAL DERIVATION FOR A SINGLE BIAS LAYER
The cosine similarity loss is deﬁned as Eq 7, and the derivative is presented as Eq 8:
[Equation]
The above equation is used a prior for our following derivations.
The single bias layer in the predictor is denoted as b .
p a =
Z a + b (note that Z a is l 2 -normalized as mentioned in the main manuscript) and P a = p a / || p a || .
The cosine similarity loss is shown as
[Equation]
The gradient on the bias vector is derived as
[Equation]
where m = cossim ( Z a , Z b ) (cid:107) p a (cid:107) .
Assuming that the training is stable and the bias vector converges to a certain value when − ∂cossim ( Z a , Z b ) ∂ b
=
Thus, the converged b satisﬁes the following constraint:
[Equation]
With a batch of samples, the average of 1 m r b and r a is expected to be close to 0 by the deﬁnition of residual vector.
Thus, the bias layer vector is expected to converge to:
[Equation]
In practice, with the training dynamics, b is expected to ﬂuctuate around 1 − mm o z .
Nonetheless, we empirically measure the cosine similarity between b and o z and ﬁnd they are around 0.99.
We further derive the gradient on Z a as follows:
[Equation]
Here, for the second part of the gradient, since the loss − Z a · Z a = − 1 is a constant, this second part is a dummy term.
Considering Eq 12 and the value of m , we have b = (
[Equation]
Then the above equation is equivalent to
[Equation]
With Z b as the basic symmetric gradient, the extra gradient component G e = − (1 − cossim ( Z a , Z b ) (cid:107) p a (cid:107) )
o z , p a =
Z a + b and (cid:107) Z a (cid:107) = 1 , thus (cid:107) p a (cid:107) <
1 only when Z a is negatively correlated with b .
In practice, however, Z a and b are often positively correlated to some extent due to their shared center vector component.
In other words, (cid:107) p a (cid:107) > 1 .
Moreover, cossim ( Z a , Z b is smaller than 1, thus − (1 − cossim ( Z a , Z b ) (cid:107) p a (cid:107) ) < 0 , suggesting G e consists of negative o z with the effect of de-centerization.
This derivation shows why a single bias layer can help avoid collapse.
B D ISCUSSION : DOES BN HELP AVOID COLLAPSE ?
To our knowledge, our work is the ﬁrst to revisit and refute the explanatory claims in (Chen & He, 2021).
Several works, however, have attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam.
The success has been ascribed to BN in (Fetterman & Albrecht, 2020), however, (Richemond et al., 2020) refutes their claim.
Since the role of intermediate BNs is ascribed to stabilize training (Richemond et al., 2020; Chen & He, 2021), we only discuss the ﬁnal
BN in the SimSiam encoder.
Note that with our Conjecture1, the ﬁnal BN that removes the mean of representation vector is supposed to have de-centering effect.
Why would such a BN not prevent collapse in this case?
Interestingly, we observe that such BN can actually help alleviate collapse with a simple MSE loss.
Note that the cosine loss is in essence equivalent to a MSE loss on the l 2normalized vectors.
This phenomenon can be interpreted as that the l 2 -normalization causes another mean after the BN removes it.
Thus, with such l 2 -normalization in the MSE loss, i.e . adopting the default cosine loss, it is important to remove the o e from the optimization target.
The results with the loss of − Z a · sg ( Z b + o e ) in Table 3 verify the above interpretation.