H OW
D OES
S IM S
IAM A VOID C
OLLAPSE
W ITHOUT
N EGATIVE S AMPLES ? T OWARDS A U NIFIED U NDERSTANDING OF P ROGRESS IN SSL
Anonymous authors Paper under double-blind review A BSTRACT
To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples.
Without negative samples yet achieving competitive performance, a recent work (Chen & He, 2021) has attracted signiﬁcant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse.
However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam.
After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the l 2 -normalized representation vector.
This yields a uniﬁed perspective on how negative samples and SimSiam alleviate collapse.
Such a uniﬁed perspective comes timely for understanding the recent progress in SSL.
I NTRODUCTION
Beyond the success of NLP (Lan et al., 2020; Radford et al., 2019; Devlin et al., 2019;
Su et al., 2020; Nie et al., 2020), self-supervised learning (SSL) has also shown its potential in the ﬁeld of vision tasks (Li et al., 2021;
Chen et al., 2021; El-Nouby et al., 2021).
Without the ground-truth label, the core of most SSL methods lies in learning an encoder with augmentation-invariant representation (Bachman et al., 2019;
He et al., 2020; Chen et al., 2020a; Caron et al., 2020; Grill et al., 2020).
Speciﬁcally, they often minimize the representation distance between two positive samples, i.e . two augmented views of the same image, based on a Siamese network architecture (Bromley et al., 1993).
It is widely known that for such Siamese networks there exists a degenerate solution, i.e . all outputs “collapsing” to an undesired constant (Chen et al., 2020a; Chen & He, 2021).
Early works have attributed the collapse to lacking a repulsive component in the optimization goal and adopted negative samples, i.e . views of different samples, to alleviate this problem.
Introducing momentum into the target encoder, BYOL shows that Siamese architectures can be trained with only positive pairs.
More recently, SimSiam (Chen & He, 2021) has caught great attention by further simplifying BYOL by removing the momentum encoder, which has been seen as a major milestone achievement in SSL for providing a minimalist method for achieving competitive performance.
However, the reason for such success remains unclear with a core question summarized as:
How does SimSiam avoid collapse without negative samples?
Our investigation starts with revisiting the explanatory claims in the original SimSiam paper (Chen & He, 2021).
Notably, two components, i.e . stop gradient and predictor, are essential for the success of SimSiam (Chen & He, 2021).
The reason has been mainly attributed to the stop gradient (Chen & He, 2021) by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves like alternating between optimizing each set.
Chen & He argue that the predictor h is helpful in
SimSiam because h ﬁlls the gap to approximate expectation over augmentations (EOA).
Unfortunately, the above explanatory claims are found to be ﬂawed due to reversing the two paths with and without gradient (see Sec. 2.2).
This motivates us to ﬁnd an alternative explanation, for which we introduce a simple yet intuitive framework for facilitating the analysis of collapse in SSL.
Speciﬁcally, we propose to decompose a representation vector into center and residual components.
This decomposition facilitates understanding which gradient component is beneﬁcial for avoiding collapse.
Under this framework, we show that a basic Siamese architecture cannot prevent collapse, for which an extra gradient component needs to be introduced.
With SimSiam interpreted as processing the optimization target with an inverse predictor, the analysis of its extra gradient shows that (a) its center vector helps prevent collapse via the de-centering effect; (b) its residual vector achieves dimensional de-correlation which also alleviates collapse.
Moreover, under the same gradient decomposition, we ﬁnd that the extra gradient caused by negative samples in InfoNCE (He et al., 2019; Chen et al., 2020b;a; Tian et al., 2019; Khosla et al., 2020) also achieves de-centering and de-correlation in the same manner.
It bridges the gap between InfoNCE and SimSiam and contributes to a uniﬁed understanding of the recent progress of various frameworks in SSL.
Finally, towards simplifying the predictor for more explainable SimSiam, we show that a single bias layer is sufﬁcient enough for preventing collapse.
The basic experimental settings for our analysis are detailed in Appendix A.1 with a more speciﬁc setup discussed in the context.
Overall, our work is the ﬁrst attempt for performing a comprehensive study on how SimSiam avoids collapse without negative samples.
Several works, however, have attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam.
A technical report (Fetterman & Albrecht, 2020) has suggested the importance of batch normalization (BN) in BYOL for its success, however, a recent work (Richemond et al., 2020) refutes their claim by showing BYOL works without BN, which is discussed in Appendix B.
R EVISITING S IM S IAM AND ITS EXPLANATORY CLAIMS
l 2 -normalized vector and optimization goal.
SSL trains an encoder f for learning discriminative representation and we denote such representation as a vector z , i.e . f
( x ) =
z where x is a certain input.
For the augmentation-invariant representation, a straightforward goal is to minimize the distance between the representations of two positive samples, i.e . augmented views of the same image, for which mean squared error (MSE) is a default choice.
To avoid scale ambiguity, the vectors are often l 2 -normalized, i.e .
Z =
z / ||
z ||
(Chen & He, 2021), before calculating the MSE:
[Equation]
which shows the equivalence of a normalized MSE loss to the cosine loss (Grill et al., 2020).
Collapse in SSL and solution of SimSiam .
Based on a Siamese architecture, the loss in Eqcauses the collapse, i.e. f always outputs a constant regardless of the input variance.
We refer to this Siamese architecture with loss Eq 1 as Naive Siamese in the remainder of paper.
Contrastive loss with negative samples is a widely used solution (Chen et al., 2020a).
Without using negative samples, SimSiam solves the collapse problem via predictor and stop gradient, based on which the encoder is optimized with a symmetric loss:
[Equation]
where sg ( · ) is stop gradient and P is the output of predictor h , i.e .
p =
h ( z ) and P = p / || p
|| .
2. R EVISING EXPLANATORY CLAIMS IN S IM S IAM
Interpreting stop gradient as AO.
Chen & He hypothesize that the stop gradient in Eq 2 is an implementation of Alternating between the Optimization of two sub-problems, which is denoted as AO.
Speciﬁcally, with the loss considered as L ( θ, η ) =
E x, T (cid:104)(cid:13)(cid:13)
F θ ( T ( x ))
− η x (cid:13)(cid:13) 2 (cid:105) , the optimization objective min θ,η L ( θ, η ) can be solved by alternating η t ← arg min η L ( θ t , η ) and θ t +1 ← arg min θ L ( θ, η t ) .
It is acknowledged that this hypothesis does not fully explain why the collapse is prevented (Chen & He, 2021).
Nonetheless, they mainly attribute SimSiam success to the stop gradient with the interpretation that AO might make it difﬁcult to approach a constant ∀ x .
Interpreting predictor as EOA.
The AO problem (Chen & He, 2021) is formulated independent of predictor h , for which they believe that the usage of predictor h is related to approximating EOA for ﬁlling the gap of ignoring E T [ · ] in a sub-problem of AO.
The approximation of E T [ · ] is summarized in Appendix A.2.
Chen & He support their interpretation by proof-of-concept experiments.
Specifically, they show that updating η x with a moving-average η tx ← m ∗ η tx + (1 − m ) [Equation] T [Equation]
( x )) can help prevent collapse without predictor (see Fig.
1 (b)).
Given that the training completely fails when the predictor and moving average are both removed, at ﬁrst sight, their reasoning seems valid.
2. D OES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA?
Reasoning ﬂaw.
Considering the stop gradient
[Figure]
we divide the framework into two sub-models with different paths and term them Gradient Path (GP) and Stop Gradient Path (SGP).
For SimSiam, only the sub-model with GP includes the predictor (see Fig.
1 (a)).
We point out that their reasoning ﬂaw of predictor analysis lies in the reverse of GP and SGP .
By default, the moving-average sub-model, as shown in Fig.
1 (b), is on the same side as SGP.
Note that Fig.
1 (b) is conceptually similar to
Fig.
1 (c) instead of Fig.
1 (a).
It is worth mentioning that the Mirror SimSiam in Fig.
1 (c) is what stop gradient in the original SimSiam avoids.
Therefore, it is problematic to perceive h as EOA
[Figure]
New Figure
[Figure]
[Figure]
z a z b z a z b z a z b p a p b p a p b
[Figure]
Method
[Table]
aug
Collapse
[Table] [Table]
[Table] [Table]
[Table]
Explicit EOA does not prevent collapse.
(Chen & He, 2021) points out that “in practice, it would be unrealistic to actually compute the expectation E T [ · ] .
But it may be possible for a neural network (e.g., the preditor h ) to learn to predict the expectation, while the sampling of T is implicitly distributed across multiple epochs.”
If implicitly sampling across multiple epochs is beneﬁcial, explicitly sampling sufﬁcient large N augmentations in a batch with the latest model would be more beneﬁcial to approximate E T [ · ] .
However, Table 1 shows that the collapse still occurs and suggests that the equivalence between predictor and EOA does not hold.
2. A SYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN S IM S IAM
Symmetric Predictor does not prevent collapse.
The difference between Naive Siamese and Simsiam lies in whether the gradient in backward propagation ﬂows through a predictor, however, we show that this propagation helps avoid collapse only when the predictor is not included in the SGP path.
With h being trained the same as Eq 2, we optimize the encoder f through replacing the Z in Eq 2 with P .
The results in Table.
2 show that it still leads to collapse.
Actually, this is well expected by perceiving h to be part of the new encoder F , i.e .
p =
F ( x ) =
h ( f ( x )) .
In other words, the symmetric architectures with and without predictor h both lead to collapse.
Predictor with stop gradient is asymmetric.
Clearly, how SimSiam avoids collapse lies in its asymmetric architecture, i.e . one path with h and the other without h .
Under this asymmetric architecture, the role of stop gradient is to only allow the path with predictor to be optimized with the encoder output as the target, not vice versa.
In other words, the SimSiam avoids collapse by excluding Mirror SimSiam (Fig. 1 (c)) which has a loss (mirror-like Eq 2) as L Mirror = − ( P a · Z b + P b · Z a ) , where stop gradient is put on the input of h , i.e . p a = h ( sg [ z a ]) and p b = h ( sg [ z b ]) .
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Predictor vs .
inverse predictor.
We interpret h as a function mapping from z to p , and introduce a conceptual inverse mapping h − 1 , i.e . z = h − 1 ( p ) .
Here, as shown in Table 2, SimSiam with symmetric predictor (Fig. 2 (b)) leads to collapse, while SimSiam (Fig.(a)) avoids collapse.
With the conceptual h − 1 , we interpret Fig.
1 (a) the same as Fig.
2 (c) which differs from Fig.
2 (b) via changing the optimization target from p b to z b , i.e . z
b = h − 1 ( p b ) .
This interpretation suggests that the collapse can be avoided by processing the optimization target with h − 1 .
By contrast, Fig. 1 (c) and Fig. 2 (a) both lead to collapse, suggesting that processing the optimization target with h is not beneﬁcial for preventing collapse.
Overall, asymmetry alone does not guarantee collapse avoidance, which requires the optimization target to be processed by h − 1 not h .
[Figure]
Trainable inverse predictor and its implication on
EOA.
In the above, we propose a conceptual inverse predictor h − 1 in Fig.
2 (c), however, it remains yet unknown whether such an inverse predictor is experimentally trainable .
A detailed setup for this investigation is reported in Appendix A.5.
The results in Fig.show that a learnable h − 1 leads to slightly inferior performance, which is expected because h − 1 cannot make the trainable inverse predictor output z ∗ b completely the same as z b .
Note that it would be equivalent to SimSiam if z ∗ b = z b .
Despite a slight performance drop, the results conﬁrm that h − 1 is trainable.
The fact that h − 1 is trainable provides additional evidence that the role h plays in SimSiam is not EOA because theoretically h − 1 cannot restore a random augmentation T (cid:48) from an expectation p , where p = [Equation]
V ECTOR DECOMPOSITION FOR UNDERSTANDING COLLAPSE
By default, InfoNCE (Chen et al., 2020a) and SimSiam (Chen & He, 2021)
both adopt l 2normalization in their loss for avoiding scale ambiguity.
We treat the l 2 -normalized vector, i.e . Z , as the encoder output, which signiﬁcantly simpliﬁes gradient derivation and the following analysis.
Vector decomposition.
For the purpose of analysis, we propose to decompose Z into two parts,
Z = o + r , where o , r denote center vector and residual vector respectively.
Speciﬁcally, the center vector o is deﬁned as an average of Z over the whole representation space o
z
=
E
[ Z ] .
However, we approximate it with all vectors in current mini-batch, i.e . o z = 1 M (cid:80)
Mm =1 Z m , where
M is the mini-batch size.
We deﬁne the residual vector r as the residual part of Z , i.e . r
=
Z − o z .
3. C OLLAPSE FROM THE VECTOR PERSPECTIVE
Collapse: from result to cause.
A Naive Siamese is well expected to collapse since the loss is designed to minimize the distance between positive samples, for which a constant constitutes an optimal solution to minimize such loss.
When the collapse occurs, ∀ i, Z i = 1 M (cid:80)
Mm =1 Z m =
o z , where i denotes a random sample index, which shows the constant vector is o z in this case.
This interpretation only suggests a possibility that a dominant o can be one of the viable solutions, while the optimization, such as SimSiam, might still lead to a non-collapse solution.
This merely describes o as the consequence of the collapse, and our work investigates the cause of such collapse through analyzing the inﬂuence of individual gradient components, i.e . o and r during training.
Competition between o and r .
Complementary to the Standard Deviation (Std) (Chen & He, 2021), for indicating collapse, we introduce the ratio of o in z , i.e . m o
=
||
o || /
||
z || , where || ∗ || is the L 2 norm.
Similarly, the ratio of r in z is deﬁned as m r = ||
r || /
||
z
|| .
When collapse happens, i.e . all vectors Z are close to the center vector o , m o approaches 1 and m r approaches 0, which is not desirable for SSL.
A desirable case would be a relatively small m o and a relatively large m r , suggesting a relatively small (large) contribution of o ( r ) in each Z .
We interpret the cause of collapse as a competition between o and r where o dominates over r , i.e .
m o (cid:29) m r .
For Eq 1, the derived negative gradient on Z a (ignoring Z b for simplicity due to symmetry) is shown as:
[Equation]
where the gradient component Z a is a dummy term because the loss − Z a · Z a = − 1 is a constant having zero gradient on the encoder f .
[Figure]
Conjecture1.
With Z a =
o
z
+ r a , we conjecture that the gradient component of o z is expected to update the encoder to boost the center vector thus increase m o , while the gradient component of r a is expected to behave in the opposite direction to increase m r .
A random gradient component is expected to have a relatively small inﬂuence.
To verify the above conjecture, we revisit the dummy gradient term Z a .
We design loss − Z a · sg ( o z ) and − Z a · sg ( Z a − o z ) to show the inﬂuence of gradient component o and r a respectively.
The results in
Fig.
4 show that the gradient component o z has the effect of increasing m o while decreasing m r .
On the contrary, r a helps increase m r while decreasing m o .
Overall, the results verify Conjecture1.
3. E XTRA GRADIENT COMPONENT FOR ALLEVIATING COLLAPSE
Revisit collapse in a symmetric architecture.
Based on Conjecture1, here, we provide an intuitive interpretation on why a symmetric Siamese architecture, such as Fig.
2 (a) and (b), cannot be trained without collapse.
Take Fig.
2 (a) as example, the gradient in Eq 3 can be interpreted as two equivalent forms, from which we choose Z b − Z a =
( o z + r b ) − ( o z + r a ) =
r b − r a .
Since r b comes from the same positive sample as r a , it is expected that r b also increases m r , however, this effect is expected to be smaller than that of r a , thus causing collapse.
Basic gradient and Extra gradient components.
The negative gradient on Z a in Fig.
2 (a) is derived as Z b , while that on P a in Fig.
2 (b) is derived as P b .
We perceive Z b and P b in these basic Siamese architectures as the Basic Gradient .
Our above interpretation shows that such basic components cannot prevent collapse, for which an E xtra G radient component, denoted as G e , needs to be introduced to break the symmetry .
As the term suggests, G e is deﬁned as a gradient term that is relative to the basic gradient in a basic Siamese architecture.
For example, negative samples can be introduced to Naive Siamese (Fig.
2 (a))
for preventing collapse, where the extra gradient caused by negative samples can thus be perceived as G e with Z b as the basic gradient.
Similarly, we can also disentangle the negative gradient on P a in SimSiam (Fig. 1 (a)), i.e . Z b , into a basic gradient (which is P b ) and G e which is derived as Z b − P b (note that Z b
=
P b + G e ).
We analyze how G e prevents collapse via studying the independent roles of its center vector o e and residual vector r e .
3. A TOY EXAMPLE EXPERIMENT WITH NEGATIVE SAMPLE
Which repulsive component helps avoid collapse?
Existing works often attribute the collapse in
Naive Siamese to lacking a repulsive part during the optimization.
This explanation has motivated previous works to adopt contrastive learning, i.e . attracting the positive samples while repulsing the negative samples.
We experiment with a simple triplet loss 1 , L tri = − Z a · sg ( Z b − Z n ) , where Z n indicates the representation of a N egative sample.
The derived negative gradient on Z a is Z b − Z n , where Z b is the basic gradient component and thus G e = − Z n in this setup.
For a sample representation, what determines it as a positive sample for attracting or a negative sample for repulsing is the residual component, thus it might be tempting to interpret that r e is the key component of repulsive part that avoids the collapse .
However, the results in Table 3 show that the component beneﬁcial for preventing collapse inside G e is o e instead of r e .
Speciﬁcally, to explore the individual inﬂuence of o e and r e in the G e , we design two experiments by removing one component while keeping the other one.
In the ﬁrst experiment, we remove the r e in G e while keeping the o e .
By contrast, the o e is removed while keeping the r e in the second experiment.
In contrast to what existing explanations may expect, we ﬁnd that the residual component o e prevents collapses.
With Conjecture1, a gradient component alleviates collapse if it has negative center vector.
In this setup, o e = − o z , thus o e has the de-centering role for preventing collapse.
On the contrary, r e does not prevent collapse and keeping r e even decreases the performance ( 36 . 21% < 47 .
Since the negative sample is randomly chosen, r e just behaves like a random noise on the optimization to decrease performance.
[Table]
Removing r e − Z a · sg ( Z b + o e ) 0.
× 47.
Removing o e − Z a · sg ( Z b + r e )   (cid:88)
3. D ECOMPOSED GRADIENT ANALYSIS IN S IM S IAM
It is challenging to derive the gradient on the encoder output in SimSiam due to a nonlinear MLP module in h .
The negative gradient on P a for L SimSiam in Eq 2 can be derived as
[Equation]
[Table]
[Equation]
[Table]
where G e indicates the aforementioned extra gradient component.
To investigate the inﬂuence of o e and r e on the collapse, similar to the analysis with the toy example experiment in Sec. 3.3, we design the experiment by removing one component while keeping the other.
The results are reported in Table 4.
As expected, the model collapses when both components in G e are removed and the best performance is achieved when both components are kept.
Interestingly, the model does not collapse when either o e or r e is kept.
To start, we analyze how o e affects the collapse based on Conjecture1.
How o e alleviates collapse in SimSiam.
Here, o p is used to denote the center vector of P to differentiate from the above introduced o z for denoting that of Z .
In this setup G e = Z b − P b , thus the residual gradient component is derived to be o e =
o
z − o p .
With Conjecture1, it is well expected that o e helps prevent collapse if o e contains negative o p since the analyzed vector is P a .
To determine the amount of component of o p existing in o e , we measure the cosine similarity between o e − η p o p and o p for a wide range of η p .
The results in Fig. 5 (a) show that their cosine similarity is zero when η p is around − 0 .
5 , suggesting o e has ≈ − 0 .
5 o p .
With Conjecture1, this negative η p explains why SimSiam avoids collapse from the perspective of de-centering.
How o e causes collapse in Mirror SimSiam.
As mentioned above, the collapse occurs in Mirror
SimSiam, which can also be explained by analyzing its o e .
Here, o e =
o p − o z , for which we evaluate the amount of component
o z existing in o e via reporting the similarity between o e − η
z
o z and o z .
The results in Fig. 5 (a) show that their cosine similarity is zero when η z is set to around 0.2.
This positive η z explains why Fig.
1(c) causes collapse from the perspective of de-centering.
Overall, we ﬁnd that processing the optimization target with h − 1 , as in Fig.
2 (c), alleviates collapse ( η p ≈ − 0 . 5 ), while processing it with h , as in Fig. 1(c), actually strengthens the collapse ( η z ≈ 0 . 2 ).
In other words, via the analysis of o e , our results help explain how SimSiam avoids collapse as well as how Mirror SimSiam causes collapse from a straightforward de-centering perspective.
[Figure]
Relation to prior works.
Motivated from preventing the collapse to a constant, multiple prior works, such as W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021), explicitly adopt de-centering to prevent collapse.
Despite various motivations, we ﬁnd that they all implicitly introduce an o e that contains a negative center vector.
The success of their approaches aligns well with our Conjecture1 as well as our above empirical results.
Based on our ﬁndings, we argue that the effect of de-centering can be perceived as o e having a negative center vector.
With this interpretation, we are the ﬁrst to demonstrate that how SimSiam with predictor and stop gradient avoids collapse can be explained from the perspective of de-centering.
Beyond de-centering for avoiding collapse.
In the toy example experiment in Sec. 3.3, r e is found to be not beneﬁcial for preventing collapse and keeping r e even decreases the performance.
Interestingly, as shown in Table 4, we ﬁnd that r e alone is sufﬁcient for preventing collapse and achieves comparable performance as G e .
This can be explained from the perspective of dimensional de-correlation, which will be discussed in Sec. 3.5.
3. D IMENSIONAL DE - CORRELATION HELPS PREVENT COLLAPSE
Conjecture2 and motivation.
We conjecture that dimensional de-correlation increases m r for preventing collapse.
The motivation is straightforward as follows.
The dimensional correlation would be minimum if only a single dimension has a very high value for every individual class and the dimension changes for different classes.
In another extreme case, when all the dimensions have the same values, equivalent to having a single dimension, which already collapses by itself in the sense of losing representation capacity.
Conceptually, r e has no direct inﬂuence on the center vector, thus we interpret that r e prevents collapse through increasing m r .
To verify the above conjecture, we train SimSiam normally with the loss in Eq 2 and train for several epochs with the loss in Eq 1 for intentionally decreasing the m r to close to zero.
Then, we train the loss with only a correlation regularization term, which is detailed in Appendix A.6.
The results in
Fig.
5 (b) show that this regularization term increases m r at a very fast rate.
Dimensional de-correlation in SimSiam.
Assuming h only has a single FC layer to exclude the inﬂuence of o e , the weights in FC are expected to learn the correlation between different dimensions for the encoder output.
This interpretation echos well with the ﬁnding that the eigenspace of h weight aligns well with that of correlation matrix (Tian et al., 2021).
In essence, the h is trained to minimize the cosine similarity between h ( z a ) and I ( z b ) , where I is identity mapping.
Thus, h that learns the correlation is optimized close to I , which is conceptually equivalent to optimizing with the goal of de-correlation for Z .
As shown in Table 4, for SimSiam, r e alone also prevents collapse, which
[Figure]
is attributed to the de-correlation effect since r e has no de-centering effect.
We observe from Fig.
6 that except in the ﬁrst few epochs, SimSiam decreases the covariance during the whole training.
Fig. 6 also reports the results for InfoNCE which will be discussed in Sec. 4.
T OWARDS A UNIFIED UNDERSTANDING OF RECENT PROGRESS IN SSL
De-centering and de-correlation in InfoNCE.
InfoNCE loss is a default choice in multiple seminal contrastive learning frameworks (Sohn, 2016;
Wu et al., 2018;
Oord et al., 2018; Wang & Liu, 2021).
The derived negative gradient of InfoNCE on Z a is proportional to Z b + (cid:80) Ni =
− λ i
Z i , where λ i =
[Equation]
[Equation]
[Equation]
Ni =0 exp( Z a · Z i /τ ) , and Z 0 =
Z b for notation simplicity.
See Appendix A.7 for the detailed derivation.
The extra gradient component G e = (cid:80) Ni =0 − λ i
Z i = − o z − (cid:80) Ni
=
λ i r i , for which o e = − o z and r e = − (cid:80) Ni =
λ i r i .
Clearly, o e contains negative o z as de-centering for avoiding collapse, which is equivalent to the toy example in Sec.
3.3 when the r e is removed.
Regarding r e , the main difference between L tri in the toy example and InfoNCE is that the latter exploits a batch of negative samples instead of a random one.
λ i is proportional to exp( Z a · Z i ) , indicating that a large weight is put on the negative sample when it is more similar to the anchor Z a , for which, intuitively, its dimensional values tend to have a high correlation with Z a .
Thus, r e containing such negative representation with a high weight tends to decrease dimensional correlation.
To verify this intuition, we measure the cosine similarity between r e and the gradient on Z a induced by a correlation regularization loss.
The results in Fig.
5 (c) show that their gradient similarity is high for a wide range of temperature values, especially when τ is around 0.1 or 0.2, suggesting r e achieves similar role as an explicit regularization loss for performing de-correlation.
Replacing r e with o e leads to a low cosine similarity, which is expected because o e has no de-correlation effect.
The results of InfoNCE in Fig.
6 resembles that of SimSiam in terms of the overall trend.
For example, InfoNCE also decreases the covariance value during training.
Moreover, we also report the results of InfoNCE where r e is removed for excluding the de-correlation effect.
Removing r e from the InfoNCE loss leads to a high covariance value during the whole training.
Removing r e also leads to a signiﬁcant performance drop, which echos with the ﬁnding in (Bardes et al., 2021) that dimensional de-correlation is essential for competitive performance.
Regarding how r e in InfoNCE achieves de-correlation, formally, we hypothesize that the de-correlation effect in InfoNCE arises from the biased weights ( λ i ) on negative samples.
This hypothesis is corroborated by the temperature analysis in Fig.
We ﬁnd that a higher temperature makes the weight distribution of λ i more balanced indicated a higher entropy of λ i , which echos with the ﬁnding in (Wang & Liu, 2021).
Moreover, we observe that a higher temperature also tends to increase the covariance value.
Overall, with temperature as the control variable, we ﬁnd that more balanced weights among negative samples decrease the de-correlation effect, which constitutes an evidence for our hypothesis.
Unifying InfoNCE and SimSiam.
At ﬁrst sight, there is no conceptual similarity between InfoNCE and SimSiam, and this is why the community is intrigued by the success of SimSiam without negative samples.
Through decomposing the G e into o e and r e , we ﬁnd that for both, their o e plays the role of de-centering and their r e behaves like de-correlation.
In this sense, we bring two seemingly irrelevant frameworks into a uniﬁed perspective with disentangled de-centering and de-correlation.
Beyond InfoNCE and SimSiam.
A literature survey on the recent progress in SSL shows that there is a trend of performing explicit manipulation of de-centering and de-correlation, for which
W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021) are three representative works.
They often achieve performance comparable to those with InfoNCE or
SimSiam.
Towards a uniﬁed understanding of recent progress in SSL, our work is most similar to a
[Figure]
concurrent work (Bardes et al., 2021).
Their work is mainly inspired by Barlow-twins (Zbontar et al., 2021) but decomposes its loss into three explicit components.
By contrast, our work is motivated to answer the question of how SimSiam prevents collapse without negative samples.
Their work claims that variance component (equivalent to de-centering) is an indispensable component for preventing collapse, while we ﬁnd that de-correlation itself alleviates collapse.
Overall, demystifying InfoNCE and SimSiam helps understand various frameworks in SSL from a more uniﬁed perspective.
T OWARDS SIMPLIFYING THE PREDICTOR IN S IM S IAM
Towards simplifying the predictor.
Based on our understanding of how SimSiam prevents collapse, we demonstrate that simple components (instead of a non-linear MLP in SimSiam) in the predictor are sufﬁcient for preventing collapse.
For example, to achieve dimensional de-correlation, a single FC layer might be sufﬁcient because a single FC layer can realize the interaction among various dimensions.
On the other hand, to achieve de-centering, a single bias layer might be sufﬁcient because a bias vector can represent the center vector.
Attaching an l 2 -normalization layer at the end of the encoder, i.e . before the predictor, is found to be critical for achieving the above goal.
[Table]
Pridictor with FC layers.
To learn the dimensional correlation, an FC layer is sufﬁcient theoretically but can be difﬁcult to train in practice.
Inspired by the property that Multiple FC layers make the training more stable even though they can be mathematically equivalent to a single FC layer (Bell-Kligler et al., 2019), we adopt two consecutive FC layers which are equivalent to removing the BN and ReLU in the original predictor.
The training can be made more stable if a Tanh layer is applied on the adopted single FC after every iteration.
Table 5 shows that they achieve performance comparable to that with a non-linear MLP.
[Table]
Predictor with a bias layer.
A predictor with a single bias layer can be utilized for preventing collapse (see
Table 5) and the trained bias vector is found to have a cosine similarity of 0.99 with the center vector (see
Table 6).
A bias in the MLP predictor also has a high cosine similarity of 0.89, suggesting that it is not a coincidence.
A theoretical derivation for justifying such a high similarity as well as how this single bias layer prevents collapse are discussed in Appendix A.8.
C ONCLUSION
We point out a hidden ﬂaw in prior works for explaining the success of SimSiam and propose to decompose the representation vector and analyze the decomposed components of extra gradient.
We ﬁnd that its center vector gradient helps prevent collapse via the de-centering effect and its residual gradient achieves de-correlation which also alleviates collapse.
Our further analysis reveals that
InfoNCE achieve the two effects in a similar manner, which bridges the gap between InfoNCE and
SimSiam and contributes to a uniﬁed understanding of recent progress in SSL.
Towards simplifying the predictor we have also found that a single bias layer is sufﬁcient for preventing collapse.
A A PPENDIX A. E XPERIMENTAL SETTINGS
Self-supervised encoder training : Below are the settings for self-supervised encoder training.
For simplicity, we mainly use the default settings in a popular open library termed solo-learn (da Costa et al., 2021).
Data augmentation and normalization: We use a series of transformations including RandomResizedCrop with scale [0.2, 1.0], bicubic interpolation.
ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4), hue (0.1)) is randomly applied with the probability of 0.8.
Random gray scale RandomGrayscale is applied with p = 0 .
2 Horizontal ﬂip is applied with p = 0 .
The images are normalized with the mean (0.4914, 0.4822, 0.4465) and Std (0.247, 0.243, 0.261).
Network architecture and initialization:
The backbone architecture is ResNet-18.
The projection head contains three fully-connected (FC) layers followed by Batch Norm (BN) and ReLU, for which
ReLU in the ﬁnal FC layer is removed, i.e . FC 1 + BN + ReLU + FC 2 + BN + ReLU + FC 3 + BN .
All projection FC layers have 2048 neurons for input, output as well as the hidden dimensions.
The predictor head includes two FC layers as follows:
FC 1 + BN + ReLU + FC 2 .
Input and output of the predictor both have the dimension of 2048, while the hidden dimension is 512.
All layers of the network are by default initialized in Pytorch.
Optimizer: SGD optimizer is used for the encoder training.
The batch size M is 256 and the learning rate is linearly scaled by the formula lr × M/ 256 with the base learning rate lr set to 0.5.
The schedule for learning rate adopts the cosine decay as SimSiam.
Momentum 0.9 and weight decay 1 .
× 10 − 5 are used for SGD.
We use one GPU for each pre-training experiment.
Following the practice of SimSiam, the learning rate of the predictor is ﬁxed during the training.
We use warmup training for the ﬁrst 10 epochs.
If not speciﬁed, by default we train the model for 1000 epochs.
Online linear evaluation: For the online linear revaluation, we also follow the practice in the solo-learn library (da Costa et al., 2021).
The frozen features (2048 dimensions) from the training set are extracted (from the self-supervised pre-trained model) to feed into a linear classiﬁer (1 FC layer with the input 2048 and output of 100).
The test is performed on the validation set.
The learning rate for the linear classiﬁer is 0.1.
A. T WO SUB - PROBLEMS IN AO OF S IM S IAM
In the sub-problem η t ← arg min η L ( θ t , η ) , η t indicating latent representation of images at step t is actually obtained through η tx ← E T (cid:104) F θ t ( T ( x )) (cid:105) , where they in practice ignore E T [ · ] and sample only one augmentation T (cid:48) , i.e .
η tx ← F θ t ( T (cid:48) ( x )) .
Conceptually, Chen & He equate the role of predictor to EOA.
A. E XPERIMENTAL DETAILS FOR E XPLICIT EOA IN T ABLE
In the Moving average experiment, we follow the setting in SimSiam (Chen & He, 2021) without predictor.
In the Same batch experiment, multiple augmentations, 10 augmentations for instance, are applied on the same image.
With multi augmentations, we get the corresponding encoded representation, i.e .
z i , i ∈ [1 , 10] .
We minimize the cosine distance between the ﬁrst representation zand the average of the remaining vectors, i.e . ¯ z = 19 (cid:80) 10 i =
z i .
The gradient stop is put on the averaged vector.
We also experimented with letting the gradient backward through more augmentations, however, they consistently led to collapse.
A. E XPERIMENTAL SETUP AND RESULT TREND FOR T ABLE
Mirror SimSiam.
Here we provide the pseudocode for Mirror SimSiam.
In the Mirror SimSiam experiment which relates to Fig.
1 (c).
Without taking symmetric loss into account, the pseudocode is shown in Algorithm 1.
Taking symmetric loss into account, the pseudocode is shown in Algorithm 2.
[Table] [Table]
-normalize return -(z*p).sum(dim=1).mean()
Symmetric Predictor.
To implement the SimSiam with Symmetric Predictor as in Fig.
2 (b), we can just perceive the predictor as part of the new encoder, for which the pseudocode is provided in
Algorithm 3.
Alternatively, we can additionally train the predictor similarly as that in SimSiam, for which the training involves two losses, one for training the predictor and another for training the new encoder (the corresponding pseudocode is provided in Algorithm 4).
Moreover, for the second implementation, we also experiment with another variant that ﬁxes the predictor while optimizing the new encoder and then train the predictor alternatingly.
All of them lead to collapse with a similar trend as long as the symmetric predictor is used for training the encoder.
For avoiding redundancy, in Fig.
8 we only report the result of the second implementation.
# f: encoder (backbone + projector)
# h: predictor for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x)
# augmentation z_a, z_b = f(x_a), f(x_b)
# projections p_a, p_b = h(z_a), h(z_b)
# predictions
L = D(p_a, p_b)/2 + D(p_b, p_a)/2 # loss
L.backward() # back-propagate update(f, h) # SGD update def D(p, z): # negative cosine similarity z = z.detach() # stop gradient p = normalize(p, dim=1)
# l2-normalize z = normalize(z, dim=1)
# l2-normalize return -(p*z).sum(dim=1).mean()
# f: encoder (backbone + projector)
# h: predictor for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x)
# augmentation z_a, z_b = f(x_a), f(x_b)
# projections p_a, p_b = h(z_a), h(z_b)
# predictions d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach())
# detached predictor output # predictor training loss L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/ # encoder training loss L_enc =
D(p_a, d_p_b)/2 + D(p_b, d_p_a)/
L = L_pred + L_enc
L.backward() # back-propagate update(f, h) # SGD update def D(p, z): # negative cosine similarity with detach on z z = z.detach() # stop gradient p = normalize(p, dim=1)
# l2-normalize z = normalize(z, dim=1)
# l2-normalize return -(p*z).sum(dim=1).mean()
Result trend. The result trend of SimSiam, Naive Siamese, Mirror SimSiam, Symmetric Predictor are shown in Fig.
We observe that all architectures lead to collapse except for SimSiam.
Mirroe
SimSiam was stopped in the middle because a NaN value was returned from the loss.
[Figure]
, Symmetric Predictor.
A. E XPERIMENTAL DETAILS FOR INVERSE PREDICTOR .
In the inverse predictor experiment which relates to Fig.
2 (c), we introduce a new predictor which has the same structure as that of the original predictor.
The training loss consists of 3 parts: predictor training loss, inverse predictor training and new encoder (old encoder+predictor) training.
The new encoder F consists of the old encoder f + predictor h .
The practice of gradient stop needs to be considered in the implementation.
We provide the pseudocode in Algorithm 5.
A. R EGULARIZATION LOSS
Following Zbontar et al.
(2021), we compute covariance regularization loss of encoder output along the mini-batch.
The pseudocode for de-correlation loss calculation is put in Algorithm 6.
# f: encoder (backbone + projector)
# h: predictor # h_inv: inverse predictor for x in loader: # load a minibatch x with n samples x_a,
x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b)
# projections p_a, p_b = h(z_a), h(z_b)
# predictions d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach())
# detached predictor output # predictor training loss
L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/2 # to train h inv_p_a, inv_p_b = h_inv(p_a.detach()), h_inv(p_b.detach())
# to train h_inv # inverse predictor training loss
L_inv_pred = D(inv_p_a, z_a)/2 + D(inv_p_b, z_b)/ # encoder training loss L_enc =
D(p_a, h_inv(p_b))/2 + D(p_b, h_inv(p_a))
L = L_pred + L_inv_pred + L_enc
L.backward() # back-propagate update(f, h, h_inv) # SGD update def D(p, z): # negative cosine similarity with detach on z z = z.detach() # stop gradient p = normalize(p, dim=1)
# l2-normalize z = normalize(z, dim=1)
# l2-normalize return -(p*z).sum(dim=1).mean() like Pseudocode: De-correlation loss
[Table]
A. G RADIENT DERIVATION AND TEMPERATURE ANALYSIS FOR I NFO NCE
With · indicating the cosine similarity between vectors, the InfoNCE loss can be expressed as
[Equation]
where N indicates the number of negative samples and Z
=
Z b for simplifying the notation.
By treating Z a · Z i as the logit in a normal CE loss, we have the corresponding probability for each negative sample as λ i =
[Equation]
) , where i = 0 , 1 , 2 , ..., N and we have (cid:80) Ni =
λ i = 1 .
The negative gradient of the InfoNCE on the representation Z a is shown as
[Equation]
(6) where 1 τ can be adjusted through learning rate and is omitted for simple discussion.
With Z b as the basic gradient, G e = − o z − (cid:80) Ni =
λ i r i , for which o e = − o z and r e = − (cid:80) Ni =
λ i r i .
When the temperature is set to a large value, λ i =
[Equation]
approaches 1 N +1 , indicated by a high entropy value (see Fig. 7). InfoNCE will degenerate to a simple contrastive loss, i.e . L simple = − Z a · Z b +
[Equation]
N
[Equation]
{+Ni =
+}
Z a · Z i , which repulses every negative sample with an equal force.
In contrast, a relative smaller temperature will give more relative weight, i.e . larger λ , to negative samples that are more similar to the anchor ( Z a ).
The inﬂuence of the temperature on the covariance and accuracy is shown in Fig. 7 (b) and (c).
We observe that a higher temperature tends to decrease the effect of de-correlation, indicated by a higher covariance value, which also leads to a performance drop.
This veriﬁes our hypothesis regarding on how r e in InfoNCE achieves de-correlation because a large temperature causes more balanced weights λ i , which is found to alleviate the effect of de-correlation.
For the setup, we note that the encoder is trained for 200 epochs with the default setting in Solo-learn for the SimCLR framework.
A. T HEORETICAL DERIVATION FOR A SINGLE BIAS LAYER
With the cosine similarity loss deﬁned as Eq 7 Eq 8: [Equation] for which the derived gradient on the vector a is shown as
[Equation]
The above equation is used as a prior for our following derivations.
As indicated in the main manuscript, the encoder output z a is l 2 -normalized before feeding into the predictor, thus p a = Z a + b p , b p denotes the bias layer in the predictor.
The cosine similarity loss (ignoring the symmetry for simplicity) is shown as
[Equation]
The gradient on p a is derived as
[Equation]
m = cossim ( Z
[Equation]
, Z [Equation] Given that p a = Z a + b p , the negative gradient on b p is the same as that on p a as
[Equation] We assume that the training is stable and the bias layer converges to a certain value when − ∂cossim ( Z a , Z b ) ∂ b p = 0 .
Thus, the converged b p satisﬁes the following constraint:
[Equation]
With a batch of samples, the average of 1 m r b and r a is expected to be close to 0 by the deﬁnition of residual vector.
Thus, the bias layer vector is expected to converge to:
[Equation]
Rational behind the high similarity between b p and o z . The above theoretical derivation shows that the parameters in the bias layer are excepted to converge to a vector 1 − mm o z .
This theoretical derivation justiﬁes why the empirically observed cosine similarity between b p and o z is as high as 0.99.
Ideally, it should be 1, however, such a small deviation is expected with the training dynamics taken into account.
Rational behind how a single bias layer prevents collapse.
Given that p a = Z a + b p , the negative gradient on Z a is shown as
[Equation]
Here, we highlight that since the loss − Z a · Z a = − 1 is a constant having zero gradients on the encoder, − cossim ( Z a , Z b ) (cid:107) p a (cid:107) Z a can be seen as a dummy term.
Considering Eq 13 and m = cossim ( Z a , Z b ) (cid:107) p a (cid:107) , we have b =
( (cid:107) p a (cid:107) cossim ( Z a , Z b ) − 1) o z .
The above equation is equivalent to
[Equation]
With Z b as the basic gradient, the extra gradient component G e = − (1 − cossim ( Z a , Z b ) (cid:107) p a (cid:107) )
o z . Given that p a =
Z a + b p and (cid:107) Z a (cid:107) = 1 , thus (cid:107) p a (cid:107) < 1 only when Z a is negatively correlated with b p .
In practice, however, Z a and b p are often positively correlated to some extent due to their shared center vector component.
In other words, (cid:107) p a (cid:107) > 1 .
Moreover, cossim ( Z a , Z b ) is smaller than 1, thus − (1 − cossim ( Z a , Z b ) (cid:107) p a (cid:107) ) < 0 , suggesting G e consists of negative o z with the effect of de-centerization.
This above derivation justiﬁes the rationale why a single bias layer can help alleviate collapse.
B D ISCUSSION : DOES BN HELP AVOID COLLAPSE ?
To our knowledge, our work is the ﬁrst to revisit and refute the explanatory claims in (Chen & He, 2021).
Several works, however, have attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam.
The success has been ascribed to BN in (Fetterman & Albrecht, 2020), however, (Richemond et al., 2020) refutes their claim.
Since the role of intermediate BNs is ascribed to stabilize training (Richemond et al., 2020; Chen & He, 2021), we only discuss the ﬁnal
BN in the SimSiam encoder.
Note that with our Conjecture1, the ﬁnal BN that removes the mean of representation vector is supposed to have de-centering effect.
BY default SimSiam has such a
BN at the end of its encoder, however, it still collapses with the predictor and stop gradient.
Why would such a BN not prevent collapse in this case?
Interestingly, we observe that such BN can help alleviate collapse with a simple MSE loss.
Note that the cosine loss is in essence equivalent to a MSE loss on the l 2 -normalized vectors.
This phenomenon can be interpreted as that the l 2 -normalization causes another mean after the BN removes it.
Thus, with such l 2 -normalization in the MSE loss, i.e . adopting the default cosine loss, it is important to remove the o e from the optimization target.
The results with the loss of − Z a · sg ( Z b + o e ) in Table 3 show that this indeed prevents collapse and veriﬁes the above interpretation.