Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-supervised Learning
Abstract
Recent non-contrastive methods for self-supervised representation learning showpromising performance.
While they are attractive since they do not need negativesamples, it necessitates some mechanism to avoid collapsing into a trivial solution.
Currently, there are two approaches to collapse prevention.
One uses an asymmetricarchitecture on a joint embedding of input, e.g., BYOL and SimSiam, and the otherimposes decorrelation criteria on the same joint embedding, e.g., Barlow-Twins and VICReg.
The latter methods have theoretical support from information theory as towhy they can learn good representation.
However, it is not fully understood why theformer performs equally well.
In this paper, focusing on BYOL/SimSiam, whichuses the stop-gradient and a predictor as asymmetric tricks, we present a novelinterpretation of these tricks; they implicitly impose a constraint that encouragesfeature decorrelation similar to Barlow-Twins/VICReg.
We then present a novelnon-contrastive method, which replaces the stop-gradient in BYOL/SimSiam withthe derived constraint; the method empirically shows comparable performance tothe above SOTA methods in the standard benchmark test using ImageNet.
Thisresult builds a bridge from BYOL/SimSiam to the decorrelation-based methods,contributing to demystifying their secrets.
Introduction
Recently, many methods have been proposed for self-supervised learning of visual representation [1–6, 8–10, 12, 16]
They share a fundamental idea: to learn a visual representation of images invariantto a range of image transformations maintaining their semantics.
This idea is implemented as theoptimization of an objective that the joint embeddings of different views of an input image, e g.,different image subregions cut from a single image and subjected to additional data augmentation,should be close to each other.
The methods are categorized into contrastive and non-contrastive methods.
The latter needs onlypositive samples (i e., different views of the same images), whereas the former also needs negativesamples (i.e., different images).
Therefore, although it stabilizes learning, contrastive methods tendto need large memory or additional measures (e.g., a momentum encoder) to alleviate it.
On theother hand, the non-contrastive methods are potentially more efﬁcient.
However, their objective hasa trivial solution: to map all inputs into a single point in the feature space.
Thus, non-contrastivemethods need additional measures to prevent this feature collapse.
To do so, BYOL/SimSiam employs asymmetric structure into the joint embeddings, i.e., stop-gradientand a predictor; see Fig. 2(a).
The predictor is a subnetwork placed on top of one of the two jointembedding pipelines; it predicts the output of the other pipeline from its input.
BYOL/SimSiam usesthe loss of minimizing the prediction error, aiming at the above objective of learning invariant featurerepresentation.
It back-propagates the gradient only to the ﬁrst pipeline having the predictor while itstops the gradient ﬂow to the second pipeline.
(While BYOL inherited a momentum encoder for thesecond pipeline, later it was recognized [6, 14] that it is not indispensable.)
However, it has not been well understood why the above asymmetric structure prevents collapse andfurther enables to learn good representation.
Tian et al.
theoretically tackle this question, deriving [Figure] BYOL/SimSiam makes Σ approach to an identity matrix I , so do feature decorrelation methods (i.e., Barlow-Twins/VICReg).
This is also the case with the proposed method. See Sec. 5.2 for more details. [Figure]
/IHz
[Figure]
QPrTY/m</latexit> Backbone
[Figure]
[Table]
/NIpWKxuNdZQr0IDwULGcHaSL5drnb85KHPhKaSYI7Mq+rbFafmzIFWiZuTCuRo+vZXfxCTNKJCE46V6rlOor0JlpoRTqelfqpogskYD2nPUIEjqrzJfPkpOjXKAIWxNCU0mqu/JyY4UiqLAtMZYT1Sy95M/M/rpTq88iZMJKmmgiw+ClOOdIxmSaABk5RonhmCiWRmV0RGWGJiolAlE4K7fPIqaddr7kXt/K5eaVzncRThGE7gDFy4hAbcQhNaQCCDZ3iFN+vJerHerY9Fa8HKZ8rwB9bnD72JlDA=</latexit> W | p W p (d) Ours some results helping to understand the tricks [14].
Nonetheless, we still lack a complete understandingof the working mechanism of the asymmetry tricks, especially why they lead to the learning of goodrepresentation.
In this paper, we show that the asymmetry tricks employed by BYOL/SimSiam have an additionalimplicit effect, such that the features extracted from different images will be decorrelated.
Speciﬁcally, we extend Tian et al.’s
analyses under similar assumptions.
We then show that the updatingdynamics of the predictor’s weights and the features (i.e., the predictor’s inputs) indicates that theminimization implicitly imposes a constraint achieving feature decorrelation; see Fig.
Finally, weshow through experiments that we can eliminate stop-gradient by using the constraint as an explicitregularizer with the standardization of features.
Speciﬁcally, using the regularizer and the standardinvariance loss leads to learning as good representation as the state-of-the-art SSL methods, including
BYOL/SimSiam.
This result provides a link from BYOL/SimSiam to the other group of non-contrastive methods,i.e., Barlow-Twins [16] and VICReg [1].
These methods do not employ an asymmetric structureand instead incorporate an explicit objective of decorrelating features of different input images.
Asexplained in [16], information theory supports the goodness of feature decorrelation for representationlearning, explaining why it helps learn a good representation.
The link between the two groups ofmethods implies that the same is true for BYOL/SimSiam.
In short, it implies that the asymmetrytricks implicitly achieve feature decorrelation, leading to the learning of good representation withoutcollapse.
Related Work
2.1 Asymmetry Tricks: BYOL/SimSiam
BYOL [10] and SimSiam [6] employ asymmetric architectures; see Fig. 2(a).
Let f 1 , f 2 ∈ R d bethe outputs of an identical network for two views of an identical image x .
BYOL minimizes thefollowing loss:
[Equation]
where p ( · ) is a predictor, e.g., a two-layer MLP with ReLU and batch normalization in the intermediatelayer; and StopGrad indicates that the gradients are not back-propagated with the second path for f 2 .
There are several differences between BYOL and SimSiam; while BYOL uses a momentum encoderto compute f 2 , SimSiam does not, and SimSiam uses a loss based on cosine similarity.
It is unclear why BYOL and SimSiam can avoid collapsing to a trivial solution even though theyonly impose augmentation invariance.
In [14], incorporating some simpliﬁcation of the model andassumptions on the data, the authors analyze the dynamics of how the above minimization updates thenetwork weights.
They then show that i) there is an implicit balancing effect between the projector’sweights and the predictor’s weight, which may help prevent collapse; ii) stop-gradient is indispensablefor collapse prevention.
They then present a method named DirectPred, which directly sets the weightof a linear predictor based on the principal component analysis of the extracted features.
However,these do not fully explain the working mechanism of BYOL/SimSiam, especially why they can learngood representation.
2.2 Decorrelation-based: Barlow -Twins/VICReg
As shown in Fig. 2(b), Barlow-Twins [16] minimizes the following loss:
[Equation]
where F 1 ( ∈ R d × n ) and F 2 ( ∈ R d × n ) are matrices storing all the f 1 i ’s and f 2 i ’s of a batch of inputsx
i ’s
( i = 1 , . . . , n ) , respectively; corr( F 1 , F 2 )( ∈ R d × d ) is the cross-correlation between F 1 and
F 2 along the batch dimension and I is the identity matrix.
Note that Zbontar et al.
separate the lossinto the terms of diagonal and non-diagonal components of corr( F 1 , F 2 ) with different weightingconstants, which is omitted in (2) for brevity.
As shown in Fig. 2(c), VICReg [1] minimizes the following loss:
[Equation]
where C ( F 1 ) and C ( F 2 ) are the auto-covariance matrices of f 1 and f 2 , respectively, and µ and ν areweighting constants.
As shown above, VICReg separates the diagonal and non-diagonal componentsof the matrices and assigns different weights, as in Barlow-Twins [16], and employs a hinge loss forthe diagonal term.
Barlow-Twins and VICReg share a similar objective, i.e., decorrelating features of different inputs,although the targets are slightly different (correlation vs. covariance).
Roles of Asymmetry Tricks
In this section, we consider what role(s) the asymmetry tricks, i.e., a predictor and stop-gradient, play.
We extend the results of the study [14], showing a new interpretation.
3.1 Problem Statement
We succeed the assumptions/settings employed in [14] except an assumption on the data distribution.
Namely, we approximate the backbone + projector (i.e., the mapping from an input image x to itsfeature f ) to be a linear mapping.
We then consider (a variant of) BYOL with a linear predictorminimizing the following (cid:96) 2 loss:
[Equation]
where E x is the expectation over the distribution of an input image x ; f 1 and f 2 ( ∈ R d ) are theoutputs of the projector for two different views of x ; W p is the weight of the linear predictor.
Notethat the original version of BYOL employs a different loss (1), and a nonlinear predictor, i.e., an MLPhaving two or more layers with ReLUs.
We do not use the assumption on x ’s distribution employedin [14] and instead assume each x to be normalized, i.e., x (cid:62) x = 1 .
In the implementation of BYOL/SimSiam and others, E x [ · ] is replaced with the average over amini-batch.
Using F 1 = [ f 11 , . .
, f 1 n ] / √ n and F 2 =
[ f 21 , . .
. , f 2 n ] / √ n , where n is the mini-batchsize, we have F 1 F 1 (cid:62)
≈ E
x [ f 1 f 1 (cid:62) ] etc.
As a result, (4) is rewritten as
[Equation]
[Equation] BYOL/SimSiam use not only the orientation F 1 → F 2 but also the other way F 2 → F 1 ; letting L (cid:48)be the loss for the second way, they minimize L + L (cid:48) .
For brevity, we show only L in what followsunless otherwise noted.
Finally, our goal is to understand how W p are updated and how F i ’s behave as a result whenminimizing (4), or equivalently (5).
3.2 Dynamics of W p and F 1
Following [14], we regard the network weights varying during loss minimization as time-dependentvariables, e.g., W p = W p
( t ) .
We do this also for intermediate layer outputs, e.g., F 1 = F 1 ( t ) .
Theorem 3.1.
When minimizing the loss (5) with weight decay η , W p ( t ) and F 1 ( t ) satisfy
[Equation]
where C is a constant matrix determined by the initial weights.
To prove Theorem 3.1, we need the following lemmas.
Lemma 3.2.
The derivatives of the loss (5) with respect to W p and F 1 can be respectively given asfollows:
[Equation]
Lemma 3.3.
When minimizing the loss (5) with weight decay η under the above assumptions, F 1 ( t )is updated with the following velocity:
[Equation]
We show the proof of these lemmas in the supplementary material.
Now we prove Theorem 3.1.
Proof of Theorem 3.1.
When minimizing the loss (5) with gradient descent and weight decay, thevelocity of W p and F 1 during the minimization are respectively given by
[Equation]
We then calculate W (cid:62) p ˙ W p − ˙ F 1 (cid:62) F 1 , to which we substitute (7) and (8), yielding [Equation]
[Equation]
Then, adding the above with its transposed version and then taking an integral over t , we have
[Equation]
13) where C is a constant matrix determined by the initial values W p (0) and F 1 (0) .
The multiplicationwith e − 2 ηt yields (6).
For a large t , (6) reduces to
[Equation]
This means that minimizing (5) makes W (cid:62) p W p approach to F 1 F 1 (cid:62) =
(1 /n ) (cid:80) ni
{+=
+}
f i f (cid:62) i , theuncentered covariance of f 1 , . . . , f n .
3.3 Relation to Feature Decorrelation
The equation (14) implies the decorrelation of features if W p ( t ) W p ( t ) (cid:62) is a diagonal matrix.
Weempirically found this is true; W p quickly converges to a diagonal matrix as t increases.
We showdetailed results in the supplementary material due to the lack of space.
This empirical observationtogether with the above theoretical result (14) implies the connection between BYOL/SimSiam withthe decorrelation-based methods.
Deriving a Method without StopGrad
4.1 Basic Idea
Aiming to validate the above implication, we consider eliminating StopGrad from (5) and insteadimpose (14) explicitly in training.
Speciﬁcally, we eliminate StopGrad and incoporate (cid:107) W (cid:62) p W p −
F 1 F 1 (cid:62) (cid:107) 2F as a regularization term as
[Equation]
The underlying thought is that since the presence of the asymmetric tricks leads to (14) in the analysisof the dynamics, we wish to see if the explicit imposition of (14) can substitute StopGrad, whichis the major difference between the two categories of non-contrastive methods.
We will show ourempirical ﬁndings below that this approach works well when used with either the standardizationof features (by adding a standardization layer on the projector’s output) or the elimination of thepredictor.
We will refer to the former as STD and the latter as I-PRED (i.e., identity predictor).
4.2 Some Consideration for Implementation
BYOL/SimSiam consider the symmetric loss, i.e., the sum of the losses in the both ways (i.e.,
F 1 ↔ F 2 ).
Incorporating the proposed regularizer in each way, the total loss will be
L
[Equation]
[Equation]
Now, we consider replacing F 1 and F 2 in the third and fourth terms as F 1 → W p F 2 and F 2 →
W p F 1 , respectively.
Assuming the ﬁrst and second terms are sufﬁciently small, we can say that thismodiﬁcation will not change the objective here.
Nevertheless, we found through our experimentsthat this modiﬁcation leads to better convergence.
We think that the modiﬁed loss must have somepositive effect on the updating dynamics of W p , particularly at the initial stage of training.
Furtheranalysis is left to a future study.
Going back to one-way expression (i.e., F 1 → F 2 ), we write the modiﬁed loss by pairing the ﬁrstand the fourth terms of (16) (and the second and the third terms in the other way) as follows:
[Equation]
where c inv and c cov are weighting constants of the two terms, and d is the feature dimension size (i.e.,the column size of F 1 ); the division by d makes the effective range of c cov similar to c inv .
We summarize the interpretation of the above loss here.
The ﬁrst term enforces augmentationinvariance of f 1 i and f 2 i for any
i (= 1 , . . . , n ) , and the second term enforces covariance maintaining,or speciﬁcally, avoiding the collapse of (uncentered) covariance of f 11 , . . . , f 1 n .
It should be noted again that the loss (17) does not have a factor directly preventing feature collapse,as with BYOL/SimSiam.
An obvious collapsing case is when the projector always outputs zero,i.e., f 1 i = f 2 i = 0 for any i .
In this case, the loss vanishes when W p
= O .
To avoid this, we applystandardization to the output of the projector, i.e., F 1 =
[ f 11 , . .
. , f 1 n ] .
Speciﬁcally, we compute
[Equation]
where µ = 1 n (cid:80) nj =1 f 1 j and σ 2 = 1 n − 1 (cid:80) nj =
( f 1 j − µ ) 2 , and set F 1 ← [ ¯ f 11 , . .
, ¯ f 1 n ] / √ n .
Weperform the same for F 2 .
Note that SimSiam [6] employs the same strategy, and interestingly, so does
Barlow-Twins; it is reported [1] that Barlow-Twins performs slightly worse without a standardizationlayer.
However, it is also known that standardization alone is not enough; the space of f 1 can bedegenerate, as reported in [13].
As will be shown later, we empirically found that the loss (17) plusthe feature standardization successfully prevents W p from being rank-deﬁcient.
4.3 (Dis)similarity with Other Methods
Our method does not use StopGrad , which is a stark difference from BYOL/SimSiam/DirectPred.
In this sense, our method may be more similar to decorrelation-based methods, i.e., BarlowTwins/VICReg.
Roughly speaking, we can say that Barlow-Twins enforces the cross-correlation between F 1 and
F 2 to be an identity matrix, and VICReg enforces the auto-covariance of F 1 (and of F 2 ) to be anidentity matrix.
In the same perspective, our method deals with the auto-correlation of F 1 (and F 2 );speciﬁcally, it enforces them to be the same (i.e., W (cid:62) p W p ).
This does not mean that our methoddecorrelates features unless W (cid:62) p W p is a diagonal matrix.
However, similar to BYOL as discussed in
Sec. 3.3, W p approaches a diagonal matrix with training iteration t in our method as well, althoughits effect is weaker than BYOL and others; see Fig. 1.
When W p =
I , this effectively eliminates the predictor, resulting in our method getting closer to
VICReg.
The remaining differences are as follows: hinged loss vs. (cid:96) 2 norm for the correlationconstraint term, a different treatment of diagonal/non-diagonal elements of the covariance matrix vs.their equal treatment, and the use of auto-covariance vs. auto-correlation.
As discussed in [16], there is an explanation based on information theory as to why decorrelatingfeatures leads to learning a good representation.
However, it is not a rigorous proof since the lossesof Barlow-Twins and VICReg differ from the ideal objective function suggested by the theory.
Thesame is true of our method.
The implication of the goodness of feature decorrelation may be effectiveon our method as well.
4.4 Partitioned Correlation Constraint Term
We found through preliminary experiments that partitioning the correlation constraint term yieldsbetter results.
To be speciﬁc, we partition F 1 , the set of features from a single batch, into k subsetsof an equal size n (cid:48)
= n/k , as F 1 = (1 / √ k )
[ F 11 , F 12 , ..., F 1 k ] .
We then modify the loss (17) as
[Equation]
[Table]
[Table]
We will experimentally show the effectiveness of this partitioned loss later.
Its theoretical validationwill be left to future research.
A small remark is that the partitioned loss is suitable for distributedtraining of the model in a data-parallel manner.
We need only to compute each partitioned termseparately on a single computational node, e.g., a GPU.
Note that this method differs from thosefound to be effective in [15].
Speciﬁcally, we do not introduce different variances for each partition,while theirs use different variances for multiple branches of the Siamese networks.
Our methodsynchronizes the mean and variance among all the nodes, unlike their asymmetrically synchronized“AsymBN.”
See the pseudo-code shown in the supplementary material for more details.
Experimental Results
5.1 Experimental Setting
Architecture We follow the previous studies for the network architecture.
Speciﬁcally, we use
Resnet-50 [11] without the last fully-connected layer (i.e., the classiﬁcation layer) as a backbone.
Following [1, 16], we zero-initialize the weights of the last batch normalization layer in each residualbranch (i.e. zero_init_residual=
True in PyTorch).
We then employ a projector on top of the backbone,which is a MLP with two or three layers basically having the same width, each intermediate layerhaving BatchNorm and ReLU.
Following the previous methods but BYOL, we synchronize all thebatch norm layers, including the standardization layer used for F 1 and F 2 , across different devices,e.g., using SyncBatchNorm of Pytorch.
Our method uses a predictor as with BYOL/SimSiam.
Notehowever that ours is a linear predictor, in contrast with BYOL/SimSiam whose default settings employnon-linear predictor, i.e., a two-layer MLP with ReLU at the intermediate layer.
Devices We use a machine equipped with two Intel Xeon Platinum 8360Y Processors, eight
NVIDIA A100 GPU and 520 GiB DDR4 RAM.
Augmentation We follow the procedure of BYOL [10] and Barlow-Twins [16] for data augmentation creating two views of input images.
Speciﬁcally, we build an augmentation pipeline in thefollowing order: random cropping followed by resizing to 224 × 224 , horizontal ﬂipping, colorjittering, grayscale convertion, Gaussian blurring, and solarization.
While random cropping withresizing is always applied, others are probabilistic ally applied with randomly chosen parameters.
Following the above papers, the application of blurring and solarization is asymmetric, i.e., theirprobabilities differ between the two views.
SimSiam and VICReg employ symmetric setting.
Optimization We also follow BYOL and Barlow-Twins for the optimizer and its hyperparametersettings.
To be speciﬁc, LARS is used along with cosine learning rate decay with 10-epoch linearwarm-up; weights in BatchNorm and bias are excluded from weight decay and LARS adaptation.
We set the base learning rate = 0 .
3 for the whole model, which is multiplied by BatchSize / 256 .
Wetrain our model for 100 epochs, and the employed learning rate decay to 0 .
at the 100-th epoch.
Weset the batch size to 2048 .
Evaluation For evaluation of methods, we follow the previous studies [1, 6, 10, 16].
Speciﬁcally,we use ImageNet (ILSVRC2012)
[7] for all the experiments.
We use the proposed self-supervisedlearning method to train the above backbone, along with a projector and a predictor, using all the [Table] images of the training split, where the above augmentation and optimization procedures are employed.
We then evaluate the method’s performance by training and testing a linear classiﬁer using the featuresextracted by the backbone.
We follow the standard evaluation protocol employed in the previousstudies.
Speciﬁcally, we train a linear classiﬁer with SGD with momentum = 0 .
9 , batch size = 256 ,weight decay = 10 − 6 , and learning rate subject to cosine scheduling with the base rate = 0 .
3 , for100 epochs.
5.2 Feature Decorrelation
The previous sections show that the asymmetry tricks implicitly enforce feature decorrelation.
Toexperimentally validate this, we analyze how the auto-correlation matrix Σ of extracted features (i.e.,f 1 i ’s) changes during training.
For input images, we use all the images of the ImageNet validationsplit without applying random data augmentation.
Figure 1 shows how (cid:107) Σ − I (cid:107) 2 changes during the 249 training of a network by different methods.
We can see that all the tested methods decrease (cid:107) Σ − I (cid:107) 2 250 with epochs, including those using the asymmetry tricks (i.e., BYOL and SimSiam) and those usingdifferent forms of explicit decorrelation constraint (i.e., Barlow-Twins, VICReg, and ours).
It isnoteworthy that this applies to the BYOL with a linear predictor (i.e., the assumed conﬁguration)
andalso BYOL/SimSiam with a non-linear predictor.
5.3 Performance with Different Conﬁgurations
Our method has three hyperparameters, i.e., the partition number k , the weight c inv of the invarianceterm, and the weight c cov of the correlation term, as in (19).
We conduct experiments to examine howtheir choice affects the performance using the above evaluation procedure (i.e., ImageNet and linearprobe evaluation).
We use a projector whose size is 2048 - 2048 - 2048 here.
Table 1 shows the resultsfor several combinations of these parameters.
We can observe that partitioning the correlation lossand setting the weights as c inv > c cov yield better performance.
The decorrelation-based methods, such as Barlow-Twins and VICReg, are reported to show betterperformance with a larger projector.
We examine our method’s dependency on the size of the projector.
Table 2 shows the results.
It is observed that a larger projector leads to better performance, similar to
Barlow-Twins/VICReg; this tendency persists for different hyperparameter settings.
Table 2 reportsa case where the training did not converge when using a small projector of 4096 - 256 and settingk = 32 , c inv = 25 , and c cov = 1 .
5.4 Feature Standardization, Predictor, and StopGrad
As explained in Sec.
4.2, we apply standardization to the features, intending to prevent collapse.
Weconduct experiments to verify its effects.
As shown in Fig.
1, our method shows a similar effect of feature decorrelation to existing methods,which is caused by W p ’s behaviour that W p W (cid:62) p approaches diagonal.
This raises a question what ifwe set W p
=
I from the beginning.
Setting W p
= I means that we do not use a predictor, makingour method further closer to the decorrelation methods.
Thus, we additionally examine this in theexperiments.
[Table]
Table 3 shows the results.
(We ﬁxed k = 1 and c inv = 1 and set c cov =
or 1 for simplicity.
Theprojector size is set to 2048-2048-2048.)
First, we can conﬁrm that the feature standardization helpsprevent collapse independently of the presence of the correlation term (i.e., c cov = 0 or 1).
Second, we can see that ﬁxing W p
=
I , or equivalently eliminating the predictor, works well at leastwith this speciﬁc conﬁguration.
However, our method having a learnable predictor with a differentconﬁguration achieves the best performance, as will be shown later.
Third, “ StopGrad ” in the table indicates the case of applying StopGrad similar to BYOL/SimSiam.
Comparing the result with/without it under the same conﬁguration (i.e., c inv = 1 and c cov = 0 ), wecan see StopGrad improves the performance, i.e., 50.9% → 55.0%.
Switching StopGrad with ourcorrelation term yields further better results, i.e., 55.0% → 61.4%.
Finally, we can see that simply removing StopGrad from BYOL/SimSiam will lead to collapse.
We need an additional method to prevent collapsing, i.e., either a standardization layer or COV +
I -PRED (i.e., the correlation constraint term plus the elimination of the predictor).
5.5 Comparison with Previous Methods
Table 4 compares the best performance of our method with other methods 1 .
As methods’ performancevaries depending on the size of projectors, we compare them for a ﬁxed projector size.
Withthe projector size of 2048 - 2048 - 2048 , our method achieves 67.3%, which is better than BarlowTwins (63.4%) and VICReg (65.1%) and slightly lower than SimSiam (68.1%); SimSiam’s higherperformance may be attributable to the use of a non-linear predictor.
With 8192 - 8192 - 8192 , ourmethod outperforms Barlow-Twins and VICReg again, i.e., 69.0% vs. 68.6%.
BYOL tends to learnslower due to the use of a momentum encoder and performs worse for epochs = 100 .
Conclusion and Discussion
In this paper, we ﬁrst theoretically showed (under some assumptions) that minimizing the loss (4)with StopGrad and a predictor (i.e., BYOL/SimSiam) leads to that (a) W (cid:62) p W p → F 1 F 1 (cid:62) as t → ∞ .
We then showed an empirical observation with BYOL (and many others including ours) that (b) W (cid:62) p W p → diagonal with increasing t .
We proposed a novel method incorporating an explicitregularizer to enforce (a) while removing StopGrad.
Finally, we presented experimental resultsthat the method works; it learns as good representation as BYOL/SimSiam even though it does notuse StopGrad.
We have also presented some extensions (Sec. 4.2 and 4.4) that empirically helpconvergence and better learning.
These all point to the connection between BYOL/SimSiam and thedecorrelation-based methods.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.4.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See supplementarymaterial.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] But we provide
our pseudo-code in Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]