Z ERO -CL: I NSTANCE AND F EATURE DECORRELATION FOR NEGATIVE - FREE SYMMETRIC CONTRASTIVE LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
For self-supervised contrastive learning, models can easily collapse and generate trivial constant solutions.
The issue has been mitigated by recent improvement on objective design, which however often requires square complexity either for the size of instances ( O ( N 2 ) ) or feature dimensions ( O ( d ) 2 ).
To prevent such collapse, we develop two novel methods by decorrelating on different dimensions on the instance embedding stacking matrix, i.e., I nstance-wise (ICL) and F eaturewise (FCL) C ontrastive L earning.
The proposed two methods (FCL, ICL) can be combined synthetically, called Zero-CL, where “Zero” means negative samples are zero relevant, which allows Zero-CL to completely discard negative pairs i.e., with zero negative samples.
Compared with previous methods, Zero-CL mainly enjoys three advantages: 1) Negative free in symmetric architecture.
By whitening transformation, the correlation of the different features is equal to zero, alleviating information redundancy.
3) Zero-CL remains original information to a great extent after transformation, which improves the accuracy against other whitening transformation techniques.
Extensive experimental results on CIFAR-10/ and ImageNet show that Zero-CL outperforms or is on par with state-of-the-art symmetric contrastive learning methods.
I NTRODUCTION
One of the current main bottlenecks in deep network training is the dependence on heavy annotated training data, and this motivates the recent surge of interests in unsupervised (Donahue &
Simonyan, 2019) and self-supervised (Chen & He, 2021; Chen et al., 2020) methods.
Speciﬁcally, in self-supervised representation learning (SSL), a network is pre-trained without any form of manual annotation, thus providing a means to extract information from unlabeled data sources (e.g., text corpora, videos, images from the Internet, etc.).
In self-supervision, label-based information is replaced by a prediction problem using a certain context or using a pretext task.
Pretext task in SSL can mainly be divided into three categories: 1) Generative based approaches (Donahue & Simonyan, 2019) learn to generate or otherwise model pixels in the input space.
However, pixel-level generation is computationally expensive and may not be necessary for representation learning.
Contextual based methods (Vincent et al., 2008; Pathak et al., 2016; Ye et al., 2019) design pretext tasks (denoising auto-encoders (Vincent et al., 2008), context auto encoders (Zhang et al., 2016; 2017), etc).
3) Contrastive based methods (Chen et al., 2020;
Grill et al., 2020; Caron et al., 2020; Asano et al., 2019) take augmented views of the same image as positive pair and others as negative pairs.
Generally, one positive sample corresponds to lots of negative samples.
In recent works, contrastive based methods have shown great promise, achieving state-of-the-art results in image classiﬁcation (Chen et al., 2020), video classiﬁcation (Han et al., 2020) and other downstream tasks (Chen & He, 2021).
However, trivial constant solutions (different samples get the same representation) is easily happening without the proper design of architecture and objective function.
The well-known solutions to avoid this problem can be summarized into two parts: asymmetric model architecture and proper objective function.
1) Model architecture: MoCo (He et al., 2020), BYOL (Grill et al., 2020) update encoders separately and stopping gradient operation is adopted to avoid such problem.
Then,
BYOL and SimSiam (Chen & He, 2021) introduce a predictor module to avoid collapse, which is composed by MLP (Goodfellow et al., 2016).
Current mainstream interpretation is using the predic- tor to construct an asymmetric structure, which is useful to alleviate trivial solutions.
Objective function: SimCLR uses symmetric framework in contrastive learning.
They prevent trivial solutions by using negative pairs and InfoNCE, where InfoNCE can be divided into an alignment term and a uniformity term (Arora et al., 2019).
The uniformity term pulls different samples to a hyper-sphere uniformly, forcing obtaining different representations and avoiding trivial solutions.
Recently, Barlow Twins (Zbontar et al., 2021) designs a new objective function from the information redundancy perspective, which also has two terms (an invariance term and a redundancy reduction term).
The invariance term maximizes the correlation of the same feature across different views, and the redundancy term reduces information redundancy.
However, for the symmetric framework, both SimCLR and Barlow Twins require square order complexity in objective functions, and the main complexity comes from the uniformity and redundancy term.
In this paper, we propose two methods, named
Zero-ICL and Zero-FCL, where Zero-ICL discards the uniformity term and only requires O ( N ) complexity by instance-wise whitening.
Correspondingly, Zero-FCL discards the redundancy term by feature-wise whitening and only requires O ( d ) complexity.
Our contributions are :
1) We propose a new contrastive learning framework to prevent trivial solutions, Zero-CL, which includes two parts, i.e., Zero-ICL (instance-wise) and Zero-FCL (feature-wise), either of which can
work
independently and only requires linear order complexity (without negative pairs).
To our best knowledge, Zero-ICL is the ﬁrst attempt of instance-wise whitening, which is conceptually comprehensible for preventing collapses in contrastive learning.
Note that most previous methods (including other domains beyond vision) e.g. (Eldar & Oppenheim, 2003; Kessy et al., 2018) only use whitening transformation to reduce the information redundancy on feature-wise. 3)
We give empirical analysis on the relationship between previous methods and our Zero-CL, where previous negative sample consuming methods (Zbontar
et al., 2021; Chen et al., 2020) can be regarded as our method with Lagrangian transformation.
Then, we theoretically introduce ZCA-based whitening from the maximal correlation (Kessy et al., 2018) perspective.
4) Experimental results on standard image benchmarks (CIFAR-10/100 and ImageNet-100/1k) show our method achieves new state-of-the-art results for symmetric contrastive learning compared with (Chen et al., 2020; Zbontar et al., 2021), especially for small hidden dimension and batch size.
R ELATED W ORK
We review recent contrastive learning works.
We mainly divide the previous contrastive methods into feature-wise and instance-wise according to different contrastive dimensions.
We particularly discuss the similarity and differences of some most related methods in Section 5 in detail.
instance-wise contrastive learning.
instance-wise contrastive learning aims to attract positive pairs and repulse negative pairs (Chen et al., 2020;
He et al., 2020), where each pair is composed of two instance views.
One of the widely used objective functions in instance-wise learning is InfoNCE, and extensive experiments in (Oord et al., 2018) show that InfoNCE requires negative pairs to avoid trivial solutions.
SimCLR (Chen et al., 2020) regards views augmented from different images in a mini-batch as negative pairs.
However, SimCLR requires a large batch size to improve the accuracy, which is GPU intensive.
To address this issue, MoCo (He et al., 2020) proposes storing negative samples in the memory bank and updates the bank by the ﬁrst-in-ﬁrst-out principle.
MoCo further proposes two operations to prevent trivial solutions, i.e., momentum update key encoder and stop gradient.
In recent works (Chen et al., 2020; Grill et al., 2020), the mainstream explanation of the two operations is they construct an asymmetric framework, which can avoid trivial solutions.
Inspired by this, BYOL (Grill et al., 2020) adopts stop gradient operation and EMA algorithm to update target encoder.
Further, BYOL introduces a predictor module to build a more asymmetric framework.
Then, SimSiam (Chen & He, 2021) empirically shows that stop gradient and the predictor is the key component to avoid trivial solutions.
Another approach to preventing such solutions is introduced in
W-MSE (Ermolov et al., 2021).
Different from previous works
(Chen et al., 2020; Chen & He, 2021;
He et al., 2020), W-MSE requires multiple augmented views to improve the accuracy.
Besides, WMSE simply borrows Cholesky-based whitening transformation in SSL and the multi-view objective requires O ( NM 2 ) complexity, where M is the number of views.
feature-wise contrastive learning.
Different from instance-wise learning, feature-wise methods contrast on feature dimension.
Barlow Twins (Zbontar et al., 2021) is the ﬁrst feature-wise method, which pushes the cross-correlation matrix to identity matrix I .
Its objective includes both invariance
[Figure]
term and redundancy reduction term, where the latter term can avoid trivial solutions.
Inspired by
Barlow Twins, VICReg (Bardes et al., 2021) proposes a new regularization term named invariancecovariance on feature dimension, and achieves similar results to Barlow Twins.
However, both
Barlow Twins and VICReg require large contrastive dimensions and the covariance regularization requires O ( d 2 ) complexity, which could be also regarded as requiring negative pairs on feature dimension (each negative pair is composed of features in different dimensions across images).
Whitening transformation.
Whitening, or sphering, is a common pre-processing step to transform random variables to orthogonality.
However, due to rotational freedom, there are inﬁnitely many possible whitening procedures.
There is a diverse range of sphering methods, e.g. principal component analysis (PCA)
(J ´ egou & Chum, 2012), Cholesky matrix decomposition (Higham, 1990) and zero-phase component analysis (ZCA) (Bell & Sejnowski, 1997).
T HE P ROPOSED Z ERO -CL
In this section, we present the framework of Zero-CL in detail, starting with the whitening transformation procedure, followed by the model’s framework as well as the objective function.
At last, we provide theoretical analysis on whitening from maximal correlation perspective.
3. W HITENING TRANSFORMATION
Given a set of images { x } Ni =1 , we extract an embedding in contrastive space z i
= g ( f ( x i , θ ) , γ ) using an encoder network (He et al., 2016)
f ( · , θ ) and a MLP module g ( · , γ ) , where z i ∈ R 1 × d .
Denote the set of embeddings as Z ∈ R N × d , whitening transformation can be formulated as:
[Equation]
where H ∈ R d × N is the whitened embedding matrix.
The square matrix W ∈ R d × d is the socalled whitening matrix.
Since var ( H )
=
I , it follows that WΣW =
I and thus W ( ΣW (cid:62) W ) =
W , which is fulﬁlled if W satisﬁes W (cid:62) W = Σ − 1 .
However, the constraint does not uniquely determine the whitening matrix W .
Speciﬁcally, for any W rot = RW with orthogonal matrix, R will also be a whitening transformation matrix.
Here we present ZCA whitening matrix in detail.
ZCA-based whitening takes E (stacks together eigenvectors of the covariance matrix) as the orthogonal matrix R .
Then ZCA whitening matrix can be written as:
[Equation]
By rotation matrix E , the whitened data will be as close as possible to original data, to which we will give theoretical analysis later (see Theorem 1).
3. S YMMETRIC OBJECTIVE FUNCTION BACKGROUND
For instance-wise contrastive learning, to avoid trivial solutions in the symmetric framework, the objective function usually can be divided into alignment and uniformity terms (Arora et al., 2019).
𝑋
[Figure]
𝑋 𝑋 !
𝑋 " 𝑇 ~
𝒯
[Figure]
𝑋 𝑋 !
𝑋 "
[Figure]
~ 𝒯
[Figure]
𝑋 𝑋 !
[Figure]
We now review the loss formulated in SimCLR (Chen et al., 2020) as:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Equation]
where τ is the temperature hyper-parameter.
x
+ and x − are positive and negative samples, respectively.
The ﬁrst term is the so-called alignment, which maximizes the similarity between positive pairs.
The second term is the key to prevent trivial solution, pushing all the data points on a unit hyper-sphere uniformly.
Then, we rewrite the feature-wise objective of Barlow Twins as:
[Equation]
where C ∈ R d × d is the covariance matrix.
The ﬁrst term encourages correlation of the same feature across different views, while the second term minimizes correlation of different features.
We can observe that the objective of both instance-wise and feature-wise methods requires square order calculation complexity, where the ﬁrst term (alignment, invariance) requires linear order complexity and the second term (uniformity, redundancy reduction) requires square order complexity.
We wonder if we can discard the second term, i.e., completely discard negative pairs.
3. INSTANCE - WISE AND FEATURE - WISE WHITENING
Inspired by W-MSE (Ermolov et al., 2021), we propose two efﬁcient methods in SSL, which are named Zero-ICL and Zero-FCL.
Zero-ICL can replace the uniformity term in Eq.
3, while ZeroFCL can replace the redundancy reduction term in Eq.
Instance-wise whitening.
Given a set of centered embeddings Z ∈ R N × d , different from traditional whitening methods, we calculate the afﬁnity matrix by S = ZZ (cid:62) , where S ∈ R N × N .
Then
the
ZCA whitening matrix W I − ZCA can be calculated by:
[Equation]
(5) where E ∈ R N × N stacks eigenvector of the afﬁnity matrix and Λ S represents the diagonal variance matrix.
One of the questions is how this transformation replaces the uniformity term.
We denote the whitened embedding matrix as H , we have:
[Equation]
where E is a rotation orthogonal matrix, i.e., E (cid:62) E
= EE (cid:62)
=
I .
Thus, the result of Eq.
6 is
I ∈ R N × N , which means embeddings of different samples are strictly orthogonal.
Then, we can replace the second part in InfoNCE with instance-wise whitening transformation.
Feature-wise whitening.
Similar to instance-wise whitening transformation, it is calculated by:
[Equation]
where C is the covariance matrix, Λ C and E ∈
R d × d are the diagonal variance matrix and eigenmatrix of C , respectively.
Denote whitened embedding matrix as H ∈ R N × d , we have E [ H (cid:62) H ] =
I , i.e., the correlation of the different features is equal to 0.
Hence, redundancy reduction term in
Eq.
4 can be replaced with feature-wise whitening transformation.
3. F RAMEWORK AND OBJECTIVE FUNCTION
Like other methods in SSL (Chen et al., 2020; Zbontar et al., 2021; Chen & He, 2021), given a batch { X i } Ki =1 sampled from a dataset, we ﬁrst augment data via a distribution of data augmentation
T .
We term the two views as X A and X B .
The two batches are then fed to an encoder (He et al., 2016; Kipf & Welling, 2016)
f ( · , θ ) , producing batches of embeddings T A and T B .
Then, a projector module g ( · , γ ) is used to map representation to contrastive space.
The two batches of embeddings are then fed into projector g , producing Z A ∈ R K × d and Z B ∈ R K × d .
Since whitening transformation requires centered vector, we use standard scaling method to each batch of embeddings.
Note that to calculate instance-wise and feature-wise whitening transformation matrix, we must center embeddings on feature dimension and instance dimension, respectively.
We denote feature-wise and instance-wise centered embeddings as Z Fea and Z Ins as follow:
[Equation]
where Z i, · is the i -th vector in Z and Z · ,i is the vector composed of each value at dimension i in all embeddings in Z .
µ and σ are mean and variance of one vector.
(cid:15) is set 1e-4 as default.
Instance-wise objective.
Now we have two batches of instance-wise centered embeddings Z A,Ins and Z B,Ins .
By Eq.
5, we can obtain instance-wise whitened embeddings H A,Ins and H B,Ins .
Then, the instance-wise objective can be formulated as:
[Equation]
where H i,d represents the d -th feature value of the i -th instance.
Feature-wise objective.
Similar to Eq.
9, we can obtain feature-wise whitened embeddings
H A,C − ZCA and H B,C − ZCA via Eq.
Then the feature-wise objective can be formulated as:
[Equation]
where K denotes the batch size.
The overall loss can be formulated as:
[Equation]
We set the hyper-parameter λ = 1 in this paper.
Note that both two losses can work independently and we term L Ins only as Zero-ICL (instance-wise) and L Fea only as Zero-FCL (feature-wise).
3. E MPIRICAL ANALYSIS
Here for simplicity, we discuss feature-wise whitening.
From the redundancy reduction perspective, high-quality self-supervised embeddings require that: 1) embeddings are in a non-trivial constant distribution, where trivial distribution means all the embeddings collapse to a single point; 2) positive image pairs share similar semantics; and 3) correlation of the different feature should be zero.
Considering feature-wise SSL, for simplicity, denote i -th feature in Z as Z · ,i , where Z ∈ R N × d and
Z · ,i ∈ R N × 1 , we can formulate this problem as follows:
[Equation]
12) where dist ( · , · ) can be quantiﬁed as any similarity metric.
To solve the conditional optimization problem, previous works (Zhang et al., 2021a; Zbontar et al., 2021) transform the hard condition to soft condition via the Lagrangian multiplier.
The objective becomes:
[Equation]
However, the second term of the above objective function requires square order complexity.
Hence, we propose another approach to solve this hard conditional optimization problem, i.e., whitening transform the representations before calculating loss.
Then the objective of Eq.
12 is equal to
[Equation]
14) where H is the whitened embeddings.
Finally, our objective only uses O ( d ) complexity to solve the conditional problem, which is similar to the instance-wise branch.
3. T HEORETICAL ANALYSIS ON WHITENING
By Eq. 1, for any W rot = RW , which satisﬁes R (cid:62) R
=
I , W rot can be a whitening matrix.
For
SSL, whitening transformation aims to make different instances or features orthogonal, and also remains as much original information as possible.
The problem can be formulated as:
[Equation]
where R , Σ are the rotation matrix and covariance matrix.
Z and H are embeddings before and after whitening, respectively.
Since H and Z are two centered scaling matrices, we can rewrite the constraint as follows, and thus minimizing the variance between whitened data and original data equals to maximizing the second term:
[Equation]
Theorem 1 Maximization of tr ( REΛ − 1 / 2 E (cid:62) ) uniquely determines the rotation matrix R to be the identity matrix I .
Proof 1 Since Λ is a diagonal matrix, we can write Eq.
16 as:
[Equation]
= (cid:88) i Λ − 1 / 2 ii
A ii (17) where A = E (cid:62) RE .
Since R and E are both orthogonal, A is also an orthogonal matrix.
So we have A ii ≤ 1 .
Note that E is a rotation matrix, i.e., E (cid:62) E
= I .
Thus, if and only if R =
I , we can obtain the maximum of Eq.
Then, we can complete the proof.
(cid:3)
E XPERIMENTS
We use the following datasets.
The implementation and protocol details are given in Appendix.
[Table]
1) CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), two small-scale datasets, are composed of 32 × 32 images with 10 andclasses, respectively.
2) ImageNet-100 and ImageNet-1k
(Deng et al., 2009) sets include 100 and 1k classes, respectively.
The datasets are well-balanced in class distribution and the images contain an iconic view of objects, which are the commonly used benchmark in SSL (Zbontar et al., 2021).
Main comparison.
We mainly conduct experiments on CIFAR-10/100 and ImageNet-100/1k datasets.
For fair comparison, we set batch size as 128 on ImageNet-100, 1024 on ImageNet-1k, and 256 on CIFAR-10/100.
Table 1 and 2 show the best results on different datasets across different methods, where our method outperforms most of the prior arts with symmetric architecture and less complexity.
We modify our code based on Barlow Twins, and the reported results are strictly followed ofﬁcial code in this site 1 .
Note that on CIFAR-10 and CIFAR-100 datasets, we remove the ﬁrst maxpool layer and modify the ﬁrst convolutional layer with kernel size 3 and strides 1 in
ResNet-18, which are commonly used tricks in low-resolution datasets (Chen et al., 2020).
The MLP
[Table]
For fair comparisons, we set the dimension of Projection head as 128-128-128 and batch size as 256.
[Table] [Table]
(Projector) in our methods is quantiﬁed as three linear layers with two BNs (Ioffe & Szegedy, 2015) and ReLU activation function.
Table 2 shows the results on CIFAR and ImageNet-100 datasets.
We compare our Zero-FCL with Barlow Twins with the same hidden dimension.
With 256 hidden dimension, Zero-FCL outperforms Barlow Twins by 8.89% and 8.46% in top-1 accuracy on
CIFAR-100 and ImageNet-100, respectively.
For hidden dimension 2048, Zero-FCL outperforms
Barlow Twins 1.07% and 0.7% top-1 accuracy on CIFAR-100 and ImageNet-100 datasets.
Note that although Zero-ICL is an instance-wise method and the batch size is set only 128 or 256, Zero-ICL still gets 69.33% top-1 accuracy on CIFAR-100, which is 3.94% higher than SimCLR.
Convergence rate.
Many experiments (Gutmann & Hyv¨arinen, 2010; Chen et al., 2020) have shown that contrasting on high dimension sphere converge faster than low dimension sphere.
Hence, to explore if our methods can converge fast on a low contrastive dimension, we conduct extensive experiments with 100 and 1000 epochs on CIFAR-10 dataset.
For fair comparisons, we choose two symmetric methods, i.e., SimCLR (instance-wise) and Barlow Twins (feature-wise) as baselines.
Table 3 illustrates our method outperforms the mentioned two methods.
Speciﬁcally, Zero-FCL outperforms Barlow Twins and SimCLR by a large range ( 4.36% , 6.63% top-1 accuracy) withepochs, while the improvement decreases after 1000 epochs.
We conjecture the reason may be that
[Figure]
A cc @
Barlow
[Figure]
L ins only L Fea
[Figure]
32 64 128 256 512 1024 2048 4096 hidden size 72.
A cc @
Barlow Twins SimCLR L ins only L Fea only
[Figure]
[Figure]
A cc @
Barlow Twins SimCLR L ins only L Fea only
[Figure]
[Figure]
A cc @
[Figure]
Barlow
[Figure]
L ins
[Figure]
L Fea
[Figure]
Barlow Twins and SimCLR have two terms to optimize, where the ﬁrst term is the key to learn good embeddings (Arora et al., 2019), and the second term (uniformity or redundancy reduction) is the key to prevent trivial solutions.
However, the second term is hard to converge (Wang & Isola, 2020).
In contrast, our methods exactly discard the second term via whitening transformation.
Thus, our method converges at a faster speed.
We also report the results of different whitening transformation methods, i.e., Cholesky and ZCA.
We ﬁnd that for instance-wise contrastive learning, Choleskybased decomposition gets much lower accuracy scores rather than the ZCA-based method.
For feature-wise contrastive learning, Cholesky gets a slightly lower accuracy than ZCA-based.
We conjecture this is because ZCA-based whitening remains maximal information of original data (see
Theorem 1), which is important for instance-wise learning (slight rotation on instance dimension will heavily inﬂuence linear evaluation), while for feature-wise learning, when the features are permuted or rotated, it may not cause serious inﬂuence for linear evaluation (Mitra et al., 2002).
Sensitivities to batch size.
To explore our method’s sensitiveness to batch size, we conduct experiments with different batch sizes from 32 ∼ 1024.
For a fair comparison, each method with a different batch size is trained with 50K iterations .
Fig. 3(c) shows the results on CIFAR-100.
We can observe SimCLR and Zero-ICL get lower accuracy when batch size is small and this phenomenon is not surprising since instance-wise contrastive learning methods usually require a large batch size to create negative pairs (SimCLR) or exact statistics (Zero-ICL).
Another observation is the accuracy of Zero-FCL saturates when BatchSize = 256 , which is consistent with the results in (Zbontar et al., 2021).
Note that under 1024 batch size, Zero-ICL outperforms SimCLR by about 3.1% topaccuracy and Zero-FCL outperforms Barlow Twins 3.8% top-1 accuracy.
We guess that’s because of the diagonal property of whitened embeddings, which makes our methods robust to batch size.
Sensitivities to hidden dimensions.
We also explore the sensitivity of our method to hidden dimensions, which are set from 32 ∼ 4096.
Note that the three linear layers in projection MLP g are set as the same input and output dimension.
For feature-wise methods (Barlow Twins, Zero-FCL), the batch size is set as 256 (best hyper-parameter in Fig. 3(c)).
For instance-wise methods (SimCLR,
Zero-ICL), we set batch size as 1024.
All the results are get by 50K iterations.
Fig.
3(a) shows classiﬁcation accuracy with different hidden dimensions.
For Barlow Twins and Zero-FCL, both two methods are heavily inﬂuenced by hidden dimensions and Zero-FCL outperforms Barlow Twins under all hidden dimensions settings.
Oppositely, because the objective functions of SimCLR and
Zero-ICL are instance wise, making them more robust to hidden dimensions than Barlow Twins.
Ablation to negatives.
To show Zero-CL is indeed a negative-free method, we conduct an ablation study w/ and w/o negative samples and compare our methods with SimCLR, Barlow Twins.
For our methods w/ negatives, we add the off-diag entry of cross-correlation matrix in our objective function, which is followed Barlow Twins (Zbontar et al., 2021).
Table 4 shows results of four symmetric methods w/ and w/o negatives pairs.
Performance of Barlow Twins signiﬁcantly decreases and
SimCLR directly crashes w/o negative pairs.
Zero-FCL and Zero-ICL do not require negatives and perform the best without negatives.
After adding negatives, both Zero-ICL and Zero-FCL lose some accuracy.
We analyze that is because whitening transformation can be regarded as a constraint, while using negatives in the objective function is also a constraint.
Such two constraints will trade off the weight of alignment part (Arora et al., 2019), where the alignment part is the key to learn representations in contrastive learning (Grill et al., 2020; Wang & Isola, 2020).
Thus, the accuracy of Zero-ICL and Zero-FCL w/ negatives is lower than those w/o negatives.
Breaking Symmetry.
In line with Barlow Twins (Zbontar et al., 2021), we conduct extensive experiments with different asymmetric methods, i.e., stop-gradient and predictor.
For Zero-ICL, the objective only has an alignment part, which has been shown easily to collapse without the asym- metric framework (Grill et al., 2020; Chen & He, 2021), while the proposed instance whitening can solve this problem well.
Table 5 gives classiﬁcation accuracy on ImageNet-100 with ResNetbackbone.
In our experiments, the predictor module is composed of two linear layers with batch normalization.
Similar to Barlow Twins, we ﬁnd these asymmetries slightly decrease the performance of our network.
For stop gradient, it is easy to understand the accuracy dropping, i.e., without considering collapse, the information provided by two views will be more precise than single view (Caron et al., 2020; Ermolov et al., 2021), while for predictor, we guess that’s because overﬁtting by the non-linear transformation power of deep MLP (Goodfellow et al., 2016).
F URTHER D ISCUSSION
We further compare our work with some existing decorrelation methods and highlight our differences to better elaborate on the contributions and discuss the broad impacts.
Relation to CCA-SSG (Zhang et al., 2021a).
Inspired by classical Canonical Correlation Analysis,
CCA-SSG proposes a new loss function for graph data learning, which is similar to Barlow Twins.
[Equation]
where the ﬁrst term is similar to L BT .
However, the decorrelation term of L CCA aims to minimize the intra -correlation of different features, while the redundancy term of L BT minimizes cross -correlation of different features.
Our feature-wise whitening with L Fea only requires the invariance term of CCA-SSG or Barlow Twins, while the other term of L CCA is replaced by intra-feature-wise whitening, i e., whitening transformation is conducted inside each batch view.
Inspired by Barlow Twins, We expect out method can perform whitening transformation across two batch views, which we leave for future work.
[Table]
Relation to W-MSE (Ermolov et al., 2021).
W-MSE ﬁrstly combines classical whitening transformation with contrastive learning.
They create multiple views from the same images and perform the whitening transformation in each batch view on the feature dimension.
Then W-MSE objective is proposed to maximize similarities between every two views (instance-wise).
Compared with this whitening method, ZeroCL mainly has three differences.
1) Method.
W-MSE directly borrows whitening transformation on feature-wise, which is widely used in other domains (Huang et al., 2018;
Zhang et al., 2021b).
However, after feature-wise whitening, the instancewise objective function is adopted, which is groundless and also limits their performance.
Depart from W-MSE, ZeroCL aims to replace the second term of InfoNCE and L BT .
Thus, Zero-ICL uses instance-wise objective function with instance-wise whitening, and Zero-FCL uses feature-wise objective function with feature-wise whitening, which is meaningful and more comprehensible.
Such methods also leverage the performance (See Table 2).
2) Technology.
Instead of directly using
Cholesky or PCA-based whitening, we ﬁrst analyze which whitening transformation method is we really need.
Then, we theoretically analyze why ZCA whitening can remain maximal information.
3) SSL settings.
W-MSE requires multiple views to boost the accuracy (the version with two views gets much lower accuracy), while Zero-CL can get higher accuracy with only two views.
C ONCLUSION
We have proposed two new methods to prevent degenerate solutions in the symmetric architecture of SSL.
We ﬁrst analyze existing objective functions in SSL ( L Info , L BT and L CCA ).
Then, we show why Zero-FCL and Zero-ICL can replace the uniformity term of L Info and redundancy term of L BT , respectively.
We further give theoretical analysis on how ZCA based whitening remains maximal information rather than other whitening transformation methods.
Finally, we conduct experiments on three image benchmarks and results show that our methods can outperform or be on par with prior arts.
Further, to the best of our knowledge, Zero-ICL is the ﬁrst attempt for instance-wise whitening and we hope it can bring more inspiration to other tasks.
A A PPENDIX A. I MPLEMENTATION IN VISION
Image augmentations.
Each input image is transformed twice to generate two different views mentioned before.
The image augmentation pipeline follows Barlow Twins (Zbontar et al., 2021), composed of: random cropping, resizing to 224 × 224 (32 × 32 for CIFAR), horizontal ﬂipping, color jittering, converting to gray-scale, Gaussian blurring, and solarization.
The last ﬁve are applied randomly on two views with different probabilities.
Architecture.
Followed by recent works (Chen et al., 2020; Zbontar et al., 2021), the encoder consists of ResNet-50 and ResNet-18 (He et al., 2016) (without the ﬁnal classiﬁcation layer, 2048, 512 output units, respectively) followed by a MLP module.
In line with Barlow Twins (Zbontar et al., 2021), the MLP module contains three linear layers, each with the same output units.
The ﬁrst two layers of the MLP are followed by a BatchNorm layer (Ioffe & Szegedy, 2015).
Optimization.
Similar to previous works (Grill et al., 2020; Zbontar et al., 2021), we use the LARS optimizer (You et al., 2017) on both three image datasets.
We use a learning rate of 0.2 for the weights and 0.005 for the biases and batch normalization parameters.
We multiply the learning rate by batch size and divide it by 256.
We use a learning rate warm-up period of the ﬁrst 10 epochs, after which we reduce the learning rate by a factor of 1000 using a cosine decay scheduler (Loshchilov & Hutter, 2016).
For CIFAR-10 and CIFAR-100, we use single 1080 GPU.
For ImageNet-100, the batch size is set as 128 as default, and we use 8 Tesla V100 16G GPUs.
For ImageNet, we use1080Ti GPUs.
Evaluation.
We train a linear classiﬁer on three vision datasets on top of ﬁxed representations of
ResNets pre-trained by Zero-CL.
Speciﬁcally, the linear classiﬁer is trained for 100 epochs with a learning rate of 0.3 and a cosine learning rate scheduler.
We minimize the cross-entropy loss with
SGD optimizer with momentum 0.9 and weight decay 1e-6.
In line with previous arts (Zbontar et al., 2021; Chen et al., 2020), we set batch size 256.
At the inference stage, we resize the image to× 256 and center crop it to a size of 224 × 224.