Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
Abstract
Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations.
Although SSL has recentlyreached a milestone: outperforming supervised methods in many modalities.
..thetheoretical foundations are limited, method-specific, and fail to provide principleddesign guidelines to practitioners.
In this paper, we propose a unifying frameworkunder the helm of spectral manifold learning.
Through the course of this study,we will demonstrate that VICReg, SimCLR, BarlowTwins et al.
correspond toeponymous spectral methods such as Laplacian Eigenmaps, ISOMAP et al.
Fromthis unified viewpoint, we obtain (i) the close-form optimal representation, (ii)the close-form optimal network parameters in the linear regime, (iii) the impactof the pairwise relations used during training on each of those quantities andon downstream task performances, and most importantly, (iv) the first theoreticalbridge between contrastive and non-contrastive methods to global and local spectralmethods respectively hinting at the benefits and limitations of each.
For example, ifthe pairwise relation is aligned with the downstream task, all SSL methods produceoptimal representations for that downstream task.
Introduction
Self-Supervised Learning (SSL) is one of the most promising method to learn data representationsthat generalize across downstream tasks.
SSL places itself in-between supervised and unsupervisedlearning as it does not require labels but does require knowledge of what makes some samplessemantically close to others.
Hence, where unsupervised learning relies on a collection of inputs ( X ) ,and supervised learning relies on inputs and outputs ( X , Y ) , SSL relies on inputs and inter-samplerelations ( X , G ) that indicate semantic similarity.
The latter matrix G is often constructed byaugmenting X through data-augmentations known to preserve input semantics [1–3] e.g. horizontalflip for an image, although recent methods have went away from DA by using videos from whichconsecutive frames can be seen as semantically equivalent [4–6].
Although SSL originated decades ago [7], recent advances have pushed SSL performances beyondexpectations [8–10].
Due to those rapid empirical advances, an urgent need for a principled theoreticalunderstanding of those methods has emerged [11, 12].
Studies in this direction often take one ofthe three following approaches: (i) studying the training dynamics and optimization landscapes ofexisting methods e.g. validating some empirically found tricks as necessary conditions for stablegradient dynamics [13–16], (ii) studying the role of each SSL component e.g. the projector andpredictor networks [17–19], or (iii) developing novel SSL criteria that often combine multipleinterpretable objectives that a SSL model must fulfill [20–25].
While those branches have led to novel understandings and even stem novel SSL methods, some fundamental questions remain open
[Figure]
/NNCLR [Figure] [Figure] Preserving
[Figure]
Linear DiscriminantAnalysis ℓ 2 Regression -Principal Component Analysis -Multi-Dimensional Scaling -ISOMAP ( z -space) -Kernel ISOMAP ( x -space) a l w a y s li n ea r f θ + G =
X T X s up e r v i s e d
G li n ea r f θ + G =
X T X li n ea r f θ li n ea r f θ
+ s up e r
v i s e d G a [-r = L c o v = 
-]l w a y s li
[Figure]
r f θ + s up e r v
i s e d G li n ea r f θ a l w a y s ≈ ( li n ea r)
[Figure]
More recently, a few focused studies have started to provide theoretical works e.g. Tian [26] oncontrastive losses with deep linear networks, HaoChen et al.
[27, 28] on SimCLR [8], [29] on theprojector of contrastive models, [14] on BYOL [30] and SimSIAM [8].Those studies paved our wayforward as we propose in this paper a much broader analysis that applies to most (if not all) existing
SSL methods allowing us for the first time to provide provable design guildelines to practitionersin their choice of architecture and methods .
We do so by unifying most SSL methods as differentflavors of spectral methods for embedding and clustering as summarized in Fig.
The instrumentalresults we obtain allow us to answer some long-standing questions which we summarize as part ofour contributions below:
Closed-form optimal representation for SSL losses.
The DN representation Z of inputs X learned by minimizing any SSL loss given a sample relation matrix G is obtained in
closed-form, shedding light to many spectral properties of those representations e.g. SSL only constrains the left singular vectors and singular values of Z to align with the ones of G (Sections 3.1, 4.1 and 5.1 for VICReg, SimCLR and BarlowTwins).
Closed-form optimal network parameters for SSL losses with linear networks.
The
linear representation Z = XW + b parameters obtained by minimizing any SSL loss given a sample relation matrix G are obtained in closed-form, providing insights into the type of input statistics that a network parameters focuses on to produce the optimal input mapping
(Sections 3.2 and 5.2 for VICReg and BarlowTwins).
Exact equivalence between SSL and spectral embedding methods.
SSL methods employ diverse criterion that can be tied to eponymous spectral analysis methods both in embedding space and in data space, and with a nonlinear or a linear DN e.g. Laplacian Eigenmaps
(VICReg, Section 3.2), ISOMAP (SimCLR/NNCLR, Section 4.3), Canonical Correlation
Analysis (BarlowTwins, Section 5.2) and when employing a linear network as Locality
Preserving Projection (VICReg), Cannonical Correlation Analysis (BarlowTwins), and
Linear Discriminant Analysis for both VICReg and BarlowTwins (summarized in Fig. 1)
We also relegate to Appendix A a study quantifying the relationship between the optimal representation of each SSL method and the downstream task performances e.g. when the correct data relationmatrix is given (Appendix A.3) as those results, although not crucial to our contributions, followdirectly from the above ties between SSL and spectral embedding.
We carefully prove each statementof this study in Appendix D and provide the code to reproduce the main figures in the supplementarymaterials .
Notations and Background on Self-Supervised Learning
Dataset, Embedding and Relation Matrix Notations.
Regardless of the loss and method employed,
SSL relies on having access to a set of observations
i.e. input samples X ≜
[ x 1 , . . .
, x N ] T ∈ R N × Dand a known pairwise positive relation between those samples e.g. in the form of a symmetric matrix
G ∈ ( R + ) N × N where ( G ) i,j >
iff samples x i and x j are known to be semantically related, and with 0 in the diagonal.
Commonly, one is only given a dataset X ′ and artificially constructs X and
G from augmentations of X ′ e.g. rotated, noisy versions of the original samples and turning thecorresponding entries of G to be positive for the samples that have been augmented form the sameoriginal sample.
VICReg.
With the above notations out of the way, we can directly remind the VICReg loss as definedin Bardes et al.
[25] as a function of X and G in the following triplet loss
L vic =
[Equation]
with Z ∈ R N × K being the matrix of feature maps obtained from a model f θ : R D (cid:55)→ R K —commonly a Deep Network— as Z ≜ [ f θ ( x 1 ) , . . . , f θ ( x N )] T .
SimCLR.
The SimCLR loss [8] is slightly different and first produces an estimated relation matrix(cid:98)
G ( Z ) generally using the cosine similarity ( CoSim ) via
[Equation]
with τ > 0 a temperature parameter.
Then SimCLR encourages the elements of (cid:98) G ( Z ) and G tomatch.
The most popular solution to achieve that is to leverage the infoNCE loss given by
[Equation]
The only difference between SimCLR and its varients e.g. NNCLR [31] lies in defining G .
BarlowTwins.
Lastly, BarlowTwins, DINO, BYOL, and SimSIAM [30, 32, 24, 10] propose yet aslightly different approach where z i must be close to z j if G i,j > 0 .
They do so with different flavorsof losses and constraints to facilitate training.
Hence, and for those models only, it is common toexplicitly group X into two subsets X left and X right based on G .
This does not loose any generality.
In fact, suppose that we have 5 samples a, b, c, d, e , and that G says that a, b, c are related to eachother, and that d, e are related to each other.
Then, we can create the two data matrices as
[Equation]
Denoting by C ∈ [ - 1 , 1] K × K the cross-correlation between Z left and Z right , we obtain the loss
[Equation]
Notice that ( C ) i,j falls back to measuring the cosine similarity between the i th column of Z left andthe j th column of Z right as in ( C ) i,j = CoSim(( Z left ) .,i , ( Z right ) .,j ) .
Our goal in the following sections (Section 3 for VICReg, Section 4 for SimCLR/NNCLR, and
Section 5 for BarlowTwins) will be to find the optimal representations Z of X in variosu regimes.
Three surprising facts will emerge: (i) all existing methods recover exactly some flavors of famousspectral methods, (ii) the spectral properties of Z can be provably characterized for each methods, and(iii) from those properties, we will provide necessary and sufficient conditions for a SSL representationto perfectly solve a downstream task.
We provide linear algebra notations in Appendix B that mightbe useful for readers unfamiliar with methods such as the Singular Value Decomposition (SVD) [33].
VICReg Minimizes the Dirichlet Energy to Produce Smooth Signals on theGraph G While Preventing Dimensional Collapse
Recall from Section 2 and Eq.
(1) that VICReg is defined as a triplet loss (variance/invariance/covariance).
We first demonstrate in Section 3.1 that the optimal VICReg representation can be obtained inclosed form (Theorem 1) and that turning the VICReg optimization as a constrained problem recovers
Laplacian Eigenmaps in embedding space and Kernel Locality Preserving Projection in data space(Section 3.2).
We also consider the linear network regime where we obtain the analytical form of theoptimal network’s parameters (Theorem 3).
3.1 Close-Form Optimal Representation for VICReg
First, we build up some insights into VICReg by demonstrating how the invariance term correspondsto the Dirichlet energy of the signal Z on the graph G .
Then, replacing the variance hinge loss at 1with the squared loss at 1 as in (cid:80) Kk =1 (1 − Cov( Z ) k,k ) 2 , notice that minimizing the latter impliesminimizing the former.
With that, we are able to obtain the close-form optimal representation Z ∗minimizing Eq.
(1) only as a function of G and the loss’ hyperparameters.
From invariance to trace minimization.
The first insight that we propose into VICReg is obtainedby rewriting the invariance loss of VICReg as the energy of the signal Z on the graph G [34] sincewe have (derivations in Appendix D.5)
[Equation]
where L is the graph Laplacian matrix L
= D − G with D the diagonal degree matrix of G i.e.( D ) i,j = (cid:80)
j
( G ) i,j and ( D ) i,j = 0 , ∀ i ̸
= j .
From Eq.
(6) it is clear that the invariance termdepends on the matching between the left singular vectors of Z and the eigenvectors of L .
Hence,non-contrastive learning aims at producing non-degenerate signals
Z that are smooth on G .
Optimal representation.
To gain further insights into VICReg, we ought to obtain the analyticalform of the optimal representation Z ∗ minimizing Eq.
(1) —although this optimum is not unique e.g.adding a constant entry to each column of Z does not change the loss value (details in Appendix D.6).
We can now obtain the following characterization of Z ∗ as a function of the spectral decompositionof the matrix that combines two Laplacian matrices (details in Appendix D.7).
The first, (left of
Eq.
(7)) comes form the variance+covariance term is the Laplacian of a complete graph i.e. whereeach node/sample is connected all others.
The second comes from the SSL graph G to form thefollowing (with its eigen-decomposition)
[Equation]
where the eigenvalues/eigenvectors are in descending orders.
The eigenvectors of the combined
Laplacians will be key to produce the optimal VICReg representation as formalized below.
Theorem 1.
A global minimizer of the VICReg loss ( α = β, ∀ γ ) denoted by Z ∗ α,γ is obtained from
Eq.
(7) along with the minimal achievable loss which are given by
Z ∗ α,γ =
( P α,γ (diag( λ α,γ ) N ) 1 / 2 ) : , 1: K and
[Equation]
and any K -out-of- N columns of P α,γ ( Λ α,γ N ) 1 / 2 is a local minimum.
(Proof in Appendix D.7.)
The above result provides a few key insights.
First, only the ratio γ/α governs the VICReg representation.
Second, there exists many local minimum, some of which can be explicitly found by takingvarious K -out-of- N columns of P α,γ ( Λ α,γ N ) 1 / 2 which we display in Fig.
2 along with the losslandscape of L VIC around the optimal representation Z ∗ .
We also depict in Fig.
3 the evolution ofthe eigenvalues ( λ α,γ ) 1: K for varying γ along with the downstream task (induced by G ) trainingperformance.
We observe that VICReg benefits from a sweet-spot where it can both preserve afull-rank representation Z ∗ and incorporate enough information about G to solve the task at handperfect.
Prior to moving to other SSL methods, we first emphasize the ability of VICReg to recover localspectral methods in the following sections.
3.2 VICReg Recovers Local Spectral Embedding Methods
The goal of this section is to relate VICReg to known local spectral embedding methods both infeature space ( Z ) and in data space ( X ). In feature space.
Laplacian Eigenmaps (LE) [35] is a non-parametric method searching for arepresentation Z by minimizing the following Brockett [36] optimization problem
[Equation]
8) with D the diagonal degree matrix of G (recall Eq.
m i n
L V I C / t r a i n acc .
( % ) α
= 1 ,γ =α = 1 ,γ =
.convex
interp.
between top [ k : K + k ] eigenvectors of Eq.
(7) r a n k ( G ) =
K r a n k (
G )
=
α =1, γ =α =1, γ = 0 .α
=1, γ =α =1, γ = 0 .
k m i n
L v i c /t
r a i n acc .
[Figure]
Hence, VICReg hyper
[Table]
Theorem 2.
Given a dataset X and relation matrix G solving the LE optimization problem Eq.
(8)produces a representation that minimizes the VICReg loss with constraint that the variance and covariance loss are 0 as in L vic ( Z ∗ LE )
=
min Z L inv ( Z ) s.t.
L var =0 and L cov =0.
(Proof in Appendix D.8.)
We ought to highlight however that a crucial part of LE lies in the design of that matrix G , oftenfound from a k -NN graph [37] of the samples X in the input space, while in SSL it is constructedfrom data-augmentations, or given.
In data space with DNs.
The difficulty to produce new representations z for new data samples (ashared difficulty among non-parametric methods) led to the development of a two-step modelingprocess [38] as x ( ∈ R D )
(cid:55)→ h
= ϕ ( x ) ( ∈ R S )
(cid:55)→ z = W T h ( ∈ R K ) with S ≫ K, W ∈ R M × K ,and where ϕ ’s goal is to learn a generic input embedding that can be reused on new samples x .
To seethis, we collect those mappings for all the training set into the matrix Φ ∈ R N × S .
With that, the LEproblem in data space —known as the kernel Locality Preserving Projection (kLPP) [39] problem—becomes
[Equation]
so that the original LE representation Z can be obtained simply as Z = Φ W .
And more importantly,given a new sample x , one directly computes z = W T ϕ ( x ) .
The crucial result of interest for ourstudy is the following one that simply combines Theorem 2 with a result from He and Niyogi [39]demonstrating the equivalence between LE in feature space and KLLE in data space.
Proposition 1.
VICReg with variance/covariance constraint solves LE in embedding space and
KLLE in input space (recall Eq.
(9) ) employing a DN for ϕ .
In data space with linear models.
We now consider a linear mapping Z =
XW (the offset is takencare of by adding a 1 column to X ).
In that case, and in the same setting as for Theorem 3, VICRegrecovers two known spectral methods: Locality Preserving Projections (LPP) [39] for an arbitraryrelation matrix G , and Linear Discriminant Analysis (LDA) [40, 41] when G is the supervisedrelation matrix.
In both cases we obtain the analytical form of the optimal weights W .
Theorem 3.
Linear VICReg recovers LPP for any G and K ≤ N , and recovers LDA for supervised
G and K = C , in both cases the optimal parameter W ∗ is given by the top- K eigenvectors of( X T ( D − G ) X ) − 1 X T GX .
(Proofs in Appendices D.13 and D.14.)
Interestingly, the eigenvalues associated to the eigenvectors of W ∗ exactly recover the multivariateanalysis of variance (MANOVA) sufficient statistics of the data [42, 43].
Hence, although not furtherexplored in this study, we believe that important statistical results could be further obtained e.g. toassess the goodness-of-fit of the model without requiring a downstream task [44–46].
We now turn toanother important SSL loss which is SimCLR and its variants.
SimCLR Solves a Generalized Multidimensional Scaling Problem à laISOMAP
Recall from Section 2 and Eq.
(2) that SimCLR first computes a similarity matrix (cid:98) G of some flavorthat depends on the representation Z , and then matches it against the known data relation G .
Different
Z (cid:55)→ (cid:98) G methods lead to different variants of SimCLR [8] such as NNCLR [31] or MeanShift [47].
The goal of this section is two-fold.
First, we demonstrate in Section 4.1 that different Z (cid:55)→ (cid:98) Gmappings are solutions of different optimization problems (Theorem 4) —all trying to estimate thesimilarity matrix G from the signals Z akin to Laplacian estimation in Graph Signal Processing.
Second, we demonstrate in Section 4.2 that SimCLR and its variants force Z ’s spectrum to alignwith the one of G as the training task falls back to a generalized (kernel) Multi-Dimensional Scalingmethod (Proposition 2).
4.1 Step 1: SimCLR Pairwise Similarities Solve a Graph Laplacian Estimation Problem
Let’s first define the minimization problem that given a set of signals i.e. rows of Z produces a relationestimate (cid:98) G of G .
To ease notations, we gather in the N × N matrix D all the pairwise distances( D ) i,j = d ( f θ ( x i ) , f θ ( x j )) with d any preferred metric.
The standard problem of estimating (cid:98) Gfrom Z can be cast as an optimization problem [48, 49] as
[Equation]
with G the set (or subset) of symmetric matrices with nonnegative entries and zero diagonal, and with
R a regularizer preventing (cid:98) G to be the trivial zero matrix e.g.
[Equation]
We provide in Fig.
7 a depiction of the impact of R ( G ) which pushes the entries of the weight matrixto be close to 1 with strength depending on the temperature parameter τ .
Hence (cid:98) G from Eq.
(10) isthe optimal graph —expressed as a weight matrix— for which the signal Z = f θ ( X ) on that graphis smooth.
For example one can solve Eq.
(10) only on G rsto , the space of right-stochastic matricesi.e.
a subset of G that only contains matrices whose rows sum to 1 i.e. G rsto = { G ∈ G : G 1 = 1 } .
Theorem 4.
Using R from Eq.
(11) leads to the following graph weight estimate
[Equation]
and thus if d is the cosine distance, Eq.
(12) recovers SimCLR’s case.
(Proof in Appendix D.9.)
( Σ Z ) k , k i n l og - s ca l e [Figure] Based on this result, deriving novel, principled and interpretable variations of SimCLR is streamlined. To do so, one could solve Eq.
(10) but now with a different constraint than the one recovering SimCLRin Theorem 4. For example (derivations in Appendix D.9) the following variations can be obtained [Equation]
with G , τ > 0 )
[Equation]
[Equation] [Equation] only by using a different regularizer. Now that we understood the first part of the SimCLR method,we move to the second part, matching that graph estimate to the given one.
4.2 Step 2: SimCLR Fits the Estimated Graph (cid:98) G to the Known Graph G
After SimCLR estimates the graph with (cid:98) G as per the previous section, it employs a loss to enforce (cid:98) Gto be as close as possible to G with some desired metric.
That metric should reflect the propertiesthat G fulfills e.g. being a doubly-stochastic, right-stochastic or else.
We demonstrate in this sectionthat in doing so, SimCLR forces Z to have the same nonzero left singular vectors as the nonzeroeigenvectors of G , and that as opposed to VICReg, the rank of Z ∗ τ and G always matches.
Let’s first denote the SimCLR contrastive loss to be one of the two following variants (depending onthe type of constraints put on (cid:98) G and G
[Equation]
When minimizing Eq.
(15), SimCLR will learn an embedding Z so that the graph estimate matchesclosely the known graph G .
This already brings a contrast from non-contrastive methods showingthat contrastive methods learn to produce signals such that the graph estimate is close to the knowngraph . We now characterize the optimal SimCLR representation Z ∗ τ , recalling that we denote by U G and Σ G the left singular vectors and singular values of G respectively.
Notice that since G issymmetric semi-definite positive, U also corresponds to its eigenvectors and Σ 2 G to its eigenvalues.
Theorem 5.
A global minimizer of the SimCLR loss denoted by Z ∗ along with the minimal achievableloss using Eq.
(13) or Eq.
(14) , τ ≥ max i,j ( D ) i,j and with the LHS of Eq.
(15) are given by
[Equation]
U G Σ
[Equation]
[Equation]
G )
[Equation]
K and
[Equation]
up to permutations of the singular vectors associated to the same singular value.
Also, for any loss of
Eq.
(15) and graph estimation, the rank of Z ∗ τ is min( K, rank( G )) .
(Proof in Appendix D.10.)
We illustrate the above theorem for many combinations of distances and regularizers in Fig.
4 wherewe see that in all cases, SimCLR forces the representations Z to have a dimensional collapse, aphenomenon first observed in Hua et al.
[17] and that has been one of the unanswered phenomenonin SSL [20, 29].
In our goal to unify SSL methods under the realm of spectral embedding methods, we now proposethe following section that ties SimCLR and its variants to global spectral methods.
4.3 SimCLR Recovers Global Spectral Embedding Methods
We now propose to tie the SimCLR method along with its variants e.g. NNCLR to known globalspectral methods, e.g. ISOMAP [50] which is in contrast to VICReg which was tied to local spectralmethods (recall Section 3.2).
In feature space.
Let’s first recall that ISOMAP is a variation of Multi-Dimensional Scaling (MDS)[51] also known as Principal Coordinates Analysis.
Classical MDS tries to learn embedding vectorsthat have similar pairwise distance (usually ℓ 2 ) than the pairwise distance of the given input data.
Often, MDS does this by using similarities instead of distances and thus by solving the followingoptimization problem min Z
∥
G − ZZ T ∥ 2 F .
At the most general level, ISOMAP simply correspondsto solving that same optimization problem but after redefining G to better capture the geometricinformation of X e.g. using the shortest path distance of the k -NN graph of X [52].
The surprisingresult that we formalize below is that SimCLR and its variants recover ISOMAP.
Proposition 2.
SimCLR, using the settings of Theorem 5, recovers ISOMAP (and MDS for the correctchoice of G ).
(Proof in Appendix D.11.)
In data space with DNs.
From the above, we can extend Proposition 2 but in input space, in a verysimilar way as was done in Section 3.2.
In fact, originating in Webb [53], there was a search to extend
MDS, and ISOMAP to an input space formulation to solve the out-of-bag problem.
In this setting,and taking MDS as an example, the original similarity matrix ZZ T is replaced with Φ W T W Φ Tusing the same notations as in Eq.
(9) and already known relationship between those models, weobtain the following.
Proposition 3 ([54]) .
Whenever SimCLR recovers ISOMAP or MDS in feature space, it recoverskernel ISOMAP or kernel PCA [55] in input space.
We now ought to turn to BarlowTwins, another non-contrastive method akin to VICReg (both ofwhich fall back to LDA in the linear regime and with supervised G ).
BarlowTwins Solves a (Kernel) Canonical Correlation Analysis Problemand Can Recover VICReg
Our last step in our journey to unify SSL methods under spectral embedding methods deals with
BarlowTwins.
Akin to the development for VICReg and SimCLR, BarlowTwins will also fall back toa known spectral method in embedding space (Section 5.1) and in data space (Section 5.2) where inthe later case we again obtain the close-form optimal network parameters in the linear regime.
5.1 BarlowTwins Recovers Kernel Canonical Correlation Analysis
Recall from Section 2 and Eq.
(5) that the BarlowTwins loss is based on a cross-correlation matrixbetween positive pairs of samples.
As we did for VICReg and SimCLR, our goal here is to tie
BarlowTwins to a known spectral method known as Kernel Canonical Correlation Analysis.
Although we focus here on BarlowTwins for clarity, we ought ot recall that other methods such as DINO [10] are akin to BarlowTwins, but simply employ a different metric to compare Z left and Z right , the embeddings of X left and X right respectively.
We hope that our results on BarlowTwinswill stem the unification of those methods too in future works.
Going back to BarlowTwins, we nowobtain the following result that nicely parallels with the ones we obtained for VICReg and SimCLR.
In data space, BarlowTwins can be regarded (put in perspective with Section 5.2) as a nonlinearcanonical correlation analysis (NLCA) [56] and in particular Kernel CCA (KCCA) [57, 58] akin tohow VICReg recovered Kernel Locality Preserving Projection and SimCLR Kernel ISOMAP.
Weleverage the same notations as in Section 3.2.
Theorem 6.
BarlowTwins recovers Kernel Canonical Correlation Analysis with a DN as the featurizerϕ and produce a representation with rank min( K, D ) for any value of α (recall Eq.
(5) ) and withorthogonal columns.
(Proof in Appendix D.12.).
In addition to the above tie, we obtain further interpretation of the components of BarlowTwins loss.
For example, the additional ϵ constant added in the denominator of the BarlowTwins loss furthercorresponds to a ridge-type regularization which as been introduced in Gretton et al.
[59] as a meanto introduce numerical stability.
The above statement also brings yet another flavor of SSL methods.
In fact, where VICReg allows tocontrol the rank of Z to be in-between K and rank( G ) throug the loss hyper-parameters, and where
SimCLR enforces the rank of Z to be exactly the rank of G , BarlowTwins enforces to have a full-rankrepresentation.
We depict in Fig.
8 the evolution of rank( Z ) depending on the rank of the initializedrepresentation rank( Z init ) using a gradient descent optimizer.
We see that if the initialization is fullrank but G is lower rank, the BarlowTwins loss does not collapse the extra nonzero singular values of
Z .
Vice-versa, if rank( Z init ) < rank( G )
then BarlowTwins loss will increase the rank of Z .
lastly,although not further studied here, we should poitn out to the reader that regularized forms of KCCAcan be shown to include kernel ridge regression and regularized kernel Fisher LDA as special cases[60], further tying the special cases for which different SSL methods would fall back to the samemodel.
In the following section we will demonstrate how BarlowTwins in the linear regime exactly recovers
Canonical Correlation Analysis.
5.2 With a Linear Network BarlowTwins Recovers Canonical Correlation Analysis andLinear Discriminant Analysis
The goal of this section is to further demonstrate the benefits of connecting SSL methods to spectralmethods by exploiting the known techniques of the latter to help answer questions on the former.
As was done VICReg (we use linear settings of Section 3.2), we now obtain the optimal weightsfor BarlowTwins in the linear regime.
We can even provide additional insights in this case since
BarlowTwins is often seen as a key method that allows the use of different parameters/architectures toprocess X left and X right .
We now show under what conditions on G sharing parameters is sufficientby first demonstrating how BarlowTwins recovers exactly CCA, and even LDA for supervised G .
Tostreamline notations, we assume that our data is already centered, and thus define the covariance andcross-covariance matrices as C ll = X T left X right ,
C lr = X T left X right and so on.
Theorem 7.
In the linear regime BarlowTwins recovers CCA with optimal weights given by
[Equation]
[Equation]
Proof Appendix D
The above result opens new venues to extend current SSL methods (BarlowTwins in this case).
Forexample, penalized matrix decomposition (PMD) from Witten et al.
[61] formulates a novel sparseformulation of CCA.
In our context, this could lead to a new variation of BarlowTwins, in boththe linear and nonlinear regimes.
With the above results, we now connected most SSL methodsto spectral methods, and found key properties that their representations/parameters inherit.
Weprovide for completeness a few direct results in Appendix A that directly leverage the above resultsto characterize downstream performances of each method.
Conclusions
We provided a unifying analysis of the major self-supervised learning methods covering VICReg(Section 3), SimCLR (Section 4) and BarlowTwins (Section 5).
In doing so, we were able to not onlytie each of those methods and their variants to common spectral embedding techniques, but we werealso able to find the commonalities between all those methods.
At a more general level, we were ableto parallel the contrastive versus non-contrastive dichotomy in SSL to the global versus local methodsin spectral methods respectively.
Among the many insights that we obtained, the most crucial one isthat VICReg enables a continue control on collapsing the representation’s rank versus encapsulatinginformation about G .
This is in contrast to BarlowTwins and SimCLR that either always maintainfull-rank, or always collapse the representation none of which would be ideal.
Another angle that should prove insightful for future work concerns the combining of spectral methodsdesign of G with the one of SSL methods. In particular, some techniques in the spectral embeddingliterature have already proposed to combine an unsupervised G defined from the data geometry e.g.from a k -NN graph, and a supervised G [62]. Extending such method for a SSL G could provebeneficial in various cases. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] Each section of the study answers a specific aspect of the abstract (b) Did you describe the limitations of your work?
[Yes] We discussed the assumptions of
each result.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do
not believe that our work has any potential negative societal impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
For each
method we carefully state our assumptions and provide background and citations for
known results/assumptions
(b) Did you include complete proofs of all theoretical results?
[Yes] All the proofs
are carefully expressed in multiple appendices with additional details to ensure full
transparency on our results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Code to
reproduce the figures is provided in the supplementary materials
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] For each figure we provided the exact hyper-parameters used in
each case within the caption
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No] We only provide empirical validation that do not require
errors
bars
(d) Did you include the total amount of compute and the type of resources used (e
g.,
type of GPUs, internal cluster, or cloud provider)?
[No] We only provide empirical
validation on toy settings that do not require GPU or cluster computations
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] we did not use
existing assets
(b) Did you mention the license of the assets?
[Yes] No licence provided
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include in the supplementary material the code to reproduce the figures provides in
this paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used
crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]