U NDERSTANDING D IMENSIONAL C OLLAPSE IN C ONTRASTIVE S ELF - SUPERVISED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Self-supervised visual representation learning aims to learn useful representations without relying on human annotations.
Joint embedding approach bases on maximizing the agreement between embedding vectors from different views of the same image.
Various methods have been proposed to solve the collapsing problem where all embedding vectors collapse to a trivial constant solution.
Among these methods, contrastive learning prevents collapse via negative sample pairs.
It has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space.
Here, we show that dimensional collapse also happens in contrastive learning.
In this paper, we shed light on the dynamics at play in contrastive learning that leads to dimensional collapse.
Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR , which directly optimizes the representation space without relying on a trainable projector.
Experiments show that
DirectCLR outperforms SimCLR with a trainable linear projector on ImageNet.
I NTRODUCTION
Self-supervised learning aims to learn useful representations of the input data without relying on human annotations.
Recent advances in self-supervised visual representation learning based on joint embedding methods (Misra & Maaten, 2020b;
He et al., 2020; Chen et al., 2020a; Chen & He, 2020;
Grill et al., 2020; Zbontar et al., 2021;
Bardes et al., 2021;
Chen et al., 2020b; Dwibedi et al., 2021;
Li et al., 2021; Misra & Maaten, 2020a;
HaoChen et al., 2021; Assran et al., 2021; Caron et al., 2021) show that self-supervised representations have competitive performances compared with supervised ones.
These methods generally aim to learn representations invariant to data augmentations by maximizing the agreement between embedding vectors from different distortions of the same images.
As there are trivial solutions where the model maps all input to the same constant vector, known as the collapsing problem, various methods have been proposed to solve this problem that rely on different mechanisms.
Contrastive methods like Chen et al.
(2020a)
and
He et al.
(2016) deﬁne ‘positive’ and ‘negative’ sample pairs which are treated differently in the loss function.
Non-contrastive methbods like Grill et al.
and Chen & He (2020) use stop-gradient, and an extra predictor to prevent collapse without negative pairs; Caron et al.
(2018; 2020) use an additional clustering step; and Zbontar et al.
(2021) minimize the redundant information between two branches.
These self-supervised learning methods are successful in preventing complete collapse whereby all representation vectors shrink into a single point.
However, it has been observed empirically in noncontrastive learning methods (Hua et al., 2021; Tian et al., 2021) that while embedding vectors do not completely collapse; they collapse along certain dimensions.
This is known as dimensional collapse (Hua et al., 2021), whereby the embedding vectors only span a lower-dimensional subspace.
In contrastive methods that explicitly use positive and negative pairs in the loss function, it seems intuitive to speculate that the repulsive effect of negative examples should prevent this kind of dimensional collapse and make full use of all dimensions.
However, contrary to intuition, contrastive learning methods still suffer from dimensional collapse (See Fig. 7).
In this work, we theoretically study the dynamics behind this phenomenon.
We show there are two different mechanisms that cause collapsing: (1) along the feature direction where the variance caused by the data augmentation is larger than the variance caused by the data distribution, the weight collapses.
Moreover, (2) even if the covariance of data augmentation has a smaller magnitude than the data variance along all dimensions, the weight will still collapse due to the interplay of weight matrices at different layers known as implicit regularization. This kind of collapsing happens only in over-parametrized networks where the network has more than one layer.
Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR , which directly optimizes the encoder (i.e., representation space) without relying on a trainable projector.
DirectCLR outperforms SimCLR with a linear trainable projector on ImageNet.
We summarize our contributions as follows:
• We empirically show that contrastive self-supervised learning suffers from dimensional collapse whereby all the embedding vectors fall into a lower-dimensional subspace instead of the entire available embedding space.
• We showed that there are two mechanisms causing the dimensional collapse in contrastive learning: (1) strong augmentation along feature dimensions (2) implicit regularization driving models toward low-rank solutions.
• We propose DirectCLR , a novel contrastive learning method that directly optimizes the representation space without relying on a trainable projector.
DirectCLR outperforms SimCLR with a linear trainable projector.
R ELATED W ORKS 2. S ELF - SUPERVISED L EARNING M ETHODS
Joint embedding methods are a promising approach in self-supervised learning, whose principle is to match the embedding vectors of augmented views of a training instance.
Contrastive methods (Chen et al., 2020a; He et al., 2016) directly compare training samples by effectively viewing each sample as its own class, typically based on the InfoNCE contrastive loss (van den Oord et al., 2018) which encourages representations from positive pairs of examples to be close in the embedding space while representations from negative pairs are pushed away from each other.
In practice, contrastive methods are known to require a large number of negative samples.
Non-contrastive methods do not directly rely on explicit negative samples.
These include clustering-based methods (Caron et al., 2018; 2020), redundancy reduction methods (Zbontar et al., 2021; Bardes et al., 2021) and methods using special architecture design (Grill et al., 2020; Chen & He, 2020).
2. T HEORETICAL U NDERSTANDING OF S ELF - SUPERVISED L EARNING
Although self-supervised learning models have shown success in learning useful representations and have outperformed their supervised counterpart in several downstream transfer learning benchmarks (Chen et al., 2020a), the underlying dynamics of these methods remains somewhat mysterious and poorly understood.
Several theoretical works have attempted to understand it.
Arora et al. (2019b);
Lee et al.
Tosh et al.
(2021) theoretically proved that the learned representations via contrastive learning are useful for downstream tasks.
Tian et al.
(2021) explained why non-contrastive learning methods like BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2020) work: the dynamics of the alignment of eigenspaces between the predictor and its input correlation matrix play a key role in preventing complete collapse.
2. I MPLICIT R EGULARIZATION
It has been theoretically explained that gradient descent will drive adjacent matrices aligned in a linear neural network setting (Ji & Telgarsky, 2019).
Under the aligned matrix assumption, Gunasekar et al.
(2018) prove that gradient descent can derive minimal nuclear norm solution.
Arora et al.
(2019a) extend this concept to the deep linear network case by theoretically and empirically demonstrating that a deep linear network can derive low-rank solutions.
In general, over-parametrized neural networks tend to ﬁnd ﬂatter local minima (Saxe et al., 2019; Neyshabur et al., 2019; Soudry et al., 2018; Barrett & Dherin, 2021).
D IMENSIONAL C OLLAPSE
Self-supervised learning methods learn useful representation by minimizing the distances between embedding vectors from augmented images (Figure 1a).
On its own, this would result in a collapsed solution where the produced representation becomes constant (Figure 1b).
Contrastive methods prevent complete collapse via the negative term that pushes embedding vectors of different input images away from each other.
In this section, we show that while they prevent complete collapse, contrastive methods still experience a dimensional collapse in which the embedding vectors occupy a lower-dimensional subspace than their dimension (Figure 1c). [Figure] [Equation] [Figure]
[Equation]
[Figure]
We train a SimCLR model (Chen et al.
(2020a))
with a two-layer MLP projector.
We followed the standard recipe and trained the model on ImageNet for 100 epoch.
We evaluate the dimensionality by collecting the embedding vectors on the validation set.
Each embedding vector has a size of d = 128 . We compute the covariance matrix C ∈ R d × d of the embedding layer (here ¯ z := (cid:80) Ni =
1 z i /N and N is the total number of samples): [Equation]
Figure 2 shows singular value decomposition on this matrix ( C = USV T , S = diag ( σ k ) ). in sorted order and logarithmic scale ( { log( σ k ) } ).
We observe that a number of singular values collapse to zero, thus representing collapsed dimensions.
[Figure]
r v a l u e s
D IMENSIONAL C OLLAPSE CAUSED BY S TRONG A UGMENTATION 4. L INEAR M ODEL
In this section, we explain one scenario for contrastive learning to have collapsed embedding dimensions, where the augmentation surpasses the input information. We focus on a simple linear network setting.
We denote the input vector as x and the augmentation is an additive noise.
The network is a single linear layer with weight matrix is W .
Hence, the embedding vector is z
= W x .
We focus on a typical contrastive loss, InfoNCE (van den Oord et al., 2018):
[Equation]
where z i and z (cid:48)
i are a pair of embedding vectors from the two branches, z j indicates the negative samples within the minibatch.
When all z i and z (cid:48) i are normalized to be unit vector, the negative distance −| z i − z (cid:48)
i | 2 / 2 can be replaced by inner products z Ti z (cid:48) i .
The model is trained with a basic stochastic gradient descent without momentum or weight decay.
4. G RADIENT F LOW D YNAMICS
We study the approximate dynamics via gradient ﬂow, i.e., gradient descent with an inﬁnitesimally small learning rate.
Lemma 1.
The weight matrix in a linear contrastive self-supervised learning model evolves by:
[Equation]
where G = (cid:80) i ( g z i x Ti + g z (cid:48) i x (cid:48)
Ti ) , and g z i is the gradient on the embedding vector z i (similarly g z (cid:48) i ).
This can be easily proven based on the chain rule.
See proof in Appendix B.1.
For InfoNCE loss deﬁned in Eqn 2, the gradient of the embedding vector for each branch can be written as
[Equation]
where { α ij } are the softmax of similarity of between z i and { z j } , deﬁned by α ij =
exp( −| z i − z j | 2 / 2) /Z i , α ii = exp( −| z i − z
cid:48) i | 2 / 2) , and Z i = (cid:80) j (cid:54) = i exp( −| z i − z
j | 2 / 2) + exp( −| z i − z (cid:48) i | 2 / 2) .
Hence, (cid:80) j α ij = 1 . Since z i = W x i , we have G
[Equation] (5) where X [Equation]
Lemma 2.
{+X is a subtraction of two PSD matrices: [Equation] Here ˆΣ 
+}
= (cid:80) i,j α ij ( x i − x j )( x i − x j ) T is a weighted data distribution covariance matrix and ˆΣ 1 = (cid:80) i (1 − α ii )( x (cid:48)
i − x i )( x (cid:48)
i − x i ) T is a weighted augmentation distribution covariance matrix. See proof in Appendix B.2. Therefore, the amplitude of augmentation determines whether X is a positive deﬁnite matrix. Next, we look into the dynamics of weight matrix W given property of X .
Theorem 1.
With strong augmentation such that ˆΣ 0 − ˆΣ 1 has negative eigenvalues, the weight matrix W has vanishing singular values.
See proof in Appendix B.3.
Corollary 1 (Dimensional Collapse Caused by Strong Augmentation) .
With strong augmentation, the embedding space covariance matrix becomes low-rank.
The embedding space is identiﬁed by the singular value spectrum of the covariance matrix on the embedding (Eqn. 1), C = (cid:80) i
( z i − ¯ z )( z i − ¯ z ) T /N = (cid:80) i W ( x i − ¯ x )( x i − ¯ x ) T W T /N . Since W has vanishing singular values, C is also low-rank, indicating collapsed dimensions.
D IMENSIONAL C OLLAPSE CAUSED BY I MPLICIT R EGULARIZATION 5. T WO - LAYER LINEAR MODEL
With strong augmentation, a linear model under InfoNCE loss will have dimensional collapse.
However, such scenarios rely on the condition that the network has a limited capacity which may not hold for real cases.
On the other hand, when there is no strong augmentation ( ˆΣ 1 ≺ ˆΣ 0 ) and thus X matrix remains PSD, a single linear model won’t have dimensional collapsing.
However, interestingly, for deep networks, dimensional collapsing still happens in practice.
In the following, we will show that it stems from a different nature: implicit regularization, where over-parametrized networks tend to ﬁnd low-rank solutions.
[Figure] To understand this counter-intuitive phenomena, we start with the simplest over-parametrized setting by choosing the network as a two-layer linear MLP without bias. The weight matrices of these two layers are denoted by W 1 ∈ R d × d and W 2 ∈ R d × d . Similar to the setting in Sec 4, the input vector is denoted as x and the augmentation is an additive noise.
The embedding vector from each branch is z
= W 2 W 1 x , hence z ∈ R n .
We do not normalize z .
See Figure 3.
We use InfoNCE loss deﬁned in
Eqn 2.
The model is trained with a basic stochastic gradient descent without momentum or weight decay.
5. G RADIENT F LOW D YNAMICS
Similar to Lemma 1, we derive the gradient ﬂow on the two weight matrices W 1 and W 2 . Lemma 3. The weight matrices of the two layer linear contrastive self-supervised learning model evolves by ( G = (cid:80) i ( g z i x Ti + g z (cid:48)
i x (cid:48) Ti ) is deﬁned in Lemma 1):
[Equation]
This can be easily proven based on the chain rule.
See proof in Appendix B.4.
For the two layer case, similar to Eqn 5, we have the speciﬁc form of G :
[Equation]
where X is deﬁned in Eqn 6.
According to Lemma 2, we know that with small augmentation,
X = ˆΣ 0 − ˆΣ 1 (cid:31) 0 is a positive-deﬁnite matrix.
5. W EIGHT A LIGNMENT
Since we have two matrices W 1 and W 2 , the ﬁrst question is how they interact with each other.
We apply singular value decomposition on both matrices W 1 and W 2 , i.e., W 1 = U 1 S 1 V T 1 , W 2 =
U 2 S 2 V T 2 and S 1 = diag ([ σ k 1 ]) , S 2 = diag ([ σ k 2 ]) .
The alignment is now governed by the interaction between the adjacent orthonormal matrices V 2 := [ v k 2 ] and U 1 = [ u k 1 ] .
This can be characterized by the alignment matrix
A = V T 2 U 1 , whose ( k, k (cid:48) )
-entry represents the alignment between the k -th right singular vector v k 2 of W 2 and the k (cid:48) -th left singular vector u k (cid:48) 1 of W 1 . The following shows that indeed W 1 and W 2 aligns.
Theorem 2 (Weight matrices align) .
If for all t , W 2 ( t ) W 1 ( t ) (cid:54) = 0 , X ( t ) is positive-deﬁnite and W 1 (+ ∞ ) , W 2 (+ ∞ ) have distinctive singular values, then the alignment matrix A = V T 2 U 1 → I .
See proof in Appendix B.5.
Here, we also empirically demonstrate that under InfoNCE loss, the absolute value of the alignment matrix A approximately converges to an identity matrix.
See Figure 4.
The alignment effect has been studied in other scenarios (Ji & Telgarsky, 2019; Radhakrishnan et al., 2020).
In the real case, when some of our assumptions are not satisﬁed, e.g., there are degenerate singular values in weight matrices, we will not observe a perfect alignment.
This can be easily understood by the fact that the singular decomposition is no longer unique given degenerate singular values.
In our toy experiment, we speciﬁcally initialize the weight matrices to have non-degenerate singular values.
In real scenario, when weight matrices are randomly initialized, we will only observe the alignment matrix to converge to a block-diagonal matrix, with each block representing a group of degenerate singular values.
Given the fact that singular vectors corresponding to the same singular value approximately align, we can now study the dynamics of the singular values of each weight matrix W 1 and W 2 .
Theorem 3.
If W 2 and W 1 are aligned (i.e., V 2 = U T 1 ), then the singular values of the weight matrices W 1 and W 2 under InfoNCE loss evolve by:
[Equation]
See proof in Appendix B.6.
Since X is a positive deﬁnite matrix, the term v k 1 T X v k 1 is always nonnegative.
This shows that each singular value has a gradient proportional to itself and the paired ones (singular values with same ranking from the other matrix).
Since all singular values are sorted, this explains why we observe that the smallest group of singular values grow signiﬁcantly slower.
See demonstrative experiment results in Figure 5a and 5b.
[Figure]
Corollary 2 (Dimensional Collapse Caused by Implicit Regularization) .
With small augmentation and over-parametrized networks, the embedding space covariance matrix becomes low-rank.
The embedding space is identiﬁed by the singular value spectrum of the covariance matrix on the embedding vectors, C = (cid:80) ( z − ¯ z )( z − ¯ z ) T /N
=
(cid:80) W 2 W 1 ( x − ¯ x )( x − ¯ x ) T W T 1 W T 2 /N .
As
W 2 W 1 evolves to be low-rank, C is approximately low-rank, indicating collapsed dimensions.
See Figure 5c for experimental veriﬁcation.
5. E FFECT OF M ORE L AYERS AND N ONLINEARITY
In our toy model, we focused on a two-layer linear MLP setting.
Here, we empirically show that our theory extends to multilayer and nonlinear cases, as shown in Figure 6a.
Stronger over-parametrization leads to a stronger collapsing effect, which has been shown theoretically (Arora et al., 2019a; Barrett & Dherin, 2021) and empirically (Jing et al., 2020).
This can be explained by the fact that more adjacent matrices getting aligned, and the collapsing in the product matrix gets ampliﬁed.
Note that for a single-layer case, L = 1 , there is no dimensional collapse in the embedding space, which is consistent with our analysis.
[Figure]
We empirically show that the collapsing effect also applies to the nonlinear scenario.
We insert
ReLU between linear layers ( L = 2 ) and observe a similar singular value collapse compared to the linear case.
See Figure 6b.
D IRECT CLR 6. M OTIVATION
We now leverage our theoretical ﬁnding to design novel algorithms.
Here we are targeting the projector component in contrastive learning.
Empirically, adding a projector substantially improves the quality of the learned representation and downstream performance (Chen et al., 2020a).
Checking the spectrum of the representation layer also reveals a difference with/without a projector.
To see this, we train two SimCLR models with and without a projector.
The representation space spectrum are shown in Figure 7b.
The dimensional collapse in representation space happens when the model is trained without a projector.
Thus, the projector prevents the collapse in the representation space.
[Figure]
The projector in contrastive learning is essential to prevent dimensional collapse in the representation space.
Based on our theory, we claim the following propositions regarding the role of projector in contrastive learning models.
The gradient will drive the projector weight matrix aligned with the last layer of the encoder backbone.
We suspect that such alignment effect V T 2 U 1 → I only requires one of V 2 and U 1 to evolve.
Therefore, the projector weight matrix only needs to be diagonal .
The projector only applies a gradient to a subspace to the representations.
Therefore, the projector weight matrix only needs to be low-rank . M AIN I DEA
[Figure] We propose to remove the projector in contrastive learning by directly sending a sub-vector of the representation vector to the loss function.
We call our method DirectCLR .
In contrast to all recent state-of-the-art self-supervised learning methods, our method directly optimizes the representation space.
See Figure 8, DirectCLR picks a subvector of the representation z
= r [0 : d 0 ] , where dis a hyperparameter.
Then, it applies a standard InfoNCE loss on this normalized subvector ˆ z
= z / | z | ,
L = (cid:80) i log
[Equation]
We train DirectCLR with a standard recipe of SimCLR for 100 epochs on ImageNet.
The backbone encoder is a ResNet50.
DirectCLR demonstrates better performance compared to SimCLR with a trainable linear projector on ImageNet.
The linear probe accuracies for each model are listed in
Table 1.
Loss function
[Table]
1 Accuracy
SimCLR 2-layer nonlinear projector 66.
SimCLR 1-layer linear projector 61.
SimCLR no projector 51.
DirectCLR no projector 62.
We visualize the learnt representation space spectrum in Figure 9.
DirectCLR prevents dimensional collapse in the representation space similar to the functionality of a trainable projector in SimCLR.
[Figure]
One may suspect that the contrastive loss in DirectCLR does not apply a gradient on the rest part of the representation vector r [ d 0 :] , then why these dimensions would contain useful information?
In fact, the entire representation vector r is trained and contains useful information.
The rest part of the representation vector is copied from the layer before the last residual block via residual connection.
This part of the representation does not experience a gradient directly from the loss function but is updated by the gradient through the convolution block.
Thus, DirectCLR takes advantage of the residual connection in the ResNet backbone.
See Figure 10.
[Figure]
6. A BLATION S TUDY
Fixed low-rank projector : DirectCLR is equivalent to SimCLR with a ﬁxed linear projector with a diagonal matrix.
We observe that it performs similarly on ImageNet when using a ﬁxed low-rank matrix as a projector, which achieves 62 .
3% linear probe accuracy. The singular values of this lowrank matrix are set to have d 0 numbers of 1 and 0 for the rest.
The corresponding singular vectors of this matrix are all randomly generated.
Therefore, their only difference is that this ﬁxed projector has an extra ﬁxed orthogonal matrix in between.
This shows that the representation space (or equivalently the previous layer) has evolved to be aligned to this orthogonal matrix during training.
Trainable diagonal projector : We trained a SimCLR model with a trainable projector that is constrained only to have diagonal terms.
The model achieves 60 .
2% linear probe accuracy on ImageNet, which is comparable to a SimCLR with a 1-layer linear projector.
This can also be explained by the alignment phenomenon that the projector gets aligned with the previous layer in the backbone.
We suspect that initialization is the reason why this model is slightly worse than a SimCLR with a 1layer linear trainable projector: the singular value of a random matrix is different from the uniform distribution.
Linear probe on subvector z : For DirectCLR , we perform a linear probe only on the sub-vector z and get 47 .
9% accuracy on ImageNet.
This shows that the rest of r still contains useful information even though it does not see gradient directly coming from the loss function.
C ONCLUSIONS
In this work, we showed that contrastive self-supervised learning suffers from dimensional collapse, where the embedding vectors only span a lower-dimensional subspace.
We provided the theoretical understanding of this phenomenon and showed that there are two mechanisms causing dimensional collapse: strong augmentation and implicit regularization.
Inspired by our theory, we proposed a novel contrastive self-supervised learning method DirectCLR that directly optimizes the representation space without relying on a trainable projector.
DirectCLR outperforms SimCLR with a linear projector on ImageNet.
R EPRODUCIBILITY S TATEMENT
We provide detailed proof for every lemma and theorem in the Appendices.
We provide an anonymous and executable code for reproducing the results of the experiment section in the supplementary material.
We will upload the code to a public Github repo after publication.
A U SEFUL L EMMAS
We adapt two useful lemmas from Arora et al.
(2019a)
Lemma 4.
Given a matrix W and the dynamics that W evolves by ˙ W , the singular values of this matrix evolve by:
[Equation]
where u k and v k are singular value σ k ’s corresponding left and right singular vectors.
i e. the k -th column of matrices U and V respectively.
Proof.
Given a matrix W and its singular value decomposition W = USV T .
We have the dynamics of the matrix
[Equation]
Multiplying U T from the left and multiplying V from the right, considering U and V are orthogonal matrices, we have
[Equation]
Since S = diag ( σ k ) is a diagonal matrix, we have
[Equation]
Again, considering u k and v k have unit-norm, we have u kT ˙ u k = 0 and ˙ v kT v k = 0 .
Therefore, we derive
[Equation]
Lemma 5.
Given a matrix W and the dynamics that W evolves by ˙ W , the singular vectors of this matrix evolve by:
[Equation]
where (cid:12) represents Hadamard element-wise multiplication.
H is a skew-symmetric matrix
[Equation]
Proof.
Same as proof for Lemma 1, we start from the following equation
[Equation]
Considering the fact that U T ˙ U and ˙ V T V are skew-symmetric matrices, whose diagonal terms are all zero, we Hadamard-multiply ¯
I to both sides of the equation.
Here, ¯ I has all diagonal values equal zeros and all off-diagonal values equal to one, we have [Equation]
[Equation]
Taking transpose, we have
[Equation]
[Equation]
Right-multiplying S to Eqn 16 and left-multiplying S to Eqn 17, then adding them up, we have
[Equation]
Therefore, we have
[Equation]
where
[Equation]
Similar proof applies to Eqn
[Equation] Lemma 6 (Alignment matrix dynamics) .
The alignment matrix A , deﬁned by A =
V T U 1 , evolves by:
[Equation]
where (cid:12) represents Hadamard (element-wise) multiplication.
H l is a skew-symmetric matrix, whose ( k, k (cid:48) )
-entry is given by
[Equation]
and F is deﬁned by
[Equation]
Proof.
According to Lemma. [Equation] we have
[Equation]
Plugging the above two equations and Eqn 8, Eqn [Equation] the dynamics of the alignment matrix A =
V T 2 U 1 can be written as
[Equation]
where
[Equation]
Lemma 7 (Singular value dynamics) . The singular values of the weight matrices W 1 and W 2 evolve by: [Equation]
Proof.
According to Lemma
[Equation] Plugging in Eqn 8, we have [Equation] Similar proof applies to Eqn 22.
B D ELAYED P ROOFS B. P ROOF OF L EMMA
The gradient on matrix W is dL
[Equation]
∂W )
We denote the gradient on z i and z (cid:48)
i as g z
i and g z (cid:48) i , respectively.
Since ∂ z i ∂W
= x i and ∂ z (cid:48)
i ∂W
= x (cid:48)
i , we get
[Equation]
B. P ROOF OF L EMMA
Proof.
X is deﬁned in Eqn 6.
[Equation]
Given the fact that (cid:80) j (cid:54) =
i α ij = 1 − α ii , we have (cid:80) i (cid:80) j (cid:54) =
i α ij x (cid:48)
i x Ti = (cid:80) i (1 − α ii ) x (cid:48)
i x Ti .
Also, since (cid:80) i (cid:80)
j
(cid:54) =
i iterates all pairs of i, j , we can replace the index between i and j , we have
[Equation]
Therefore
[Equation]
Proof.
According to Lemma 1, we have
[Equation]
Apply SVD on W , we have W = USV T where S = diag ( σ k ) . Therefore, W T W = V S 2 V T =
[Equation]
k v k σ k 2 v kT , where v k is the k -th column of matrix V . Let w k = σ k v k , Eqn 23 can be written as
[Equation]
[Equation] Given the fact that w k are orthogonal to each other and w k · w k = ( σ k ) 2 , we left-multiply w kT and right-multiply w k to Eqn 24, and we derive
[Equation]
Because X is not positive deﬁnite, there exists σ k such that
[Equation]
Therefore, W have vanishing singular values. B. P ROOF OF L EMMA Proof.
The gradient on matrix W 2 is [Equation] [Equation] We denote the gradient on z i and z (cid:48)
i as g z i and g z (cid:48) i , respectively. Since ∂ z i ∂W 2 =
W 1 x i and ∂ z (cid:48)
i ∂W 2 =
W 1 x (cid:48)
i , we get
[Equation]
Similar proof applies to W 1 .
B. P ROOF OF T HEOREM
Here, we prove that under the assumption that singular values are non-degenerate, the alignment matrix
A = V T 2 U 1 converges to identity matrix.
Proof.
According to Lemma 3, we have
[Equation]
therefore,
[Equation]
or
[Equation]
Next, we show that the Frobenius norm of each weight matrix grow to inﬁnitely.
[Equation]
According to Eqn 10, G = − W 2 W 1 X , we have
[Equation]
Because X is a positive deﬁnite matrix and for all t , W 2 ( t ) W 1 ( t ) (cid:54) = 0 , we know B := W 2 W 1 XW T 1 W T 2 is positive semi-deﬁnite and B (cid:54) = 0 . Therefore, tr ( B ) = (cid:80)
k λ k ( B ) > 0 since not all eigenvalues of B are zero.
Therefore, we know
|| W
|| 2 F → + ∞ (similarly ||
W [-
-]2 || 2 F → + ∞ ). In the limit t − > + ∞ , we have
[Equation]
Plug in the singular value decomposition of W 1 and W 2 , we have U 1 S 21 U T 1 =
V 2 S 22 V T 2 .
Assuming
W 1 and W 2 have non-degenerate singular values, due to the uniqueness of eigen-decomposition, we have
[Equation]
therefore,
[Equation]
Remark . Note that when the non-degenerate singular value assumption does not hold, the corresponding singular vectors are not unique and we will not observe the corresponding dimensions becoming aligned.
B. P ROOF OF T HEOREM
Proof.
According to Theorem 2, for σ k 1 and σ k 2 with same index, the corresponding singular vector pairs v k 2 and u k 1 will get aligned, i.e., v k (cid:48) 2 T u k 1 → δ i,j . Therefore, Eqn 21 and Eqn 22 can be simpliﬁed to
[Equation]
Insert Eqn 10 and considering the alignment, we derive
[Equation]