Extended Abstract Track
On Rotational Symmetry in the Loss landscape of Self-Supervised Learning
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
We derive an analytically tractable theory of SSL landscape and show that it accurately captures an array of collapse phenomena and identifies their causes.
Self-supervised learning (SSL) methods have achieved remarkable results in learning good representations without labeled data.
SSL loss functions are designed to promote representational similarity between pairs of related samples while using explicit penalties or asymmetric dynamics to ensure that the distance between unrelated samples remains large.
In practice, however, SSL training often experiences the failure mode of dimensional collapse (Jing et al., 2021; Tian et al., 2021; Pokle et al., 2022), where the learned representation spans a low dimensional subspace of the overall available space.
In the extreme case, this failure mode instantiates as a complete collapse , where the learned representation becomes zero-rank, and no informative features can be extracted.
Additional related works are discussed in Appendix A.
In this work, we analytically solve the effective landscapes of linear models trained on several popular losses used in self-supervised learning.
Combining theory and empirical results, a key insight we offer is: collapses of representations are strongly dependent on the stability of the last layer at the origin and happens when a broken symmetry is restored.
1. A Landscape Theory of Self-Supervised-Learning
Let { ˆ x i } Ni be a dataset with N data points.
For two data points ˆ x and ˆ chi , we augment it with an i.i.d. noise ϵ such that x ∶= ˆ x + ϵ and χ ∶= ˆ χ + ϵ ′ are independently sampled data with independent augmentations.
To be concrete, we start with considering the standard contrastive loss, InfoNCE (Oord et al., 2018):
[Equation]
Extended Abstract Track where f ( x ) ∈ R d 1 is the model output; all x , x ′ and χ are augmented data points for some independent additive noise ϵ such that E ϵ [ x ] =
ˆ x = E ϵ
[ x ′ ]
≠ E ϵ
[ χ ] = ˆ χ .
We decompose the model output into a general function ϕ ( x
) ∈ R d 0 and the last-layer weight matrix W ∈ R
d 1 × d 0 : f ( x ) =
W ϕ ( x ) .
The covariance of ϕ ( ˆ x ) is A 0 ∶= E ˆ x
[ ϕ ( ˆ x ) ϕ ( ˆ x ) T ] , and the covariance of the data-augmented penultimate layer representation
is Σ ∶= E x
[ ϕ ( x ) ϕ ( x ) T ] .
The effect of data augmentation on the learned representation is captured through a symmetric matrix
C ∶= Σ − A 0 .
For a general ϕ , the eigenvalues of C can be either positive or negative.
When ϕ is the identity mapping, A 0 becomes the empirical data covariance, C becomes PSD and is the covariance of the noise ϵ , and Σ is the covariance of the augmented data.
In some sense, this loss function captures the essence of SSL: the numerator encourages the representation f ( x ) to be closer to the representation of similar data, and the denominator encourages a separation between dissimilar data.
For a fixed set of noises, we can write the InfoNCE in a cleaner form:
[Equation]
where we used E ˆ x to denote an averaging over the training set.
In this notation, we have
E ϵ E
ˆ x [ x ] =
E x [ x ] and E ϵ
[ L ϵ ] =
L .
For a quantitative understanding, we mainly focus on the case when ϕ is the identity function.
We discuss the general nonlinear case in Section 1.3.
The proofs are presented in Appendix E.
1.1. Landscape of a Linear Model
NT-xent .
As in Tian (2022), we note InfoNCE can be generalized as follows:
[Equation]
Different from InfoNCE, one term in the denominator is weighted by a factor α ≥ 0.
Two interesting limits are α = 1, where we recover the InfoNCE loss, and α = 0, where we obtain the popular NT-xent loss used in SimCLR (Chen et al., 2020).
For general α , we refer to this loss as the weighted InfoNCE .
For a perceptron, the leading terms of the loss function is L
[Equation]
In fact, for the losses functions we consider, the leading order terms of the loss function all take the following rather universal form, for some symmetric matrix B ,
[Equation]
Landscape Analysis .
When training ends, one expects the model to locate at (at least close to) a stationary point of the loss.
It is thus important to identify all the stationary points of this loss function.
Theorem
Let d ∗ ∶= min ( d 0 , d 1 ) .
Let the data and noise be Gaussian.
All stationary points W of Eq.
(4) satisfy W T W = 12 Σ − 1 / 2 UM Λ U T Σ − 1 / 2 , where U Λ U T is the eigenvalue decomposition of Σ − 1 / 2 B Σ − 1 / 2 , and M is an arbitrary (masking) diagonal matrix containing only zero or one
such that (1) M ii = 0 if Λ ii < 0 and (2) contain at most d ∗ nonzero terms.
Additionally, if C and A 0 commute, all stationary points satisfy W T W = 12 Σ − 1 B M Σ − 1 , where B M denotes the matrix obtained by masking the eigenvalues of B with M . Extended Abstract Track
This stationary-point condition implies the direct cause of the dimensional collapse.
Namely, dimensional collapse happens when the eigenvalues of the matrix B become negative.
The eigenvalues of B , in turn, depend on the competition between data augmentation and the data feature.
Comparing the commuting case with the noncommuting case, we see that the main difference is that when CA 0 ≠ A 0 C , the augmentation can also change the orientation of the learned representation; otherwise, augmentation only affects the eigenvalues.
To focus on the most important terms, we now assume that the augmentation is well-aligned with the features such that the augmentation covariance commute with the data covariance.
From now on, we assume CA
=
A 0 C .
For the case of weighted InfoNCE, we have that B =
A 0 − 1 − αN C .
Let a i denote the i -th eigenvalue of the A and c i that of C viewed in a predetermined order; then, the i th subspace collapses when 1 − αN c
i ≥ a i , namely, when the variation introduced by the noise dominates that of the original data.
Importantly, this collapse is a property shared by all stationary points of the landscape, and one cannot hope to fix the problem by, say, biasing the gradient descent towards a certain type of local minima.
When weight decay is used, the condition for collapse becomes
1 − αN c
i + γ ≥ a i .
It becomes easier to cause a collapse when weight decay is used.
Because the stationary points contain collapsed solutions where the eigenvalues of W T W are zero, one is naturally interested in how likely it is to converge to these solutions.
The following proposition implies that the loss landscape of contrastive SSL (with a linear model) is rather benign because all local minima must achieve a maximum possible rank.
Proposition( W T W achieves maximum possible rank)
Let m denote the number of positive eigenvalues B .
Then, rank ( W T W ) =
min ( m, d ∗ ) for any local minimum.
1.2. Landscape with Normalization
It is common in practice to normalize the learned representation such that ∣∣ f
( x )∣∣ 2 = c .
When the normalization is applied, only the direction of the learned representation matters.
While this is a simple trick in practice, its implication on the landscape is poorly understood.
In this section, we extend our theory to analyze the effect of normalization.
We model the effect of normalization as a regularization term: R ∶=
( E x ∣∣ f
( x )∣∣ 2 − c ) 2 :
L
[Equation]
This regularization term achieves two things: (1) ∣∣ f
( x )∣∣
= c is a minimizer of the loss function; (2) the regularization is invariant to a rotation of the representation.
This loss function can also be seen as a mathematical model of the VICReg loss (Bardes et al., 2021), where R effectively models the variance regularization term of VICReg loss and κ is its strength.
This modeling is necessary because the variance term of the original
VICReg is not differentiable and thus cannot be expanded.
The proposed term R captures the essence of the variance term because it also encourages the representation to have a constant variance.
Our theory also explains why the VICReg is observed to experience collapses when κ is not large enough.
As κ tends to infinity, this constraint will become perfectly satisfied.
We thus take the infinite κ limit to study the effect of normalization.
The following proposition gives a condition that all stationary points of Eq.
(5) satisfy.
Proposition 3 Let ρ ( W ) ∶=
Tr [ W Σ W T ] , B ′ ∶= B + 2 κ ( c − ρ ) Σ , and let Λ i be the eigevalues of B ′ .
Then, every stationary point of Eq.
(5) satisfy W T W = 12 Σ − 1 B ′ M Σ − 1 , where M is an arbitrary diagonal mask of the eigenvalues of B ′ containing only zero or one such that (1)
M ii = 0 if Λ i < 0 and (2) contain at most d ∗ nonzero terms.
Compared with the unnormalized case, the term 2 κ ( 1 − ρ ) Σ M emerges due to normalization.
The effect of normalization is as expected: it shrinks the norm of the model if ρ > 1, and it expands the model if ρ < 1, and it does not have any effect if we have already achieved ρ = 1.
Interestingly, this rescaling effect is anisotropic and stronger along the directions of larger eigenvalues of the covariance of the augmented data Σ. Section D.3 directly finds the solution of ρ .
For a finite κ , these results suggest that collapses can still happen.
For VICReg, B = − A 0 , and the complete collapse can happen when κ ≪ ∣∣
A 0 ∣∣/ c ∣∣ Σ ∣∣ – this explains the experimental observation of collapses for small values of κ in (Bardes et al., 2021).
Lastly, to understand normalization, we are interested in the case of κ → ∞ .
We discuss this case in detail in Appendix D.1.
Our result can also be applied to analyze spectral contrastive loss (SCL)
(HaoChen et al., 2021) and Barlow Twins, which are discussed in Appendix D.
1.3. Relevance to Nonlinear Models
An important question is how much the analysis connects to deep nonlinear models.
In fact, the loss landscape we have studied is close to the most general landscape one can have.
Let L ( f ( x )) be a general SSL loss function for data point x .
The quality of the learned representation should be independent of the population-level orientation of the representation.
Therefore, the loss function should be rotationally invariant: for any rotation matrix R , L ( x )
=
L ( Rf ( x )) .
This invariance implies that the loss expands as
L ( f ( x ))
=
af ( x ) T f ( x ) + b [ f ( x ) T f ( x )]
2 + O ( f ( x ) 6 ) .
Note that all the odd-order terms of f ( x ) vanish due to the rotational symmetry.
Substituting f ( x ) =
W ϕ ( x ) in the loss function, we obtain the general form of landscape that W obeys:
[Equation]
where A and Z are dependent on ϕ .
All the examples we have studied take this form.
For
W , its collapse depends on the stability of the matrix A .
Thus the study of the stability of the matrix A is crucial for our understanding.
To illustrate, we train a Resnet18 on
CIFAR10 with the SimCLR loss with normalization and with weight decay strength 10 −until convergence to obtain the converged weights W ∗ .
We inject independent Gaussian noises with variance σ 2 as data augmentation.
The representation has a dimension 128.
We Extended Abstract Track rescale the weight matrix of the last layer W ∗ last by a factor of a and compute the loss as a function of a .
See Figure 1-a.
We then partition the singular values of W ∗ last into the larger and smaller half.
We rescale the larger half by a factor r 1 and the smaller half by r 2 .
We plot the loss as a 2 d function of ( r 1 , r 2 ) in Figure 1.
See Appendix B for more experiments that validate our theory on both linear and nonlinear models.
Those familiar with statistical physics should note that the proposed theory mimics the
Landau theory of second-order phase transitions.
When treating the loss function as the free energy, the square root of the eigenvalues √ λ of W T W are the order parameters of the system, and the phase transitions happen when λ turns from 0 to positive.
These transitions (collapses) happen because of symmetry breaking : the loss function (2) is symmetric in the sign of W .
Yet, for any nontrivial learning, W must be nonzero.
A symmetry breaking of the sign of W needs to happen for learning.
This phase transition phenomenon with the 0 .
5 scaling is also in line with the neural collapse phenomenon in supervised learning (Ziyin and Ueda, 2022).
In this work, we approached the problem of collapses in SSL from a landscape and symmetry breaking perspective. We analytically solved an effective landscape that can be extended to understand the effect of normalization. Our result suggests that dimensional collapse can be well understood in the minimal setting and is something neutral to learning on its own.
We showed that when task-irrelevant dimensions are targeted, dimensional collapse can result in dramatically improved performance, whereas an uninformative noise will (without good luck) leads to collapses in the dimensions that are relevant to the task.
It is thus important for practitioners to devise targeted data augmentation mechanisms that incorporate the correct domain knowledge.
The proposed theory can serve as a theoretical foundation and baseline of any advanced theory of collapses because a correct theory should agree with our results when restricting to the case of a linear model.
We advocated the thesis that the local geometry of the loss landscape around the origin is an essential component for understanding collapses, and this should invite more future work to understand the landscape around the origin.
The limitation of our work is clear; our result only identifies the causes of the collapse that can be directly attributed to the low-rank structure of the local minima of the landscape.
One possible alternative cause of the collapse is dynamics.
For example, having a large learning rate and small batch can sometimes cause a convergence towards the saddle points in the landscape (Ziyin et al., 2021), which, as we have shown, are the collapsed solutions.
Investigating the role of dynamics in the collapse is thus a crucial future problem.
Appendix A. Related Works
SSL and Collapses.
Prior literature has often argued collapse as a harmful phenomenon that can deteriorate downstream task performance (Jing et al., 2021; Zbontar et al., 2021).
Preventing such collapsed representations is a frequently discussed topic in literature (Hua et al., 2021;
Jing et al., 2021; Pokle et al., 2022; Tian et al., 2021) and has motivated the design of several SSL techniques (Zbontar et al., 2021;
Bardes et al., 2021; Ermolov et al., 2021).
However, in direct contrast, Cosentino et al.
(2022) empirically showed that dimensional collapses under strong augmentations could dramatically improve generalization performance.
Our work demystifies these conflicting results by finding analytic solutions to loss landscapes of several standard SSL techniques.
Theoretical Advances in SSL.
Recently, several advances have been made towards understanding the success of SSL techniques from different perspectives: e.g., learning theory (Arora et al., 2019;
HaoChen et al., 2021; Saunshi et al., 2022; Nozawa and Sato, 2021;
Ash et al., 2021; Wei et al., 2021), information theory (Tsai et al., 2021a,b; Tosh et al., 2021a,b), causality and data-generating processes (Zimmerman et al., 2021; Kugelgen et al., 2021; Trivedi et al., 2022; Tian et al., 2020; Mitrovic et al., 2020; Wang et al., 2022), dynamics (Wang and Isola, 2020; Tian et al., 2021; Tian, 2022; Wang and Liu, 2021), and loss landscapes (Jing et al., 2021; Pokle et al., 2022).
These advances have unveiled practically useful properties of SSL, such as robustness to dataset imbalance (Liu et al., 2021) and principled solutions to avoid spurious correlations (Robinson et al., 2021).
The work by Jing et al.
(2021) is the closest to ours in problem setting.
In that paper, the authors focused on studying the linearized learning dynamics and suggested that a competition between the feature signal strength and augmentation strength can lead to dimensional collapse.
In contrast, our focus is on the landscape and our result implies that this feature-augmentation competition on its own is insufficient to cause a dimensional collapse.
In fact, we show that there will be no collapse in the setting studied by Jing et al.
Interpretability of linear models.
Previous works have demonstrated how linear models are often sufficient to reproduce phenomena observed in non-linear deep networks (Saxe et al., 2013; Kawaguchi, 2016).
If an observed phenomenon does not occur for linear models, one can conclude that the use of nonlinearity is a necessary condition for this phenomenon.
If a phenomenon happens for both models, one naturally concludes that non-linearity is not necessary to cause the phenomena.
For example, posterior collapses in
Bayesian deep learning were first thought to be caused by expressivity due to non-linearity (Alemi et al., 2018); however, recent works (Lucas et al., 2019; Wang and Ziyin, 2022) found that models without non-linearity can still induce the collapse.
Appendix B. Additional Numerical Results
In this section, we validate our theory with numerical results.
Unless specified otherwise, the dimension of the learned representation is set to be equal to the input dimension: d 0 = d 1 .
No Collapse for InfoNCE .
We showed that there is no collapse at all for the vanilla
InfoNCE, no matter how strong the augmentation is.
Our result implies that the smallest singular of the model W scales as σ 4 where σ 2 is the strength (namely, the variance) of the augmentation.
See the left panel of Fig.
We use the vanilla InfoNCE loss defined in (1) with a linear model.
The training set is sampled from N ( 0 , I 32 ) .
The training proceeds with Adam with a learning rate of 6 e − 4 with full batch training for 5000 iterations.
We use a simple diagonal Gaussian noise with variance σ 2 for data augmentation.
We see that the singular values scale as σ 4 and never vanishes, as the theory predicts.
Nonrobust Collapses of Weighted InfoNCE .
We now demonstrate that, as the theory predicts, collapses of weighted InfoNCE depend strongly on the dataset size.
We use the same dataset and training procedure as the previous experiment.
We set α = 0 .
1 and change the size of the training set.
Theory suggests that for a collapse in the i − th subspace to happen, the size of the dataset needs to obey
[Equation]
See the middle panel of Figure 2.
We show the smallest three eigenvalues of W T W (roughly having similar magnitudes), and the critical dataset size for the smallest eigenvalue.
We see that the theoretical threshold of collapse agrees well with where the collapse actually happens.
Collapses in β -InfoNCE .
With β < 1, one can cause collapses in a predictable and controllable way.
In this experiment, we let d 0 =
and we plot all five eigenvalues of W T W as we increase the strength of an isotropic augmentation.
As the numerical results show, collapses happen at the points predicted by the theory.
Normalization Causes Dimensional Collapse .
We also plot the three smallest eigenvalues of W T W when we apply the standard representation normalization in practice: f ( x ) → f ( x )/∣∣
f ( x )∣∣ .
To facilitate comparison, we also use the same dataset and training procedure as before.
See Figure 3.
We see that normalization does cause a collapse in the smallest eigenvalues at an augmentation strength much smaller than the feature variation.
Appendix C. Landscape of a Nonlinear Model
In this section, we plot the landscape of the layer of nonlinear models on the same synthetic dataset we outlined in the previous section.
We train a three-layer nonlinear network with output dimension 2 with SGD until convergence.
We then rescale the optimized weight of the last by a factor a : W last → aW last and plot the loss function along this direction.
See the top panel of Figure 4 for both the tanh and the ReLU nonlinearity.
We then rescale the two rows of the weight matrix of the model by r 1 and r 2 respectively:
W = ( w 1 , W 2 ) T → ( r 1 w 1 , r 2 w 2 ) .
Extended Abstract Track
Appendix D. Additional Theoretical Concerns D.1. Collapse condition for normalization
Combining Proposition 3 and 5, one obtains lim κ →∞ W Tκ W κ = 12 Σ − 1 [ B M + 2 c − Tr
[ Σ − 1 B M ] d M
[Equation]
Because the eigenvalues of W W T must be positive, the following condition holds for all solutions:
[Equation]
) where λ i are the eigenvalues of Σ − 1 B M and ¯ λ is its average.
Namely, for the i − th dimension not to collapse, it must be smaller than the average eigenvalues by at most 2 c / d M .
Any smaller eigenvalues must collapse.
Compared to the case without normalization, normalization makes collapses dependent on the relative strength of each feature and augmentation.
One finds that the condition for collapse becomes heavily dependent on the data structure, and there are cases where collapses become harder, and there are cases where collapses become much easier.
Importantly, it also becomes the case that a sufficiently strong augmentation can always cause a collapse in the corresponding subspace.
The important condition for collapse in Eq.
(8) can be better understood by considering the extreme cases.
First of all, note that the eigenvalues of Σ B M are bounded between −and
[Equation]
and − 1 is achieved when c i ≫ a i , and 1 is achieved when a i ≫ c i .
When the augmentation is negligibly small, Σ − 1 B M ≈ M , and λ i ≈ ¯ λ = 1, the condition thus becomes
[Equation]
which always holds.
Thus, a sufficiently small augmentation will never cause collapse.
Next, when we apply very strong augmentation to the j -th subspace and zero augmentation to the others, the condition for the non-augmented spaces becomes
[Equation]
meaning that the collapse will not happen.
For the j -th space, the condition is
[Equation]
which is only possible when d M = 1, namely, the strongly augmented space is the only space that does not collapse.
This is reasonable when the original data is rank-1 because the normalization will ensure that this space does not collapse, but when the original data is not rank-1, this stationary point will be a saddle and will not be preferred by gradient descent.
In different word, a strong enough augmentation will cause a collapse in the corresponding subspace, as is the case without normalization.
It is also interesting to note that having c i ≥
a
i is no longer sufficient to cause a collapse.
For example, let c 1 = 0 and c j = a j for j ≠ 1.
The condition for j ≠ 1 becomes
[Equation]
Extended Abstract Track
which always holds.
At the same time, it does not mean that collapsing has become harder in general.
For example, it is also possible for c
i < a i to cause a collapse.
Suppose we add a weak augmentation only to the first subspace such that a i − c i = ϵ > 0, the condition for this dimension to not to collapse is
[Equation]
which can be violated whenever ϵ < ( a i + c i )
( d M − 3 )
a i + c i + d m .
Namely, in some cases, normalization can in fact facilitate collapse.
D.2. Effect of Bias
Effect of Bias .
Lastly, we study the effect of explicitly having a bias term: W x → W x + b .
First of all, when there is no normalization, the bias term does not affect the solution because the loss landscape is invariant to a translation in the learned representation.
However, this effect dramatically changes if we apply normalization at the same time.
This is because normalization removes the translation symmetry of the effective loss, and the trivial solution
W = 0, b
= 1 becomes the simplest way to achieve the norm − 1 constraint.
Our result shows that the addition of bias dramatically affects the stationary points.
Theorem 4 Let f ( x ) =
W x + b and E [ x ] = 0 .
Then, all stationary points satisfy W T W = 12 Σ − 1 B M Σ − 1 , subject to the constraint that ρ ( W ) =
Tr [ W T Σ W ] ≤ c .
Namely, the solution reverts to the case where there is no normalization at all, except that the norm of the solution can no longer be larger than c .
This upper bound can make collapses much easier to happen.
For example, if c < ( a i − c i )/
( a i + c i ) for all i , a complete collapse can happen despite normalization.
When c = 1 and c i ≪ a i , ρ ≈ d M / 2 and the constraint indicates that d M ≤ 2: when the augmentation is very weak, there are at mostnontrivial subspaces.
This is too restrictive for learning a meaningful representation, which helps us understand why dimensional collapse can harm learning in practice.
The fact that simple normalization cannot prevent collapse has been noticed for a while for the simplest case of a cosine-similarity loss, and our result explains why previous works have tried to introduce asymmetry to cosine similarity to avoid collapses (Grill et al., 2020;
Chen and
He, 2021).
D.3. Solution of ρ
The next theorem gives the explicit form of ρ at the stationary points.
Proposition 5 For any stationary point W ∗ ,
[Equation]
where d M is the number of non-zero eigenvalues of B ′ M .
Extended Abstract Track
D.4. Relevant Loss Functions
Having developed a framework for understanding normalization, we show that other common loss functions in SSL can also be written in the form given in Eq.
The spectral contrastive loss (SCL) (HaoChen et al., 2021) reads
[Equation]
Let f ( x ) =
W x be linear, the distributions are zero-mean Gaussian, and ignore the normalization.
This loss function becomes
[Equation]
When normalization exists, we can apply the result in Section 1.2.
By our argument, there is no collapse in this loss function.
The difference with InfoNCE loss is that the learned feature spreads along the directions of the augmentation C , not along the directions of the feature
A 0 .
The case of Barlow Twin (BT) (Zbontar et al., 2021) is similar.
While the fourth-order term of BT is much more complicated due to the imbalance created by the λ term.
The second-order term can be identified easily: L BT = − 2Tr
[ W Σ W T ]
+ O (∣∣ W ∣∣ 4 ) .
This also does not collapse.
A difference between the SCL loss and InfoNCE is that the learned representation has a spread that aligns with the combination of the feature and the augmentation strength.
Appendix E. Proofs E.1. Proposition
Before proving the main results, we first prove a proposition that we will rely on to prove the main results.
The following proposition shows that the variance term of the loss takes a specific form when the data is Gaussian.
Proposition 6 Let the data and noise be Gaussian.
Then, L = − Tr [ W BW T ]+ Tr [ W Σ W T W Σ W T ] .
Proof .
The second term in Eq.
(4) can be written as
[Equation]
where we have used the definition Σ
=
A 0 + C .
The first term is [ first term
[Equation] [Equation]
[Equation]
Combining the above expressions, we see that Eq.
(4) can be written as
[Equation]
This finishes the proof.
◻
Extended Abstract Track
E.2. Proof of Theorem
Proof .
All stationary points have a zero gradient:
[Equation]
Multiplying by W T on the left and B − 1 on the right,
[Equation]
Defining H ∶= Σ 1 / 2 W T W Σ 1 / 2 , we obtain
[Equation]
Because both H and Σ 1 / 2 Σ B − 1 Σ 1 / 2 are symmetric, one can take the transpose of Eq.
(27) to find that H and Σ 1 / 2 B − 1 Σ 1 / 2 commute with each, which implies that H has the same eigenvectors as Σ
1 / 2 B − 1 Σ 1 / 2 / 2.
Eq.
(28) then implies that the eigenvalues of H is either the inverse of that of Σ 1 / 2 B − 1 Σ 1 /or zero.
This implies that any stationary point of H can be written in the form
[Equation]
29) where U is a unitary matrix, Λ is diagonal matrix containing the eigenvalues of Σ 1 / 2 B − 1 Σ 1 / 2 , and M is an arbitrary (masking) diagonal matrix containing only zero or one
such that (1)
M ii = 0 if Λ ii < 0 and (2) contain at most d ∗ nonzero terms.
This then implies that the weight matrix W satisfies
[Equation]
Lastly, when Σ and B commute, we can compactly write the result as
[Equation]
(31) where B M denotes the matrix obtained by masking the eigenvalues of B with M .
This finishes the proof.
◻
E.3. Proof of Proposition
Proof .
For all stationary points, W T W commutes with B and Σ, which means that at these stationary points, one can simultaneously diagonalize all the matrices and the loss function (4) can be written as
[Equation]
(32) where λ i , b i , s i are the eigenvalues of W T W , B , and Σ respectively.
We can thus consider each i separately.
When b i > 0, λ i = 0 cannot be a local minimum because the local Hessian is − b i < 0.
When b i ≤ 0, the only stationary point is λ i = 0.
This sum covers at most d ∗ summands, and so, at the local minima, λ i ≠ if and only if b i > 0, and so the number of non-zero eigenvalues of W T W is min ( m, d ∗ ) . ◻
Extended Abstract Track
E.4. Proof of Proposition
Proof .
The regularization can be written as
[Equation]
By Proposition 6, Eq.
(5) reads
[Equation]
The derivative of ρ is
[Equation]
[Equation]
The zero-gradient gradient is thus
[Equation]
We can define B ′ ∶= B + 2 κc Σ − 2 κρ Σ to see that this condition is the same as Eq.
(24) in the proof of Theorem 1.
The rest of the proof thus follows from the arguments.
We thus arrive at the theorem statement:
[Equation]
We are done.
◻
E.5. Proof of Proposition
Proof .
Recalling that ρ =
Tr [ W Σ W T ] , we multiply Σ from the right to both sides of the solution in Proposition 3 and take trace:
[Equation]
[Equation]
The first line further simplifies to
[Equation]
43) where d M ∶=
Tr [ M ] is the number of nonzero eigenvalues of B ′ M .
This gives an equation of ρ that solves to
[Equation]
This proves the proposition.
◻