Original: T OWARDS A U NIFIED T HEORETICAL U NDERSTANDING OF N ON - CONTRASTIVE L EARNING VIA R ANK D IFFERENTIAL M ECHANISM
Revised: T OWARDS A U NIFIED T HEORETICAL U NDERSTANDING OF N ON - CONTRASTIVE L EARNING VIA R ANK D IFFERENTIAL M ECHANISM

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, a variety of methods under the name of non-contrastive learning (like
Revised: Recently, a variety of methods under the name of non-contrastive learning (like

Original: BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric architectural designs, aligning positive pairs alone is sufficient to attain good performance in self-supervised visual learning.
Revised: BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric architectural designs, aligning positive pairs alone is sufficient to attain good performance in self-supervised visual learning.

Original: Despite some understandings of some specific modules (like the predictor in BYOL), there is yet no unified theoretical understanding of how these seemingly different asymmetric designs can all avoid feature collapse, particularly considering methods that also work without the predictor (like DINO).
Revised: Despite some understandings of some specific modules (like the predictor in BYOL), there is yet no unified theoretical understanding of how these seemingly different asymmetric designs can all avoid feature collapse, particularly considering methods that also work without the predictor (like DINO).

Original: In this work, we propose a unified theoretical understanding for existing variants of non-contrastive learning.
Revised: In this work, we propose a unified theoretical understanding for existing variants of non-contrastive learning.

Original: Our theory named Rank Differential Mechanism (RDM) shows that all these asymmetric designs create a consistent rank difference in their dual-branch output features.
Revised: Our theory named Rank Differential Mechanism (RDM) shows that all these asymmetric designs create a consistent rank difference in their dual-branch output features.

Original: This rank difference will provably lead to an improvement of effective dimensionality and alleviate either complete or dimensional feature collapse.
Revised: This rank difference will provably lead to an improvement of effective dimensionality and alleviate either complete or dimensional feature collapse.

Original: Different from previous theories, our RDM theory is applicable to different asymmetric designs (with and without the predictor), and thus can serve as a unified understanding of existing non-contrastive learning methods.
Revised: Different from previous theories, our RDM theory is applicable to different asymmetric designs (with and without the predictor), and thus can serve as a unified understanding of existing non-contrastive learning methods.

Original: Besides, our RDM theory also provides practical guidelines for designing many new non-contrastive variants.
Revised: Besides, our RDM theory also provides practical guidelines for designing many new non-contrastive variants.

Original: We show that these variants indeed achieve comparable performance to existing methods on benchmark datasets, and some of them even outperform the baselines.
Revised: We show that these variants indeed achieve comparable performance to existing methods on benchmark datasets, and some of them even outperform the baselines.

Original: Our code is available at https://github.com/PKU-ML/
Revised: Our code is available at https://github.com/PKU-ML/

Original: Rank-Differential-Mechanism .
Revised: Rank-Differential-Mechanism .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning of visual representations has undergone rapid progress in recent years, particularly due to the rise of contrastive learning (CL)
Revised: Self-supervised learning of visual representations has undergone rapid progress in recent years, particularly due to the rise of contrastive learning (CL)

Original: (Oord et al., 2018).
Revised: (Oord et al., 2018; Wang et al., 2021).

Original: Canonical contrastive learning methods like SimCLR (Chen et al., 2020) and MoCo (He et al., 2020) utilize both positive samples (for feature alignment) and negative samples (for feature uniformity).
Revised: Canonical contrastive learning methods like SimCLR (Chen et al., 2020) and MoCo (He et al., 2020) utilize both positive samples (for feature alignment) and negative samples (for feature uniformity).

Original: Surprisingly, researchers notice that CL can also work well by only aligning positive samples, which isoften referred to as non-contrastive learning.
Revised: Surprisingly, researchers notice that CL can also work well by only aligning positive samples, which is referred to as non-contrastive learning.

Original: Without the help of negative samples, various techniques are proposed to prevent feature collapse, for example, stop-gradient, momentum encoder, predictor (BYOL (Grill et al., 2020), SimSiam (Chen & He, 2021)), Sinkhorn iterations (SwAV (Caron et al., 2020)), feature centering and sharpening (DINO (Caron et al., 2021)).
Revised: Without the help of negative samples, various techniques are proposed to prevent feature collapse, for example, stop-gradient, momentum encoder, predictor (BYOL (Grill et al., 2020), SimSiam (Chen & He, 2021)), Sinkhorn iterations (SwAV (Caron et al., 2020)), feature centering and sharpening (DINO (Caron et al., 2021)).

Original: These above designs all create a certain of asymmetry between the online branch (with gradient) and the target branch (without gradient) (Wang et al., 2022a).
Revised: These above designs all create a certain of asymmetry between the online branch (with gradient) and the target branch (without gradient) (Wang et al., 2022a).

Original: Empirically, these tricks can successfully alleviate feature collapse and obtain comparable or even superior performance than canonical contrastive learning.
Revised: Empirically, these tricks can successfully alleviate feature collapse and obtain comparable or even superior performance than canonical contrastive learning.

Original: Despite this progress, it is still not clear why these different heuristics can reach the same goal.
Revised: Despite this progress, it is still not clear why these different heuristics can reach the same goal.

Original: Some existing works are proposed to understand some specific non-contrastive techniques, mostly focusing on the predictor head proposed by BYOL (Grill et al., 2020).
Revised: Some existing works are proposed to understand some specific non-contrastive techniques, mostly focusing on the predictor head proposed by BYOL (Grill et al., 2020).

Original: From an empirical side, Chen & He (2021) think that the predictor helps approximate the expectation over augmentations, and
Revised: From an empirical side, Chen & He (2021) think that the predictor helps approximate the expectation over augmentations, and

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022a) take a center-residual decomposition of representations for analyzing the collapse.
Revised: (2022a) take a center-residual decomposition of representations for analyzing the collapse.

Original: From a theoretical perspective, Tian et al.
Revised: From a theoretical perspective, Tian et al.

Original: (2021) analyze the dynamics of predictor weights
Revised: (2021) analyze the dynamics of predictor weights

Original: [Figure]
Revised: [Figure]

Original: OnlineTarget under simple linear networks, and Wen & Li (2022) obtain optimization guarantees for two-layer nonlinear networks.
Revised: OnlineTarget under simple linear networks, and Wen & Li (2022) obtain optimization guarantees for two-layer nonlinear networks.

Original: These theoretical discussions often need strong assumptions on the data distribution ( e.g., standard normal (Tian et al., 2021)) and augmentations ( e.g., random masking (Wen &
Revised: These theoretical discussions often need strong assumptions on the data distribution ( e.g., standard normal (Tian et al., 2021)) and augmentations ( e.g., random masking (Wen &

Original: Li, 2022)).
Revised: Li, 2022)).

Original: Besides, their analyses are often problem-specific, which is hardly extendable to other non-contrastive variants without a predictor, e.g., DINO.
Revised: Besides, their analyses are often problem-specific, which is hardly extendable to other non-contrastive variants without a predictor, e.g., DINO.

Original: Therefore, a natural question is raised here:
Revised: Therefore, a natural question is raised here:

Original: Are there any basic principles behind these seemingly different techniques?
Revised: Are there any basic principles behind these seemingly different techniques?

Original: In this paper, we make the first attempt in this direction by discovering a common mechanism behind these non-contrastive variants.
Revised: In this paper, we make the first attempt in this direction by discovering a common mechanism behind these non-contrastive variants.

Original: To get a glimpse of it, in Figure 1, we measure the effective rank (Roy & Vetterli, 2007) of four different non-contrastive methods (BYOL, SimSiam, SwAV, and DINO).
Revised: To get a glimpse of it, in Figure 1, we measure the effective rank (Roy & Vetterli, 2007) of four different non-contrastive methods (BYOL, SimSiam, SwAV, and DINO).

Original: We find the following phenomenons: 1) among different methods, the target branch (orange line) has consistently higher rank than the online branch (blue line); 2) after the initial warmup stage, the rank of the online branch (blue line) consistently improves along the training process.
Revised: We find the following phenomenons: 1) among different methods, the target branch (orange line) has consistently higher rank than the online branch (blue line); 2) after the initial warmup stage, the rank of the online branch (blue line) consistently improves along the training process.

Original: Inspired by this observation, we propose a new theoretical understanding of non-contrastive methods, dubbed
Revised: Inspired by this observation, we propose a new theoretical understanding of non-contrastive methods, dubbed

Original: Rank Differential Mechanism (RDM), where we show that these different techniques essentially behave as a low-pass spectral filter, which is guaranteed to induce the rank difference above and avoid feature collapse along the training.
Revised: Rank Differential Mechanism (RDM), where we show that these different techniques essentially behave as a low-pass spectral filter, which is guaranteed to induce the rank difference above and avoid feature collapse along the training.

Original: We summarize the contribution of this work as follows:
Revised: We summarize the contribution of this work as follows:

Original: • Asymmetry matters for feature diversity .
Revised: • Asymmetry matters for feature diversity .

Original: In contrast to common beliefs, we show that even a symmetric architecture can provably alleviate complete feature collapse.
Revised: In contrast to common beliefs, we show that even a symmetric architecture can provably alleviate complete feature collapse.

Original: However, it still suffers from low feature diversity, collapsing to a very low dimensional subspace.
Revised: However, it still suffers from low feature diversity, collapsing to a very low dimensional subspace.

Original: It indicates the key role of asymmetry is to avoid dimensional feature collapse.
Revised: It indicates the key role of asymmetry is to avoid dimensional feature collapse.

Original: • Asymmetry induces low-pass filters that provably avoid dimensional collapse.
Revised: • Asymmetry induces low-pass filters that provably avoid dimensional collapse.

Original: Based on theoretical and empirical evidence on real-world data, we point out the common underlying mechanism of asymmetric designs in BYOL, SimSiam, SwAV, DINO is that they behave as low-pass online-branch filters, or equivalently, high-pass target-branch filters.
Revised: Based on theoretical and empirical evidence on real-world data, we point out the common underlying mechanism of asymmetric designs in BYOL, SimSiam, SwAV, DINO is that they behave as low-pass online-branch filters, or equivalently, high-pass target-branch filters.

Original: We further show that the asymmetry-induced low-pass filter can provably yield the rank collapse (Figure 1) and prevent feature collapse along the training process.
Revised: We further show that the asymmetry-induced low-pass filter can provably yield the rank collapse (Figure 1) and prevent feature collapse along the training process.

Original: • Principled designs of asymmetry.
Revised: • Principled designs of asymmetry.

Original: Following the principle of RDM, we design a series of non-contrastive variants to empirically verify the effectiveness of our theory.
Revised: Following the principle of RDM, we design a series of non-contrastive variants to empirically verify the effectiveness of our theory.

Original: For the online encoder, we show that different variants of low-pass filters can also attain fairly good performance.
Revised: For the online encoder, we show that different variants of low-pass filters can also attain fairly good performance.

Original: We also design a new kind of target predictors with high-pass filters .
Revised: We also design a new kind of target predictors with high-pass filters .

Original: Experiments show that SimSiam with our target predictors can outperform DirectPred (Tian et al., 2021) and achieve comparable or even superior performance to the original SimSiam.
Revised: Experiments show that SimSiam with our target predictors can outperform DirectPred (Tian et al., 2021) and achieve comparable or even superior performance to the original SimSiam.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Non-contrastive Learning .
Revised: Non-contrastive Learning .

Original: Among existing methods, BYOL (Grill et al., 2020) is the first to show we can alleviate the feature collapse of aligning positive samples along with an online predictor and a momentum encoder.
Revised: Among existing methods, BYOL (Grill et al., 2020) is the first to show we can alleviate the feature collapse of aligning positive samples along with an online predictor and a momentum encoder.

Original: Later, SimSiam (Chen & He, 2021) further simplifies this requirement and shows that only the online predictor is enough.
Revised: Later, SimSiam (Chen & He, 2021) further simplifies this requirement and shows that only the online predictor is enough.

Original: As for another thread, SwAV (Caron et al., 2020) applies Sinkhorn-Knopp iterations (Cuturi, 2013) to the target output from an optimal transport view.
Revised: As for another thread, SwAV (Caron et al., 2020) applies Sinkhorn-Knopp iterations (Cuturi, 2013) to the target output from an optimal transport view.

Original: DINO (Caron et al., 2021) further simplifies this approach by simply combining feature centering and feature sharpening.
Revised: DINO (Caron et al., 2021) further simplifies this approach by simply combining feature centering and feature sharpening.

Original: Remarkably, all these methods adopt an online-target dual-branch architecture and gradients from the target branch are detached.
Revised: Remarkably, all these methods adopt an online-target dual-branch architecture and gradients from the target branch are detached.

Original: Our theory provides a unified understanding of these designs and reveals their common underlying mechanisms.
Revised: Our theory provides a unified understanding of these designs and reveals their common underlying mechanisms.

Original: Additional comparison with related work is included in Appendix F.
Revised: Additional comparison with related work is included in Appendix F.

Original: [Figure]
Revised: [Figure]

Original: Dimensional Collapse of Self-supervised Representations.
Revised: Dimensional Collapse of Self-supervised Representations.

Original: Prior to ours, several works also explore the dimensional collapse issue in contrastive learning.
Revised: Prior to ours, several works also explore the dimensional collapse issue in contrastive learning.

Original: Ermolov et al.
Revised: Ermolov et al.

Original: (2021), Hua et al.
Revised: (2021), Hua et al.

Original: Weng et al.
Revised: Weng et al.

Original: (2022) and Zhang et al.
Revised: (2022) and Zhang et al.

Original: (2022b)
Revised: (2022b)

Original: propose whitening techniques to alleviate dimensional collapse, similar in spirit to Barlow Twins (Zbontar et al., 2021) with a feature decorrelation regularization.
Revised: propose whitening techniques to alleviate dimensional collapse, similar in spirit to Barlow Twins (Zbontar et al., 2021) with a feature decorrelation regularization.

Original: Jing et al.
Revised: Jing et al.

Original: (2022) point out the dimensional collapse of contrastive learning without using the projector
Revised: (2022) point out the dimensional collapse of contrastive learning without using the projector

Original: [Figure]
Revised: [Figure]

Original: and propose DirectCLR as a direct replacement.
Revised: and propose DirectCLR as a direct replacement.

Original: Instead, our work mainly focuses on understanding the role of asymmetric designs on overcoming dimensional collapse.
Revised: Instead, our work mainly focuses on understanding the role of asymmetric designs on overcoming dimensional collapse.

Original: Theoretical Analysis on Contrastive Learning .
Revised: Theoretical Analysis on Contrastive Learning .

Original: Saunshi et al.
Revised: Saunshi et al.

Original: (2019) first establish downstream guarantees for contrastive learning, which are later gradually refined (Nozawa & Sato, 2021; Ash et al., 2022; Bao et al., 2022).
Revised: (2019) first establish downstream guarantees for contrastive learning, which are later gradually refined (Nozawa & Sato, 2021; Ash et al., 2022; Bao et al., 2022).

Original: Tosh et al.
Revised: Tosh et al.

Original: (2021) and Lee et al.
Revised: (2021) and Lee et al.

Original: (2021) also propose similar guarantees on downstream tasks.
Revised: (2021) also propose similar guarantees on downstream tasks.

Original: However, these methods mostly rely on the conditional independence assumption that is far from practice.
Revised: However, these methods mostly rely on the conditional independence assumption that is far from practice.

Original: Recently, HaoChen et al.
Revised: Recently, HaoChen et al.

Original: (2021) and Wang et al.
Revised: (2021) and Wang et al.

Original: (2022b) propose an augmentation graph perspective with more practical assumptions, and contribute the generalization ability to the existence of augmentation overlap (which also exists for non-contrastive learning).
Revised: (2022b) propose an augmentation graph perspective with more practical assumptions, and contribute the generalization ability to the existence of augmentation overlap (which also exists for non-contrastive learning).

Original: Wen & Li (2021) analyze the feature dynamics of contrastive learning with shadow ReLU networks.
Revised: Wen & Li (2021) analyze the feature dynamics of contrastive learning with shadow ReLU networks.

Original:  A SYMMETRY IS THE K EY TO A LLEVIATE D IMENSIONAL C OLLAPSE
Revised:  A SYMMETRY IS THE K EY TO A LLEVIATE D IMENSIONAL C OLLAPSE

Original: Prior works tend to believe that asymmetric designs are necessary for avoiding complete feature collapse (Zhang et al., 2022a), while we show that a fully symmetric architecture, dubbed SymSimSiam (Symmetric Simple Siamese network), can also avoid complete collapse.
Revised: Prior works tend to believe that asymmetric designs are necessary for avoiding complete feature collapse (Zhang et al., 2022a), while we show that a fully symmetric architecture, dubbed SymSimSiam (Symmetric Simple Siamese network), can also avoid complete collapse.

Original: Specifically, we simply align the positive pair ( x, x + ) with a symmetric alignment loss,
Revised: Specifically, we simply align the positive pair ( x, x + ) with a symmetric alignment loss,

Original: [Equation]
Revised: [Equation]

Original: where we apply feature centering on the output of an encoder f , i.e., f ′ θ ( · )
Revised: where we apply feature centering on the output of an encoder f , i.e., f ′ θ ( · )

Original: =
Revised: =

Original: f θ ( · ) − µ .
Revised: f θ ( · ) − µ .

Original: The feature average µ = E x f θ ( x ) can be computed via mini-batch estimate or exponential moving average as in Batch Normalization (Ioffe & Szegedy, 2015)
Revised: The feature average µ = E x f θ ( x ) can be computed via mini-batch estimate or exponential moving average as in Batch Normalization (Ioffe & Szegedy, 2015)

Original: (see Algorithm 1).
Revised: (see Algorithm 1).

Original: Theorem 1 states that SymSimSiam can avoid complete collapse by simultaneously maximizing feature variance Var( f θ ( x )) .
Revised: Theorem 1 states that SymSimSiam can avoid complete collapse by simultaneously maximizing feature variance Var( f θ ( x )) .

Original: Theorem 1.
Revised: Theorem 1.

Original: When f θ ( x ) is ℓ 2 -normalized, the SymSimSiam objective is equivalent to
Revised: When f θ ( x ) is ℓ 2 -normalized, the SymSimSiam objective is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Empirically shown in Figures 2(a) & 2(b), compared to vanilla Siamese network, SymSimSiam indeed alleviates complete collapse, i.e., the feature variance is maximized along training and a good linear probing accuracy is achieved.
Revised: Empirically shown in Figures 2(a) & 2(b), compared to vanilla Siamese network, SymSimSiam indeed alleviates complete collapse, i.e., the feature variance is maximized along training and a good linear probing accuracy is achieved.

Original: However, the accuracy of SymSimSiam is also obviously lower than the asymmetric SimSiam (Figure 2(a)).
Revised: However, the accuracy of SymSimSiam is also obviously lower than the asymmetric SimSiam (Figure 2(a)).

Original: This indicates that symmetric design could alleviate complete collapse, but it may be not enough to prevent dimensional feature collapse .
Revised: This indicates that symmetric design could alleviate complete collapse, but it may be not enough to prevent dimensional feature collapse .

Original: Intuitively, features uniformly distributed on a great circle of a unit sphere have maximal variance while being dimensionally collapsed.
Revised: Intuitively, features uniformly distributed on a great circle of a unit sphere have maximal variance while being dimensionally collapsed.

Original: As further shown in Figure 2(c), SymSimSiam indeed suffers from more severe dimensional collapse compared to SimSiam.
Revised: As further shown in Figure 2(c), SymSimSiam indeed suffers from more severe dimensional collapse compared to SimSiam.

Original: With few effective dimensions, the encoder network has limited ability to encode rich semantics (HaoChen et al., 2021).
Revised: With few effective dimensions, the encoder network has limited ability to encode rich semantics (HaoChen et al., 2021).

Original: The above SymSimSiam experiments show that with a symmetric architecture, we can easily prevent complete collapse while hardly improving the effective feature dimensionality for overcoming
Revised: The above SymSimSiam experiments show that with a symmetric architecture, we can easily prevent complete collapse while hardly improving the effective feature dimensionality for overcoming

Original: [Figure]
Revised: [Figure]

Original: dimensional collapse.
Revised: dimensional collapse.

Original: Instead, we also notice that SimSiam with asymmetric designs can alleviate dimensional collapse and achieve better performance.
Revised: Instead, we also notice that SimSiam with asymmetric designs can alleviate dimensional collapse and achieve better performance.

Original: It reflects the fact that the asymmetry in existing non-contrastive methods is the key to alleviating dimensional collapse , which leads us to the main focus of our paper, i.e., demystifying asymmetric designs.
Revised: It reflects the fact that the asymmetry in existing non-contrastive methods is the key to alleviating dimensional collapse , which leads us to the main focus of our paper, i.e., demystifying asymmetric designs.

Original:  T HE R ANK D IFFERENCE M ECHANISM OF A SYMMETRIC D ESIGNS
Revised:  T HE R ANK D IFFERENCE M ECHANISM OF A SYMMETRIC D ESIGNS

Original: In Figure 1, we have observed a common mechanism behind the non-contrastive learning methods: these asymmetric designs create positive rank differences between the target and online outputs consistently throughout training.
Revised: In Figure 1, we have observed a common mechanism behind the non-contrastive learning methods: these asymmetric designs create positive rank differences between the target and online outputs consistently throughout training.

Original: Here, we provide a formal analysis of this phenomenon from both theoretical and empirical sides and show how it helps alleviate dimensional feature collapse.
Revised: Here, we provide a formal analysis of this phenomenon from both theoretical and empirical sides and show how it helps alleviate dimensional feature collapse.

Original: Problem Setup.
Revised: Problem Setup.

Original: Given a set of natural training data ¯ X = { ¯ x | ¯ x ∈ R d } , we can draw a pair of positive samples ( x, x + ) from independent random augmentations of a natural example ¯ x with distribution A ( ·| ¯ x ) .
Revised: Given a set of natural training data ¯ X = { ¯ x | ¯ x ∈ R d } , we can draw a pair of positive samples ( x, x + ) from independent random augmentations of a natural example ¯ x with distribution A ( ·| ¯ x ) .

Original: Their joint distribution satisfies P ( x, x + ) =
Revised: Their joint distribution satisfies P ( x, x + ) =

Original: P ( x + , x ) =
Revised: P ( x + , x ) =

Original: E ¯ x A ( x | ¯ x ) A ( x + | ¯ x ) .
Revised: E ¯ x A ( x | ¯ x ) A ( x + | ¯ x ) .

Original: Without loss of generality, we consider a finite sample space |X| =
Revised: Without loss of generality, we consider a finite sample space |X| =

Original: n (can be exponentially large) following HaoChen et al.
Revised: n (can be exponentially large) following HaoChen et al.

Original: (2021), and denote the collection of online and target outputs as p, z ∈ R n × k whose x -th row is p x , z x , respectively.
Revised: (2021), and denote the collection of online and target outputs as p, z ∈ R n × k whose x -th row is p x , z x , respectively.

Original: We consider a general alignment loss
Revised: We consider a general alignment loss

Original: [Equation]
Revised: [Equation]

Original: to cover different variants of non-contrastive methods.
Revised: to cover different variants of non-contrastive methods.

Original: The online and target outputs p x , z x are either ℓ 2 -normalized (BYOL and SimSiam) or softmax -normalized (SwAV and DINO).
Revised: The online and target outputs p x , z x are either ℓ 2 -normalized (BYOL and SimSiam) or softmax -normalized (SwAV and DINO).

Original: For the loss function ℓ , BYOL and SimSiam adopt the mean square error (MSE) loss, while SwAV and
Revised: For the loss function ℓ , BYOL and SimSiam adopt the mean square error (MSE) loss, while SwAV and

Original: DINO adopt the cross entropy (CE) loss.
Revised: DINO adopt the cross entropy (CE) loss.

Original: sg( · ) denotes stopping the gradients from the operand.
Revised: sg( · ) denotes stopping the gradients from the operand.

Original: For an encoder f , we define its feature correlation matrix C =
Revised: For an encoder f , we define its feature correlation matrix C =

Original: E x f ( x ) f ( x ) ⊤ , whose spectral decomposition is C = V Λ V ⊤ , where V contains unit eigenvectors in columns, and Λ is the diagonal matrix with descending eigenvalues λ 1 ≥ · · · ≥ λ k ≥ 0 .
Revised: E x f ( x ) f ( x ) ⊤ , whose spectral decomposition is C = V Λ V ⊤ , where V contains unit eigenvectors in columns, and Λ is the diagonal matrix with descending eigenvalues λ 1 ≥ · · · ≥ λ k ≥ 0 .

Original: Measure of Dimensional Collapse.
Revised: Measure of Dimensional Collapse.

Original: A well-known measure of the effective feature dimensionality is the effective rank (erank) of the feature correlation matrix C =
Revised: A well-known measure of the effective feature dimensionality is the effective rank (erank) of the feature correlation matrix C =

Original: E x f ( x ) f ( x ) ⊤ (Roy & Vetterli, 2007).
Revised: E x f ( x ) f ( x ) ⊤ (Roy & Vetterli, 2007).

Original: Specifically, erank( C ) = exp( H ( q )) , where q = ( q 1 , . . . , q k ) , q i
Revised: Specifically, erank( C ) = exp( H ( q )) , where q = ( q 1 , . . . , q k ) , q i

Original: = λ
Revised: = λ

Original: i / (cid:80)
Revised: i / (cid:80)

Original: i
Revised: i

Original: λ
Revised: λ

Original: i are the normalized eigenvalues as a probability distribution, and H ( q )
Revised: i are the normalized eigenvalues as a probability distribution, and H ( q )

Original: = − (cid:80)
Revised: = − (cid:80)

Original: i q i log( q i ) is its Shannon entropy.
Revised: i q i log( q i ) is its Shannon entropy.

Original: Compared to the canonical rank, the effective rank is real-valued and invariant to feature scaling.
Revised: Compared to the canonical rank, the effective rank is real-valued and invariant to feature scaling.

Original: A more uniform distribution of eigenvalues has a larger effective rank, and vice versa.
Revised: A more uniform distribution of eigenvalues has a larger effective rank, and vice versa.

Original: Thus, the effective rank of C is a proper metric to measure the degree of dimensional feature collapse.
Revised: Thus, the effective rank of C is a proper metric to measure the degree of dimensional feature collapse.

Original: Spectral Filters.
Revised: Spectral Filters.

Original: In the signal processing literature, a spectral filtering process G of a signal f is to apply a scalar function ( i.e., a spectral filter) g : R → R element-wisely on its eigenvalues in its spectral domain, i.e., u x = Gf ( x ) =
Revised: In the signal processing literature, a spectral filtering process G of a signal f is to apply a scalar function ( i.e., a spectral filter) g : R → R element-wisely on its eigenvalues in its spectral domain, i.e., u x = Gf ( x ) =

Original: V g (Λ) V ⊤ f ( x ) , where G = V g (Λ) V ⊤ is also known as a spectral convolution operator.
Revised: V g (Λ) V ⊤ f ( x ) , where G = V g (Λ) V ⊤ is also known as a spectral convolution operator.

Original: The filtered signal admits C u
Revised: The filtered signal admits C u

Original: =
Revised: =

Original: E x u x u ⊤ x
Revised: E x u x u ⊤ x

Original: = V g (Λ)Λ V T .
Revised: = V g (Λ)Λ V T .

Original: Depending on the property of g , a filter can be categorized as low-pass, high-pass, band-pass or band-stop.
Revised: Depending on the property of g , a filter can be categorized as low-pass, high-pass, band-pass or band-stop.

Original: Generally speaking, a low-pass filter will amplify large eigenvalues and diminish smaller ones ( e.g., a monotonically increasing function), and a high-pass filter does the opposite.
Revised: Generally speaking, a low-pass filter will amplify large eigenvalues and diminish smaller ones ( e.g., a monotonically increasing function), and a high-pass filter does the opposite.

Original: Many canonical algorithms can be seen as special cases of spectral filtering, e.g., PCA-based image denoising is low-pass filtering.
Revised: Many canonical algorithms can be seen as special cases of spectral filtering, e.g., PCA-based image denoising is low-pass filtering.

Original: 4. A SYMMETRIC D ESIGNS B EHAVE AS S PECTRAL F ILTERS
Revised: 4. A SYMMETRIC D ESIGNS B EHAVE AS S PECTRAL F ILTERS

Original: First of all, we notice that regardless of the existence of asymmetry, the alignment of two-branch outputs in non-contrastive learning will enforce the two-branch output features of the positive pairs to be close to each other.
Revised: First of all, we notice that regardless of the existence of asymmetry, the alignment of two-branch outputs in non-contrastive learning will enforce the two-branch output features of the positive pairs to be close to each other.

Original: Therefore, from a spectral viewpoint, a natural hypothesis is that the online and target features will be aligned into the same eigenspace, and only differ slightly in the eigenvalues.
Revised: Therefore, from a spectral viewpoint, a natural hypothesis is that the online and target features will be aligned into the same eigenspace, and only differ slightly in the eigenvalues.

Original: We describe this hypothesis formally below.
Revised: We describe this hypothesis formally below.

Original: Definition 1 (Eigenspace alignment) .
Revised: Definition 1 (Eigenspace alignment) .

Original: For two matrices A and B , they have aligned eigenspace if
Revised: For two matrices A and B , they have aligned eigenspace if

Original: [Equation]
Revised: [Equation]

Original: where V is an orthogonal matrix of eigenvectors and Λ a , Λ b consist non-increasing eigenvalues.
Revised: where V is an orthogonal matrix of eigenvectors and Λ a , Λ b consist non-increasing eigenvalues.

Original: Hypothesis 1.
Revised: Hypothesis 1.

Original: During training, non-contrastive learning aligns the eigenspace of three correlation matrices of output features: the online correlation C p =
Revised: During training, non-contrastive learning aligns the eigenspace of three correlation matrices of output features: the online correlation C p =

Original: E x p x p ⊤ x , the target correlation C z =
Revised: E x p x p ⊤ x , the target correlation C z =

Original: E x z x z ⊤ x , and the feature correlation of positive samples C + = E x,x + z x z ⊤ x + .
Revised: E x z x z ⊤ x , and the feature correlation of positive samples C + = E x,x + z x z ⊤ x + .

Original: Next, we validate this hypothesis from both theoretical and empirical aspects.
Revised: Next, we validate this hypothesis from both theoretical and empirical aspects.

Original: To begin with, we consider a simplified setting adopted in prior work (Tian et al., 2021) for the ease of analysis: 1) data isotropy, where the natural data distribution p (¯ x ) has zero mean and identity covariance, and the augmentation A ( x | ¯ x ) has mean ¯ x and covariance σ 2 I ; 2) linear encoder
Revised: To begin with, we consider a simplified setting adopted in prior work (Tian et al., 2021) for the ease of analysis: 1) data isotropy, where the natural data distribution p (¯ x ) has zero mean and identity covariance, and the augmentation A ( x | ¯ x ) has mean ¯ x and covariance σ 2 I ; 2) linear encoder

Original: z x = f ( x ) =
Revised: z x = f ( x ) =

Original: W f x, W f ∈
Revised: W f x, W f ∈

Original: R d × k ; 3) linear online predictor p x =
Revised: R d × k ; 3) linear online predictor p x =

Original: Wz x , W ∈ R k × k .
Revised: Wz x , W ∈ R k × k .

Original: Under this setting, the following lemma shows that for an arbitrary encoder f , the eigenspace of three correlation matrices indeed align well:
Revised: Under this setting, the following lemma shows that for an arbitrary encoder f , the eigenspace of three correlation matrices indeed align well:

Original: Lemma 1.
Revised: Lemma 1.

Original: With the assumptions above as in Tian et al. (2021), when the predictor W ∗ minimizes the alignment loss (Eq. 3), we have
Revised: With the assumptions above as in Tian et al. (2021), when the predictor W ∗ minimizes the alignment loss (Eq. 3), we have

Original: [Equation]
Revised: [Equation]

Original: where V is an orthogonal matrix and Λ p , Λ z , Λ + are diagonal matrices consisting of descending eigenvalues λ pi , λ zi , λ
Revised: where V is an orthogonal matrix and Λ p , Λ z , Λ + are diagonal matrices consisting of descending eigenvalues λ pi , λ zi , λ

Original: + i , i = 1 , . . .
Revised: + i , i = 1 , . . .

Original: , k, respectively.
Revised: , k, respectively.

Original: Next, we provide an empirical examination of Hypothesis 1 on real-world data.
Revised: Next, we provide an empirical examination of Hypothesis 1 on real-world data.

Original: From Figures 3, we can see that there is a consistently high degree of eigenspace alignment between C p and C z among all non-contrastive methods.
Revised: From Figures 3, we can see that there is a consistently high degree of eigenspace alignment between C p and C z among all non-contrastive methods.

Original: In Appendix E.1, we further verify the alignment w.r.t.
Revised: In Appendix E.1, we further verify the alignment w.r.t.

Original: C + .
Revised: C + .

Original: Therefore, these methods indeed attain a fairly high degree of eigenspace alignment.
Revised: Therefore, these methods indeed attain a fairly high degree of eigenspace alignment.

Original: A Spectral Filter View.
Revised: A Spectral Filter View.

Original: As a result of eigenspace alignment, the alignment loss essentially works on mitigating the difference in eigenvalues .
Revised: As a result of eigenspace alignment, the alignment loss essentially works on mitigating the difference in eigenvalues .

Original: Therefore, we can take a spectral perspective on noncontrastive methods, where the asymmetric designs are equivalent to a spectral filtering process applied to the target output z x , or a target spectral filtering process applied to the target output p x .
Revised: Therefore, we can take a spectral perspective on noncontrastive methods, where the asymmetric designs are equivalent to a spectral filtering process applied to the target output z x , or a target spectral filtering process applied to the target output p x .

Original: As the two cases are equivalent, we mainly take the online filter as an example in the discussion below.
Revised: As the two cases are equivalent, we mainly take the online filter as an example in the discussion below.

Original: Lemma 2.
Revised: Lemma 2.

Original: Denote an online filter function g : λ z → λ g that satisfies λ gi =
Revised: Denote an online filter function g : λ z → λ g that satisfies λ gi =

Original: (cid:112) λ pi /λ zi , i = 1 , . . .
Revised: (cid:112) λ pi /λ zi , i = 1 , . . .

Original: , k .
Revised: , k .

Original: We can apply a spectral filtering on z x with g , and get ˜ p x
Revised: We can apply a spectral filtering on z x with g , and get ˜ p x

Original: =
Revised: =

Original: W
Revised: W

Original: g
Revised: g

Original: z x , W g
Revised: z x , W g

Original: =
Revised: =

Original: V g (Λ z ) V ⊤ .
Revised: V g (Λ z ) V ⊤ .

Original: Then, we have C p = C ˜
Revised: Then, we have C p = C ˜

Original: p
Revised: p

Original: =
Revised: =

Original: E x ˜ p x ˜ p ⊤ x .
Revised: E x ˜ p x ˜ p ⊤ x .

Original: In other words, p x , ˜ p
Revised: In other words, p x , ˜ p

Original: x have the same feature correlation.
Revised: x have the same feature correlation.

Original: This spectral filter view reveals the key difference between symmetric and asymmetric designs in non-contrastive learning.
Revised: This spectral filter view reveals the key difference between symmetric and asymmetric designs in non-contrastive learning.

Original: In the symmetric case, the two branches yield almost equal eigenvalues (Figure 2(d)).
Revised: In the symmetric case, the two branches yield almost equal eigenvalues (Figure 2(d)).

Original: Thus, the alignment loss will quickly diminish and the representations collapse dimensionally (Figure 2(c)).
Revised: Thus, the alignment loss will quickly diminish and the representations collapse dimensionally (Figure 2(c)).

Original: Instead, in asymmetric designs, asymmetric components create a difference in eigenvalues such that the target output generally has a higher rank than the online output (Figure 4(a)).
Revised: Instead, in asymmetric designs, asymmetric components create a difference in eigenvalues such that the target output generally has a higher rank than the online output (Figure 4(a)).

Original: Therefore, the alignment loss will not easily diminish (not necessarily decrease; see Figure 9).
Revised: Therefore, the alignment loss will not easily diminish (not necessarily decrease; see Figure 9).

Original: Instead, the alignment improves feature diversity in an implicit way, as we will show later.
Revised: Instead, the alignment improves feature diversity in an implicit way, as we will show later.

Original: 4. L OW - PASS P ROPERTY OF A SYMMETRY - INDUCED S PECTRAL F ILTERS
Revised: 4. L OW - PASS P ROPERTY OF A SYMMETRY - INDUCED S PECTRAL F ILTERS

Original: The discussion above reveals that a specific asymmetric design behaves as a spectral filter when applied to non-contrastive learning.
Revised: The discussion above reveals that a specific asymmetric design behaves as a spectral filter when applied to non-contrastive learning.

Original: To gain some insights for a unified understanding, we further investigate whether there is a common pattern behind the filters of different asymmetric designs.
Revised: To gain some insights for a unified understanding, we further investigate whether there is a common pattern behind the filters of different asymmetric designs.

Original: To achieve this, we calculate and plot the corresponding online filter g ( λ z )
Revised: To achieve this, we calculate and plot the corresponding online filter g ( λ z )

Original: =
Revised: =

Original: (cid:112) λ pi /λ zi of each non-contrastive method.
Revised: (cid:112) λ pi /λ zi of each non-contrastive method.

Original: From Figure 4(b), we find that the spectral filters indeed look very similar,
Revised: From Figure 4(b), we find that the spectral filters indeed look very similar,

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: particularly in the sense that all filter functions are roughly monotonically increasing w.r.t.
Revised: particularly in the sense that all filter functions are roughly monotonically increasing w.r.t.

Original: λ z .
Revised: λ z .

Original: This kind of filter is usually called a low-pass filter because it (relatively) enlarges low-frequency components (large eigenvalues) and shrinks high-frequency components (small eigenvalues).
Revised: This kind of filter is usually called a low-pass filter because it (relatively) enlarges low-frequency components (large eigenvalues) and shrinks high-frequency components (small eigenvalues).

Original: Based on this empirical finding, we propose the following hypothesis on the low-pass nature of asymmetry.
Revised: Based on this empirical finding, we propose the following hypothesis on the low-pass nature of asymmetry.

Original: Hypothesis 2.
Revised: Hypothesis 2.

Original: Asymmetric modules in non-contrastive learning behave as low-pass online filters.
Revised: Asymmetric modules in non-contrastive learning behave as low-pass online filters.

Original: Formally, the corresponding spectral filter g ( λ z ) =
Revised: Formally, the corresponding spectral filter g ( λ z ) =

Original: (cid:112) λ
Revised: (cid:112) λ

Original: pi /λ zi is monotonically increasing.
Revised: pi /λ zi is monotonically increasing.

Original: We note that we are not suggesting any asymmetric design behaves as low-pass filters, as someone could easily apply a high-pass filter to the online output deliberately (which, as we have observed, will likely fail).
Revised: We note that we are not suggesting any asymmetric design behaves as low-pass filters, as someone could easily apply a high-pass filter to the online output deliberately (which, as we have observed, will likely fail).

Original: Therefore, our hypothesis above only applies to asymmetric designs that work well in practice.
Revised: Therefore, our hypothesis above only applies to asymmetric designs that work well in practice.

Original: In the discussion below, we further provide theoretical and empirical investigations of why existing asymmetric modules have a low-pass filtering effect.
Revised: In the discussion below, we further provide theoretical and empirical investigations of why existing asymmetric modules have a low-pass filtering effect.

Original: Case I. Online Predictor.
Revised: Case I. Online Predictor.

Original: One popular kind of non-contrastive method, including BYOL and SimSiam, utilizes a learnable online predictor g θ : R k
Revised: One popular kind of non-contrastive method, including BYOL and SimSiam, utilizes a learnable online predictor g θ : R k

Original: → R k for architectural asymmetry.
Revised: → R k for architectural asymmetry.

Original: One would wonder why such a learnable predictor will behave as a low-pass filter (Lemma 2).
Revised: One would wonder why such a learnable predictor will behave as a low-pass filter (Lemma 2).

Original: Here, we provide some theoretical insights in the following theorem
Revised: Here, we provide some theoretical insights in the following theorem

Original: Theorem 2.
Revised: Theorem 2.

Original: Under Hypothesis 1, assume the invertibility of C z , the optimal predictor is given by
Revised: Under Hypothesis 1, assume the invertibility of C z , the optimal predictor is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ω
Revised: ω

Original: [Equation]
Revised: [Equation]

Original: = Ω
Revised: = Ω

Original: [Equation]
Revised: [Equation]

Original: = λ
Revised: = λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, the spectral property of the learnable filter is determined by the filter function λ
Revised: Therefore, the spectral property of the learnable filter is determined by the filter function λ

Original: + i /λ zi .
Revised: + i /λ zi .

Original: Theorem 2 shows that the predictor essentially learns to predict the correlation C + between positive samples from one augmented view C z , i.e., eliminating the augmentation noise and predicting the common features.
Revised: Theorem 2 shows that the predictor essentially learns to predict the correlation C + between positive samples from one augmented view C z , i.e., eliminating the augmentation noise and predicting the common features.

Original: The following lemma further reveals that 1) the correlation between positive samples is equivalent to the correlation of their underlying natural data ¯ x , and 2) the difference between C + and C z is equal to the conditional covariance induced by data augmentation A ( x | ¯ x ) .
Revised: The following lemma further reveals that 1) the correlation between positive samples is equivalent to the correlation of their underlying natural data ¯ x , and 2) the difference between C + and C z is equal to the conditional covariance induced by data augmentation A ( x | ¯ x ) .

Original: Lemma 3.
Revised: Lemma 3.

Original: The following equalities hold:
Revised: The following equalities hold:

Original: C + = ¯ C :=
Revised: C + = ¯ C :=

Original: E ¯ x
Revised: E ¯ x

Original: z ¯ x z ⊤ ¯ x , where z ¯ x = E x | ¯ x z x ; 2.
Revised: z ¯ x z ⊤ ¯ x , where z ¯ x = E x | ¯ x z x ; 2.

Original: C z = ¯ C + V x
Revised: C z = ¯ C + V x

Original: | ¯ x , where V x | ¯ x =
Revised: | ¯ x , where V x | ¯ x =

Original: E ¯ x E x
Revised: E ¯ x E x

Original: | ¯ x ( z x − z ¯ x ) ( z x − z ¯ x ) ⊤ is the conditional covariance.
Revised: | ¯ x ( z x − z ¯ x ) ( z x − z ¯ x ) ⊤ is the conditional covariance.

Original: In practice, data augmentations mainly cause high-frequency noises in the feature space, therefore, the denoising predictor will behave as a low-pass filter.
Revised: In practice, data augmentations mainly cause high-frequency noises in the feature space, therefore, the denoising predictor will behave as a low-pass filter.

Original: Indeed, Figures 5(a) and 5(b) empirically show that the filter derived from our theory, λ
Revised: Indeed, Figures 5(a) and 5(b) empirically show that the filter derived from our theory, λ

Original: + i /λ zi , aligns well with the actual learned predictor in
Revised: + i /λ zi , aligns well with the actual learned predictor in

Original: Figure 4(b) as a low-pass filter.
Revised: Figure 4(b) as a low-pass filter.

Original: Case II.
Revised: Case II.

Original: Target Transformation.
Revised: Target Transformation.

Original: Another kind of non-contrastive method is to apply hand-crafted (instead of learned) transformations in the target branch, such as the Sinkhorn-Knopp (SK) iteration in SwAV (Caron et al., 2020) and centering-sharpening operators in DINO (Caron et al., 2021).
Revised: Another kind of non-contrastive method is to apply hand-crafted (instead of learned) transformations in the target branch, such as the Sinkhorn-Knopp (SK) iteration in SwAV (Caron et al., 2020) and centering-sharpening operators in DINO (Caron et al., 2021).

Original: In this case, we further study how these transformations behave as high-pass filters applied to the target output (see footnote of Hypothesis 2).
Revised: In this case, we further study how these transformations behave as high-pass filters applied to the target output (see footnote of Hypothesis 2).

Original: Since it is generally hard to analyze these spatial transformations in the spectral domain, here we empirically study the role of each transformation on the resulting filter.
Revised: Since it is generally hard to analyze these spatial transformations in the spectral domain, here we empirically study the role of each transformation on the resulting filter.

Original: Figure 5(c) shows that SK iterations indeed act like low-pass filters, and one iteration is enough.
Revised: Figure 5(c) shows that SK iterations indeed act like low-pass filters, and one iteration is enough.

Original: This explains why a few SK iterations already work well in SwAV.
Revised: This explains why a few SK iterations already work well in SwAV.

Original: As for
Revised: As for

Original: DINO, we notice that centering operation alone is not enough to produce a high-pass filter, which agrees with empirical results in DINO.
Revised: DINO, we notice that centering operation alone is not enough to produce a high-pass filter, which agrees with empirical results in DINO.

Original: Meanwhile, we notice that in order to obtain a high-pass filter (monotonically decreasing), it is necessary for DINO to apply a temperature smaller than the online branch ( < 0 . 1 ), which is exactly the feature sharpening technique adopted in DINO (Caron et al., 2021).
Revised: Meanwhile, we notice that in order to obtain a high-pass filter (monotonically decreasing), it is necessary for DINO to apply a temperature smaller than the online branch ( < 0 . 1 ), which is exactly the feature sharpening technique adopted in DINO (Caron et al., 2021).

Original: These facts show our theory aligns well with empirical results in non-contrastive learning.
Revised: These facts show our theory aligns well with empirical results in non-contrastive learning.

Original: 4. A SYMMETRY - INDUCED L OW - PASS F ILTERS S AVE N ON - CONTRASTIVE L EARNING
Revised: 4. A SYMMETRY - INDUCED L OW - PASS F ILTERS S AVE N ON - CONTRASTIVE L EARNING

Original: In the discussion above, we observed a common pattern in existing asymmetric designs: their corresponding spectral filters are all low-pass.
Revised: In the discussion above, we observed a common pattern in existing asymmetric designs: their corresponding spectral filters are all low-pass.

Original: Here, we further show that this property is so essential that it can provably save non-contrastive learning from the risk of feature collapse by producing the rank difference between outputs and alleviating dimensional collapse during training.
Revised: Here, we further show that this property is so essential that it can provably save non-contrastive learning from the risk of feature collapse by producing the rank difference between outputs and alleviating dimensional collapse during training.

Original: First, let us take a look at its effect on the effective rank of two-branch output features.
Revised: First, let us take a look at its effect on the effective rank of two-branch output features.

Original: In the theorem below, we show that low-pass online filters are guaranteed to produce a consistently higher effective rank of target features than that of online features, as shown in Figure 1.
Revised: In the theorem below, we show that low-pass online filters are guaranteed to produce a consistently higher effective rank of target features than that of online features, as shown in Figure 1.

Original: Theorem 3.
Revised: Theorem 3.

Original: If g ( λ zi ) =
Revised: If g ( λ zi ) =

Original: (cid:112) λ pi /λ zi is monotonically increasing, or equivalently, h ( λ pi )
Revised: (cid:112) λ pi /λ zi is monotonically increasing, or equivalently, h ( λ pi )

Original: = (cid:112) λ zi /λ pi is monotonically decreasing, we have erank( C p ) ≤ erank( C z ) .
Revised: = (cid:112) λ zi /λ pi is monotonically decreasing, we have erank( C p ) ≤ erank( C z ) .

Original: Further, as long as g ( λ zi ) or h ( λ pi ) is non-constant, the inequality holds strictly, erank( C p ) <
Revised: Further, as long as g ( λ zi ) or h ( λ pi ) is non-constant, the inequality holds strictly, erank( C p ) <

Original: erank( C z ) .
Revised: erank( C z ) .

Original: Meanwhile, we also observe that for each output, its own effective rank is also successfully elevated along this process.
Revised: Meanwhile, we also observe that for each output, its own effective rank is also successfully elevated along this process.

Original: This is not a coincidence.
Revised: This is not a coincidence.

Original: Below, we theoretically show how the rank difference alleviates dimensional collapse.
Revised: Below, we theoretically show how the rank difference alleviates dimensional collapse.

Original: As the training dynamics of deep neural networks is generally hard to analyze, prior works (Tian et al., 2021; Wen & Li, 2021) mainly deal with linear or shadow networks with strong assumptions on data distribution, which could be far from practice.
Revised: As the training dynamics of deep neural networks is generally hard to analyze, prior works (Tian et al., 2021; Wen & Li, 2021) mainly deal with linear or shadow networks with strong assumptions on data distribution, which could be far from practice.

Original: As overparameterized deep neural networks are very expressive, we instead adopt the unconstrained feature setting (Mixon et al., 2022; HaoChen et al., 2021) and consider gradient descent directly in the feature space R k .
Revised: As overparameterized deep neural networks are very expressive, we instead adopt the unconstrained feature setting (Mixon et al., 2022; HaoChen et al., 2021) and consider gradient descent directly in the feature space R k .

Original: Taking the MSE loss ℓ ( u, v ) = 1 2 ∥ u − v ∥ 2 as an example, the following theorem shows that the rank difference indeed helps improve the effective rank of online output p .
Revised: Taking the MSE loss ℓ ( u, v ) = 1 2 ∥ u − v ∥ 2 as an example, the following theorem shows that the rank difference indeed helps improve the effective rank of online output p .

Original: Theorem 4.
Revised: Theorem 4.

Original: Under Hypothesis 1, when we apply an online spectral filter p x =
Revised: Under Hypothesis 1, when we apply an online spectral filter p x =

Original: Wz x (Lemma 2), gradient descent with step size 0 < α < 1 gives the following update at the t -th step,
Revised: Wz x (Lemma 2), gradient descent with step size 0 < α < 1 gives the following update at the t -th step,

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: σ − 0 .77.
Revised: σ − 0 .77.

Original: h ( σ ) =
Revised: h ( σ ) =

Original: σ − 0 .78.
Revised: σ − 0 .78.

Original: h ( σ ) =
Revised: h ( σ ) =

Original: σ − 0 .77.
Revised: σ − 0 .77.

Original: h ( σ ) = σ −77.
Revised: h ( σ ) = σ −77.

Original: where h ( λ ) = 1 /g ( λ ) is a high-pass filter because g ( λ ) is a low-pass filter (Hypothesis 2).
Revised: where h ( λ ) = 1 /g ( λ ) is a high-pass filter because g ( λ ) is a low-pass filter (Hypothesis 2).

Original: Then, the update λ pi,t +1 /λ pi,t will nicely correspond to a high-pass filter under either of the two conditions:
Revised: Then, the update λ pi,t +1 /λ pi,t will nicely correspond to a high-pass filter under either of the two conditions:

Original: the learned encoder is nearly optimal, i.e., W ≈ W ∗ in Theorem 2. 2.
Revised: the learned encoder is nearly optimal, i.e., W ≈ W ∗ in Theorem 2. 2.

Original: λ
Revised: λ

Original: + i,t ≈ λ zi,t , which naturally holds with good positive alignment,
Revised: + i,t ≈ λ zi,t , which naturally holds with good positive alignment,

Original: i.e., z x ≈ z x + .
Revised: i.e., z x ≈ z x + .

Original: Then, according to Theorem 3, we have erank( C p ( t +1) ) >
Revised: Then, according to Theorem 3, we have erank( C p ( t +1) ) >

Original: erank( C p ( t ) ) .
Revised: erank( C p ( t ) ) .

Original: In other words, the effective rank of online output will keep improving after gradient descent.
Revised: In other words, the effective rank of online output will keep improving after gradient descent.

Original: Intuitively, the improvement of effective rank is a natural consequence of the rank difference.
Revised: Intuitively, the improvement of effective rank is a natural consequence of the rank difference.

Original: As the online output has a lower effective rank than the target output, optimizing its alignment loss w.r.t.
Revised: As the online output has a lower effective rank than the target output, optimizing its alignment loss w.r.t.

Original: the gradient-detached target sg( z x ) will enforce the online output p x to improve its effective rank in order to match the target output z x .
Revised: the gradient-detached target sg( z x ) will enforce the online output p x to improve its effective rank in order to match the target output z x .

Original: In this way, the rank difference becomes a ladder (created by asymmetric designs) for non-contrastive methods to gradually improve its effective feature dimensionality and get rid of dimensional feature collapse eventually.
Revised: In this way, the rank difference becomes a ladder (created by asymmetric designs) for non-contrastive methods to gradually improve its effective feature dimensionality and get rid of dimensional feature collapse eventually.

Original: Note on Stop Gradient.
Revised: Note on Stop Gradient.

Original: Our analysis above also reveals the importance of the stop gradient operation.
Revised: Our analysis above also reveals the importance of the stop gradient operation.

Original: In particular, when gradients from the target branch are not detached, in order to match the rank of two outputs, minimizing the alignment loss can also be fulfilled by simply pulling down the rank of the target output.
Revised: In particular, when gradients from the target branch are not detached, in order to match the rank of two outputs, minimizing the alignment loss can also be fulfilled by simply pulling down the rank of the target output.

Original: In this case, the feature rank will never be improved without stop gradient.
Revised: In this case, the feature rank will never be improved without stop gradient.

Original:  P RINCIPLED A SYMMETRIC D ESIGNS B ASED ON RDM
Revised:  P RINCIPLED A SYMMETRIC D ESIGNS B ASED ON RDM

Original: The discussions above show that the rank differential mechanism provides a unified theoretical understanding of different non-contrastive methods.
Revised: The discussions above show that the rank differential mechanism provides a unified theoretical understanding of different non-contrastive methods.

Original: Besides, it also provides a general recipe for designing new non-contrastive variants.
Revised: Besides, it also provides a general recipe for designing new non-contrastive variants.

Original: As Theorem 3 points out, the key requirement is that the asymmetry can produce a low-pass filter on the online branch, or equivalently, a high-pass filter on the target branch.
Revised: As Theorem 3 points out, the key requirement is that the asymmetry can produce a low-pass filter on the online branch, or equivalently, a high-pass filter on the target branch.

Original: Below, we propose some new variants directly following this principle.
Revised: Below, we propose some new variants directly following this principle.

Original: 5. V ARIANTS OF O NLINE L OW - PASS F ILTERS
Revised: 5. V ARIANTS OF O NLINE L OW - PASS F ILTERS

Original: Despite the learnable predictor (Section 4.2), we can also directly design online predictors with fixed low-pass filters.
Revised: Despite the learnable predictor (Section 4.2), we can also directly design online predictors with fixed low-pass filters.

Original: For numerical stability, we adopt the singular value decomposition (SVD) of the output to compute the eigenvalues and the eigenspace, e.g., z = U Σ
Revised: For numerical stability, we adopt the singular value decomposition (SVD) of the output to compute the eigenvalues and the eigenspace, e.g., z = U Σ

Original: z V ⊤ with singular values σ zi ’s.
Revised: z V ⊤ with singular values σ zi ’s.

Original: As λ z i = ( σ z i ) 2 , a filter that is monotone in λ is also monotone in σ , and vice versa.
Revised: As λ z i = ( σ z i ) 2 , a filter that is monotone in λ is also monotone in σ , and vice versa.

Original: Specifically, for
Revised: Specifically, for

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: an online encoder f : X → R k , we assign W =
Revised: an online encoder f : X → R k , we assign W =

Original: V g
Revised: V g

Original: (Σ z ) V ⊤ , where g (Σ z ) ii =
Revised: (Σ z ) V ⊤ , where g (Σ z ) ii =

Original: g ( σ zi ) is a low-pass filter that is monotonically increasing w.r.t.
Revised: g ( σ zi ) is a low-pass filter that is monotonically increasing w.r.t.

Original: σ .
Revised: σ .

Original: We note that DirectPred proposed by Tian et al.
Revised: We note that DirectPred proposed by Tian et al.

Original: (2021) is a special case with g 0 ( σ ) = σ .
Revised: (2021) is a special case with g 0 ( σ ) = σ .

Original: Additionally, we consider three variants: 1) g 1 ( σ ) =
Revised: Additionally, we consider three variants: 1) g 1 ( σ ) =

Original: log( σ ) ; 2) g 2 ( λ ) =
Revised: log( σ ) ; 2) g 2 ( λ ) =

Original: log( σ + 1) ; 3) g 3 ( σ ) = log( σ 2 + 1) .
Revised: log( σ + 1) ; 3) g 3 ( σ ) = log( σ 2 + 1) .

Original: These three new variants are low-pass filters because they are monotonically increasing with σ ≥ 0 .
Revised: These three new variants are low-pass filters because they are monotonically increasing with σ ≥ 0 .

Original: We evaluate their performance on four datasets, i.e., CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), ImageNet-100 and ImageNet-1k (Deng et al., 2009).
Revised: We evaluate their performance on four datasets, i.e., CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), ImageNet-100 and ImageNet-1k (Deng et al., 2009).

Original: We use ResNet-18 (He et al., 2016) as the backbone encoder for CIFAR-10, CIFAR-100 and ImageNet-100 and adopt ResNet-50 for
Revised: We use ResNet-18 (He et al., 2016) as the backbone encoder for CIFAR-10, CIFAR-100 and ImageNet-100 and adopt ResNet-50 for

Original: Imagenet-1k following standard practice.
Revised: Imagenet-1k following standard practice.

Original: And the projector is a three-layer MLP in which BN (Ioffe & Szegedy, 2015) is applied to all layers.
Revised: And the projector is a three-layer MLP in which BN (Ioffe & Szegedy, 2015) is applied to all layers.

Original: We adopt the linear probing task for evaluating the learned representations.
Revised: We adopt the linear probing task for evaluating the learned representations.

Original: More details are included in Appendix B.2.
Revised: More details are included in Appendix B.2.

Original: Table 1 shows that our designs of online predictors with different low-pass filters work well in practice, and achieve comparable performance to SimSiam with learnable predictors.
Revised: Table 1 shows that our designs of online predictors with different low-pass filters work well in practice, and achieve comparable performance to SimSiam with learnable predictors.

Original: In particular, it also significantly outperforms SymSimSiam (Figure 2(a)), showing that rank differences indeed help alleviate dimensional collapse (Figure 6).
Revised: In particular, it also significantly outperforms SymSimSiam (Figure 2(a)), showing that rank differences indeed help alleviate dimensional collapse (Figure 6).

Original: 5. V ARIANTS OF T ARGET H IGH - PASS F ILTERS
Revised: 5. V ARIANTS OF T ARGET H IGH - PASS F ILTERS

Original: In turn, we also consider applying a high-pass filter on the target branch using a target predictor .
Revised: In turn, we also consider applying a high-pass filter on the target branch using a target predictor .

Original: Compared to the online predictors above, target predictors have additional advantages: with stop gradient, we do not require backpropagation through SVD, which could reduce time overhead (Table 5).
Revised: Compared to the online predictors above, target predictors have additional advantages: with stop gradient, we do not require backpropagation through SVD, which could reduce time overhead (Table 5).

Original: Specifically, we consider polynomial high-pass target filters h ( σ )
Revised: Specifically, we consider polynomial high-pass target filters h ( σ )

Original: =
Revised: =

Original: σ p with different − 1 ≤ p < 0 , which are all monotonically decreasing (see Algorithm 2).
Revised: σ p with different − 1 ≤ p < 0 , which are all monotonically decreasing (see Algorithm 2).

Original: We further evaluate the target filters following the same protocols above.
Revised: We further evaluate the target filters following the same protocols above.

Original: As shown in Table 2, our high-pass target filters can often outperform SimSiam by a large margin with a relatively short training time.
Revised: As shown in Table 2, our high-pass target filters can often outperform SimSiam by a large margin with a relatively short training time.

Original: Especially, on CIFAR-10, the filter h ( σ ) =
Revised: Especially, on CIFAR-10, the filter h ( σ ) =

Original: σ − 0 .
Revised: σ − 0 .

Original: 5 improves the baseline by 5.75% with 100 epochs and the filter h ( σ ) =
Revised: 5 improves the baseline by 5.75% with 100 epochs and the filter h ( σ ) =

Original: σ − 1 improves the baseline by 2.77% with 200 epochs.
Revised: σ − 1 improves the baseline by 2.77% with 200 epochs.

Original: Notably, on CIFAR-100, our methods outperform the baseline by a large margin (8.07%, 3.18%, and 0.81% improvements with 100, 200, and 400 epochs, respectively).
Revised: Notably, on CIFAR-100, our methods outperform the baseline by a large margin (8.07%, 3.18%, and 0.81% improvements with 100, 200, and 400 epochs, respectively).

Original: Additional results based on the BYOL framework can be found in Appendix C.
Revised: Additional results based on the BYOL framework can be found in Appendix C.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented a unified theoretical understanding of non-contrastive learning via the rank differential hypothesis.
Revised: In this paper, we presented a unified theoretical understanding of non-contrastive learning via the rank differential hypothesis.

Original: In particular, we showed that existing non-contrastive learning all produce a consistent rank difference between the online and the target outputs.
Revised: In particular, we showed that existing non-contrastive learning all produce a consistent rank difference between the online and the target outputs.

Original: Digging deeper into this phenomenon, we theoretically proved that low-pass online filters can yield such a rank difference and improve the effective feature dimensionality along the training.
Revised: Digging deeper into this phenomenon, we theoretically proved that low-pass online filters can yield such a rank difference and improve the effective feature dimensionality along the training.

Original: Meanwhile, we provided theoretical and empirical insights on how existing asymmetric designs produce low-pass filters.
Revised: Meanwhile, we provided theoretical and empirical insights on how existing asymmetric designs produce low-pass filters.

Original: At last, following the principle of our theory, we designed a series of new online and target filters, and showed that they achieve comparable or even superior to existing asymmetric designs.
Revised: At last, following the principle of our theory, we designed a series of new online and target filters, and showed that they achieve comparable or even superior to existing asymmetric designs.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Jinwen Ma is supported by the National Key Research and Development Program of China under grant 2018AAA0100205.
Revised: Jinwen Ma is supported by the National Key Research and Development Program of China under grant 2018AAA0100205.

Original: Yisen Wang is partially supported by the National Natural Science Foundation of China (62006153), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), and
Revised: Yisen Wang is partially supported by the National Natural Science Foundation of China (62006153), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), and

Original: Huawei Technologies Inc
Revised: Huawei Technologies Inc.

Original: A O MITTED P ROOFS
Revised: A O MITTED P ROOFS

Original: In this section we present proofs for all lemmas and theorems in the main paper.
Revised: In this section, we present proofs for all lemmas and theorems in the main paper.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Since the output of f θ is ℓ 2 -normalized and E x f ( x ) =
Revised: Since the output of f θ is ℓ 2 -normalized, and E x f ( x ) =

Original: E x
Revised: E x

Original: + f ( x + ) = µ  we have L sym ( θ ) =−
Revised: + f ( x + ) = µ , we have L sym ( θ ) =

Original:  x,x + f ′ θ ( x ) ⊤ f
Revised: − E x,x + f ′ θ ( x ) ⊤ f

Original: ′ θ ( x + ) =
Revised: ′ θ ( x + ) =

Original: − E x,x + ( f θ ( x ) − µ ) ⊤ ( f θ ( x + ) − µ ) =
Revised: − E x,x + ( f θ ( x ) − µ ) ⊤ ( f θ ( x + ) − µ ) =

Original: − E x,x + f θ ( x ) ⊤ f θ ( x + ) + ∥ µ ∥ 2 , and Var( f ( x ))
Revised: − E x,x + f θ ( x ) ⊤ f θ ( x + ) + ∥ µ ∥ 2 , and Var( f ( x ))

Original: =
Revised: =

Original: E x
Revised: E x

Original: ∥ f ( x ) ∥ 2 − ∥ µ ∥ 2 = 1 − ∥ µ ∥ 2 .
Revised: ∥ f ( x ) ∥ 2 − ∥ µ ∥ 2 = 1 − ∥ µ ∥ 2 .

Original: Thus we conclude that L sym ( θ ) =
Revised: Thus, we conclude that L sym ( θ ) =

Original: − E x,x + f θ ( x ) ⊤ f θ ( x + ) − Var( f ( x ))
Revised: − E x,x + f θ ( x ) ⊤ f θ ( x + ) − Var( f ( x ))

Original: + 1 .
Revised: + 1 .

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: Let z x = W f x and z x + = W
Revised: Let z x = W f x and z x + = W

Original: f
Revised: f

Original: x
Revised: x

Original: + , the loss functions is
Revised: + , the loss functions is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Notice that
Revised: Notice that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  and 
Revised: 18) and [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original:  Taking partial derivative with respect to W [Equation] we get
Revised: [Equation] Taking partial derivative with respect to W , we get

Original: [Equation]
Revised: [Equation]

Original: With additional weight decay
Revised: With additional weight decay, we have

Original: [Equation]
Revised: [Equation]

Original: we have

Original: [Equation]

Original: [Equation]

Original: where η > 0 is the coefficient of weight decay.
Revised: where η > 0 is the coefficient of weight decay.

Original: Suppose the spectral decomposition W f W ⊤ f is
Revised: Suppose the spectral decomposition W f W ⊤ f is

Original: W f W ⊤ f = V Λ V ⊤ , where V is an orthogonal matrix and Λ is a diagonal matrix consisting of decending eigenvalues λ 1 , · · · , λ k .
Revised: W f W ⊤ f = V Λ V ⊤ , where V is an orthogonal matrix and Λ is a diagonal matrix consisting of decending eigenvalues λ 1 , · · · , λ k .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Let ∂ L ( W ) ∂W = 0 , we get W ∗ =
Revised: Let ∂ L ( W ) ∂W = 0 , we get W ∗ =

Original: W f W ⊤ f (cid:16) (1 + σ 2 ) W f W ⊤ f + ηI (cid:17) − 1 = V Λ w V ⊤ , where Λ w = diag
Revised: W f W ⊤ f (cid:16) (1 + σ 2 ) W f W ⊤ f + ηI (cid:17) − 1 = V Λ w V ⊤ , where Λ w = diag

Original: λ 1 / ((1 + σ 2 ) λ 1 + η ) , · · · , λ k / ((1 + σ 2 ) λ k + η ) } .
Revised: λ 1 / ((1 + σ 2 ) λ 1 + η ) , · · · , λ k / ((1 + σ 2 ) λ k + η ) } .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Hence, C p , C z , C + share the same eigenspace.
Revised: Hence, C p , C z , C + share the same eigenspace.

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: By the definition of ˜ p x = W g z x , we know that
Revised: By the definition of ˜ p x = W g z x , we know that

Original: [Equation]
Revised: [Equation]

Original: Since W g = V g (Λ z ) V ⊤ [Equation] C z = V Λ z V ⊤ [Equation]
Revised: Since W g = V g (Λ z ) V ⊤ , C z = V Λ z V ⊤ , C p = V Λ p V ⊤ and g ( λ zi ) =

Original: C p =

Original: V Λ p V ⊤ and g ( λ zi ) =

Original: (cid:112) λ pi /λ zi [Equation] we have
Revised: (cid:112) λ pi /λ zi , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: For the first equality, we have
Revised: For the first equality, we have

Original: [Equation]
Revised: [Equation]

Original: For the first part,
Revised: For the first part,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Similar to the derivation in the proof of Lemma 1, we have
Revised: Similar to the derivation in the proof of Lemma [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Notice that C z
Revised: Notice that C z

Original: =
Revised: =

Original: E x
Revised: E x

Original: z x z ⊤ x = E x + z x + z ⊤ x + and
Revised: z x z ⊤ x = E x + z x + z ⊤ x + and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Taking partial derivative with respect to W , we get∂ L ( W [Equation] ∂W
Revised: Taking partial derivative with respect to W , we get

Original: [Equation]
Revised: [Equation]

Original: Let ∂ L ( W ) ∂W = 0 , we have W ∗ =
Revised: Let ∂ L ( W ) ∂W = 0 , we have W ∗ =

Original: ¯ CC − 1 z .
Revised: ¯ CC − 1 z .

Original: Since C z , ¯ C have aligned eigenspace V , we have
Revised: Since C z , ¯ C have aligned eigenspace V , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: (47) which implies that the corresponding eigenvalues satisfy ε i =
Revised: 47) which implies that the corresponding eigenvalues satisfy ε i =

Original: λ zi − ¯
Revised: λ zi − ¯

Original: λ
Revised: λ

Original: i ≥ 0 , where ¯ λ i ,
Revised: i ≥ 0 , where ¯ λ i ,

Original: ε i denote the i -th eigenvalues of ¯ C , V x
Revised: ε i denote the i -th eigenvalues of ¯ C , V x

Original: | ¯ x , respectively.
Revised: | ¯ x , respectively.

Original: And ¯ CC − 1 z = V ¯Λ V ⊤ (cid:0) V Λ z V ⊤ (cid:1) − 1 =
Revised: And ¯ CC − 1 z = V ¯Λ V ⊤ (cid:0) V Λ z V ⊤ (cid:1) − 1 =

Original: V ¯ΛΛ − 1 z V ⊤ =
Revised: V ¯ΛΛ − 1 z V ⊤ =

Original: V Ω V ⊤ , where Ω is a diagonal matrix and Ω ii = ¯ λ
Revised: V Ω V ⊤ , where Ω is a diagonal matrix and Ω ii = ¯ λ

Original: i λ zi ∈ [0 , 1] , i = 1 , 2 , . .
Revised: i λ zi ∈ [0 , 1] , i = 1 , 2 , . .

Original: . , k .
Revised: . , k .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Before the proof of Theorem 3, we first introduce the following useful lemma.
Revised: Before the proof of Theorem 3, we first introduce the following useful lemma.

Original: Lemma 4.
Revised: Lemma 4.

Original: Assume that (cid:80) ki
Revised: Assume that (cid:80) ki

Original: =1 q i = 1 and q 1 ≥ q 2 ≥ · · · ≥ q k > 0 , then for any 1 ≤ i <
Revised: =1 q i = 1 and q 1 ≥ q 2 ≥ · · · ≥ q k > 0 , then for any 1 ≤ i <

Original: j ≤ k and any ∆ ∈ (0 , p j ) , it holds that
Revised: j ≤ k and any ∆ ∈ (0 , p j ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We first note that
Revised: We first note that

Original: [Equation]
Revised: [Equation]

Original: Define f ( x ) = − ( q i + x ) log( q i + x ) − ( q j − x ) log( q j − x ) and its first order derivative satisfies
Revised: Define f ( x ) = − ( q i + x ) log( q i + x ) − ( q j − x ) log( q j − x ) and its first order derivative satisfies

Original: [Equation]
Revised: [Equation]

Original: Hence, Equation (50) holds.
Revised: Hence, Equation (50) holds.

Original: This completes the proof of the lemma.
Revised: This completes the proof of the lemma.

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Let q
Revised: Let q

Original: zi
Revised: zi

Original: =
Revised: =

Original: λ zi / ( (cid:80) k
Revised: λ zi / ( (cid:80) kl

Original: l =1 λ z l ) and q pi =λ p i /
Revised: =1 λ zl ) and q pi =

Original:  ( (cid:80) k l =1 λ pl ) , where i = 1 , 2 , . . 
Revised: λ pi / ( (cid:80) kl =1 λ pl ) , where i = 1 , 2 , . . .

Original: , k , then (cid:80) ki
Revised: , k , then (cid:80) ki

Original: =1 q zi =
Revised: =1 q zi =

Original: (cid:80)
Revised: (cid:80)

Original: ki =1 q pi = 1 , q z 1 ≥ q z 2 ≥ · · · ≥ q zk and q z 1 ≥ q z 2 ≥ · · · ≥ q zk .
Revised: ki =1 q pi = 1 , q z 1 ≥ q z 2 ≥ · · · ≥ q zk and q z 1 ≥ q z 2 ≥ · · · ≥ q zk .

Original: Without loss of generality, we assume that q pk , q zk > 0 .
Revised: Without loss of generality, we assume that q pk , q zk > 0 .

Original: Because g ( λ zi ) =
Revised: Because g ( λ zi ) =

Original: (cid:112) λ pi /λ zi is monotonically increasing and λ zi ≥ λ zj , for any 1 ≤ i < j ≤ k , we have
Revised: (cid:112) λ pi /λ zi is monotonically increasing and λ zi ≥ λ zj , for any 1 ≤ i < j ≤ k , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: If g ( λ zi ) is constant, it follows that
Revised: If g ( λ zi ) is constant, it follows that

Original: [Equation]
Revised: [Equation]

Original: Combining with (cid:80) ki =1 q zi = (cid:80) ki =1 q pi = 1 , we get q pi =
Revised: Combining with (cid:80) ki =1 q zi = (cid:80) ki =1 q pi = 1 , we get q pi =

Original: q zi , i = 1 , 2 , . . .
Revised: q zi , i = 1 , 2 , . . .

Original: , k .
Revised: , k .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: 54) which implies that erank( C p )
Revised: 54) which implies that erank( C p )

Original: =
Revised: =

Original: erank( C z ) .
Revised: erank( C z ) .

Original: If g ( λ zi ) is non-constant, then g ( λ z 1 ) >
Revised: If g ( λ zi ) is non-constant, then g ( λ z 1 ) >

Original: g ( λ zk ) .
Revised: g ( λ zk ) .

Original: And it follows that
Revised: And it follows that

Original: [Equation]
Revised: [Equation]

Original: Arming with Equations (52) and (55), we get1 =
Revised: Arming with Equations (52) and (55), we get

Original: [Equation]
Revised: [Equation]

Original: (56) which indicates q pk < q zk .
Revised: which indicates q pk < q zk .

Original: Similarly, we have q p 1 > q z 1 .
Revised: Similarly, we have q p 1 > q z 1 .

Original: Hence, m = max { i | q pi ≥ q zi , i = 1 , 2 , . .
Revised: Hence, m = max { i | q pi ≥ q zi , i = 1 , 2 , . .

Original: . , k } exists and m < 1 .
Revised: . , k } exists and m < 1 .

Original: Using Equation (52), we have
Revised: Using Equation (52), we have

Original: [Equation]
Revised: [Equation]

Original: Directly applying (cid:80) ki
Revised: Directly applying (cid:80) ki

Original: =1 q zi =
Revised: =1 q zi =

Original: (cid:80)
Revised: (cid:80)

Original: ki =1 q pi = 1 gives that
Revised: ki =1 q pi = 1 gives that

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: i
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 4 and Equation (58), if we transport the redundancy from right to left, the entropy of the distribution will decrease.
Revised: According to Lemma 4 and Equation (58), if we transport the redundancy from right to left, the entropy of the distribution will decrease.

Original: The transportation process is described as following:
Revised: The transportation process is described as following:

Original: Step 1 Let i ← 1 and j ← k .
Revised: Step 1 Let i ← 1 and j ← k .

Original: Step 2{+
+} If ∆ = min { q pi − q zi , q zj − q pj } > 0 , then q zi ← q zi + ∆ and q zj ← q zj − ∆ .
Revised: Step {+
+} If ∆ = min { q pi − q zi , q zj − q pj } > 0 , then q zi ← q zi + ∆ and q zj ← q zj − ∆ .

Original: Step 

Revised: Step

Original: If q pi =
Revised: If q pi =

Original: q zi , i ← i + 1 .
Revised: q zi , i ← i + 1 .

Original: Else j ← j − 1 .
Revised: Else j ← j − 1 .

Original: Step 4{+
+} If i ≥ j , we finish this process.
Revised: Step {+
+} If i ≥ j , we finish this process.

Original: Else, we return step 2.
Revised: Else, we return step 2.

Original: After at most k − 1 loops, ( q z 1 , q z 2 , . . .
Revised: After at most k − 1 loops, ( q z 1 , q z 2 , . . .

Original: , q zk ) becomes ( q p 1 , q p 2 , . . .
Revised: , q zk ) becomes ( q p 1 , q p 2 , . . .

Original: , q pk ) .
Revised: , q pk ) .

Original: Equation (58) ensures the correctness of the transportation process.
Revised: Equation (58) ensures the correctness of the transportation process.

Original: According to Lemma 4, we have
Revised: According to Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: (59) which implies that erank( C p ) <
Revised: which implies that erank( C p ) <

Original: erank( C z ) .
Revised: erank( C z ) .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Since p ( t +1) x
Revised: Since p ( t +1) x

Original: [Equation]
Revised: [Equation]

Original: and 
Revised: and p

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides of Equation (67) by λ pi,t , we have
Revised: Dividing both sides of Equation (67) by λ pi,t , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Revised: where h t

Original: [Equation]
Revised: [Equation]

Original: where h t ( λ pi,t ) = 1 /λ wi,t =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: λ 
Revised: λ zi

Original: [Equation]
Revised: [Equation]

Original:  pi
Revised: λ pi

Original: [Equation]
Revised: [Equation]

Original: • If W = W ∗ =
Revised: • If W = W ∗ =

Original: C ( t )
Revised: C ( t )

Original: + ( C ( t ) z ) − 1 , then h t ( λ pi,t ) =
Revised: + ( C ( t ) z ) − 1 , then h t ( λ pi,t ) =

Original: λ zi,t λ + i,t .
Revised: λ zi,t λ + i,t .

Original: Hence, λ pi,t +1 λ pi,t = 1 − α 2 + α 2 h 2 t ( λ pi,t ) is monotonically decreasing and non-constant.
Revised: Hence, λ pi,t +1 λ pi,t = 1 − α 2 + α 2 h 2 t ( λ pi,t ) is monotonically decreasing and non-constant.

Original: • If λ zi,t = λ
Revised: • If λ zi,t = λ

Original: + i,t , then λ pi,t +1 λ pi,t
Revised: + i,t , then λ pi,t +1 λ pi,t

Original: =
Revised: =

Original: (1 − α ) 2 + α 2 h 2 t ( λ pi,t )
Revised: (1 − α ) 2 + α 2 h 2 t ( λ pi,t )

Original: + 2 α (1 − α ) h t ( λ pi,t ) is monotonically decreasing and non-constant ( 0 < α < 1 ).
Revised: + 2 α (1 − α ) h t ( λ pi,t ) is monotonically decreasing and non-constant ( 0 < α < 1 ).

Original: According to Theorem 3, we have erank( C p ( t +1) ) >
Revised: According to Theorem 3, we have erank( C p ( t +1) ) >

Original: erank( C p ( t) ) .
Revised: erank( C p ( t

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: In this section, we provide the details and hyperparameters for SymSimSiam and variants of spectral filters.
Revised: In this section, we provide the details and hyperparameters for SymSimSiam and variants of spectral filters.

Original: B. D ATASETS
Revised: B. D ATASETS

Original: We evaluate the performance of our methods on four benchmark datasets: CIFAR-10, CIFAR-
Revised: We evaluate the performance of our methods on four benchmark datasets: CIFAR-10, CIFAR-

Original: (Krizhevsky et al., 2009), ImageNet-100 and ImageNet-1k (Deng et al., 2009).
Revised: (Krizhevsky et al., 2009), ImageNet-100 and ImageNet-1k (Deng et al., 2009).

Original: CIFAR-10 and
Revised: CIFAR-10 and

Original: CIFAR-100 are small-scale datasets, composed of 32 × 32 images with 10 and 100 classes, respectively.
Revised: CIFAR-100 are small-scale datasets, composed of 32 × 32 images with 10 and 100 classes, respectively.

Original: ImageNet-100 is a subset of ImageNet-1k containing 100 classes.
Revised: ImageNet-100 is a subset of ImageNet-1k containing 100 classes.

Original: B. I MPLEMENTATION D ETAILS
Revised: B. I MPLEMENTATION D ETAILS

Original: Unless specified otherwise, we follow the default settings in solo-learn (da Costa et al., 2022) on
Revised: Unless specified otherwise, we follow the default settings in solo-learn (da Costa et al., 2022) on

Original: CIFAR-10, CIFAR-100 and ImageNet-100.
Revised: CIFAR-10, CIFAR-100 and ImageNet-100.

Original: For ImageNet, our implementation follows the official code of SimSiam (Chen & He, 2021), and we use the same settings.
Revised: For ImageNet, our implementation follows the official code of SimSiam (Chen & He, 2021), and we use the same settings.

Original: For a fair comparison, SimSiam with a learnable linear predictor is adopted as our baseline.
Revised: For a fair comparison, SimSiam with a learnable linear predictor is adopted as our baseline.

Original: With the original projector, SimSiam with a learnable linear predictor could not work, so we delete the last BN in the projector in this case.
Revised: With the original projector, SimSiam with a learnable linear predictor could not work, so we delete the last BN in the projector in this case.

Original: And we also list the results of SimSiam with the default predictor (we refer it as a learnable nonlinear predictor).
Revised: And we also list the results of SimSiam with the default predictor (we refer it as a learnable nonlinear predictor).

Original: Data augmentations.
Revised: Data augmentations.

Original: The augmentation pipeline is RandomResizedCrop with scale in (0.2, 1.0),
Revised: The augmentation pipeline is RandomResizedCrop with scale in (0.2, 1.0),

Original: RandomHorizontalFlip with probability 0.5, ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4), hue (0.1)) with probability 0.8 and RandomGray with probability 0.2.
Revised: RandomHorizontalFlip with probability 0.5, ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4), hue (0.1)) with probability 0.8 and RandomGray with probability 0.2.

Original: For ImageNet-100 and
Revised: For ImageNet-100 and

Original: ImageNet-1k, Gaussian blurring (Chen et al., 2020) with an applying probability 0.5 is also used.
Revised: ImageNet-1k, Gaussian blurring (Chen et al., 2020) with an applying probability 0.5 is also used.

Original: Optimization.
Revised: Optimization.

Original: SGD is used as the optimizer with momentum 0.9 and weight decay 1 . 

Revised: SGD is used as the optimizer with momentum 0.9 and weight decay 1 .

Original: × 10 −( 1 . 

Revised: × 10 −( 1 .

Original: × 10 − 4 for ImageNet-1k).
Revised: × 10 − 4 for ImageNet-1k).

Original: The learning rate adopts the linear scaling rule (lr×BatchSize/256) with a base learning rate of 0.5 (0.05 for ImageNet-1k).
Revised: The learning rate adopts the linear scaling rule (lr×BatchSize/256) with a base learning rate of 0.5 (0.05 for ImageNet-1k).

Original: After 10 epochs of warmup training, we use the cosine learning rate decay (Loshchilov & Hutter, 2017).
Revised: After 10 epochs of warmup training, we use the cosine learning rate decay (Loshchilov & Hutter, 2017).

Original: We use a batch size 256 on CIFARand CIFAR-100; 128 on ImageNet-100 and 512 on ImageNet-1k.
Revised: We use a batch size 256 on CIFARand CIFAR-100; 128 on ImageNet-100 and 512 on ImageNet-1k.

Original: Linear evaluation.
Revised: Linear evaluation.

Original: For the linear evaluation, we evaluate the pre-trained backbone network by training a linear classifier on the frozen representation.
Revised: For the linear evaluation, we evaluate the pre-trained backbone network by training a linear classifier on the frozen representation.

Original: For CIFAR-10, CIFAR-100 and ImageNet100, the linear classifier is trained using SGD optimizer with momentum = 0.9, batch size = 256 and initial learning rate = 30.0 for 100 epochs.
Revised: For CIFAR-10, CIFAR-100 and ImageNet100, the linear classifier is trained using SGD optimizer with momentum = 0.9, batch size = 256 and initial learning rate = 30.0 for 100 epochs.

Original: The learning rate is divided by 10 at epochs 60 and 80.
Revised: The learning rate is divided by 10 at epochs 60 and 80.

Original: For ImageNet-1k, following the official code of SimSiam (Chen & He, 2021), we train the linear classifier for 90 epochs with a LARS optimizer (You et al., 2017) with momentum = 0.9, batch size = 4096, weight decay = 0, initial learning rate = 0.1 and cosine decay of the learning rate.
Revised: For ImageNet-1k, following the official code of SimSiam (Chen & He, 2021), we train the linear classifier for 90 epochs with a LARS optimizer (You et al., 2017) with momentum = 0.9, batch size = 4096, weight decay = 0, initial learning rate = 0.1 and cosine decay of the learning rate.

Original: B. E XPERIMENTAL S ETTING FOR F IGURES
Revised: B. E XPERIMENTAL S ETTING FOR F IGURES

Original: For the implementations of BYOL, SimSiam, SwAV and DINO on CIFAR-10 and CIFAR-100, we use codes and all default settings in the open-source library solo-learn (da Costa et al., 2022).
Revised: For the implementations of BYOL, SimSiam, SwAV and DINO on CIFAR-10 and CIFAR-100, we use codes and all default settings in the open-source library solo-learn (da Costa et al., 2022).

Original: For
Revised: For

Original: SwAV, we do not store features from the previous batches to augment batch features (queue size = 0) for the consistency of training loss.
Revised: SwAV, we do not store features from the previous batches to augment batch features (queue size = 0) for the consistency of training loss.

Original: We adopt ResNet-18 as the backbone encoder.
Revised: We adopt ResNet-18 as the backbone encoder.

Original: The dimensions of the outputs of BYOL, SimSiam, SwAV and DINO are 256, 2048, 3000 and 4092, respectively.
Revised: The dimensions of the outputs of BYOL, SimSiam, SwAV and DINO are 256, 2048, 3000 and 4092, respectively.

Original: In Figure 3, we use the alignment metric to measure the eigenspace alignment of the online and target outputs, i.e., C p and C z .
Revised: In Figure 3, we use the alignment metric to measure the eigenspace alignment of the online and target outputs, i.e., C p and C z .

Original: The intuition is that, given u i as an eigenvector of C z , if it is also an eigenvector of C p , then C p u
Revised: The intuition is that, given u i as an eigenvector of C z , if it is also an eigenvector of C p , then C p u

Original: i = λ ′
Revised: i = λ ′

Original: i
Revised: i

Original: u i is in the same direction as u i .
Revised: u i is in the same direction as u i .

Original: Thus, the higher the cosine similarity between u i and C p u i , the more aligned between the eigenspace of C p and C z .
Revised: Thus, the higher the cosine similarity between u i and C p u i , the more aligned between the eigenspace of C p and C z .

Original: Formally, we define the alignment between the eigenspace of C p and C z as
Revised: Formally, we define the alignment between the eigenspace of C p and C z as

Original: [Equation]
Revised: [Equation]

Original: where u i is eigenvector corresponding to the i -th largest eigenvalue λ zi of C z , i = 1 , 2 , . .
Revised: where u i is eigenvector corresponding to the i -th largest eigenvalue λ zi of C z , i = 1 , 2 , . .

Original: , m .
Revised: , m .

Original: And m is set to 512 for SimSiam, SwAV and DINO and 128 for BYOL.
Revised: And m is set to 512 for SimSiam, SwAV and DINO and 128 for BYOL.

Original: The sum of the first m eigenvalues is greater than 99.99% of the sum of all eigenvalues.
Revised: The sum of the first m eigenvalues is greater than 99.99% of the sum of all eigenvalues.

Original: Therefore, we can think that the space span by the first m eigenvectors is a good approximation to the original eigenspace.
Revised: Therefore, we can think that the space span by the first m eigenvectors is a good approximation to the original eigenspace.

Original: And in
Revised: And in

Original: Figures 4, 5 & 8, the first 512 point pairs are displayed for SimSiam, SwAV and DINO (128 for
Revised: Figures 4, 5 & 8, the first 512 point pairs are displayed for SimSiam, SwAV and DINO (128 for

Original: BYOL).
Revised: BYOL).

Original: B. P SEUDOCODE
Revised: B. P SEUDOCODE

Original: Here, we provide the pseudocode for SymSimSiam (Algorithm 1) and variants of spectral filters (Algorithm 2).
Revised: Here, we provide the pseudocode for SymSimSiam (Algorithm 1) and variants of spectral filters (Algorithm 2).

Original: # f : backbone + p r
Revised: # f : backbone + p r

Original: o j e c t
Revised: o j e c t

Original: i o n mlp # c : c e n t e r (1 −by−k ) # m: momentum 0.
Revised: i o n mlp # c : c e n t e r (1 −by−k ) # m: momentum 0.

Original: f o r x in l o a d e r : # load a minibatch x with n samples x1 , x2 =
Revised: f o r x in l o a d e r : # load a minibatch x with n samples x1 , x2 =

Original: aug ( x ) , aug ( x ) # two random augmentations z1 , z2 = f ( x1 ) , f ( x2 )
Revised: aug ( x ) , aug ( x ) # two random augmentations z1 , z2 = f ( x1 ) , f ( x2 )

Original: # p r
Revised: # p r

Original: o j e c t
Revised: o j e c t

Original: i o n s , n−by−k z1 = normalize ( z1 , dim =1) # l2 − normalize z2 = normalize ( z2 , dim =1) # l2 − normalize u p d a t e c e n t e r ( c a t ( z1 , z2 ) ) z1 , z2 =
Revised: i o n s , n−by−k z1 = normalize ( z1 , dim =1) # l2 − normalize z2 = normalize ( z2 , dim =1) # l2 − normalize u p d a t e c e n t e r ( c a t ( z1 , z2 ) ) z1 , z2 =

Original: z1−c , z2−c l o s s = −( z1*z2 ) .
Revised: z1−c , z2−c l o s s = −( z1 * z2 ) .

Original: sum ( dim = 1 ).
Revised: sum ( dim = 1 ).

Original: mean ( ) l o s s .
Revised: mean ( ) l o s s .

Original: backward ( ) update ( f . params )
Revised: backward ( ) update ( f . params )

Original: # SGD update @torch .
Revised: # SGD update @torch .

Original: no grad ( ) def u p d a t e c e n t e r ( z ) :
Revised: no grad ( ) def u p d a t e c e n t e r ( z ) :

Original: c = m * c + (1 −m) *
Revised: c = m * c + (1 −m) *

Original: z .
Revised: z .

Original: mean ( dim =0)
Revised: mean ( dim =0)

Revised: C A DDITIONAL R ESULTS B ASED ON BYOL F RAMEWORK

Revised: In the main paper, we conduct experiments based on the SimSiam framework.

Revised: Here, we also gather some results of variants of target high-pass filters based on the BYOL framework.

Revised: We adopt ResNet18 as the backbone encoder and use the default setting in da Costa et al.

Original:  # f : backbone + p r
Revised: Table 4 shows that our target high-pass filters can often outperform BYOL with a large margin at earlier epochs. In particular, on CIFAR-10, the target filter h ( σ ) = σ − 0 . 3 improves BYOL with learnable linear predictor by 4.92% # f : backbone + p r

Original: o j e c t
Revised: o j e c t

Original: i o n mlp # g : s p e c t r a l f i l t e r # l o c a t i o n : the l o c a t i o n of s p e c t r a l f i l t e r ( o n l i n e or t a r g e t ) f
Revised: i o n mlp # g : s p e c t r a l f i l t e r # l o c a t i o n : the l o c a t i o n of s p e c t r a l f i l t e r ( o n l i n e or t a r g e t ) f

Original: o r x in l o a d e r : # load a minibatch x with n samples x1 , x2 =
Revised: o r x in l o a d e r : # load a minibatch x with n samples x1 , x2 =

Original: aug ( x ) , aug ( x ) # two random augmentations z1 , z2 = f ( x1 ) , f ( x2 )
Revised: aug ( x ) , aug ( x ) # two random augmentations z1 , z2 = f ( x1 ) , f ( x2 )

Original: # p r
Revised: # p r

Original: o j e c t
Revised: o j e c t

Original: i o n s , n−by−k l o s s =
Revised: i o n s , n−by−k l o s s =

Original: L( z1 , z2 , l o c a t i o n ) / 2 + L( z2 , z1 , l o c a t i o n ) /l
Revised: L( z1 , z2 , l o c a t i o n ) / 2 + L( z2 , z1 , l o c a t i o n ) /l

Original: o s s .
Revised: o s s .

Original: backward ( ) update ( f . params )
Revised: backward ( ) update ( f . params )

Original: # SGD update def L( p , z , l o c a t i o n ) :
Revised: # SGD update def L( p , z , l o c a t i o n ) :

Original: # l o s s f u n c t
Revised: # l o s s f u n c t

Original: i o n i f
Revised: i o n i f

Original: l o c a t i o n == ”
Revised: l o c a t i o n == ”

Original: o n l i n e ” : # o n l i n e p r
Revised: o n l i n e ” : # o n l i n e p r

Original: e
Revised: e

Original: d i c t
Revised: d i c t

Original: o r with t o r c h .
Revised: o r with t o r c h .

Original: no grad ( ) : u , s , vh = svd ( p ) w = vh .
Revised: no grad ( ) : u , s , vh = svd ( p ) w = vh .

Original: T @
Revised: T @

Original: diag ( g ( s ) ) @
Revised: diag ( g ( s ) ) @

Original: vh p = p @
Revised: vh p = p @

Original: w. detach ( ) e l s e : # t a r g e t p r e d i c t o r with t o r c h .
Revised: w. detach ( ) e l s e : # t a r g e t p r e d i c t o r with t o r c h .

Original: no grad ( ) : u , s , vh = svd ( z ) z
Revised: no grad ( ) : u , s , vh = svd ( z ) z

Original: = u @ diag ( g1 ( s ) ) @
Revised: = u @ diag ( g1 ( s ) ) @

Original: vh
Revised: vh

Original: # g1 ( t ) =
Revised: # g1 ( t ) =

Original: tg ( t ) z
Revised: tg ( t ) z

Original: = z .
Revised: = z .

Original: detach ( ) # stop g r a d i e n t p = normalize ( p , dim =1)
Revised: detach ( ) # stop g r a d i e n t p = normalize ( p , dim =1)

Original: # l2 − normalize z = normalize ( z , dim =1) # l2 − normalize r e t u r n −(p*z ) .
Revised: [Table]

Original: sum ( dim =1 ). mean ( )
Revised: =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: In the main paper
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: we conduct experiments based on the SimSiam framework. Here, we also gather some results of variants of target high-pass filters based on the BYOL framework. We adopt ResNet18 as the backbone encoder and use the default setting in da Costa et al.
Revised: [Table]

Original: Table 4 shows that our target high-pass filters can often outperform BYOL with a large margin at earlier epochs.
Revised: [Table] [Table] BYOL (linear predictor) 80.

Original: In particular, on CIFAR-10, the target filter h ( σ ) = σ − 0 .
Revised: h ( σ ) = σ − 0 .85. h ( σ ) = σ − 0 .85. h ( σ ) = σ − 0 .85.

Original: 3 improves BYOL with learnable linear predictor by 4.92%, 2.68% and 1.00% with 100, 200 and 400 epochs, respectively.
Revised: h ( σ ) = σ −84.

Original: On CIFAR-100, the filter h ( σ ) = σ − 1 is better 8.75% than the baseline with 100 epochs, the filter h ( σ ) =
Revised: On CIFAR-100, the filter h ( σ ) = σ − 1 is better 8.75% than the baseline with 100 epochs, the filter h ( σ ) =

Original: σ − 0 .
Revised: σ − 0 .

Original: 7 improves the baseline by 5.05% with 200 epochs and h ( σ ) =
Revised: 7 improves the baseline by 5.05% with 200 epochs and h ( σ ) =

Original: σ − 0 .
Revised: σ − 0 .

Original: 3 improves the baseline by 5.22% with 400 epochs.
Revised: 3 improves the baseline by 5.22% with 400 epochs.

Original: D C OST C OMPARISON
Revised: D C OST C OMPARISON

Original: We compare the training speeds and GPU memory usages of different methods (SimSiam, the online filter g ( σ ) =
Revised: We compare the training speeds and GPU memory usages of different methods (SimSiam, the online filter g ( σ ) =

Original: log(1 + σ ) and the target filter h ( σ ) =
Revised: log(1 + σ ) and the target filter h ( σ ) =

Original: σ − 0 . 3 ) on CIFAR-10.
Revised: σ − 0 . 3 ) on CIFAR-10.

Original: We perform our experiments on a single RTX 2080ti GPU.
Revised: We perform our experiments on a single RTX 2080ti GPU.

Original: As shown in Table 5, compared to the original SimSiam, the GPU memory usage of our methods only increase a little (26 MiB).
Revised: As shown in Table 5, compared to the original SimSiam, the GPU memory usage of our methods only increase a little (26 MiB).

Original: SimSiam and the target filter h ( σ) =
Revised: SimSiam and the target filter h ( σ

Original: σ − 0 .
Revised: [Table]

Original: 3 have the same training time for 20 epochs.
Revised: [Table]

Original: E A DDITIONAL V ISUALIZATION E. A DDITIONAL E MPIRICAL E VIDENCE FOR E IGENSPACE A LIGNMENT
Revised: E A DDITIONAL V ISUALIZATION E. A DDITIONAL E MPIRICAL E VIDENCE FOR E IGENSPACE A LIGNMENT

Original: In the main paper, we have shown that C p and C z share the same eigenspace.
Revised: In the main paper, we have shown that C p and C z share the same eigenspace.

Original: Here, we add empirical evidence for eigenspace alignment between C z and C + on CIFAR-10 and CIFAR-100.
Revised: Here, we add empirical evidence for eigenspace alignment between C z and C + on CIFAR-10 and CIFAR-100.

Original: As shown in Table 6, we can see that all methods have very high alignment between the eigenspace of C z and C
Revised: As shown in Table 6, we can see that all methods have very high alignment between the eigenspace of C z and C

Original: + .
Revised: + .

Original: E. R ESULTS ON CIFAR- 
Revised: E. R ESULTS ON CIFAR-

Original: We conduct some experiments on CIFAR-100.
Revised: We conduct some experiments on CIFAR-100.

Original: Figure 7 demonstrates that the target branch always has higher effective rank than the online branch and the rank of the online branch continues to increase after the warmup state in all methods.
Revised: Figure 7 demonstrates that the target branch always has higher effective rank than the online branch and the rank of the online branch continues to increase after the warmup state in all methods

Original: In Figure 8(a), we compare the eigenvalues computed from two branch outputs.
Revised: In Figure 8(a), we compare the eigenvalues computed from two branch outputs.

Original: There is a clear difference in eigenvalues, and the eigenvalue distribution of the target branch is more biased towards larger values.
Revised: There is a clear difference in eigenvalues, and the eigenvalue distribution of the target branch is more biased towards larger values.

Original: Figure 8(b) shows the spectral filter g ( λ zi ) =
Revised: Figure 8(b) shows the spectral filter g ( λ zi ) =

Original: (cid:112) λ pi /λ zi , where λ pi , λ zi are eigenvalues of online and target correlation matrices C p , C z , respectively.
Revised: (cid:112) λ pi /λ zi , where λ pi , λ zi are eigenvalues of online and target correlation matrices C p , C z , respectively.

Original: The spectral filters of all methods are nearly monotonically increasing w.r.t.
Revised: The spectral filters of all methods are nearly monotonically increasing w.r.t.

Original: λ zi .
Revised: λ zi .

Original: E. T RAINING D YNAMIC
Revised: E. T RAINING D YNAMIC

Original: In Figure 9, we compare the training loss between SimSiam and SymSimSiam on CIFAR-10 and
Revised: In Figure 9, we compare the training loss between SimSiam and SymSimSiam on CIFAR-10 and

Original: CIFAR-100.
Revised: CIFAR-100.

Original: We can see that the loss of SimSiam is always larger than SymSimsiam and does not consistently decrease.
Revised: We can see that the loss of SimSiam is always larger than SymSimsiam and does not consistently decrease.

Original: Instead, the alignment loss of SymSimSiam continues to decrease.
Revised: Instead, the alignment loss of SymSimSiam continues to decrease.

Revised: [Figure]

Revised: ff e c t i v e r a n k

Original: [Figure]
Revised: [Figure]

Original: F C OMPARISON TO R ELATED W ORK
Revised: F C OMPARISON TO R ELATED W ORK

Original: Comparison with Tian et al.
Revised: Comparison with Tian et al.

Original: Although we assume eigenspace alignment as in Tian et al.
Revised: Although we assume eigenspace alignment as in Tian et al.

Original: , we take very different techniques and arrive at different conclusions, as highlighted below:
Revised: , we take very different techniques and arrive at different conclusions, as highlighted below:

Original: • Difference Perspectives, Techniques, and Conclusions .
Revised: • Difference Perspectives, Techniques, and Conclusions .

Original: As for the goal, Tian et al.
Revised: As for the goal, Tian et al

Original: only consider how predictor helps avoid full collapse.
Revised: only consider how predictor helps avoid full collapse.

Original: Instead, we first point out that avoiding full collapse is not the key role of asymmetric designs (also achievable with symmetric designs).
Revised: Instead, we first point out that avoiding full collapse is not the key role of asymmetric designs (also achievable with symmetric designs).

Original: Thus, we focus on the more general dimensional collapse issue and analyze this  quantitatively through the change of the effective rank along training.
Revised: Thus, we focus on the more general dimensional collapse issue and analyze this [Figure] [Figure] quantitatively through the change of the effective rank along training.

Original: As for the techniques, Tian et al.
Revised: As for the techniques, Tian et al.

Original: mainly analyze the linear learning dynamics under strong architectural and data assumptions, while ours focus on the common spectral filtering property that also holds for nonlinear modules and general data distributions.
Revised: mainly analyze the linear learning dynamics under strong architectural and data assumptions, while ours focus on the common spectral filtering property that also holds for nonlinear modules and general data distributions.

Original: As for the conclusion, we formally show that asymmetric designs will improve effective dimensionality, while Tian et al.
Revised: As for the conclusion, we formally show that asymmetric designs will improve effective dimensionality, while Tian et al.

Original: only discuss how it avoids full collapse (which is an extreme case of dimensional collapse, and a non-full-collapse encoder may still suffer from dimensional collapse).
Revised: only discuss how it avoids full collapse (which is an extreme case of dimensional collapse, and a non-full-collapse encoder may still suffer from dimensional collapse).

Original: • A unified framework for various asymmetric designs .
Revised: • A unified framework for various asymmetric designs .

Original: Tian et al.
Revised: Tian et al.

Original: ’s analysis only focuses on the predictor in BYOL and SimSiam, and they cannot explain why SwAV and DINO also work without predictors.
Revised: ’s analysis only focuses on the predictor in BYOL and SimSiam, and they cannot explain why SwAV and DINO also work without predictors.

Original: Our RDM applies to all existing asymmetric designs through the unified spectral filter perspective.
Revised: Our RDM applies to all existing asymmetric designs through the unified spectral filter perspective.

Original: • General principles for predictor design.
Revised: • General principles for predictor design.

Original: Tian et al.
Revised: Tian et al.

Original: propose DirectPred, which is only a specific filter.
Revised: propose DirectPred, which is only a specific filter.

Original: Instead, we point out the core underlying principle, that as long as the online filter is a low-pass filter, it could theoretically avoid feature collapse.
Revised: Instead, we point out the core underlying principle, that as long as the online filter is a low-pass filter, it could theoretically avoid feature collapse.

Original: We also empirically verify this point by showing that various online low-pass filters can avoid feature collapse.
Revised: We also empirically verify this point by showing that various online low-pass filters can avoid feature collapse

Original:  A More Effective Asymmetric Design through Target Predictor .
Revised: • A More Effective Asymmetric Design through Target Predictor .

Original: Based on our RDM theory, we also propose a new kind of asymmetric design in non-contrastive learning: applying a predictor to the target branch.
Revised: Based on our RDM theory, we also propose a new kind of asymmetric design in non-contrastive learning: applying a predictor to the target branch.

Original: We show that target predictors achieve better results than online predictors while being more computationally efficient.
Revised: We show that target predictors achieve better results than online predictors while being more computationally efficient.

Original: Therefore, our analysis improves Tian et al.
Revised: Therefore, our analysis improves Tian et al.

Original: (2021) in many aspects and apply to a wider context.
Revised: (2021) in many aspects and apply to a wider context

Original: And we achieve this with new perspectives and techniques that are quite distinctive from Tian et al.
Revised: And we achieve this with new perspectives and techniques that are quite distinctive from Tian et al.

Original: [Figure]

Original: [Figure]