Original: α -ReQ : Assessing representation quality by measuring eigenspectrum decay
Revised: ↵ -ReQ : Assessing representation quality by measuring eigenspectrum decay

Original: Abstract
Revised: Abstract

Original: Self-Supervised Learning (SSL) with large-scale unlabelled datasets enables learning useful representations for multiple downstream tasks.
Revised: Self-Supervised Learning (SSL) with large-scale unlabelled datasets enables learning useful representations for multiple downstream tasks.

Original: However, efficiently assessing the quality of such representations poses nontrivial challenges.
Revised: However, efﬁciently assessing the quality of such representations poses nontrivial challenges.

Original: Existing approaches train linear probes (with frozen features) to evaluate performance on a given task.
Revised: Existing approaches train linear probes (with frozen features) to evaluate performance on a given task.

Original: This is expensive both computationally , since it requires retraining a new prediction head for each downstream task, and statistically , which requires task-specific labels for multiple tasks.
Revised: This is expensive both computationally , since it requires retraining a new prediction head for each downstream task, and statistically , which requires task-speciﬁc labels for multiple tasks.

Original: This poses a natural question, how do we efficiently determine the "goodness" of representations learned with SSL across a wide range of potential downstream tasks?
Revised: This poses a natural question, how do we efﬁciently determine the "goodness" of representations learned with SSL across a wide range of potential downstream tasks?

Original: In particular, a task-agnostic statistical measure of representation quality that predicts generalization without explicit downstream task evaluation would be highly desirable.
Revised: In particular, a task-agnostic statistical measure of representation quality that predicts generalization without explicit downstream task evaluation would be highly desirable.

Original: In this work, we analyze characteristics of learned representations f θ in well-trained neural networks with canonical architectures & across SSL objectives.
Revised: In this work, we analyze characteristics of learned representations f ✓ in well-trained neural networks with canonical architectures & across SSL objectives.

Original: We observe that the eigenspectrum of the empirical feature covariance Cov( f θ ) can be well approximated with the family of a power-law distribution.
Revised: We observe that the eigenspectrum of the empirical feature covariance Cov( f ✓ ) can be well approximated with the family of a power-law distribution.

Original: We analytically and empirically (using multiple datasets, e.g. CIFAR, STL10, MIT67, ImageNet) demonstrate that the decay coefficient α serves as a measure of representation quality for tasks that are solvable with a linear readout, that is, there exist welldefined intervals for α where models exhibit excellent downstream generalization.
Revised: We analytically and empirically (using multiple datasets, e.g. CIFAR, STL10, MIT67, ImageNet) demonstrate that the decay coefﬁcient ↵ serves as a measure of representation quality for tasks that are solvable with a linear readout, that is, there exist welldeﬁned intervals for ↵ where models exhibit excellent downstream generalization.

Original: Furthermore, our experiments suggest that key design parameters in SSL algorithms, such as BarlowTwins [1], implicitly modulate the decay coefficient of the eigenspectrum ( α ).
Revised: Furthermore, our experiments suggest that key design parameters in SSL algorithms, such as BarlowTwins [1], implicitly modulate the decay coefﬁcient of the eigenspectrum ( ↵ ).

Original: As α depends only on the features themselves, this measure for model selection with hyperparameter tuning for BarlowTwins enables the search with less compute.
Revised: As ↵ depends only on the features themselves, this measure for model selection with hyperparameter tuning for BarlowTwins enables the search with less compute.

Original:  Introduction
Revised:  Introduction

Original: The recent success of self-supervised learning (SSL) has changed the landscape of deep learning significantly.
Revised: The recent success of self-supervised learning (SSL) has changed the landscape of deep learning signiﬁcantly.

Original: With well-engineered architectures and training objectives, SSL models learn useful representations from large datasets without relying on any labels [1, 2, 3].
Revised: With well-engineered architectures and training objectives, SSL models learn useful representations from large datasets without relying on any labels [1, 2, 3].

Original: Despite this progress, quantifying the representation quality for models trained with SSL is still an open problem.
Revised: Despite this progress, quantifying the representation quality for models trained with SSL is still an open problem.

Original: The most obvious (and common) solution is to assess performance of models using these representations on downstream tasks.
Revised: The most obvious (and common) solution is to assess performance of models using these representations on downstream tasks.

Original: However, if the goal is general representations that can be used across many domains then this either requires a large investment of time and energy to be done well (in order to assess performance on many tasks and datasets).
Revised: However, if the goal is general representations that can be used across many domains then this either requires a large investment of time and energy to be done well (in order to assess performance on many tasks and datasets).

Original: Alternatively, if models are assessed on only a small number of datasets and tasks, then it is hard to be confident in the assessment.
Revised: Alternatively, if models are assessed on only a small number of datasets and tasks, then it is hard to be conﬁdent in the assessment.

Original: Thus, we are left with a question: Can we assess the quality of learned representations without explicitly evaluating the performance on downstream tasks?
Revised: Thus, we are left with a question: Can we assess the quality of learned representations without explicitly evaluating the performance on downstream tasks?

Original: To answer this question, we must formally define the “quality” of representations and subsequently examine statistical estimators that measure this property without needing downstream evaluation.
Revised: To answer this question we must formally deﬁne the “quality” of representations and subsequently examine statistical estimators that measure this property without needing downstream evaluation.

Original: Beyond theoretical interest, such a metric would be highly desirable for model selection and also useful for designing new SSL algorithms.
Revised: Beyond theoretical interest such a metric would be highly desirable for model selection and also useful for designing new SSL algorithms.

Original: In search of such a metric we turn our attention to one of the more efficient learning machines in existence – the mammalian brain.
Revised: In search of such a metric we turn our attention to one of the more efﬁcient learning machines in existence – the mammalian brain.

Original: The hierarchical and distributed organization of neural circuits, especially in the cortex, provides neural representations that support a wide array of behaviours across many domains.
Revised: The hierarchical and distributed organization of neural circuits, especially in the cortex, provides neural representations that support a wide array of behaviours across many domains.

Original: For example, representations in primary visual cortex (V1) of mammalian brains are used by animals to support downstream behaviours ranging from object categorization to movement detection and motor control [4, 5].
Revised: For example, representations in primary visual cortex (V1) of mammalian brains are used by animals to support downstream behaviours ranging from object categorization to movement detection and motor control [4, 5].

Original: This berth of downstream uses of V1 representations is desirable for artificial vision systems trained with SSL.
Revised: This berth of downstream uses of V1 representations is desirable for artiﬁcial vision systems trained with SSL.

Original: Thus, understanding the properties of representations in V1 is a reasonable starting point for seeking a general metric of representation quality.
Revised: Thus, understanding the properties of representations in V1 is a reasonable starting point for seeking a general metric of representation quality.

Original: Recent breakthroughs in systems neuroscience enable large-scale recordings of neural activity.
Revised: Recent breakthroughs in systems neuroscience enable large-scale recordings of neural activity.

Original: By recording and analyzing the response to visual stimuli, [6, 7] find that activations in the mouse and macaque monkey V1 exhibit a characteristic information geometric structure.
Revised: By recording and analyzing the response to visual stimuli, [6, 7] ﬁnd that activations in the mouse and macaque monkey V1 exhibit a characteristic information geometric structure.

Original: In particular, these representations are high-dimensional, yet the amount of information encoded along the different principal directions varies significantly.
Revised: In particular, these representations are high-dimensional, yet the amount of information encoded along the different principal directions varies signiﬁcantly.

Original: Notably, this variance (computed by measuring the eigenspectrum of the empirical covariance matrix) is well-approximated by a power-law distribution with decay coefficient ≈ 1 , i.e., the n th eigenvalue of the covariance matrix scales as 1 /n .
Revised: Notably, this variance (computed by measuring the eigenspectrum of the empirical covariance matrix) is well-approximated by a power-law distribution with decay coefﬁcient ⇡ 1 , i.e., the n th eigenvalue of the covariance matrix scales as 1 /n .

Original: Motivated by these results, we explore the use of the decay rate of the empirical eigenspectrum to characterize representation quality in neural networks trained with SSL (Fig. 1a).
Revised: Motivated by these results, we explore the use of the decay rate of the empirical eigenspectrum to characterize representation quality in neural networks trained with SSL (Fig. 1a).

Original: Across diverse model architectures, pretraining objectives, and downstream classification tasks, we empirically observe an eigenspectrum decay in the representation covariance matrix that roughly follows a power-law distribution (see e.g. Fig. 1b).
Revised: Across diverse model architectures, pretraining objectives, and downstream classiﬁcation tasks, we empirically observe an eigenspectrum decay in the representation covariance matrix that roughly follows a power-law distribution (see e.g. Fig. 1b).

Original: We also find that the coefficient of this decay, denoted by α , is informative of downstream generalization performance.
Revised: We also ﬁnd that the coefﬁcient of this decay, denoted by ↵ , is informative of downstream generalization performance.

Original: Importantly, α can efficiently be calculated without any labels, and thereby, could be incorporated into existing SSL pipelines for efficient model selection on fixed compute budget.
Revised: Importantly, ↵ can efﬁciently be calculated without any labels, and thereby, could be incorporated into existing SSL pipelines for efﬁcient model selection on ﬁxed compute budget.

Original: Our core contributions in this paper are:
Revised: Our core contributions in this paper are:

Original: α as a potential metric We observe that canonical pretrained architectures have representations which loosely conform to a power-law distribution in their covariance eigenspectrum.
Revised: ↵ as a potential metric We observe that canonical pretrained architectures have representations which loosely conform to a power-law distribution in their covariance eigenspectrum.

Original: Under an assumption of a power-law distribution, we prove that the convergence rate and upper bound on generalization error are related to decay-coefficient ( α ) of the corresponding power-law distribution, with best values of α being neither too large nor too small.
Revised: Under an assumption of a power-law distribution, we prove that the convergence rate and upper bound on generalization error are related to decay-coefﬁcient ( ↵ ) of the corresponding power-law distribution, with best values of ↵ being neither too large nor too small.

Original: α as a label-free measure for representation quality : We empirically validate our theoretical results by demonstrating a relationship between α and downstream generalization.
Revised: ↵ as a label-free measure for representation quality : We empirically validate our theoretical results by demonstrating a relationship between ↵ and downstream generalization.

Original: In particular, we find that either too high or too low an α value implies poor generalization, both in-distribution and out-of-distribution.
Revised: In particular, we ﬁnd that either too high or too low an ↵ value implies poor generalization, both in-distribution and out-of-distribution.

Original: Generally, the best representations are those where α is in a range that is close to 1, as observed in V1 of the real brain.
Revised: Generally, the best representations are those where ↵ is in a range that is close to 1, as observed in V1 of the real brain.

Original: Furthermore, these results hold irrespective of the choice of network architecture or pretraining objective.
Revised: Furthermore, these results hold irrespective of the choice of network architecture or pretraining objective.

Original: α for model selection in SSL : We establish α as a reliable metric for model selection in a specific case of SSL, Barlow Twins [1].
Revised: ↵ for model selection in SSL : We establish ↵ as a reliable metric for model selection in a speciﬁc case of SSL, Barlow Twins [1].

Original: Notably, we show that α allows us to identify model hyper-parameters that lead to representations that generalize well without any labels, more so than the actual loss function used to train the network.
Revised: Notably, we show that ↵ allows us to identify model hyper-parameters that lead to representations that generalize well without any labels, more so than the actual loss function used to train the network.

Original: Altogether, our results show that α is a promising task/architecture/data agnostic metric for assessing 
Revised: Altogether, our results show that ↵ is a promising task/architecture/data agnostic metric for assessing representation quality in SSL.

Original: representation quality in SSL.

Original: We publicly release our results and code † .
Revised: We publicly release our results and code † .

Original:  Theoretical Framework to Assess Representations in SSL
Revised:  Theoretical Framework to Assess Representations in SSL

Original: We are interested in evaluating the quality of high dimensional representations learned by neural networks.
Revised: We are interested in evaluating the quality of high dimensional representations learned by neural networks.

Original: Formally, we consider the overparameterized setting, with datasets X ⊂ R d , and learned mappings f : R d → R D such that f ( x ) is a vector of D -dimensional features.
Revised: Formally, we consider the overparameterized setting, with datasets X ⇢ R d , and learned mappings f : R d ! R D such that f ( x ) is a vector of D -dimensional features.

Original: We consider DNNs as our function approximators, where each architecture implicitly defines a function class F = { f θ : θ ∈ Θ } where Θ ⊂ R p is the feasible set of model parameters (e.g bounded Θ , [ − B, B ] p for some B ∈ R ) .
Revised: We consider DNNs as our function approximators, where each architecture implicitly deﬁnes a function class F = { f ✓ : ✓ 2 ⇥ } where ⇥ ⇢ R p is the feasible set of model parameters (e.g bounded ⇥ , [  B, B ] p for some B 2 R ) .

Original: The search for good representations poses the following optimization problem: with a dataset D pretrain from some data distribution P pretrain (potentially with labels), search for optimal parameters θ ∗ such that with pretrain objective L pretrain ( f , D )
Revised: The search for good representations poses the following optimization problem: with a dataset D pretrain from some data distribution P pretrain (potentially with labels), search for optimal parameters ✓ ⇤ such that with pretrain objective L pretrain ( f , D )

Original: [Equation]
Revised: [Equation]

Original: The above optimization problem is usually non-convex, and often uses gradient based optimizer to find an approximate solution ˆ θ .
Revised: The above optimization problem is usually non-convex, and often uses gradient based optimizer to ﬁnd an approximate solution ˆ ✓ .

Original: Typically, to measure the quality of f ˆ θ , researchers evaluate the quality of representations on a downstream task D downstream with a linear readout using some metric
Revised: Typically, to measure the quality of f ˆ ✓ , researchers evaluate the quality of representations on a downstream task D downstream with a linear readout using some metric

Original: R ( linear ( f ˆ θ ), D downstream ) .
Revised: R ( linear ( f ˆ ✓ ), D downstream ) .

Original: A concrete example is pretraining a VGG-16 model ( F ) on ImageNet dataset ( D pretrain ) and evaluating the learned representations using classification accuracy ( R ) by learning a linear classifier on the MIT67 dataset ( D downstream ).
Revised: A concrete example is pretraining a VGG-16 model ( F ) on ImageNet dataset ( D pretrain ) and evaluating the learned representations using classiﬁcation accuracy ( R ) by learning a linear classiﬁer on the MIT67 dataset ( D downstream ).

Original: For the rest of the paper, we denote feature maps as f θ ( x ) ∈ R D where f θ : X → R D , and the readout network as g ϕ : R D 
Revised: For the rest of the paper, we denote feature maps as f ✓ ( x ) 2 R D where f ✓ : X ! R D , and the readout network as g  : R D !

Original: → R k , where k is the target dimensionality.
Revised: R k , where k is the target dimensionality.

Original: For simplicity of analysis, we consider linear readouts unless explicitly mentioned, i.e. g ϕ ( x ) = x T ϕ .
Revised: For simplicity of analysis, we consider linear readouts unless explicitly mentioned, i.e. g  ( x ) = x T  .

Original: 2.1 Covariance estimation and eigenspectrum
Revised: 2.1 Covariance estimation and eigenspectrum

Original: For a parameterized function f θ : X → R D (assume centered), the ( n -sample) empirical covariance matrix is defined as:
Revised: For a parameterized function f ✓ : X ! R D (assume centered), the ( n -sample) empirical covariance matrix is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The eigenspectrum of Σ n ( f θ ) informs us about the variance explained by each principal component of the space spanned by representations f θ ( x ) .
Revised: The eigenspectrum of ⌃ n ( f ✓ ) informs us about the variance explained by each principal component of the space spanned by representations f ✓ ( x ) .

Original: Using the spectral decomposition theorem on symmetric matrices, Σ
Revised: Using the spectral decomposition theorem on symmetric matrices, ⌃ =

Original: = U Λ U T , where Λ is a diagonal matrix with nonnegative entries, and U is a matrix whose columns are the eigenvectors of Σ .
Revised: U ⇤ U T , where ⇤ is a diagonal matrix with nonnegative entries, and U is a matrix whose columns are the eigenvectors of ⌃ .

Original: Without loss of generality we assume that λ 1 ≥ λ 2 ... ≥ λ m , where m = min( n, D ) is the rank of Σ n ( f θ ) .
Revised: Without loss of generality we assume that  1   2 ...   m , where m = min( n, D ) is the rank of ⌃ n ( f ✓ ) .

Original: 2.2 Eigenspectrum Decay in Deep Representation Learning
Revised: 2.2 Eigenspectrum Decay in Deep Representation Learning

Original: Recent work in characterizing representation structure in canonical DNNs has demonstrated that the covariance eigenspectrum roughly follows a power-law [9, 10, 11].
Revised: Recent work in characterizing representation structure in canonical DNNs has demonstrated that the covariance eigenspectrum roughly follows a power-law [9, 10, 11].

Original: Specifically, the eigenspectrum of a covariance matrix follows a power-law distribution PL ( α ) , or zeta distribution if for λ j ∈ [ λ min , λ max ] , the eigenvalues λ j are all nonnegative, and λ j ∝ j − α for some α > 0 .
Revised: Speciﬁcally, the eigenspectrum of a covariance matrix follows a power-law distribution PL ( ↵ ) , or zeta distribution if for  j[  min ,  max ] , the eigenvalues  j are all nonnegative, and  j / j  ↵ for some ↵ > 0 .

Original: Here, α is the slope of the power law, and is referred to as the coefficient of decay of the eigenspectrum.
Revised: Here, ↵ is the slope of the power law, and is referred to as the coefﬁcient of decay of the eigenspectrum.

Original: Intuitively, small α (typically α ≤ 1 ) suggests a dense encoding, while a high α (rapid decay) corresponds to a sparse encoding.
Revised: Intuitively, small ↵ (typically ↵  1 ) suggests a dense encoding, while a high ↵ (rapid decay) corresponds to a sparse encoding.

Original: Insights from High-Dimensional Linear Regression : Recent work in theoretical machine learning has connected bounds on generalization error for a linear regression problem to the eigenspectrum of the feature covariance matrix.
Revised: Insights from High-Dimensional Linear Regression : Recent work in theoretical machine learning has connected bounds on generalization error for a linear regression problem to the eigenspectrum of the feature covariance matrix.

Original: Specifically, in the infinite-dimensional setting with D → ∞ , [12] studied the linear regression setting with Gaussian features, and proved that if the eigenspectrum follows a power-law distribution (up to polylogarithm factors), the min-norm solution provides good generalization performance iff α = 1 .
Revised: Speciﬁcally, in the inﬁnite-dimensional setting with D ! 1 , [12] studied the linear regression setting with Gaussian features, and proved that if the eigenspectrum follows a power-law distribution (up to polylogarithm factors), the min-norm solution provides good generalization performance iff ↵ = 1 .

Original: The asymptotic regime of infinite width is an excellent framework to study theoretical properties of DNN representations.
Revised: The asymptotic regime of inﬁnite width is an excellent framework to study theoretical properties of DNN representations.

Original: However, despite having a large number of parameters, practical DNNs always possess finite dimensional representations, making it important to investigate the implications of such results in finite width models.
Revised: However, despite having a large number of parameters, practical DNNs always possess ﬁnite dimensional representations, making it important to investigate the implications of such results in ﬁnite width models.

Original: In particular, for the finite dimensional setting we try to answer: Does α sufficiently larger or smaller than 1, still allow efficient learning and strong generalizability?
Revised: In particular, for the ﬁnite dimensional setting we try to answer: Does ↵ sufﬁciently larger or smaller than 1, still allow efﬁcient learning and strong generalizability?

Original: To answer this question, we narrow our focus to gradient-based optimization techniques usually used to train DNNs.
Revised: To answer this question, we narrow our focus to gradient-based optimization techniques usually used to train DNNs.

Original: In particular, from the optimization perspective, Advani et al. showed that for deep linear regression in high dimensions, the time required for training and the steady-state generalization error are both O ( 1 λ min ) [13].
Revised: In particular, from the optimization perspective, Advani et al. showed that for deep linear regression in high dimensions, the time required for training and the steady-state generalization error are both O ( 1  min ) [13].

Original: A key difference from our work is that they assume the inputs are drawn from an isotropic Gaussian distribution.
Revised: A key difference from our work is that they assume the inputs are drawn from an isotropic Gaussian distribution.

Original: Instead, we investigate the generalization of linear regression on f θ ( x ) , which has a power law structure in its covariates.
Revised: Instead, we investigate the generalization of linear regression on f ✓ ( x ) , which has a power law structure in its covariates.

Original: We show that the training convergence time grows exponentially with α using the following theorem:
Revised: We show that the training convergence time grows exponentially with ↵ using the following theorem:

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Let ˆ y =
Revised: Let ˆ y =

Original: f θ ( x ) Tψ be an overparameterized linear regression problem whereψ is learned using gradient descent in order to optimize the training error, E x,y
Revised: f ✓ ( x ) T be an overparameterized linear regression problem where is learned using gradient descent in order to optimize the training error, E x,y

Original: [( y − f θ ( x ) Tψ ) 2 ] , where ( x, y ) ∼ D train .
Revised: [( y  f ✓ ( x ) T ) 2 ] , where ( x, y ) ⇠ D train .

Original: If we assume a power-law distribution in the eigenspectrum of representations at f θ , i.e. λ n = cn α ∀ n ≥ n ∗ , where n ∗ ∈ { 1 , 2 ...N } , then the time required by gradient descent to minimize the training error, T convergence =
Revised: If we assume a power-law distribution in the eigenspectrum of representations at f ✓ , i.e.  n = cn ↵ 8 n  n ⇤ , where n ⇤ 2 { 1 , 2 ...N } , then the time required by gradient descent to minimize the training error, T convergence =

Original: O ( N α ) where N is number of training samples.
Revised: O ( N ↵ ) where N is number of training samples.

Original: The outline of the proof builds on key results relating to gradient descent dynamics from [14].
Revised: The outline of the proof builds on key results relating to gradient descent dynamics from [14].

Original: We show that gradient descent updates, whenψ is initialized to 0, yield a recursive relation forψ ( k ) , i.e.ψ after k update steps.
Revised: We show that gradient descent updates, when is initialized to 0, yield a recursive relation for ( k ) , i.e. after k update steps.

Original: Plugging this relation in the gradient formulation, we show that the update step length along the n th principal direction of f θ ( x ) shrinks exponentially with a decay rate proportional to λ n .
Revised: Plugging this relation in the gradient formulation, we show that the update step length along the n th principal direction of f ✓ ( x ) shrinks exponentially with a decay rate proportional to  n .

Original: Therefore, the time to convergence in training is controlled by the smallest eigenvalue which, by design, follows the power law.
Revised: Therefore, the time to convergence in training is controlled by the smallest eigenvalue which, by design, follows the power law.

Original: In sum, Theorem 2.1 provides an explanation against arbitrarily large values of α .
Revised: In sum, Theorem 2.1 provides an explanation against arbitrarily large values of ↵ .

Original: Next we show that α can’t be arbitrarily small as the upper bound on generalization error is higher for smaller values of α .
Revised: Next we show that ↵ can’t be arbitrarily small as the upper bound on generalization error is higher for smaller values of ↵ .

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: Let ˆ y =
Revised: Let ˆ y =

Original: f θ ( x ) Tψ be a linear regression problem as before.
Revised: f ✓ ( x ) T be a linear regression problem as before.

Original: Let us further assume that f θ ( x ) ∀ x ∼ D train is a representative subset of the inputs from true data distribution: ( x, y ) ∼ D .
Revised: Let us further assume that f ✓ ( x ) 8 x ⇠ D train is a representative subset of the inputs from true data distribution: ( x, y ) ⇠ D .

Original: Assuming a power-law distribution in the eigenspectrum of representations at f θ , i.e. λ n = cn α ∀ n ≥ n ∗ , where n ∗ ∈ { 1 , 2 ...N } , the generalization error after T weight update steps, G ( T ) is:
Revised: Assuming a power-law distribution in the eigenspectrum of representations at f ✓ , i.e.  n = cn ↵ 8 n  n ⇤ , where n ⇤ 2 { 1 , 2 ...N } , the generalization error after T weight update steps, G ( T ) is:

Original: [Equation]
Revised: [Equation]

Original:  where
Revised:  where

Original: [Equation]
Revised: [Equation]

Original: Here, m = min ( n, D ) is the rank of Σ n ( f θ ) .
Revised: Here, m = min ( n, D ) is the rank of ⌃ n ( f ✓ ) .

Original: Notably, r ˆ d (Σ( f θ )) can be thought of as a measure of effective rank of the covariance matrix and grows with decreasing values of α .
Revised: Notably, r ˆ d ( ⌃ ( f ✓ )) can be thought of as a measure of effective rank of the covariance matrix and grows with decreasing values of ↵ .

Original: Therefore, the upper bound for generalization error is higher for lower α .
Revised: Therefore, the upper bound for generalization error is higher for lower ↵ .

Original: Taken together, Theorems 2.1 and 2.2 suggest that α can neither be too high nor too low and there exists a tradeoff region which results in efficient learning and strong generalizability, where these representations form a good basis for gradient-based optimization on downstream task performance.
Revised: Taken together, Theorems 2.1 and 2.2 suggest that ↵ can neither be too high nor too low and there exists a tradeoff region which results in efﬁcient learning and strong generalizability, where these representations form a good basis for gradient-based optimization on downstream task performance.

Original: Another motivating example Before exploring the link between α and generalization performance of deep networks, we validate our theorems in a simple linear regression setting.
Revised: Another motivating example Before exploring the link between ↵ and generalization performance of deep networks, we validate our theorems in a simple linear regression setting.

Original: We first consider its relationship to the finite dimensional regression setting as in [12].
Revised: We ﬁrst consider its relationship to the ﬁnite dimensional regression setting as in [12].

Original: Specifically,
Revised: Speciﬁcally,

Original: Theorem 6 of their paper states that the conditions for “benign overfitting”, wherein a model can perfectly fit noisy training data without any subsequent loss of performance on testing data, may be looser in finite dimensions as opposed to the necessary and sufficient condition of α = 1 in infinite dimensions.
Revised: Theorem 6 of their paper states that the conditions for “benign overﬁtting”, wherein a model can perfectly ﬁt noisy training data without any subsequent loss of performance on testing data, may be looser in ﬁnite dimensions as opposed to the necessary and sufﬁcient condition of ↵ = 1 in inﬁnite dimensions.

Original: To empirically test this in the finite high dimensional setting, we examine linear least squares regression using different covariate structures for the input data.
Revised: To empirically test this in the ﬁnite high dimensional setting, we examine linear least squares regression using different covariate structures for the input data.

Original: Formally, we consider covariates { x i } Ni =1 , such that xi ∈ R
Revised: Formally, we consider covariates { x i } Ni =1 , such that x

Original:  d is sampled from a Gaussian distribution with covariance structure Σ =diag { λ 1 , ...
Revised: i 2 R d is sampled from a Gaussian distribution with covariance structure ⌃ =

Original: λ
Revised: diag {  1 , ...

Original:  d } where λ j ∼ PL ( α ) , i.e λ j ∝ cj − α .
Revised:  d } where  j ⇠ PL ( ↵ ) , i.e  j / cj  ↵ .

Original: We assume access to the corresponding labels { y i } Ni =1 generated under a teacher function θ ∗ , such that y i
Revised: We assume access to the corresponding labels { y i } Ni =1 generated under a teacher function ✓ ⇤ , such that y i

Original: =
Revised: =

Original: x
Revised: x

Original: Ti θ ∗ + ϵ i .
Revised: Ti ✓ ⇤ + ✏ i .

Original: We find that in this scenario there is a clear relationship between the proximity of α to 1 and the presence of benign overfitting.
Revised: We ﬁnd that in this scenario there is a clear relationship between the proximity of ↵ to 1 and the presence of benign overﬁtting.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, when α is close to 1, the training loss is low, but the validation loss is also low.
Revised: 2, when ↵ is close to 1, the training loss is low, but the validation loss is also low.

Original: Thus, when α is close to 1, the generalization properties are at their best.
Revised: Thus, when ↵ is close to 1, the generalization properties are at their best.

Original: This example thus provides another hint that α is a potential measure for how well a model will be able to generalize.
Revised: This example thus provides another hint that ↵ is a potential measure for how well a model will be able to generalize.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: Our theoretical results suggest the following: for representations with power-law characteristics, the decay-coefficient ( α ) effects the adaptivity , where the convergence rate of linear regression with gradient descent scales exponentially O ( n α ) with α .
Revised: Our theoretical results suggest the following: for representations with power-law characteristics, the decay-coefﬁcient ( ↵ ) effects the adaptivity , where the convergence rate of linear regression with gradient descent scales exponentially O ( n ↵ ) with ↵ .

Original: On the other hand, if trained sufficiently long, the upper bound for generalization error improves with larger α .
Revised: On the other hand, if trained sufﬁciently long, the upper bound for generalization error improves with larger ↵ .

Original: Together, these items imply that if one wants to use gradient descent to train a downstream task then α for the representations used should be neither too large nor too small, i.e. high quality representations are those with α in a “Goldilocks” zone.
Revised: Together, these items imply that if one wants to use gradient descent to train a downstream task then ↵ for the representations used should be neither too large nor too small, i.e. high quality representations are those with ↵ in a “Goldilocks” zone.

Original: Building on these insights, we now design experiments to empirically study the potential for α to be used as a representation quality metric.
Revised: Building on these insights, we now design experiments to empirically study the potential for ↵ to be used as a representation quality metric.

Original: We do so by measuring the relationship between α and the standard metric of representation quality, i.e. downstream task generalization performance.
Revised: We do so by measuring the relationship between ↵ and the standard metric of representation quality, i.e. downstream task generalization performance.

Original: In this work, we restrict our scope to vision tasks, specifically object classification and scene recognition.
Revised: In this work, we restrict our scope to vision tasks, speciﬁcally object classiﬁcation and scene recognition.

Original: In particular, our experiments:
Revised: In particular, our experiments:

Original: • measure the eigenspectrum decay & generalization : We systematically evaluate indistribution and out-of-distribution generalization, and simultaneously, we also measure α , across diverse choices for network architectures, pretraining learning objectives, and across multiple datasets.
Revised: • measure the eigenspectrum decay & generalization : We systematically evaluate indistribution and out-of-distribution generalization, and simultaneously, we also measure ↵ , across diverse choices for network architectures, pretraining learning objectives, and across multiple datasets.

Original: We then look at whether α values near some neighbourhood around 1 are indeed predictive of representations that generalize well to downstream tasks.
Revised: We then look at whether ↵ values near some neighbourhood around 1 are indeed predictive of representations that generalize well to downstream tasks.

Original: • explore α as a metric for model-selection in SSL:
Revised: • explore ↵ as a metric for model-selection in SSL:

Original: Measuring α is label-free and computationally inexpensive, as it requires a single forward-pass on the downstream dataset.
Revised: Measuring ↵ is label-free and computationally inexpensive, as it requires a single forward-pass on the downstream dataset.

Original: We thus ask, is α predictive of generalization capabilities for current SSL algorithms, and if so, can it be used for model selection?
Revised: We thus ask, is ↵ predictive of generalization capabilities for current SSL algorithms, and if so, can it be used for model selection?

Original: We run extensive ablations on design of the BarlowTwins [1] algorithm in an effort to answer this question.
Revised: We run extensive ablations on design of the BarlowTwins [1] algorithm in an effort to answer this question.

Original: 3.1 Measuring eigenspectrum Decay & Generalization
Revised: 3.1 Measuring eigenspectrum Decay & Generalization

Original: Evaluation Protocol: To measure α for the learned representation f
Revised: Evaluation Protocol: To measure ↵ for the learned representation f

Original: ( x i ) , we first extract features from intermediate layers of a DNN pretrained on D pretrain and estimate the corresponding covariance matrix Σ n ( f ) .
Revised: ( x i ) , we ﬁrst extract features from intermediate layers of a DNN pretrained on D pretrain and estimate the corresponding covariance matrix ⌃ n ( f ) .

Original: Next, we compute the full set of numerical eigenvalues, and estimate α by a fitting a linear model on the eigenspectrum in log-log scale.
Revised: Next, we compute the full set of numerical eigenvalues, and estimate ↵ by a ﬁtting a linear model on the eigenspectrum in log-log scale.

Original: Our pretrained models are taken from PyTorch
Revised: Our pretrained models are taken from PyTorch

Original: Hub [15] and timm [16].
Revised: Hub [15] and timm [16].

Original: Given that we observed no significant difference between the observed α values in the train and test sets, we refer to this empirical estimate as the α for the dataset.
Revised: Given that we observed no signiﬁcant difference between the observed ↵ values in the train and test sets, we refer to this empirical estimate as the ↵ for the dataset.

Original: To estimate the capacity of intermediate representations in solving the downstream task, we train a linear readout layer, g ( . ) , from representations to target logits.
Revised: To estimate the capacity of intermediate representations in solving the downstream task, we train a linear readout layer, g ( . ) , from representations to target logits.

Original: Intuitively, this comes down to establishing a relationship between the manifold geometry and linear separability of representations [17].
Revised: Intuitively, this comes down to establishing a relationship between the manifold geometry and linear separability of representations [17].

Original: Thereafter, we observe the correlation between estimated α and the linear readout performance.
Revised: Thereafter, we observe the correlation between estimated ↵ and the linear readout performance.

Original: Notably, we also tried non-linear g ( . ) and observed a similar trend in results (see Appendix B.1).
Revised: Notably, we also tried non-linear g ( . ) and observed a similar trend in results (see Appendix B.1).

Original: Inductive bias of Model Architecture In this section, we investigate the relationship between α of the representation covariance matrix and object/scene recognition performance in DNNs with different backbones and the role of depth.
Revised: Inductive bias of Model Architecture In this section, we investigate the relationship between ↵ of the representation covariance matrix and object/scene recognition performance in DNNs with different backbones and the role of depth.

Original: In order to do so, we examine varying depth configurations within network architectures across three generations of models on the STL10 and MIT67 dataset [18].
Revised: In order to do so, we examine varying depth conﬁgurations within network architectures across three generations of models on the STL10 and MIT67 dataset [18].

Original: We choose both object and scene recognition task as they fundamentally differ in the nature of the features that must be extracted to perform well.
Revised: We choose both object and scene recognition task as they fundamentally differ in the nature of the features that must be extracted to perform well.

Original: For scene recognition, the model focuses on global features in the entire image, while for object recognition, the model focuses on local features of the image containing the object[19].
Revised: For scene recognition, the model focuses on global features in the entire image, while for object recognition, the model focuses on local features of the image containing the object[19].

Original: We examine deep Convolutional Neural Networks (CNNs) without any residual connection as our first family of models.
Revised: We examine deep Convolutional Neural Networks (CNNs) without any residual connection as our ﬁrst family of models.

Original: Specifically, we choose three different configurations of VGG-Net [20], namely VGG-13, VGG-16 and VGG-19.
Revised: Speciﬁcally, we choose three different conﬁgurations of VGG-Net [20], namely VGG-13, VGG-16 and VGG-19.

Original: We inspect representations that are input to the dropout and MaxPool layers during the forward pass of the network.
Revised: We inspect representations that are input to the dropout and MaxPool layers during the forward pass of the network.

Original: Second, we consider Deep Residual
Revised: Second, we consider Deep Residual

Original: Networks [21] which are widely used in computer vision.
Revised: Networks [21] which are widely used in computer vision.

Original: We inspect the representations that are input to each of the residual blocks as well as the Adaptive average pool in ResNet-13, ResNetand ResNet-101 during their respective forward passes.
Revised: We inspect the representations that are input to each of the residual blocks as well as the Adaptive average pool in ResNet-13, ResNetand ResNet-101 during their respective forward passes.

Original: Finally, owing to the recent success of transformers in object recognition tasks we consider Vision Transformers (ViT)
Revised: Finally, owing to the recent success of transformers in object recognition tasks we consider Vision Transformers (ViT)

Original: [22] as the third family of models, namely ViT-Base/8 , ViT-Large/16 and ViT-Huge/14.
Revised: [22] as the third family of models, namely ViT-Base/8 , ViT-Large/16 and ViT-Huge/14.

Original: Unlike VGG and ResNet, we only look at features in the intermediate layers because it summarizes the entire input image and is used in practice for class prediction.
Revised: Unlike VGG and ResNet, we only look at features in the intermediate layers because it summarizes the entire input image and is used in practice for class prediction.

Original: For all model architectures, we use the weights obtained from pretraining on ImageNet.
Revised: For all model architectures, we use the weights obtained from pretraining on ImageNet.

Original: Fig.
Revised: Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: 4 illustrate the relation between performance and α for all the nine architectures across intermediate layer representations, as described above.
Revised: 4 illustrate the relation between performance and ↵ for all the nine architectures across intermediate layer representations, as described above.

Original: We found that, while most intermediate representations in CNNs (with or without residual connections) exhibit α < 1 , representations in ViTs mostly exhibit α > 1 .
Revised: We found that, while most intermediate representations in CNNs (with or without residual connections) exhibit ↵ < 1 , representations in ViTs mostly exhibit ↵ > 1 .

Original: Nevertheless, representations extracted from the deepest layers of all the models exhibit α close to 1 , irrespective of the total depth of each model (see Appendix B).
Revised: Nevertheless, representations extracted from the deepest layers of all the models exhibit ↵ close to 1 , irrespective of the total depth of each model (see Appendix B).

Original: Furthermore, the performance on downstream task increases with depth.
Revised: Furthermore, the performance on downstream task increases with depth.

Original: This is unsurprising because all networks were trained to perform object recognition on ImageNet [23] and thereby would have leveraged hierarchical processing to learn features that are tuned towards object recognition.
Revised: This is unsurprising because all networks were trained to perform object recognition on ImageNet [23] and thereby would have leveraged hierarchical processing to learn features that are tuned towards object recognition.

Original: Surprisingly enough, we observe a strong significant correlation between α and performance on the STL10 dataset, i.e. a different data distribution than the training dataset, across layers and model architectures ( ρ = − 0 .
Revised: Surprisingly enough, we observe a strong signiﬁcant correlation between ↵ and performance on the STL10 dataset, i.e. a different data distribution than the training dataset, across layers and model architectures ( ⇢ =  0 .

Original: 922 , ∗ p < 0 .
Revised: 922 , ⇤ p < 0 .

Original: for representations exhibiting α > 1 and ρ = − 0 .
Revised: for representations exhibiting ↵ > 1 and ⇢ =  0 .

Original: 922 , ∗ p < 0 .
Revised: 922 , ⇤ p < 0 .

Original: for representations exhibiting α < 1 ).
Revised: for representations exhibiting ↵ < 1 ).

Original: It is worth noting here that the correlation was weaker for the earliest layers of each model.
Revised: It is worth noting here that the correlation was weaker for the earliest layers of each model.

Original: We believe that early layers learn more task invariant features that reflect the statistics of natural images [24, 25] and therefore lack task relevant information in their representations.
Revised: We believe that early layers learn more task invariant features that reﬂect the statistics of natural images [24, 25] and therefore lack task relevant information in their representations.

Original: Taken together, this observation confirms our hypothesis that α is a good indicator of out-of-distribution generalization performance when representations possess task relevant information.
Revised: Taken together, this observation conﬁrms our hypothesis that ↵ is a good indicator of out-of-distribution generalization performance when representations possess task relevant information.

Original: Thus, α has a necessary but not sufficient relationship with generalization (Appendix C).
Revised: Thus, ↵ has a necessary but not sufﬁcient relationship with generalization (Appendix C).

Original: Learning objective In this section, we first aim to understand how the α value changes across the layers of a fixed architecture DNN when trained with different learning objectives.
Revised: Learning objective In this section, we ﬁrst aim to understand how the ↵ value changes across the layers of a ﬁxed architecture DNN when trained with different learning objectives.

Original: We take a
Revised: We take a

Original: ResNet-50 model [21] pre-trained using three different SSL algorithms, namely SimCLR [2], BYOL [26] and Barlow Twins [1], and the supervised learning loss objectives on ImageNet-1k[23] dataset.
Revised: ResNet-50 model [21] pre-trained using three different SSL algorithms, namely SimCLR [2], BYOL [26] and Barlow Twins [1], and the supervised learning loss objectives on ImageNet-1k[23] dataset.

Original: We use a similar procedure as before to extract representations from the network and estimate α .
Revised: We use a similar procedure as before to extract representations from the network and estimate ↵ .

Original: Similar to results in the previous section, all networks irrespective of the pretraining loss function, exhibit α closer to 1 in the deeper layers in contrast to intermediate layers (see Appendix Fig. 9).
Revised: Similar to results in the previous section, all networks irrespective of the pretraining loss function, exhibit ↵ closer to 1 in the deeper layers in contrast to intermediate layers (see Appendix Fig. 9).

Original: This surprising result indicates that although the pre-training loss function was different, representations extracted from deepest layers are reflective of the object/scene semantics in natural images.
Revised: This surprising result indicates that although the pre-training loss function was different, representations extracted from deepest layers are reﬂective of the object/scene semantics in natural images.

Original: Furthermore, Fig. 3 and Fig.
Revised: Furthermore, Fig. 3 and Fig.

Original: 4 illustrate the strong correlation between α and generalization performance on STL10 and MIT67 across all pre-training loss functions.
Revised: 4 illustrate the strong correlation between ↵ and generalization performance on STL10 and MIT67 across all pre-training loss functions.

Original: Together with results from the previous section, we validate our hypothesis that the representations that demonstrate good out-of-distribution generalization performance are characterized by α in the neighbourhood of 1.
Revised: Together with results from the previous section, we validate our hypothesis that the representations that demonstrate good out-of-distribution generalization performance are characterized by ↵ in the neighbourhood of 1.

Original: This suggests that high quality representations are indeed those with α in this region.
Revised: This suggests that high quality representations are indeed those with ↵ in this region.

Original: 3.2 Label-agnostic metric for Model selection
Revised: 3.2 Label-agnostic metric for Model selection

Original: With empirical evidence of α being a good measure for generalization, we now wish to study its suitability as a metric for identifying the best among models pretrained with SSL algorithms.
Revised: With empirical evidence of ↵ being a good measure for generalization, we now wish to study its suitability as a metric for identifying the best among models pretrained with SSL algorithms.

Original: A labelfree metric like α could be beneficial when we don’t have access to the downstream task annotations, and the SSL loss is not useful to distinguish models with good generalization performance (see Fig. 5).
Revised: A labelfree metric like ↵ could be beneﬁcial when we don’t have access to the downstream task annotations, and the SSL loss is not useful to distinguish models with good generalization performance (see Fig. 5).

Original: To investigate this, we exhaustively ablate the relationship between alpha and model performance across a wide range of hyper-parameters for a representative non-contrastive SSL algorithm.
Revised: To investigate this, we exhaustively ablate the relationship between alpha and model performance across a wide range of hyper-parameters for a representative non-contrastive SSL algorithm.

Original: Current SSL algorithms struggle with dimension collapse (characterized by large α ), due to pathologies in training dynamics.
Revised: Current SSL algorithms struggle with dimension collapse (characterized by large ↵ ), due to pathologies in training dynamics.

Original: To study α across a wide-range of values, for our model selection experiments, we pick the Barlow Twins [1], a non-contrastive SSL algorithm that explicitly optimizes to avoid dimension collapse.
Revised: To study ↵ across a wide-range of values, for our model selection experiments, we pick the Barlow Twins [1], a non-contrastive SSL algorithm that explicitly optimizes to avoid dimension collapse.

Original: In particular, the Barlow Twins learning objective ( L BT ) proposes imposing a soft-whitening constraint :
Revised: In particular, the Barlow Twins learning objective ( L BT ) proposes imposing a soft-whitening constraint :

Original: [Equation]
Revised: [Equation]

Original: Notably with sufficient large λ , the model would impose an α = 0 constraint on the representations [27].
Revised: Notably with sufﬁcient large  the model would impose an ↵ = 0 constraint on the representations [27].

Original: Despite the intuitive connection between α and λ , it is unclear whether this relationship holds across a wide-range of values for model hyperparameters.
Revised: Despite the intuitive connection between ↵ and  it is unclear whether this relationship holds across a wide-range of values for model hyperparameters.

Original: To empirically establish this connection, we vary λ (redundancy coeffiecient), projection head dimensionality and learning rate, and train a
Revised: To empirically establish this connection, we vary  (redundancy coefﬁecient), projection head dimensionality and learning rate and train a

Original: ResNet50 encoder using Barlow Twins learning objective.
Revised: ResNet50 encoder using Barlow Twins learning objective.

Original: We provide the results for CIFAR10 [28] and STL10 [29] in Fig.
Revised: We provide the results for CIFAR10 [28] and STL10 [29] in Fig.

Original: 5 demonstrating that α is a strong indicator of in-distribution generalization performance across a large range of hyperparameter ranges of L BT .
Revised: 5 demonstrating that ↵ is a strong indicator of in-distribution generalization performance across a large range of hyperparameter ranges of L BT .

Original: We defer the reader to Appendix
Revised: We defer the reader to Appendix

Original: Fig.
Revised: Fig.

Original: 14 & Fig.
Revised: 14 & Fig.

Original: 15 for similar trends in optimization hyperparameters.
Revised: 15 for similar trends in optimization hyperparameters.

Original: Furthermore, α is predictive of out-of-distribution generalization performance in these settings (see Appendix B.3).
Revised: Furthermore, ↵ is predictive of out-of-distribution generalization performance in these settings (see Appendix B.3)

Original: [Table]
Revised: [Table]

Original: Model Selection on a Compute Budget:
Revised: Model Selection on a Compute Budget:

Original: With a constrained compute budget, using α for model selection is significantly cheaper than evaluating the representations on a suite of downstream tasks.
Revised: With a constrained compute budget, using ↵ for model selection is signiﬁcantly cheaper than evaluating the representations on a suite of downstream tasks.

Original: To illustrate this, consider the standard alternative of training a linear classifier on the representations and evaluating test accuracy.
Revised: To illustrate this, consider the standard alternative of training a linear classiﬁer on the representations and evaluating test accuracy.

Original: Compared to training a linear probe, which requires multiple epochs of forward & backward passes through the training dataset’s features, to achieve reasonable estimates of downstream accuracy, computing α requires a single PCA step on the validation dataset’s features.
Revised: Compared to training a linear probe, which requires multiple epochs of forward & backward passes through the training dataset’s features, to achieve reasonable estimates of downstream accuracy, computing ↵ requires a single PCA step on the validation dataset’s features.

Original: In Table 1, we contrast the compute times for α and evaluating a linear probe.
Revised: In Table 1, we contrast the compute times for ↵ and evaluating a linear probe.

Original: A detailed algorithm to use α for model selection is provided in Algorithm 1 and its complexity analysis in Appendix B.5.best trained ∈ [ α min , α max ]} best_trained_ } # Update
Revised: A detailed algorithm to use ↵ for model selection is provided in Algorithm 1 and its complexity analysis in Appendix B.5.

Original: [Table]

Original: and

Original: [Table]
Revised: [Table]

Original: max [Table] [Table] m
Revised: trained

Original: [Table] m > [Table] j
Revised: = best

Original: [Table]
Revised: [Table]

Original: m

Original: [Table]
Revised: [Table]

Original: j
Revised: , alpha in alpha

Original: [Table]
Revised: [Table]

Original: ,j

Original: [Table]

Original: [Table] [Table] [Table] models_stat})
Revised: models if alpha

Original: [Table][Table]
Revised: [Table]

Original: threshold

Original: [Table]
Revised: [Table]

Original: best

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: > threshold} # Return

Original: [Table]
Revised: [Table]

Original: performance from the ‘selected’

Original: [Table][Table]
Revised: [Table]

Original: _stat}) return best_

Original: [Table]
Revised: [Table]

Original: acc  Related Work
Revised:  Related Work

Original: Evaluating representations and model quality We note a substantial body of work aiming to empirically characterize the structure of emergent representations in DNN without requiring labels [30, 31].
Revised: Evaluating representations and model quality We note a substantial body of work aiming to empirically characterize the structure of emergent representations in DNN without requiring labels [30, 31].

Original: One such index that quantifies the similarity of representations across layers (of the same or different models) is Centered Kernel Alignment (CKA) [24].
Revised: One such index that quantiﬁes the similarity of representations across layers (of the same or different models) is Centered Kernel Alignment (CKA) [24].

Original: While CKA does not provide explicit guidance for downstream performance, [32] shows that the in-distribution generalization gap can be predicted using a different index based on the model’s parameters.
Revised: While CKA does not provide explicit guidance for downstream performance, [32] shows that the in-distribution generalization gap can be predicted using a different index based on the model’s parameters.

Original: In particular, they show that
Revised: In particular, they show that

Original: the
Revised: the

Original: Empirical Spectral Density (ESD) of weight matrices for many DNNs obey a power-law, with the decay coefficient being predictive of in-distribution performance.
Revised: Empirical Spectral Density (ESD) of weight matrices for many DNNs obey a power-law, with the decay coefﬁcient being predictive of in-distribution performance.

Original: In the present work, we explore similar indices that potentially correlate with out-of-distribution generalization by examining the eigenspectrum of activations .
Revised: In the present work, we explore similar indices that potentially correlate with out-of-distribution generalization by examining the eigenspectrum of activations .

Original: Generalization in Overparameterized Models Modern neural networks often have significantly more parameters than the number of training samples, challenging the classical understanding of the bias-variance tradeoff.
Revised: Generalization in Overparameterized Models Modern neural networks often have signiﬁcantly more parameters than the number of training samples, challenging the classical understanding of the bias-variance tradeoff.

Original: Overparameterization permits neural networks to overfit to noise in training data without impairing their generalization to unseen data.
Revised: Overparameterization permits neural networks to overﬁt to noise in training data without impairing their generalization to unseen data.

Original: In recent work, Bubeck et al. proved that overparameterization is a necessary condition for smooth interpolation in neural networks [33].
Revised: In recent work, Bubeck et al. proved that overparameterization is a necessary condition for smooth interpolation in neural networks [33].

Original: Furthermore, this benign overfitting phenomenon in an overparameterized linear regression problem has been linked to the power law coefficient of the input covariance matrix [12].
Revised: Furthermore, this benign overﬁtting phenomenon in an overparameterized linear regression problem has been linked to the power law coefﬁcient of the input covariance matrix [12].

Original: Specifically, Bartlett et al. showed that benign overfitting is possible for an infinite-dimensional linear regression problem iff the eigenspectrum satisfies a power law (up to polylog factors).
Revised: Speciﬁcally, Bartlett et al. showed that benign overﬁtting is possible for an inﬁnite-dimensional linear regression problem iff the eigenspectrum satisﬁes a power law (up to polylog factors).

Original: More recently, Lee et al. found that the tail eigenvalues of infinite-width network kernels exhibit a power law decay [34].
Revised: More recently, Lee et al. found that the tail eigenvalues of inﬁnite-width network kernels exhibit a power law decay [34].

Original: Following this, Tripuraneni et al.
Revised: Following this, Tripuraneni et al.

Original: explored high-dimensional random feature regression settings and analytically showed a dependence between the eigenspectrum decay rate of the feature covariance matrix and generalization error [35].
Revised: explored high-dimensional random feature regression settings and analytically showed a dependence between the eigenspectrum decay rate of the feature covariance matrix and generalization error [35].

Original: While these characterizations provide a theoretical understanding of generalization error in the asymptotic or random feature settings, corresponding questions in the finite-dimensional DNN trained with gradient descent are open problems.
Revised: While these characterizations provide a theoretical understanding of generalization error in the asymptotic or random feature settings, corresponding questions in the ﬁnite-dimensional DNN trained with gradient descent are open problems.

Original: For deep linear networks trained using gradient descent, the eigenvalues of input covariance determine the generalization error dynamics [13].
Revised: For deep linear networks trained using gradient descent, the eigenvalues of input covariance determine the generalization error dynamics [13].

Original: Advani et al.
Revised: Advani et al.

Original: demonstrated that small eigenvalues determine the convergence of training dynamics as well as the overfitting error at convergence.
Revised: demonstrated that small eigenvalues determine the convergence of training dynamics as well as the overﬁtting error at convergence.

Original: For overpa- rameterized 2-layer neural networks, Arora et al.
Revised: For overpa- rameterized 2-layer neural networks, Arora et al.

Original: provided a fine-grained analysis of generalization bounds [36].
Revised: provided a ﬁne-grained analysis of generalization bounds [36].

Original: In contrast, we study modern DNN architectures and explore the covariance structure of their learned features on visual recognition tasks.
Revised: In contrast, we study modern DNN architectures and explore the covariance structure of their learned features on visual recognition tasks.

Original:  Discussion
Revised:  Discussion

Original: Summary Our experiments suggest a strong correlation between the decay coefficient for sample eigenspectrum of representations, α , and both in-distribution and out-of-distribution generalization performance on tasks central to computer vision.
Revised: Summary Our experiments suggest a strong correlation between the decay coefﬁcient for sample eigenspectrum of representations, ↵ , and both in-distribution and out-of-distribution generalization performance on tasks central to computer vision.

Original: Representation Quality in High-Dimensional SSL We demonstrate that assessing the quality of a pretrained model with our label-free metric ( α ) is consistent with downstream generalization.
Revised: Representation Quality in High-Dimensional SSL We demonstrate that assessing the quality of a pretrained model with our label-free metric ( ↵ ) is consistent with downstream generalization.

Original: While being computationally efficient, our procedure removes the dependence on labels, making the model selection more robust and amenable for privacy critical applications.
Revised: While being computationally efﬁcient, our procedure removes the dependence on labels, making the model selection more robust and amenable for privacy critical applications.

Original: Necessary, but not sufficient condition Notably, a task-agnostic measure like α is a necessary but not sufficient condition for assessing good performance.
Revised: Necessary, but not sufﬁcient condition Notably, a task-agnostic measure like ↵ is a necessary but not sufﬁcient condition for assessing good performance.

Original: To elaborate on this point, let us present a thought experiment.
Revised: To elaborate on this point, let us present a thought experiment.

Original: Suppose we have our ideal representation space that satisfies our claims of exhibiting alpha close to the Goldilocks zone.
Revised: Suppose we have our ideal representation space that satisﬁes our claims of exhibiting alpha close to the Goldilocks zone.

Original: If we perform a set of permutation operations on individual datapoint representations, the structure of the representation space remains the same, but the mapping from representation to label is now destroyed.
Revised: If we perform a set of permutation operations on individual datapoint representations, the structure of the representation space remains the same, but the mapping from representation to label is now destroyed.

Original: In doing this set of permutations, we have now arrived at a different set of representations that would exhibit the same (or similar) alpha but demonstrate a lower task performance when measured using a linear readout layer.
Revised: In doing this set of permutations, we have now arrived at a different set of representations that would exhibit the same (or similar) alpha but demonstrate a lower task performance when measured using a linear readout layer.

Original: A similar argument is presented as theorem C.1 in [37].
Revised: A similar argument is presented as theorem C.1 in [37].

Original: Extending the conclusions of this thought experiment to our observations, it is clear that we could have models with similar alpha values but distinctly different performances.
Revised: Extending the conclusions of this thought experiment to our observations, it is clear that we could have models with similar alpha values but distinctly different performances.

Original: But an alpha value that is not in the Goldilocks zone would be associated with inferior model performance.
Revised: But an alpha value that is not in the Goldilocks zone would be associated with inferior model performance.

Original: This property is the core of our claim and allows us to propose alpha as a measure for model selection in SSL pipelines.
Revised: This property is the core of our claim and allows us to propose alpha as a measure for model selection in SSL pipelines.

Original: Redundancy in ViT representations Unlike other models that we considered, representations from early layers of ViT had a rapid eigenspectrum decay with α >> 1 (see Fig.
Revised: Redundancy in ViT representations Unlike other models that we considered, representations from early layers of ViT had a rapid eigenspectrum decay with ↵ >> 1 (see Fig.

Original: 3 and Fig. 4).
Revised: 3 and Fig. 4).

Original: The transformer architecture has a notable difference by design, i.e. early layers possess global receptive field context via self-attention on patch embeddings.
Revised: The transformer architecture has a notable difference by design, i.e. early layers possess global receptive ﬁeld context via self-attention on patch embeddings.

Original: Raghu et al. found that early ViT layers incorporate both local and global information [31].
Revised: Raghu et al. found that early ViT layers incorporate both local and global information [31].

Original: Based on these insights, one intuitive interpretation of our results is that the representations have a low effective rank and encode redundant information relevant across multiple scales.
Revised: Based on these insights, one intuitive interpretation of our results is that the representations have a low effective rank and encode redundant information relevant across multiple scales.

Original: Limitations While the role of α and its relationship to generalization performance is better understood in the asymptotic setting for linear regression, similar questions in finite-dimensional nonlinear models are unanswered.
Revised: Limitations While the role of ↵ and its relationship to generalization performance is better understood in the asymptotic setting for linear regression, similar questions in ﬁnite-dimensional nonlinear models are unanswered.

Original: It is also worth noting that the empirical correlation was weaker for the earliest layers in each model.
Revised: It is also worth noting that the empirical correlation was weaker for the earliest layers in each model.

Original: We believe that early layers learn more task invariant features such as corners and edges [24, 25], and thereby lack rich semantic information for the fine-grained downstream task.
Revised: We believe that early layers learn more task invariant features such as corners and edges [24, 25], and thereby lack rich semantic information for the ﬁne-grained downstream task.

Original: In this case, distinguishing between poorly-trained models may be inconsistent with α .
Revised: In this case, distinguishing between poorly-trained models may be inconsistent with ↵ .

Original: Future Directions Learning efficiently at scale from unlabelled datasets poses an exciting open problem in deep representation learning.
Revised: Future Directions Learning efﬁciently at scale from unlabelled datasets poses an exciting open problem in deep representation learning.

Original: We hope this work encourages new perspectives into model selection for SSL pipelines and informs the design of learning objectives and model architectures to learn task-agnostic, adaptive features.
Revised: We hope this work encourages new perspectives into model selection for SSL pipelines and informs the design of learning objectives and model architectures to learn task-agnostic, adaptive features.

Original: Understanding the behaviour of α (notably, a scaleinvariant metric) in high-dimensional representation learning might provide insight into developing a theoretically grounded understanding of generalization in deep neural networks.
Revised: Understanding the behaviour of ↵ (notably, a scaleinvariant metric) in high-dimensional representation learning might provide insight into developing a theoretically grounded understanding of generalization in deep neural networks.

Original: Acknowledgements
Revised: Acknowledgements

Original: The authors would like to thank Zahraa Chorghay and Colleen Gillon for their aesthetic contribution to the manuscript and figures.
Revised: The authors would like to thank Zahraa Chorghay and Colleen Gillon for their aesthetic contribution to the manuscript and ﬁgures.

Original: This research was enabled in part by support provided by Mila and Compute Canada.
Revised: This research was enabled in part by support provided by Mila and Compute Canada.

Original: This work was supported by Vanier Canada Graduate scholarship (AG);
Revised: This work was supported by Vanier Canada Graduate scholarship (AG);

Original: Healthy Brains, Healthy Lives (AG & BAR); NSERC, Grant No. RGPIN-2020-05105 and RGPAS2020-00031 (BAR); and CIFAR, Canada AI Chair & Learning in Machines and Brains Fellowship (BAR).A Appendix A.1 Proofs In this section, we present a formal proof of Theorem 2.1.
Revised: Healthy Brains, Healthy Lives (AG & BAR); NSERC, Grant No. RGPIN-2020-05105 and RGPAS2020-00031 (BAR); and CIFAR, Canada AI Chair & Learning in Machines and Brains Fellowship (BAR).

Original: In order to do so, we will use a lemma pertaining to iterative expression of the linear regression parameters over training epochs.

Original: This lemma is inspired by the results presented in [14]

Original: Lemma A.1.

Original: Let ˆ y =

Original: x T w be a finite dimensional linear regression problem where w is learned using gradient descent in order to optimize the training error,

Original: [Equation]

Original: where ( x, y ) ∼ D train , i.e. the training dataset.

Original: Then w k , i.e. w after training for k epochs can be written as

Original: [Equation]

Original: where X, Y indicate the entire training dataset, i.e. X ∈ R N × d and Y ∈ R N

Original: Proof.

Original: We start with the gradient of the regression loss function for a defined training set denoted by ( X, Y ) in a vectorized notation:

Original: [Equation]

Original: We assume that the weights are initialized at 0, i.e. w 0 = 0 .

Original: Using the gradient descent update:

Original: [Equation]

Original: Let w k = ηX

Original: [Equation]

Original: u

Original: [Equation]

Original: I

Original: [Equation]

Original: +

Original: [Equation]

Original: +

Original: [Equation]

Original: Y = ⇒ u

Original: [Equation]

Original: =

Original: [Equation]

Original: −

Original: [Equation]

Original: T

Original: [Equation]

Original: + I u k =

Original: [Equation]

Original: I −

Original: [Equation]

Original: T )

Original: u k −

Original: [Equation]

Original: This proves Lemma A.1.

Original: We will now use this lemma to prove Theorem 2.1.

Original: From this result, we can also write:

Original: [Equation]

Original: We restate the theorem from the main text.

Original: Note that the notations are simplified here from the theorem statement to improve readability.

Original: Theorem A.2.

Original: Let ˆ Y =

Original: X T w be a finite dimensional linear regression problem where w is learned using gradient descent.

Original: If we assume power law distribution in eigenspectrum of X , i.e. λ n = cn α ∀ n ∈ { 1 , 2 ...N } , then the time required by gradient descent to minimize the training error,

Original: T convergence =

Original: O ( N α )

Original: Proof.

Original: Using result from Lemma A.1, it is clear that the gradient converges to 0 if λ 1 < 1 η where λis the leading eigenvalue of XX T .

Original: Thus, η < 1 λ 1 , i.e. small learning rate setting.

Original: So, we set η = ˆ ηλ 1 where ˆ η < 1 .

Original: Plugging this in

Original: Eq.

Original: [Equation]

Original: Let X =

Original: U ∧ 12 V T denote the singular value decomposition (SVD), which implies XX T =

Original: U ∧ U T .

Original: Using the SVD, we get ( I − ˆ ηλ 1 XX T ) k

Original: = ( I − ˆ ηλ 1 U ∧ U T ) k .

Original: It is worth noting that eigenvalues and eigenvectors of ( I − ˆ ηλ 1 U ∧ U T ) are related to that of XX T as shown below:

Original: [Equation]

Original: [Using U T U

Original: = I ]

Original: [Equation]

Original: Using Eq.

Original: (11), we can write ( I − ˆ ηλ 1 U ∧ U T ) in the eigendecomposition form as U ˜ ∧

Original: U T where ˜ λ

Original: i = 1 − ˆ η λ i λ 1 .

Original: Thus, (cid:16) I − ˆ ηλ 1 U ∧ U T (cid:17) k

Original: = U ˜

Original: ∧ k U T .

Original: Plugging this in Eq.

Original: (10), we get:

Original: [Equation]

Original: (12) where S = U T Y ∈ R N .

Original: For the i th element, we get ∆ w ( i ) k

Original: = ˆ ηλ 1 (cid:80)

Original: j v i,j (cid:112) λ

Original: j ˜ λ jk S

Original: j = ˆ ηλ 1 (cid:80)

Original: j v i,j (cid:112) λ j

Original: (1 − ˆ η λ j λ 1 ) k S j .

Original: Since all other factors remain constant across training, i.e. do not change with k , the convergence of gradient descent depends on the factors (cid:16) 1 − ˆ η λ j λ 1 (cid:17) k .

Original: Note that we define gradient descent to converge when ∆ w ( i ) k ≈ 0 ∀ i .

Original: Therefore, the limiting factor that determines rate of convergence is (1 − ˆ η λ j λ 1 ) k , which in turn is limited by the smallest eigenvalue factor: λ N λ 1 .

Original: Assuming λ N λ 1 ≪ 1 = ⇒ ˆ η λ N λ 1 ≪ 1 as ˆ η < 1 = ⇒ (1 − ˆ η λ N λ 1 )

Original: k

Original: ≈ 1 − k ˆ η λ N λ 1 .

Original: Hence the convergence time, k ∗ =

Original: O (ˆ η λ 1 λ N ) =

Original: O ( λ 1 λ N )

Original: If λ i follows power law, i.e, λ i = ci − α and λ N λ 1 = N − α then k ∗

Original: =

Original: O ( N α ) i.e. k ∗ grows exponentially with α .

Original: Theorem A.3.

Original: Let ˆ y =

Original: f θ ( x ) T ψ be a linear regression problem as before.

Original: Let us further assume that f θ ( x ) ∀ x ∼ D train is a representative subset of the inputs from true data distribution: ( x, y ) ∼ D .

Original: Assuming a power-law distribution in the eigenspectrum of representations at f θ , i.e. λ n = cn α ∀ n ≥ n ∗ , where n ∗ ∈ { 1 , 2 ...N } , the generalization error after T weight update steps, G ( T ) is:

Original: [Equation]

Original: where

Original: [Equation]

Original: Here, m = min ( N, d ) is the rank of Σ N ( f θ ) .

Original: Proof.

Original: We start with the gradient of the regression loss function for a defined training set denoted by ( X s , Y s ) in a vectorized notation where X ∈ R N × d and Y ∈ R N .

Original: For the brevity of notation let us denote f θ ( x ) by x .

Original: The loss/error for the regression problem can be given by:

Original: [Equation]

Original: This gives gradient of weight vector as:

Original: [Equation]

Original: where η is the learning rate.

Original: Now, let the true labels be generated by Y s =

Original: X s ϕ ∗

Original: + ϵ where ϵ is a random variable denoting noise.

Original: Also, let X s =

Original: U s ∧ 12 V T which gives X Ts X

Original: s =

Original: V ∧ V T and X

Original: Ts Y s = V ∧ 12 U

Original: Ts Y =

Original: V ∧ 12 ˜ S where ˜ S

Original: = U

Original: Ts Y

Original: =

Original: U

Original: Ts ( U s ∧ 12 V T ϕ ∗ + ϵ ) =

Original: ∧ 12 Z ∗ + ˜ ϵ ,

Original: Z ∗ =

Original: V T ϕ ∗ and ˜ ϵ

Original: = U

Original: Ts ϵ .

Original: Let ϕ =

Original: V Z , i.e. expressing ϕ using the eigenbasis V of X T X and Z as the new parameters.

Original: This gives:

Original: [Equation]

Original: This implies no mixing between the eigenmodes.

Original: Solving Eq.

Original: (16) we get:

Original: [Equation]

Original: Now, let ϕ ∗ − ϕ

Original: =

Original: δ and Z ∗ − Z

Original: = ρ = V T δ , which gives ρ i ( t ) =

Original: ρ i (0) e − ηλ i t − ˜

Original: ϵ

Original: i √ λ i (1 − e − ηλ i t ) .

Original: Then the generalization error is:

Original: [Equation]

Original: (18) where X is the datapoints from the total dataset for which we care about the generalization error and m is the rank of the covariance matrix, X T X .

Original: Without loss of generality, we can assume m =

Original: min ( N, d ) .

Original: Note that ⟨ .

Original: ⟩ denotes the expectation over different network weight initialization and label noise.

Original: We define X s to be a reliable sample of dataset X if both X T X and X

Original: Ts X s have the same eigenvectors, i.e. V .

Original: Therefore, we could write X T X =

Original: X

Original: Ts X s in Eq.

Original: Note that the reliability of the sample indicates the inherent structure of the data, i.e. the variance along the principal components in the data space have been captured by X s .

Original: Subsequently, we get the expression for generalization error for a fixed training budget upto time/epoch (T):

Original: [Equation]

Original: where σ ρ indicates the noise due to weight initialization and σ ϵ indicates the noise in labels.

Original: It is clear from Eq.

Original: (19) that directions corresponding to larger eigenvalues converge faster and contribute to the generalization error through second term.

Original: Let us, therefore, assume that the top ˜ d eigenmodes have converged.

Original: Therefore, e − ηλ i T ≤ c 0 ∀ i ≤ ˜ d < m , where c 0 is some small number.

Original: Hence, e − ηλ i T ≤ c∀ i ≤ ˜ d

Original: and e − ηλ i T > c∀ i > ˜

Original: d

Original: = ⇒ e − 2 ηλ i T ≤ c∀ i ≤ ˜ d and (1 − e − ηλ i T ) 2 ≤ (1 − c 0 )∀ i > ˜ d

Original: Also (1 − e − ηλ i T ) 2 ≤ 1 ∀ i ≤ ˜ d and e − 2 ηλ i T ≤ 1 ∀ i > ˜ d (20)

Original: Plugging inequalities from Eq.

Original: (20) in Eq.

Original: (19), we can upper bound the generalization error:

Original: [Equation]

Original: Therefore

Original: [Equation]

Original: the generalization error can be upper bounded as follows:

Original: [Equation]

Original: Now, we can choose c 0 to be such that c 20 (cid:80) ˜ di

Original: [-=
-]
Revised: [-=
-]

Original: λ i = κ 0 where κ 0 is some constant independent of

Original: T .

Original: Furthermore, we impose the power-law assumption in the eigenvalues, i.e. λ i

Original: = ci − α .

Original: With these assumptions, we can simplify Eq.

Original: (22) as follows:

Original: [Equation]

Original: where κ is another constant (used for brevity), κ = σ 2 ρ κ

Original: [Equation]

Original: + σ 2 ϵ

Original: [Equation]

Original: 1 + (

Original: [Equation]

Original: − c 0 ) 2 (cid:1) .

Original: Also, we used the relation: var ( X ) =

Original: (cid:80) i λ i

Original: = (cid:80) i ci − α .

Original: Therefore, we can set ˆ d = ˜

Original: d − 1 and subsequently prove the statement of the theorem:

Original: [Equation]

Original: B Additional Experimental results

Original: In this section we provide additional results, evidence to support our hypothesis of α being a useful measure of downstream generalization performance.

Original: B.1 Generalization & Eigenspectrum Decay on STL10 and ImageNet

Original: B.3 Visualizing design landscape for BarlowTwins

Original: As a diagnostic metric for measuring representation quality we chart the learning landscape for self-supervised learning algorithms (Barlow Twins here).

Original: To this effect, we empirically investigate the role of different critical hyperparameters in learnability and generalization.

Original: In particular, we ablate across projection-dimensionality, redundancy coefficient, weight-decay and learning rate.

Original: B.5 Average complexity analysis of Algorithm

Original: Let us assume that the compute budget is characterized by M, i.e. the number of SSL models that can be trained parallely on the compute infrastructure.

Original: Let us assume H such parallel steps are run sequentially in order to train M × H different model configurations (e.g. sweeps over different hyperparameter configurations).

Original: Let us denote the probability of performing linear evaluation on a model trained in the r th sequential step as ϵ r .

Original: Therefore, expected number of linear evaluations in the r th step will be Mϵ r .

Original: Now, clearly ϵ r ≤ 1 , and hence:

Original: [Equation]

Original: This relationship provides the worst case complexity of Algorithm 2.

Original: We can further assume that the probability of linear evaluation decays with each sequential step.

Original: Therefore, the average number of linear evaluations would be less than MH .

Original: Assuming ϵ r = O (cid:0) 1 r (cid:1) † , we can write:

Original: [Equation]

Original: Taken together, the above equation shows that the average case complexity of model evaluation using α is less than the standard complexity of model evaluation, i.e. MH linear evaluations.

Original: [Figure]

Original: C On necessary and sufficiency conditions for predicting generalization

Original: Understanding the correlation between α and generalization requires probing the natural question of whether α in a given range is sufficient for good downstream performance.

Original: To further elaborate on this question, consider the following: Let us suppose that we have our ideal representation space that exhibit alpha within the Goldilocks zone.

Original: If we perform a set of permutation operations on individual datapoint representations, the structure of the representation space remains the same but the mapping from representation to label is now destroyed.

Original: In doing this set of permutation operations, we have now arrived at a different set of representations that would exhibit the same (or similar) alpha but demonstrate a lower task performance when measured using a linear readout layer.

Original: Extending the conclusions of this thought experiment to our observations, it is clear that we could have models with similar alpha values but distinctly different performance.

Original: But an alpha value that is not in the

Original: Goldilocks zone would be associated with inferior model performance.

Original: This property is the core of our claim and allows us to propose alpha as a measure for model selection in SSL pipelines.