Original:  Assessing representation qualityin self-supervised learning by measuring eigenspectrum decay
Revised: ↵ -ReQ : Assessing representation quality by measuring eigenspectrum decay

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Self-supervised learning (SSL) with large-scale unlabelled datasets enables thelearning of useful representations for multiple downstream tasks.
Revised: Self-Supervised Learning (SSL) with large-scale unlabelled datasets enables learning useful representations for multiple downstream tasks.

Original: However, assessing the quality of these representations is a challenging open problem.
Revised: However, assessing the quality of such representations efﬁciently poses nontrivial challenges.

Original: Directlyevaluating performance on specific tasks is either inefficient (if measuring performance on many different tasks) or insufficient (if measuring performance on onlya small number of tasks).
Revised: Existing approaches train linear probes (with frozen features) to evaluate performance on a given task. This is expensive both computationally , since it requires retraining a new prediction head for each downstream task, and statistically , requires task-speciﬁc labels for multiple tasks.

Original: This leaves us with a dilemma, how do we determineif representations learned with SSLwill generalize well across a wide range ofpotential downstream tasks?
Revised: This poses a natural question, how do we efﬁciently determine the "goodness" of representations learned with SSL across a wide range of potential downstream tasks?

Original: Therefore, some task-agnostic statistical measure ofrepresentation quality for estimating generalization without explicit downstreamtask assessment would be highly desirable.
Revised: In particular, a task-agnostic statistical measure of representation quality, that predicts generalization without explicit downstream task evaluation, would be highly desirable.

Original: Here, we analyze representationslearned by deep neural networks, pretrained with multiple SSL learning objectiveswith the goal of developing a task-agnostic metric of representation quality.
Revised: In this work, we analyze characteristics of learned representations f ✓ , in welltrained neural networks with canonical architectures & across SSL objectives.

Original: In-linewith recent work, we find that the eigenspectrum of the empirical representationcovariance matrix can be  approximated with a power-law distribution.
Revised: We observe that the eigenspectrum of the empirical feature covariance Cov( f ✓ ) can be well approximated with the family of power-law distribution.

Original: Furthermore,if a task is solvable with a linear readout from therepresentations, we find thatthe decay coefficient of the eigenspectrum ( α ) is agood measure of representationquality, i.e. there exists an interval for α where representations exhibit excellentdownstream task generalization. We provide analytical and empirical evidenceacross several datasets to support this claim.
Revised: We analytically and empirically (using multiple datasets, e.g. CIFAR, STL10, MIT67, ImageNet) demonstrate that the decay coefﬁcient ↵ serves as a measure of representation quality for tasks that are solvable with a linear readout, i.e. there exist well-deﬁned intervals for ↵ where models exhibit excellent downstream generalization. Furthermore, our experiments suggest that key design parameters in SSL algorithms, such as BarlowTwins [1], implicitly modulate the decay coefﬁcient of the eigenspectrum ( ↵ ).

Original: Notably, α depends only on therepresentations themselves, and as such, it offers an efficient, general mechanismfor assessing representation quality and guiding model selection in SSL.
Revised: Notably, ↵ depends only on the features themselves, and as such, using this measure for model selection in hyperparameter tuning for BarlowTwins allows searching for optimal hyperparameters with upto ⇠ 15 ⇥ less compute.

Original:  Introduction
Revised:  Introduction

Original: The recent success of self-supervised learning (SSL) has changed the landscape of deep learningsignificantly.
Revised: The recent success of self-supervised learning (SSL) has changed the landscape of deep learning signiﬁcantly.

Original: With well-engineered architectures and training objectives, SSL models learn usefulrepresentations from large datasets without relying on any labels [1, 2, 3].
Revised: With well-engineered architectures and training objectives, SSL models learn useful representations from large datasets without relying on any labels [1, 2, 3].

Original: Despite this progress,quantifying the representation quality for models trained with SSL is still an open problem.
Revised: Despite this progress, quantifying the representation quality for models trained with SSL is still an open problem.

Original: Themost obvious (and common) solution is to assess performance of models using these representationson downstream tasks.
Revised: The most obvious (and common) solution is to assess performance of models using these representations on downstream tasks.

Original: However, if the goal is general representations that can be used across manydomains, then this either requires a large investment of time and energy to be done well (in orderto assess performance on many tasks and datasets).
Revised: However, if the goal is general representations that can be used across many domains then this either requires a large investment of time and energy to be done well (in order to assess performance on many tasks and datasets).

Original: Alternatively, if models are assessed on only asmall number of datasets and tasks, then it is hard to be confident in the assessment.
Revised: Alternatively, if models are assessed on only a small number of datasets and tasks, then it is hard to be conﬁdent in the assessment.

Original: Thus, we are leftwith a question: Can we assess the quality of learned representations without explicitly evaluating theperformance on downstream tasks?
Revised: Thus, we are left with a question: Can we assess the quality of learned representations without explicitly evaluating the performance on downstream tasks?

Original: To answer this question, we must formally define the “quality” ofrepresentations and subsequently examine statistical estimators that measure this property withoutneeding downstream evaluation.
Revised: To answer this question we must formally deﬁne the “quality” of representations and subsequently examine statistical estimators that measure this property without needing downstream evaluation.

Original: Beyond theoretical interest, such a metric would be highly desirablefor model selection and also useful for designing new SSL algorithms.
Revised: Beyond theoretical interest such a metric would be highly desirable for model selection and also useful for designing new SSL algorithms.

Original: In search of such a metric, we turn our attention to one of the more efficient learning machines inexistence – the mammalian brain.
Revised: In search of such a metric we turn our attention to one of the more efﬁcient learning machines in existence – the mammalian brain.

Original: The hierarchical and distributed organization of neural circuits,especially in the cortex, provides neural representations that support a wide array of behaviours acrossmany domains.
Revised: The hierarchical and distributed organization of neural circuits, especially in the cortex, provides neural representations that support a wide array of behaviours across many domains.

Original: For example, representations in primary visual cortex (V1) of mammalian brains areused by animals to support downstream behaviours ranging from object categorization to movementdetection and motor control [4, 5].
Revised: For example, representations in primary visual cortex (V1) of mammalian brains are used by animals to support downstream behaviours ranging from object categorization to movement detection and motor control [4, 5].

Original: This berth of downstream uses of V1 representations is desirablefor artificial vision systems trained with SSL.
Revised: This berth of downstream uses of V1 representations is desirable for artiﬁcial vision systems trained with SSL.

Original: Thus, understanding the properties of representationsin V1 is a reasonable starting point for seeking a general metric of representation quality.
Revised: Thus, understanding the properties of representations in V1 is a reasonable starting point for seeking a general metric of representation quality.

Original: Recent breakthroughs in systems neuroscience enable large-scale recordings of neural activity.
Revised: Recent breakthroughs in systems neuroscience enable large-scale recordings of neural activity.

Original: Byrecording and analyzing the response to visual stimuli, [6, 7] find that activations in the mouse andmacaque monkey V1 exhibit a characteristic information geometric structure.
Revised: By recording and analyzing the response to visual stimuli, [6, 7] ﬁnd that activations in the mouse and macaque monkey V1 exhibit a characteristic information geometric structure.

Original: In particular, theserepresentations are high-dimensional, yet the amount of information encoded along the differentprincipal directions varies significantly.
Revised: In particular, these representations are high-dimensional, yet the amount of information encoded along the different principal directions varies signiﬁcantly.

Original: Notably, this variance (computed by measuring the eigenspectrum of the empirical covariance matrix) is well-approximated by a power-law distribution withdecay coefficient ≈ 1 , i.e., the n th eigenvalue of the covariance matrix scales as 1 /n .
Revised: Notably, this variance (computed by measuring the eigenspectrum of the empirical covariance matrix) is well-approximated by a power-law distribution with decay coefﬁcient ⇡ 1 , i.e., the n th eigenvalue of the covariance matrix scales as 1 /n .

Original: Motivated by these results, we explore the use of the decay rate of the empirical eigenspectrum tocharacterize representation quality in neural networks trained with SSL (Fig. 1a).
Revised: Motivated by these results, we explore the use of the decay rate of the empirical eigenspectrum to characterize representation quality in neural networks trained with SSL (Fig. 1a).

Original: Across diversemodel architectures, pretraining objectives, and downstream classification tasks, we empiricallyobserve an eigenspectrum decay in the representation covariance matrix that roughly follows apower-law distribution (see e.g. Fig. 1b).
Revised: Across diverse model architectures, pretraining objectives, and downstream classiﬁcation tasks, we empirically observe an eigenspectrum decay in the representation covariance matrix that roughly follows a power-law distribution (see e.g. Fig. 1b).

Original: We also find that the coefficient of this decay, denotedby α , is informative as to downstream generalization performance.
Revised: We also ﬁnd that the coefﬁcient of this decay, denoted by ↵ , is informative of downstream generalization performance.

Original: Importantly, α can efficientlybe calculated without any labels, and thereby, could be incorporated into existing SSL pipelines forefficient model selection on fixed compute budget.
Revised: Importantly, ↵ can efﬁciently be calculated without any labels, and thereby, could be incorporated into existing SSL pipelines for efﬁcient model selection on ﬁxed compute budget.

Original: Our core contributions in this paper are:
Revised: Our core contributions in this paper are:

Original: α as a potential metric We observe that canonical pretrained architectures have representa- tions which loosely conform to a power-law distribution in their covariance eigenspectrum.
Revised: ↵ as a potential metric We observe that canonical pretrained architectures have representations which loosely conform to a power-law distribution in their covariance eigenspectrum.

Original: Under an assumption of a power-law distribution we prove that the convergence rate and upper bound on generalization error are related to decay-coefficient ( α ) of the corresponding power-law distribution, with best values of α being neither too large nor too small.
Revised: Under an assumption of a power-law distribution, we prove that the convergence rate and upper bound on generalization error are related to decay-coefﬁcient ( ↵ ) of the corresponding power-law distribution, with best values of ↵ being neither too large nor too small.

Original: α as a label-free measure for representation quality : We empirically validate our theoret- ical results by demonstrating a relationship between α and downstream generalization.
Revised: ↵ as a label-free measure for representation quality : We empirically validate our theoretical results by demonstrating a relationship between ↵ and downstream generalization.

Original: In particular, we find that either too high  too low an α value implies poor generalization, both in-distribution and out-of-distribution.
Revised: In particular, we ﬁnd that either too high or too low an ↵ value implies poor generalization, both in-distribution and out-of-distribution.

Original: Generally, the best representations are those where α in a range that is close to 1, as observed in V1 of the real brain.
Revised: Generally, the best representations are those where ↵ is in a range that is close to 1, as observed in V1 of the real brain.

Original: Furthermore, these results hold irrespective of the choice of network architecture or pretraining objective.
Revised: Furthermore, these results hold irrespective of the choice of network architecture or pretraining objective.

Original: α for model selection in SSL : We establish α as a reliable metric for model selection in a specific case of SSL, Barlow Twins [3].
Revised: ↵ for model selection in SSL : We establish ↵ as a reliable metric for model selection in a speciﬁc case of SSL, Barlow Twins [1].

Original: Notably, we show that α allows us to identify model hyper-parameters that lead to representations that generalize well without any labels, more so than the actual loss function used to train the network.
Revised: Notably, we show that ↵ allows us to identify model hyper-parameters that lead to representations that generalize well without any labels, more so than the actual loss function used to train the network.

Original: Altogether, our results show that α is a promising task/architecture/data agnostic metric for assessingrepresentation quality in SSL.
Revised: Altogether, our results show that ↵ is a promising task/architecture/data agnostic metric for assessing representation quality in SSL.

Revised: We publicly release our results and code † .

Original:  Theoretical Framework to Assess Representations in SSL
Revised:  Theoretical Framework to Assess Representations in SSL

Original: We are interested in evaluating the quality of high dimensional representations learned by neuralnetworks.
Revised: We are interested in evaluating the quality of high dimensional representations learned by neural networks.

Original: Formally, we consider the overparameterized setting, with datasets X ⊂ R d , and learnedmappings f : R d → R D such that f ( x ) is a vector of D -dimensional features.
Revised: Formally, we consider the overparameterized setting, with datasets X ⇢ R d , and learned mappings f : R d ! R D such that f ( x ) is a vector of D -dimensional features.

Original: We consider DNNs as our function approximators, where each architecture implicitly defines afunction class F = { f θ : θ ∈ Θ } where Θ ⊂ R p is the feasible set of model parameters (e.g boundedΘ , [ − B, B ] p for some B ∈ R ) .
Revised: We consider DNNs as our function approximators, where each architecture implicitly deﬁnes a function class F = { f ✓ : ✓ 2 ⇥ } where ⇥ ⇢ R p is the feasible set of model parameters (e.g bounded ⇥ , [  B, B ] p for some B 2 R ) .

Original: The search for good representations poses the following optimizationproblem: with a dataset D pretrain from some data distribution P pretrain (potentially with labels),search for optimal parameters θ ∗ such that with pretrain objective L pretrain ( f , D )
Revised: The search for good representations poses the following optimization problem: with a dataset D pretrain from some data distribution P pretrain (potentially with labels), search for optimal parameters ✓ ⇤ such that with pretrain objective L pretrain ( f , D )

Original: [Equation]
Revised: [Equation]

Original: The above optimization problem is usually non-convex, and often uses gradient based optimizer tofind an approximate solution ˆ θ .
Revised: The above optimization problem is usually non-convex, and often uses gradient based optimizer to ﬁnd an approximate solution ˆ ✓ .

Original: Typically, to measure the quality of f ˆ θ , researchers evaluate thequality of representations on a downstream task D downstream with a linear readout using some metric
Revised: Typically, to measure the quality of f ˆ ✓ , researchers evaluate the quality of representations on a downstream task D downstream with a linear readout using some metric

Original: R ( linear ( f ˆ θ ), D downstream ) .
Revised: R ( linear ( f ˆ ✓ ), D downstream ) .

Original: A concrete example is pretraining a VGG-16 model ( F ) on ImageNetdataset ( D pretrain ) and evaluating the learned representations using classification accuracy ( R ) bylearning a linear classifier on the MIT-67 dataset ( D downstream ).
Revised: A concrete example is pretraining a VGG-16 model ( F ) on ImageNet dataset ( D pretrain ) and evaluating the learned representations using classiﬁcation accuracy ( R ) by learning a linear classiﬁer on the MIT67 dataset ( D downstream ).

Original: For the rest of the paper, we denotefeature maps as f θ ( x ) ∈ R D where f θ : X → R D , and the readout network as g ϕ : R D 
Revised: For the rest of the paper, we denote feature maps as f ✓ ( x ) 2 R D where f ✓ : X ! R D , and the readout network as g  : R D !

Original: → R k , wherek is the target dimensionality.
Revised: R k , where k is the target dimensionality.

Original: For simplicity of analysis, we consider linear readouts unless explicitlymentioned, i.e. g ϕ ( x ) = x T ϕ .
Revised: For simplicity of analysis, we consider linear readouts unless explicitly mentioned, i.e. g  ( x ) = x T  .

Original: 2.1 Covariance estimation and eigenspectrum
Revised: 2.1 Covariance estimation and eigenspectrum

Original: For a parameterized function f θ : X → R D  the ( n -sample) empirical covariance matrix is defined as:
Revised: For a parameterized function f ✓ : X ! R D (assume centered), the ( n -sample) empirical covariance matrix is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The eigenspectrum of Σ n ( f θ ) informs us about the variance explained by each principal component ofthe space spanned by representations f θ ( x ) .
Revised: The eigenspectrum of ⌃ n ( f ✓ ) informs us about the variance explained by each principal component of the space spanned by representations f ✓ ( x ) .

Original: Using the spectral decomposition theorem on symmetricmatrices, Σ = U Λ U T , where Λ is a diagonal matrix with nonnegative entries, and U is a matrixwhose columns are the eigenvectors of Σ .
Revised: Using the spectral decomposition theorem on symmetric matrices, ⌃ = U ⇤ U T , where ⇤ is a diagonal matrix with nonnegative entries, and U is a matrix whose columns are the eigenvectors of ⌃ .

Original: Without loss of generality we assume that λ 1 ≥ λ 2 ... ≥ λ m ,where m = min( n, D ) is the rank of Σ n ( f θ ) .
Revised: Without loss of generality we assume that  1   2 ...   m , where m = min( n, D ) is the rank of ⌃ n ( f ✓ ) .

Original: 2.2 Eigenspectrum Decay in Deep Representation Learning
Revised: 2.2 Eigenspectrum Decay in Deep Representation Learning

Original: Recent work in characterizing representation structure in canonical DNNs has demonstrated that thecovariance eigenspectrum roughly follows a power-law [9, 10, 11].
Revised: Recent work in characterizing representation structure in canonical DNNs has demonstrated that the covariance eigenspectrum roughly follows a power-law [9, 10, 11].

Original: Specifically, the eigenspectrumof a covariance matrix follows a power-law distribution PL ( α ) , or zeta distribution if for λ j ∈[ λ min , λ max ] , the eigenvalues λ j are all nonnegative, and [Equation] for some α > 0 .
Revised: Speciﬁcally, the eigenspectrum of a covariance matrix follows a power-law distribution PL ( ↵ ) , or zeta distribution if for  j[  min ,  max ] , the eigenvalues  j are all nonnegative, and  j / j  ↵ for some ↵ > 0 .

Original: Here, α is the slope of the power law, and is referred to as the coefficient of decay ofthe eigenspectrum.
Revised: Here, ↵ is the slope of the power law, and is referred to as the coefﬁcient of decay of the eigenspectrum.

Original: Intuitively, small α (typically α ≤ 1 ) suggests a dense encoding, while a high α(rapid decay) corresponds to a sparse encoding.
Revised: Intuitively, small ↵ (typically ↵  1 ) suggests a dense encoding, while a high ↵ (rapid decay) corresponds to a sparse encoding.

Original: Insights from High-Dimensional Linear Regression : Recent work in theoretical machine learninghas connected bounds on generalization error for a linear regression problem to the eigenspectrum ofthe feature covariance matrix.
Revised: Insights from High-Dimensional Linear Regression : Recent work in theoretical machine learning has connected bounds on generalization error for a linear regression problem to the eigenspectrum of the feature covariance matrix.

Original: Specifically, in the infinite-dimensional setting with D → ∞ , [12]studied the linear regression setting with Gaussian features, and proved that if the eigenspectrumfollows a power-law distribution (up to polylogarithm factors), the min-norm solution providesgood generalization performance iff α = 1 .
Revised: Speciﬁcally, in the inﬁnite-dimensional setting with D ! 1 , [12] studied the linear regression setting with Gaussian features, and proved that if the eigenspectrum follows a power-law distribution (up to polylogarithm factors), the min-norm solution provides good generalization performance iff ↵ = 1 .

Original: The asymptotic regime of infinite width is an excellentframework to study theoretical properties of DNN representations.
Revised: The asymptotic regime of inﬁnite width is an excellent framework to study theoretical properties of DNN representations.

Original: However, despite having a large number of parameters, practical DNNs always possess finite dimensional representations, making itimportant to investigate the implications of such results in finite width models.
Revised: However, despite having a large number of parameters, practical DNNs always possess ﬁnite dimensional representations, making it important to investigate the implications of such results in ﬁnite width models.

Original: In particular, for thefinite dimensional setting we try to answer: Does α sufficiently larger or smaller than 1, still allowefficient learning and strong generalizability?
Revised: In particular, for the ﬁnite dimensional setting we try to answer: Does ↵ sufﬁciently larger or smaller than 1, still allow efﬁcient learning and strong generalizability?

Original: To answer this question, we narrow our focus to gradient-based optimization techniques usually usedto train DNNs.
Revised: To answer this question, we narrow our focus to gradient-based optimization techniques usually used to train DNNs.

Original: In particular, from the optimization perspective, Advani et al. showed that for deeplinear regression in high dimensions, the time required for training and the steady-state generalizationerror are both O ( 1 λ min ) [13].
Revised: In particular, from the optimization perspective, Advani et al. showed that for deep linear regression in high dimensions, the time required for training and the steady-state generalization error are both O ( 1  min ) [13].

Original: A key difference from our work is that they assume the inputs are drawnfrom an isotropic Gaussian distribution.
Revised: A key difference from our work is that they assume the inputs are drawn from an isotropic Gaussian distribution.

Original: Instead, we investigate the generalization of linear regressionon f θ ( x ) , which has a power law structure in its covariates.
Revised: Instead, we investigate the generalization of linear regression on f ✓ ( x ) , which has a power law structure in its covariates.

Original: We show that the training convergencetime grows exponentially with α using the following theorem:
Revised: We show that the training convergence time grows exponentially with ↵ using the following theorem:

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Let ˆ y =
Revised: Let ˆ y =

Original: f θ ( x ) Tψ be an overparameterized linear regression problem where ψ islearned using gradient descent in order to optimize the training error, E x,y
Revised: f ✓ ( x ) T be an overparameterized linear regression problem where is learned using gradient descent in order to optimize the training error, E x,y

Original: [( y − f θ ( x ) Tψ ) 2 ] , where( x, y ) ∼ D train .
Revised: [( y  f ✓ ( x ) T ) 2 ] , where ( x, y ) ⇠ D train .

Original: If we assume a power-law distribution in the eigenspectrum of representations atf θ , i.e. λ n = cn α ∀ n ≥ n ∗ , where n ∗ ∈ { 1 , 2 ...N } , then the time required by gradient descent tominimize the training error, T convergence =
Revised: If we assume a power-law distribution in the eigenspectrum of representations at f ✓ , i.e.  n = cn ↵ 8 n  n ⇤ , where n ⇤ 2 { 1 , 2 ...N } , then the time required by gradient descent to minimize the training error, T convergence =

Original: O ( N α ) where N is number of training samples.
Revised: O ( N ↵ ) where N is number of training samples.

Original: The outline of the proof builds on key results relating to gradient descent dynamics from [14].
Revised: The outline of the proof builds on key results relating to gradient descent dynamics from [14].

Original: Weshow that gradient descent updates, whenψ is initialized to 0, yield a recursive relation forψ ( k ) , i.e.ψ after k update steps.
Revised: We show that gradient descent updates, when is initialized to 0, yield a recursive relation for ( k ) , i.e. after k update steps.

Original: Plugging this relation in the gradient formulation, we show that the update steplength along the n th principal direction of f θ ( x ) shrinks exponentially with a decay rate proportionalto λ n .
Revised: Plugging this relation in the gradient formulation, we show that the update step length along the n th principal direction of f ✓ ( x ) shrinks exponentially with a decay rate proportional to  n .

Original: Therefore, the time to convergence in training is controlled by the smallest eigenvalue which,by design, follows the power law.
Revised: Therefore, the time to convergence in training is controlled by the smallest eigenvalue which, by design, follows the power law.

Original: In sum, Theorem 2.1 provides an explanation against arbitrarilylarge values of α .
Revised: In sum, Theorem 2.1 provides an explanation against arbitrarily large values of ↵ .

Original: Next we show that α can’t be arbitrarily small as the upper bound on generalization error is higherfor smaller values of α .
Revised: Next we show that ↵ can’t be arbitrarily small as the upper bound on generalization error is higher for smaller values of ↵ .

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: Let ˆ y =
Revised: Let ˆ y =

Original: f θ ( x ) Tψ be a linear regression problem as before.
Revised: f ✓ ( x ) T be a linear regression problem as before.

Original: Let us further assume thatf θ ( x ) ∀ x ∼ D train is a representative subset of the inputs from true data distribution: ( x, y ) ∼ D .
Revised: Let us further assume that f ✓ ( x ) 8 x ⇠ D train is a representative subset of the inputs from true data distribution: ( x, y ) ⇠ D .

Original: Assuming a power-law distribution in the eigenspectrum of representations at f θ , i.e. λ n = cn α ∀ n ≥n ∗ , where n ∗ ∈ { 1 , 2 ...N } , the generalization error after T weight update steps, G ( T ) is:
Revised: Assuming a power-law distribution in the eigenspectrum of representations at f ✓ , i.e.  n = cn ↵ 8 n  n ⇤ , where n ⇤ 2 { 1 , 2 ...N } , the generalization error after T weight update steps, G ( T ) is:

Original: [Equation]
Revised: [Equation]

Original:  where
Revised:  where

Original: [Equation]
Revised: [Equation]

Original: Here, m = min ( n, D ) is the rank of Σ n ( f θ ) .
Revised: Here, m = min ( n, D ) is the rank of ⌃ n ( f ✓ ) .

Original: Notably, r ˆ d (Σ( f θ )) can be thought of as a measure of effective rank of the covariance matrix andgrows with decreasing values of α .
Revised: Notably, r ˆ d ( ⌃ ( f ✓ )) can be thought of as a measure of effective rank of the covariance matrix and grows with decreasing values of ↵ .

Original: Therefore, the upper bound for generalization error is higherfor lower α .
Revised: Therefore, the upper bound for generalization error is higher for lower ↵ .

Original: Taken together, Theorems 2.1 and 2.2 suggest that α can neither be too high nor toolow and there exists a tradeoff region which results in efficient learning and strong generalizability,where these representations form a good basis for gradient-based optimization on downstream taskperformance.
Revised: Taken together, Theorems 2.1 and 2.2 suggest that ↵ can neither be too high nor too low and there exists a tradeoff region which results in efﬁcient learning and strong generalizability, where these representations form a good basis for gradient-based optimization on downstream task performance.

Original: Another motivating example Before exploring the link between α and generalization performanceof deep networks, we validate our theorems in a simple linear regression setting.
Revised: Another motivating example Before exploring the link between ↵ and generalization performance of deep networks, we validate our theorems in a simple linear regression setting.

Original: We first consider its relationship to the finite dimensional regression setting as in [12].
Revised: We ﬁrst consider its relationship to the ﬁnite dimensional regression setting as in [12].

Original: Specifically,
Revised: Speciﬁcally,

Original: Theorem 6 of their paper states that the conditions for “benign overfitting”, wherein a model canperfectly fit noisy training data without any subsequent loss of performance on testing data, may belooser in finite dimensions as opposed to the necessary and sufficient condition of α = 1 in infinitedimensions.
Revised: Theorem 6 of their paper states that the conditions for “benign overﬁtting”, wherein a model can perfectly ﬁt noisy training data without any subsequent loss of performance on testing data, may be looser in ﬁnite dimensions as opposed to the necessary and sufﬁcient condition of ↵ = 1 in inﬁnite dimensions.

Original: To empirically test this in the finite high dimensional setting, we examine linear leastsquares regression using different covariate structures for the input data.
Revised: To empirically test this in the ﬁnite high dimensional setting, we examine linear least squares regression using different covariate structures for the input data.

Original: Formally, we considercovariates { x i } N i =1 , such that xi ∈ R
Revised: Formally, we consider covariates { x i } Ni =1 , such that x

Original:  d is sampled from a Gaussian distribution with covariancestructure Σ = diag { λ 1 , ...
Revised: i 2 R d is sampled from a Gaussian distribution with covariance structure ⌃ = diag {  1 , ...

Original: λ d } where λ j ∼ PL ( α ) , i.e λ j ∝ cj − α .
Revised:  d } where  j ⇠ PL ( ↵ ) , i.e  j / cj  ↵ .

Original: We assume access to thecorresponding labels { y i } Ni =1 generated under a teacher function θ ∗ , such that y i
Revised: We assume access to the corresponding labels { y i } Ni =1 generated under a teacher function ✓ ⇤ , such that y i

Original: =
Revised: =

Original: x
Revised: x

Original: Ti θ ∗ + ϵ i .
Revised: Ti ✓ ⇤ + ✏ i .

Original: Wefind that in this scenario there is a clear relationship between the proximity of α to 1 and the presenceof benign overfitting.
Revised: We ﬁnd that in this scenario there is a clear relationship between the proximity of ↵ to 1 and the presence of benign overﬁtting.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, when α is close to 1, the training loss is low, but thevalidation loss is also low.
Revised: 2, when ↵ is close to 1, the training loss is low, but the validation loss is also low.

Original: Thus, when α is close to 1, the generalization properties are at their best.
Revised: Thus, when ↵ is close to 1, the generalization properties are at their best.

Original: This example thus provides another hint that α is a potential measure for how well a model will beable to generalize.
Revised: This example thus provides another hint that ↵ is a potential measure for how well a model will be able to generalize.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: Our theoretical results suggest the following: for representations with power-law characteristics, thedecay-coefficient ( α ) effects the adaptivity , where the convergence rate of linear regression withgradient descent scales exponentially O ( n α ) with α .
Revised: Our theoretical results suggest the following: for representations with power-law characteristics, the decay-coefﬁcient ( ↵ ) effects the adaptivity , where the convergence rate of linear regression with gradient descent scales exponentially O ( n ↵ ) with ↵ .

Original: On the other hand, if trained sufficiently long,the upper bound for generalization error improves with larger α .
Revised: On the other hand, if trained sufﬁciently long, the upper bound for generalization error improves with larger ↵ .

Original: Together, these items imply thatif one wants to use gradient descent to train a downstream task then α for the representations usedshould be neither too large nor too small, i.e. high quality representations are those with α in a“Goldilocks” zone.
Revised: Together, these items imply that if one wants to use gradient descent to train a downstream task then ↵ for the representations used should be neither too large nor too small, i.e. high quality representations are those with ↵ in a “Goldilocks” zone.

Original: Building on these insights, we now design experiments to empirically study thepotential for α to be used as a representation quality metric.
Revised: Building on these insights, we now design experiments to empirically study the potential for ↵ to be used as a representation quality metric.

Original: We do so by measuring the relationshipbetween α and the standard metric of representation quality, i.e. downstream task generalizationperformance.
Revised: We do so by measuring the relationship between ↵ and the standard metric of representation quality, i.e. downstream task generalization performance.

Original: In this work, we restrict our scope to vision tasks, specifically object classification andscene recognition.
Revised: In this work, we restrict our scope to vision tasks, speciﬁcally object classiﬁcation and scene recognition.

Original: In particular, our experiments:
Revised: In particular, our experiments:

Original: • measure the eigenspectrum decay & generalization : We systematically evaluate in- distribution and out-of-distribution generalization, and simultaneously, we also measure α , across diverse choices for network architectures, pretraining learning objectives, and across multiple datasets.
Revised: • measure the eigenspectrum decay & generalization : We systematically evaluate indistribution and out-of-distribution generalization, and simultaneously, we also measure ↵ , across diverse choices for network architectures, pretraining learning objectives, and across multiple datasets.

Original: We then look at whether α values near some neighbourhood around 1 are indeed predictive of representations that generalize well to downstream tasks.
Revised: We then look at whether ↵ values near some neighbourhood around 1 are indeed predictive of representations that generalize well to downstream tasks.

Original: • explore α as a metric for model-selection in SSL:
Revised: • explore ↵ as a metric for model-selection in SSL:

Original: Measuring α is label-free and computa- tionally inexpensive, as it requires a single forward-pass on the downstream dataset.
Revised: Measuring ↵ is label-free and computationally inexpensive, as it requires a single forward-pass on the downstream dataset.

Original: We thus ask, is α predictive of generalization capabilities for current SSL algorithms, and if so, can recognition performance .
Revised: We thus ask, is ↵ predictive of generalization capabilities for current SSL algorithms, and if so, can it be used for model selection?

Original: α is strongly correlated to object recognition performance on STL10 across different architectures and pretraining loss functions.

Original: it be used for model selection?

Original: We run extensive ablations on design of the BarlowTwins[3] algorithm in an effort to answer this question.
Revised: We run extensive ablations on design of the BarlowTwins [1] algorithm in an effort to answer this question.

Original: 3.1 Measuring eigenspectrum Decay & Generalization
Revised: 3.1 Measuring eigenspectrum Decay & Generalization

Original: Evaluation Protocol: To measure α for the learned representation f
Revised: Evaluation Protocol: To measure ↵ for the learned representation f

Original: ( x i ) , we first extract featuresfrom intermediate layers of a DNN pretrained on D pretrain and estimate the corresponding covariancematrix Σ n ( f ) .
Revised: ( x i ) , we ﬁrst extract features from intermediate layers of a DNN pretrained on D pretrain and estimate the corresponding covariance matrix ⌃ n ( f ) .

Original: Next, we compute the full set of numerical eigenvalues, and estimate α by a fitting alinear model on the eigenspectrum in log-log scale.
Revised: Next, we compute the full set of numerical eigenvalues, and estimate ↵ by a ﬁtting a linear model on the eigenspectrum in log-log scale.

Original: Our pretrained models are taken from PyTorch
Revised: Our pretrained models are taken from PyTorch

Original: Hub [15] and timm [16].
Revised: Hub [15] and timm [16].

Original: Given that we observed no significant difference between the observed αvalues in the train and test sets, we refer to this empirical estimate as the α for the dataset.
Revised: Given that we observed no signiﬁcant difference between the observed ↵ values in the train and test sets, we refer to this empirical estimate as the ↵ for the dataset.

Original: To estimate the capacity of intermediate representations in solving the downstream task, we traina linear readout layer, g ( . ) , from representations to target logits.
Revised: To estimate the capacity of intermediate representations in solving the downstream task, we train a linear readout layer, g ( . ) , from representations to target logits.

Original: Intuitively, this comes down toestablishing a relationship between the manifold geometry and linear separability of representations[17].
Revised: Intuitively, this comes down to establishing a relationship between the manifold geometry and linear separability of representations [17].

Original: Thereafter, we observe the correlation between estimated α and the linear readout performance.
Revised: Thereafter, we observe the correlation between estimated ↵ and the linear readout performance.

Original: Notably, we also tried non-linear g ( . ) and observed a similar trend in results (see Appendix B.1).
Revised: Notably, we also tried non-linear g ( . ) and observed a similar trend in results (see Appendix B.1).

Original: Inductive bias of Model Architecture In this section, we investigate the relationship between αof the representation covariance matrix and object/scene recognition performance in DNNs withdifferent backbones and the role of depth.
Revised: Inductive bias of Model Architecture In this section, we investigate the relationship between ↵ of the representation covariance matrix and object/scene recognition performance in DNNs with different backbones and the role of depth.

Original: In order to do so, we examine varying depth configurationswithin network architectures across three generations of models on the STL-10 and MIT67 dataset[18].
Revised: In order to do so, we examine varying depth conﬁgurations within network architectures across three generations of models on the STL10 and MIT67 dataset [18].

Original: We choose both object and scene recognition task as they fundamentally differ in the natureof the features that must be extracted to perform well.
Revised: We choose both object and scene recognition task as they fundamentally differ in the nature of the features that must be extracted to perform well.

Original: For scene recognition, the model focuses onglobal features in the entire image, while for object recognition, the model focuses on local featuresof the image containing the object[19].
Revised: For scene recognition, the model focuses on global features in the entire image, while for object recognition, the model focuses on local features of the image containing the object[19].

Original: We examine deep Convolutional Neural Networks (CNNs) without any residual connection as ourfirst family of models.
Revised: We examine deep Convolutional Neural Networks (CNNs) without any residual connection as our ﬁrst family of models.

Original: Specifically, we choose three different configurations of VGG-Net [20],namely VGG-13, VGG-16 and VGG-19.
Revised: Speciﬁcally, we choose three different conﬁgurations of VGG-Net [20], namely VGG-13, VGG-16 and VGG-19.

Original: We inspect representations that are input to the dropoutand MaxPool layers during the forward pass of the network.
Revised: We inspect representations that are input to the dropout and MaxPool layers during the forward pass of the network.

Original: Second, we consider Deep Residual
Revised: Second, we consider Deep Residual

Original: Networks [21] which are widely used in computer vision.
Revised: Networks [21] which are widely used in computer vision.

Original: We inspect the representations that areinput to each of the residual blocks as well as the Adaptive average pool in ResNet-13, ResNet-50and ResNet-101 during their respective forward passes.
Revised: We inspect the representations that are input to each of the residual blocks as well as the Adaptive average pool in ResNet-13, ResNetand ResNet-101 during their respective forward passes.

Original: Finally, owing to the recent success oftransformers in object recognition tasks we consider Vision Transformers (ViT)
Revised: Finally, owing to the recent success of transformers in object recognition tasks we consider Vision Transformers (ViT)

Original: [22] as the thirdfamily of models, namely ViT-Base/8 , ViT-Large/16 and ViT-Huge/14.
Revised: [22] as the third family of models, namely ViT-Base/8 , ViT-Large/16 and ViT-Huge/14.

Original: Unlike VGG and ResNet, weonly look at features in the intermediate layers because it summarizes the entire input image and isused in practice for class prediction.
Revised: Unlike VGG and ResNet, we only look at features in the intermediate layers because it summarizes the entire input image and is used in practice for class prediction.

Original: For all model architectures, we use the weights obtained frompretraining on ImageNet.
Revised: For all model architectures, we use the weights obtained from pretraining on ImageNet.

Original: Fig.
Revised: Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: 4 illustrate the relation between performance and α for all the nine architecturesacross intermediate layer representations, as described above.
Revised: 4 illustrate the relation between performance and ↵ for all the nine architectures across intermediate layer representations, as described above.

Original: We found that, while most intermediaterepresentations in CNNs (with or without residual connections) exhibit α < 1 , representations in
Revised: We found that, while most intermediate representations in CNNs (with or without residual connections) exhibit ↵ < 1 , representations in

Original: ViTs mostly exhibit α > 1 .
Revised: ViTs mostly exhibit ↵ > 1 .

Original: Nevertheless, representations extracted from the deepest layers ofall the models exhibit α close to 1 , irrespective of the total depth of each model (see Appendix
Revised: Nevertheless, representations extracted from the deepest layers of all the models exhibit ↵ close to 1 , irrespective of the total depth of each model (see Appendix

Original: B).
Revised: B).

Original: Furthermore, the performance on downstream task increases with depth.
Revised: Furthermore, the performance on downstream task increases with depth.

Original: This is unsurprisingbecause all networks were trained to perform object recognition on ImageNet [23] and thereby wouldhave leveraged hierarchical processing to learn features that are tuned towards object recognition.
Revised: This is unsurprising because all networks were trained to perform object recognition on ImageNet [23] and thereby would have leveraged hierarchical processing to learn features that are tuned towards object recognition.

Original: Surprisingly enough, we observe a strong significant correlation between α and performance onthe STL-10 dataset, i.e. a different data distribution than the training dataset, across layers andmodel architectures ( ρ = − 0 .
Revised: Surprisingly enough, we observe a strong signiﬁcant correlation between ↵ and performance on the STL10 dataset, i.e. a different data distribution than the training dataset, across layers and model architectures ( ⇢ =  0 .

Original: 922 , ∗ p < 0 .
Revised: 922 , ⇤ p < 0 .

Original: for representations exhibiting α > 1 and ρ = − 0 .
Revised: for representations exhibiting ↵ > 1 and ⇢ =  0 .

Original: 922 , 233 ∗ p < 0 .
Revised: 922 , ⇤ p < 0 .

Original: for representations exhibiting α < 1 ).
Revised: for representations exhibiting ↵ < 1 ).

Original: It is worth noting here that the correlation wasweaker for the earliest layers of each model.
Revised: It is worth noting here that the correlation was weaker for the earliest layers of each model.

Original: We believe that early layers learn more task invariantfeatures that reflect the statistics of natural images [24, 25] and therefore lack task relevant informationin their representations.
Revised: We believe that early layers learn more task invariant features that reﬂect the statistics of natural images [24, 25] and therefore lack task relevant information in their representations.

Original: Taken together, this observation confirms our hypothesis that α is a goodindicator of out-of-distribution generalization performance when representations possess task relevantinformation.
Revised: Taken together, this observation conﬁrms our hypothesis that ↵ is a good indicator of out-of-distribution generalization performance when representations possess task relevant information.

Revised: Thus, ↵ has a necessary but not sufﬁcient relationship with generalization (Appendix C).

Original: Learning objective In this section, we first aim to understand how the α value changes acrossthe layers of a fixed architecture DNN when trained with different learning objectives.
Revised: Learning objective In this section, we ﬁrst aim to understand how the ↵ value changes across the layers of a ﬁxed architecture DNN when trained with different learning objectives.

Original: We take a
Revised: We take a

Original: ResNet-50 model [21] pre-trained using three different SSL algorithms, namely SimCLR [1], BYOL[26] and Barlow Twins [3], and the supervised learning loss objectives on ImageNet-1k[23] dataset.
Revised: ResNet-50 model [21] pre-trained using three different SSL algorithms, namely SimCLR [2], BYOL [26] and Barlow Twins [1], and the supervised learning loss objectives on ImageNet-1k[23] dataset.

Original: We use a similar procedure as before to extract representations from the network and estimate α .
Revised: We use a similar procedure as before to extract representations from the network and estimate ↵ .

Original: Similar to results in the previous section, all networks irrespective of the pretraining loss function,exhibit α closer to 1 in the deeper layers in contrast to intermediate layers (see Appendix Fig. 9).
Revised: Similar to results in the previous section, all networks irrespective of the pretraining loss function, exhibit ↵ closer to 1 in the deeper layers in contrast to intermediate layers (see Appendix Fig. 9).

Original: Thissurprising result indicates that although the pre-training loss function was different, representationsextracted from deepest layers are reflective of the object/scene semantics in natural images.
Revised: This surprising result indicates that although the pre-training loss function was different, representations extracted from deepest layers are reﬂective of the object/scene semantics in natural images.

Original: Furthermore, Fig. 3 and Fig.
Revised: Furthermore, Fig. 3 and Fig.

Original: 4 illustrate the strong correlation between α and generalization performance on STL-10 and MIT67 across all pre-training loss functions.
Revised: 4 illustrate the strong correlation between ↵ and generalization performance on STL10 and MIT67 across all pre-training loss functions.

Original: Together with results from the previoussection, we validate our hypothesis that the representations that demonstrate good out-of-distributiongeneralization performance are characterized by α in the neighbourhood of 1.
Revised: Together with results from the previous section, we validate our hypothesis that the representations that demonstrate good out-of-distribution generalization performance are characterized by ↵ in the neighbourhood of 1.

Original: This suggests that highquality representations are indeed those with α in this region.
Revised: This suggests that high quality representations are indeed those with ↵ in this region.

Original: 3.2 Label-agnostic metric for Model selection
Revised: 3.2 Label-agnostic metric for Model selection

Original: With empirical evidence of α being a good measure for generalization, we now wish to study itssuitability as a metric for identifying the best among models pretrained with SSL algorithms.
Revised: With empirical evidence of ↵ being a good measure for generalization, we now wish to study its suitability as a metric for identifying the best among models pretrained with SSL algorithms.

Original: A labelfree metric like α could be beneficial when we don’t have access to the downstream task annotations,and the SSL loss is not useful to distinguish models with good generalization performance (see Fig. 5).
Revised: A labelfree metric like ↵ could be beneﬁcial when we don’t have access to the downstream task annotations, and the SSL loss is not useful to distinguish models with good generalization performance (see Fig. 5).

Original: To investigate this, we exhaustively ablate the relationship between alpha and model performanceacross a wide range of hyper-parameters for a representative non-contrastive SSL algorithm.
Revised: To investigate this, we exhaustively ablate the relationship between alpha and model performance across a wide range of hyper-parameters for a representative non-contrastive SSL algorithm.

Original: Current SSL algorithms struggle with dimension collapse (characterized by large α ), due to pathologies in training dynamics.
Revised: Current SSL algorithms struggle with dimension collapse (characterized by large ↵ ), due to pathologies in training dynamics.

Original: To study α across a wide-range of values, for our model selectionexperiments we pick the Barlow Twins [3], a non-contrastive SSL algorithm that explicitly optimizesto avoid dimension collapse.
Revised: To study ↵ across a wide-range of values, for our model selection experiments we pick the Barlow Twins [1], a non-contrastive SSL algorithm that explicitly optimizes to avoid dimension collapse.

Original: In particular, the Barlow Twins learning objective ( L BT ) proposesimposing a soft-whitening constraint :
Revised: In particular, the Barlow Twins learning objective ( L BT ) proposes imposing a soft-whitening constraint :

Original: [Equation]
Revised: [Equation]

Original: Notably with sufficient large λ , the model would impose an α = 0 constraint on the representations.
Revised: Notably with sufﬁcient large  , the model would impose an ↵ = 0 constraint on the representations [27].

Original: Despite the intuitive connection between α and λ , it is unclear whether this relationship holdsacross a wide-range of values for model hyperparameters.
Revised: Despite the intuitive connection between ↵ and  , it is unclear whether this relationship holds across a wide-range of values for model hyperparameters.

Original: To empirically establish this connection,we vary λ (redundancy coeffiecient), projection head dimensionality and learning rate, and train a
Revised: To empirically establish this connection, we vary  (redundancy coefﬁecient), projection head dimensionality and learning rate, and train a

Original: ResNet50 encoder using Barlow Twins learning objective.
Revised: ResNet50 encoder using Barlow Twins learning objective.

Original: We provide the results for CIFAR10[27]and STL10 [28]  in Fig.
Revised: We provide the results for CIFAR10 [28] and STL10 [29] in Fig.

Original: 5 demonstrates that α is a strong indicator of in-distribution generalizationperformance across a large range of hyperparameter ranges of L BT ,specifically the redundancycoefficient and projector dimensionality.
Revised: 5 demonstrating that ↵ is a strong indicator of in-distribution generalization performance across a large range of hyperparameter ranges of L BT .

Original: We defer the reader to the Appendix  for similar trendsin optimization hyperparameters learning rate and weight decay parameters.
Revised: We defer the reader to the Appendix Figs. 14 & 15 for similar trends in optimization hyperparameters, i.e. learning rate and weight decay parameters.

Original: Furthermore, α ispredictive of out-of-distribution generalization performance in these settings (see Appendix B.3).
Revised: Furthermore, ↵ is predictive of out-of-distribution generalization performance in these settings (see Appendix B.3).

Original: [Table]
Revised: [Table]

Original: Model Selection on a Computer
Revised: Model Selection on a Computer

Original: Budget:
Revised: Budget:

Original: With a constrained computebudget, using α for model selection issignificantly cheaper than evaluatingthe representations on a suite of downstream tasks.
Revised: With a constrained compute budget, using ↵ for model selection is signiﬁcantly cheaper than evaluating the representations on a suite of downstream tasks.

Original: To illustrate this, consider the standard alternative of training a linear classifier on the representations and evaluating test accuracy.
Revised: To illustrate this, consider the standard alternative of training a linear classiﬁer on the representations and evaluating test accuracy.

Original: Compared to training a linear probe,which requires multiple ( K ) epochs offorward & backward passes throughthe training dataset’s features, to achieve reasonable estimates of downstream accuracy, computingα requires a single PCA step on the validation dataset’s features.
Revised: Compared to training a linear probe, which requires multiple ( K ) epochs of forward & backward passes through the training dataset’s features, to achieve reasonable estimates of downstream accuracy, computing ↵ requires a single PCA step on the validation dataset’s features.

Original: In Table 1, we provide estimatedtimes for model-selection with α and downstream accuracy for CIFAR10, STL10 and ImageNet.
Revised: In Table 1, we provide estimated times for model-selection with ↵ and downstream accuracy for CIFAR10, STL10 and ImageNet.

Original: Adetailed algorithm is provided in Appendix B.4.
Revised: A detailed algorithm is provided in Algorithm 1. [Table] [Table]

Original: Related Work
Revised: trained_ [Table] = train [Table]

Revised: SSL

Revised: [Table]

Revised: (num

Revised: [Table]

Revised: =M) # evaluate ↵ for the trained models alpha

Revised: [Table]

Revised: trained

Revised: [Table]

Revised: = evaluate_alpha(trained_models) best_trained_

Revised: [Table]

Revised: = {

Revised: [Table]

Revised: : alpha for model , alpha in alpha_trained_models if alpha 2 [

Revised: [Table]

Revised: min , ↵ max ]}

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: _linear_eval

Revised: [Table]

Revised: ) for

Revised: [Table]

Revised: , alpha in best_trained_models.items()} #

Revised: [Table]

Revised: and

Revised: [Table]

Revised: max

Revised: [Table]

Revised: ({ ↵

Revised: [Table]

Revised: | ↵

Revised: [Table]

Revised: >

Revised: [Table]

Revised: j &

Revised: [Table]

Revised: j

Revised: [Table]

Revised: ,j

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: |

Revised: [Table]

Revised: m <

Revised: [Table]

Revised: j & acc m

Revised: [Table]

Revised: j

Revised: [Table]

Revised: ,j

Revised: [Table]

Revised: _stat})

Revised: # Trim

Revised: [Table]

Revised: only

Revised: [Table]

Revised: [Table]

Revised: statistics ,

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: () if

Revised: [Table]

Revised: threshold} #

Revised: [Table]

Revised: ‘selected’

Revised: [Table]

Original:  Evaluating representations and model quality We note a substantial body of work aiming toempirically characterize structure of emergent representations in DNN, without requiring labels[29, 30].
Revised: [Table] }) return best_model_acc  Related Work Evaluating representations and model quality We note a substantial body of work aiming to empirically characterize structure of emergent representations in DNN, without requiring labels [30, 31].

Original: One such index that quantifies the similarity of representations across layers (of same ordifferent models) is Centered Kernel Alignment (CKA) [24].
Revised: One such index that quantiﬁes the similarity of representations across layers (of same or different models) is Centered Kernel Alignment (CKA) [24].

Original: While CKA does not provide explicitguidance for downstream performance, [31] show that in-distribution generalization gap can bepredicted using a different index based on the model’s parameters.
Revised: While CKA does not provide explicit guidance for downstream performance, [32] show that in-distribution generalization gap can be predicted using a different index based on the model’s parameters.

Original: In particular, they show that
Revised: In particular, they show that

Original: Empirical Spectral Density (ESD) of weight matrices for many DNNs obey a power-law, with thecoefficient of decay being predictive of in-distribution performance.
Revised: Empirical Spectral Density (ESD) of weight matrices for many DNNs obey a power-law, with the coefﬁcient of decay being predictive of in-distribution performance.

Original: In the present work, we explore similar indices that potentially correlate with out-of-distribution generalization by examining theeigenspectrum of activations .
Revised: In the present work, we explore similar indices that potentially correlate with out-of-distribution generalization by examining the eigenspectrum of activations .

Original: Generalization in Overparameterized Models Modern neural networks often have significantlymore parameters than number of training samples, challenging classical understanding of the biasvariance tradeoff.
Revised: Generalization in Overparameterized Models Modern neural networks often have signiﬁcantly more parameters than number of training samples, challenging classical understanding of the biasvariance tradeoff.

Original: Overparameterization permits neural networks to overfit to noise in training data,without impairing their generalization to unseen data.
Revised: Overparameterization permits neural networks to overﬁt to noise in training data, without impairing their generalization to unseen data.

Original: In recent work, Bubeck et al. proved that forthe interpolator (function implemented by the neural network) to be smooth, overparameterization isa necessary condition [32].
Revised: In recent work, Bubeck et al. proved that for the interpolator (function implemented by the neural network) to be smooth, overparameterization is a necessary condition [33].

Original: Furthermore, this benign overfitting phenomenon in an overparameterized linear regression problemhas been linked to the power law coefficient of the input covariance matrix [12].
Revised: Furthermore, this benign overﬁtting phenomenon in an overparameterized linear regression problem has been linked to the power law coefﬁcient of the input covariance matrix [12].

Original: Specifically,
Revised: Speciﬁcally,

Original: Bartlett et al. showed that for an infinite-dimensional linear regression problem, benign overfittingis possible iff the eigenspectrum satisfies a power law (upto polylog factors).
Revised: Bartlett et al. showed that for an inﬁnite-dimensional linear regression problem, benign overﬁtting is possible iff the eigenspectrum satisﬁes a power law (upto polylog factors).

Original: More recently, Lee etal. found that the tail eigenvalues of infinite-width network kernels exhibit a power law decay [33].
Revised: More recently, Lee et al. found that the tail eigenvalues of inﬁnite-width network kernels exhibit a power law decay [34].

Original: Following this, Tripuraneni et al.
Revised: Following this, Tripuraneni et al.

Original: explored high-dimensional random feature regression settings andanalytically showed a dependence between eigenspectrum decay rate of the feature covariance matrixand generalization error [34].
Revised: explored high-dimensional random feature regression settings and analytically showed a dependence between eigenspectrum decay rate of the feature covariance matrix and generalization error [35].

Original: While these characterizations provide a theoretical understanding ofgeneralization error in the asymptotic or random feature settings, corresponding questions in thefinite dimensional DNN trained with gradient descent are open problems.
Revised: While these characterizations provide a theoretical understanding of generalization error in the asymptotic or random feature settings, corresponding questions in the ﬁnite dimensional DNN trained with gradient descent are open problems.

Original: For deep linear networks trained using gradient descent, the eigenvalues of input covariance determinethe generalization error dynamics [13].
Revised: For deep linear networks trained using gradient descent, the eigenvalues of input covariance determine the generalization error dynamics [13].

Original: Advani et al.
Revised: Advani et al.

Original: demonstrated that small eigenvalues determinethe convergence of training dynamics as well as the overfitting error at convergence.
Revised: demonstrated that small eigenvalues determine the convergence of training dynamics as well as the overﬁtting error at convergence.

Original: For overparameterized 2-layer neural networks, Arora et al.
Revised: For overparameterized 2-layer neural networks, Arora et al.

Original: provided a fine grained analysis of generalization bounds[35].
Revised: provided a ﬁne grained analysis of generalization bounds [36].

Original: In contrast, we study modern DNN architectures and explore the covariance structure of theirlearned features on visual recognition tasks.
Revised: In contrast, we study modern DNN architectures and explore the covariance structure of their learned features on visual recognition tasks.

Original:  Discussion
Revised:  Discussion

Original: Summary Our experiments suggest a strong correlation between decay coefficient for sampleeigenspectrum of representations, α , and both in-distribution, out-of-distribution generalizationperformance on tasks central to computer vision.
Revised: Summary Our experiments suggest a strong correlation between decay coefﬁcient for sample eigenspectrum of representations, ↵ , and both in-distribution, out-of-distribution generalization performance on tasks central to computer vision.

Original: Representation Quality in High-Dimensional SSL We demonstrate that assessing the quality of apretrained model with our label-free metric ( α ) is consistent with downstream generalization.
Revised: Representation Quality in High-Dimensional SSL We demonstrate that assessing the quality of a pretrained model with our label-free metric ( ↵ ) is consistent with downstream generalization.

Original: Whilebeing computationally efficient, our procedure removes the dependence on labels, making the modelselection more robust and amenable for privacy critical applications. 
Revised: While being computationally efﬁcient, our procedure removes the dependence on labels, making the model selection more robust and amenable for privacy critical applications. Necessary, but not sufﬁcient condition :

Revised: Notably, a task-agnostic measure like ↵ is a necessary but not a sufﬁcient condition for assessing good performance.

Revised: To elaborate on this point, let us present a thought experiment.

Revised: Suppose that we have our ideal representation space that satisﬁes our claims of exhibiting alpha close to the Goldilocks zone.

Original: Redundancy in ViT representations Unlike other models that we considered, representationsfrom early layers of ViT had a rapid eigenspectrum decay with α >> 1 (see Fig. 3 and Fig. 4).
Revised: If we perform a set of permutation operations on individual datapoint representations, the structure of the representation space remains the same but the mapping from representation to label is now destroyed. In doing this set of permutations, we have now arrived at a different set of representations that would exhibit the same (or similar) alpha but demonstrate a lower task performance when measured using a linear readout layer. A similar argument is presented as theorem C.1 in [37]. Extending the conclusions of this thought experiment to our observations, it is clear that we could have models with similar alpha values but distinctly different performance.

Revised: But an alpha value that is not in the Goldilocks zone would be associated with inferior model performance.

Revised: This property is the core of our claim and allows us to propose alpha as a measure for model selection in SSL pipelines.

Revised: Redundancy in ViT representations Unlike other models that we considered, representations from early layers of ViT had a rapid eigenspectrum decay with ↵ >> 1 (see Fig.

Revised: 3 and Fig. 4).

Original: The transformer architecture has a notable difference by design, i.e. early layers possess globalreceptive field context via self-attention on patch embeddings.
Revised: The transformer architecture has a notable difference by design, i.e. early layers possess global receptive ﬁeld context via self-attention on patch embeddings.

Original: Raghu et al. found that early ViTlayers incorporate both local and global information [30].
Revised: Raghu et al. found that early ViT layers incorporate both local and global information [31].

Original: Based on these insights, one intuitiveinterpretation of our results is that the representations have a low effective rank, and encode redundantinformation relevant across multiple scales.
Revised: Based on these insights, one intuitive interpretation of our results is that the representations have a low effective rank, and encode redundant information relevant across multiple scales.

Original: Limitations While the role of α and its relationship to generalization performance is better understoodin the asymptotic setting for linear regression, similar questions in finite dimensional nonlinear modelsare unanswered.
Revised: Limitations While the role of ↵ and its relationship to generalization performance is better understood in the asymptotic setting for linear regression, similar questions in ﬁnite dimensional nonlinear models are unanswered.

Original: It is also worth noting that the empirical correlation was weaker for the earliestlayers in each model.
Revised: It is also worth noting that the empirical correlation was weaker for the earliest layers in each model.

Original: We believe that early layers learn more task invariant features such as cornersand edges [24, 25], and thereby lack rich semantic information for fine-grained downstream task.
Revised: We believe that early layers learn more task invariant features such as corners and edges [24, 25], and thereby lack rich semantic information for ﬁne-grained downstream task.

Original: Inthis case, distinguishing between poorly-trained models may be inconsistent with α .
Revised: In this case, distinguishing between poorly-trained models may be inconsistent with ↵ .

Original: Future Directions Learning efficiently at scale from unlabelled datasets poses an exciting openproblem in deep representation learning.
Revised: Future Directions Learning efﬁciently at scale from unlabelled datasets poses an exciting open problem in deep representation learning.

Original: We hope this work encourages new perspectives into modelselection for SSL pipelines, and informs design of learning objectives and model architectures to learntask-agnostic, adaptive features.
Revised: We hope this work encourages new perspectives into model selection for SSL pipelines, and informs design of learning objectives and model architectures to learn task-agnostic, adaptive features.

Original: Understanding the behaviour of α (notably, a scale-invariant metric)in high-dimensional representation learning might provide insight towards developing a theoreticallygrounded understanding of generalization in deep neural networks. 
Revised: Understanding the behaviour of ↵ (notably, a scale-invariant metric) in high-dimensional representation learning might provide insight towards developing a theoretically grounded understanding of generalization in deep neural networks. Acknowledgements

Revised: The authors would like to thank Zahraa Chorghay and Colleen Gillon for their aesthetic contribution to the manuscript and ﬁgures.

Revised: This research was enabled in part by support provided by Mila and Compute Canada.

Revised: This work was supported by Vanier Canada Graduate scholarship (AG);

Revised: Healthy Brains, Healthy Lives (AG & BAR); NSERC, Grant No. RGPIN-2020-05105 and RGPAS2020-00031 (BAR); and CIFAR, Canada AI Chair & Learning in Machines and Brains Fellowship (BAR).