Original: E VALUATING R EPRESENTATIONS WITH R EADOUT M ODEL S WITCHING
Revised: E VALUATING R EPRESENTATIONS WITH R EADOUT M ODEL S WITCHING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although much of the success of Deep Learning builds on learning good representations, a rigorous method to evaluate their quality is lacking.
Revised: Although much of the success of Deep Learning builds on learning good representations a rigorous method to evaluate their quality is lacking.

Original: In this paper, we treat the evaluation of representations as a model selection problem and propose to use the Minimum Description Length (MDL) principle to devise an evaluation metric.
Revised: In this paper we treat the evaluation of representations as a model selection problem and propose to use the Minimum Description Length (MDL) principle to devise an evaluation metric.

Original: Contrary to the established practice of limiting the capacity of the readout model, we design a hybrid discrete and continuous-valued model space for the readout models and employ a switching strategy to combine their predictions.
Revised: Contrary to the established practice of limiting the capacity of the readout model, we design a hybrid discrete and continuous-valued model space for the readout models and employ a switching strategy to combine their predictions.

Original: The
Revised: The

Original: MDL score takes model complexity, as well as data efficiency into account.
Revised: MDL score takes model complexity, as well as data efficiency into account.

Original: As a result, the most appropriate model for the specific task and representation will be chosen, making it a unified measure for comparison.
Revised: As a result, the most appropriate model for the specific task and representation will be chosen, making it a unified measure for comparison.

Original: The proposed metric can be efficiently computed with an online method and we present results for pre-trained vision encoders of various architectures (ResNet and ViT) and objective functions (supervised and self-supervised) on a range of downstream tasks.
Revised: The proposed metric can be efficiently computed with an online method and we present results for pre-trained vision encoders of various architectures (ResNet and ViT) and objective functions (supervised and self-supervised) on a range of downstream tasks.

Original: We compare our methods with accuracy-based approaches and show that the latter are inconsistent when multiple readout models are used.
Revised: We compare our methods with accuracy-based approaches and show that the latter are inconsistent when multiple readout models are used.

Original: Finally, we discuss important properties revealed by our evaluations such as model scaling, preferred readout model, and data efficiency.
Revised: Finally, we discuss important properties revealed by our evaluations such as model scaling, preferred readout model and data efficiency.

Original: .  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Data representation is crucial to the performance of machine learning algorithms (Bengio et al., 2013).
Revised: Data representation is crucial to the performance of machine learning algorithms (Bengio et al., 2013).

Original: Much of the success of Deep Neural Networks (DNN) can be attributed to their capability of gradually building up more and more abstract representations (Lee et al., 2009).
Revised: Much of the success of Deep Neural Networks (DNN) can be attributed to their capability of gradually building up more and more abstract representations (Lee et al., 2009).

Original: In supervised learning, although the network is trained to predict a specific aspect of the input, the intermediate representations are often proven to be useful for many other downstream tasks (Yosinski et al., 2014).
Revised: In supervised learning, although the network is trained to predict a specific aspect of the input, the intermediate representations are often proven to be useful for many other downstream tasks (Yosinski et al., 2014).

Original: In unsupervised and self-supervised learning, the network is trained on a surrogate task, such as reconstruction (Hinton & Salakhutdinov, 2006; Kingma & Welling, 2013;
Revised: In unsupervised and self-supervised learning, the network is trained on a surrogate task, such as reconstruction (Hinton & Salakhutdinov, 2006; Kingma & Welling, 2013;

Original: He et al., 2021) and contrastive prediction (van den Oord et al., 2018; Chen et al., 2020), which is supposed to capture generic prior of the data.
Revised: He et al., 2021) and contrastive prediction (van den Oord et al., 2018; Chen et al., 2020), which is supposed to capture generic prior of the data.

Original: In recent years, there has been significant improvements in unsupervised representation learning with state-of-the-art models achieving performance comparable to its supervised counterpart (Tomasev et al., 2022).
Revised: In recent years, there has been significant improvements in unsupervised representation learning with state-of-the-art models achieving performance comparable to its supervised counterpart (Tomasev et al., 2022).

Original: Despite the importance of data representation, the evaluation method for representations is rarely discussed.
Revised: Despite the importance of data representation, the evaluation method for representations is rarely discussed.

Original: The most prevalent practice is to train a readout model on the downstream task.
Revised: The most prevalent practice is to train a readout model on the downstream task.

Original: The readout model often has a shallow architecture, e.g. linear layer, to limit its capacity, so that the task performance reflects the representation quality.
Revised: The readout model often has a shallow architecture, e.g. linear layer, to limit its capacity, so that the task performance reflects the representation quality.

Original: The problem with this approach is that the readout model cannot adapt to the nature of the representations.
Revised: The problem with this approach is that the readout model cannot adapt to the nature of the representations.

Original: Deeper models and fine-tuning alleviate this issue.
Revised: Deeper models and fine-tuning alleviate this issue.

Original: However, the representations are left with multiple metrics, each using a different readout mechanism, making the comparison extremely difficult (Nozawa & Sato, 2022).
Revised: However, the representations are left with multiple metrics, each using a different readout mechanism, making the comparison extremely difficult (Nozawa & Sato, 2022).

Original: In this paper, we treat evaluating representations as a model selection problem.
Revised: In this paper, we treat evaluating representations as a model selection problem.

Original: We propose to use Minimum Description Length (MDL) as the main evaluation metric and use model switching to accommodate the need for multiple readout models.
Revised: We propose to use Minimum Description Length (MDL) as the main evaluation metric and use model switching to accommodate the need for multiple readout models.

Original: MDL is a well-studied compression-based approach for inductive inference that provides a generic solution to the model selection problem (Rissanen, 1984;
Revised: MDL is a well-studied compression-based approach for inductive inference that provides a generic solution to the model selection problem (Rissanen, 1984;

Original: Grunwald, 2004; Wallace, 2005; Solomonoff, 1964; Rathmanner & Hutter, 2011).
Revised: Grunwald, 2004; Wallace, 2005; Solomonoff, 1964; Rathmanner & Hutter, 2011).

Original: MDL performs a similar role as held-out validation does for Emperical Risk Minimization (Vapnik, 1991), but has the advantage of being able to deal with single sequence and non-stationary data.
Revised: MDL performs a similar role as held-out validation does for Emperical Risk Minimization (Vapnik, 1991), but has the advantage of being able to deal with single sequence and non-stationary data.

Original: It is closely related to Bayesian model selection and includes a form of Occam’s Razor where the metric takes into account the model complexity.
Revised: It is closely related to Bayesian model selection and includes a form of Occam’s Razor where the metric takes into account the model complexity.

Original: The complexity term can be explicitly represented as the codelength of the model in the case of a 2-part code, as a KL-term when using a variational code, or implicitly when using prequential or Bayesian codes.
Revised: The complexity term can be explicitly represented as the codelength of the model in the case of a 2-part code, as a KL-term when using a variational code, or implicitly when using prequential or Bayesian codes.

Original: By including the model complexity in the evaluation metric, we automatically resolve the need of limiting the readout model complexity and are able to compare MDL scores freely between different readout mechanisms.
Revised: By including the model complexity in the evaluation metric, we automatically resolve the need of limiting the readout model complexity and are able to compare MDL scores freely between different readout mechanisms.

Original: Intuitively, if the induced representation is nonlinear and requires a higher capacity model for readout, the MDL score reflects this by having a larger complexity term.
Revised: Intuitively, if the induced representation is nonlinear and requires a higher capacity model for readout, the MDL score reflects this by having a larger complexity term.

Original: Note that this also applies in the case of finetuning, where the pre-trained model is allowed to adapt for the downstream tasks.
Revised: Note that this also applies in the case of finetuning, where the pre-trained model is allowed to adapt for the downstream tasks.

Original: Model switching allows multiple readout models and automatically finds the best readout model for the downstream task at each dataset size (Figure 1).
Revised: Model switching allows multiple readout models and automatically finds the best readout model for the downstream task at each dataset size (Figure 1).

Original: Therefore, MDL with readout model switching provides a unified framework for evaluating representations regardless the evaluation protocol employed.
Revised: Therefore, MDL with readout model switching provides a unified framework for evaluating representations regardless the evaluation protocol employed.

Original: An optimally switched model will have the best performance at each point and thus the lowest  description length ( area 
Revised: An optimally switched model will have the best performance at each point and thus the lowest prequential description length (= area under the curve).

Original: It is conjured that useful representations make the variability in the data more predictable and allow to efficient learning human-like data (H´enaff et al., 2019).
Revised: It is conjured that useful representations make the variability in the data more predictable and allow to efficient learning human-like data (H´enaff et al., 2019).

Original: The MDL evaluation metric formalizes the data efficiency perspective - especially evident in the form of prequential MDL.
Revised: The MDL evaluation metric formalizes the data efficiency perspective - especially evident in the form of prequential MDL.

Original: Prequential MDL (Dawid & Vovk, 1999; Poland & Hutter, 2005) turns computing the description
Revised: Prequential MDL (Dawid & Vovk, 1999; Poland & Hutter, 2005) turns computing the description

Original: length L ( D| ϕ )
Revised: length L ( D| ϕ )

Original: = − log p ( D| ϕ ) into a sequential prediction problem: log p ( D| ϕ ) =
Revised: = − log p ( D| ϕ ) into a sequential prediction problem: log p ( D| ϕ ) =

Original: (cid:80) t log p ( y t | ϕ ≤ t  y <t ) , where ϕ is an encoder for feature extraction, ϕ t := ϕ ( x t ) an encoded input, and ( x <t , y <t ) is the data seen before timestep t .
Revised: (cid:80) t log p ( y t | ϕ ≤ t , y <t ) , where ϕ is an encoder for feature extraction, ϕ t := ϕ ( x t ) an encoded input, and ( x <t , y <t ) is the data seen before timestep t .

Original: In order to achieve a short description length, it is beneficial if the representation is capable of fast learning, i.e. given a good representation, a few examples are enough to achieve good performance.
Revised: In order to achieve a short description length, it is beneficial if the representation is capable of fast learning, i.e. given a good representation, a few examples are enough to achieve good performance.

Original: As a by-product of our readout model switching based on MDL, we can visualize the predictive performance along the data sequence and inspect how efficient the downstream learning is.
Revised: As a by-product of our readout model switching based on MDL, we can visualize the predictive performance along the data sequence and inspect how efficient the downstream learning is.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: We propose readout model switching for evaluating representations.
Revised: We propose readout model switching for evaluating representations.

Original: We prove a regret bound of readout model switching and discuss assumptions made.
Revised: We prove a regret bound of readout model switching and discuss assumptions made.

Original: We use an online learning framework for efficient computation and evaluate the performance of several popular visual representation methods on the set of downstream tasks.
Revised: We use an online learning framework for efficient computation and evaluate the performance of several popular visual representation methods on the set of downstream tasks.

Original: We investigate and inspect the preferred readout models, data efficiency and model scaling.
Revised: We investigate and inspect the preferred readout models, data efficiency and model scaling.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Minimum Description Length is based on the fundamental idea that learning and comprehension correspond to compression (Rathmanner & Hutter, 2011).
Revised: Minimum Description Length is based on the fundamental idea that learning and comprehension correspond to compression (Rathmanner & Hutter, 2011).

Original: Given data D =( y t ) N 1 ∈ Y N and a hypothesis space M =
Revised: Given data D =( y t ) N 1 ∈ Y N and a hypothesis space M =

Original: { M 1 , M 2 , . . .
Revised: { M 1 , M 2 , . . .

Original: } , where each hypothesis M corresponds to a parametric probabilistic model p ( D| θ, M ) , MDL aims to identify the model that can compress the data D best.
Revised: } , where each hypothesis M corresponds to a parametric probabilistic model p ( D| θ, M ) , MDL aims to identify the model that can compress the data D best.

Original: Considering the close relationship between lossless coding and probability distributions, this can be achieved by associating a codelength function L ( D| M )
Revised: Considering the close relationship between lossless coding and probability distributions, this can be achieved by associating a codelength function L ( D| M )

Original: = − log p ( D| M ) with each hypothesis.
Revised: = − log p ( D| M ) with each hypothesis.

Original: A vast body of literature shows that models with a shorter description length have a better chance of generalizing to future data (Wallace, 2005; Gr¨unwald, 2007a; Rathmanner & Hutter, 2011).
Revised: A vast body of literature shows that models with a shorter description length have a better chance of generalizing to future data (Wallace, 2005; Gr ¨ unwald, 2007a; Rathmanner & Hutter, 2011).

Original: A crude way to obtain description lengths is to consider L ( D | M )=
Revised: A crude way to obtain description lengths is to consider L ( D | M )=

Original: L M ( θ )+ L M ( D | θ ) , where
Revised: L M ( θ )+ L M ( D | θ ) , where

Original: L M ( θ ) is the cost of encoding the parameters and L M ( D| θ )
Revised: L M ( θ ) is the cost of encoding the parameters and L M ( D| θ )

Original: = − log p ( D| θ, M ) is the cost of compressing the data with the parameterized model.
Revised: = − log p ( D| θ, M ) is the cost of compressing the data with the parameterized model.

Original: This two-part code approach is intuitive but sub- optimal and ambiguous because it does not specify how to encode the parameters.
Revised: This two-part code approach is intuitive but suboptimal and ambiguous because it does not specify how to encode the parameters.

Original: This crude MDL approach has been refined in three distinct but closely related ways:
Revised: This crude MDL approach has been refined in three distinct but closely related ways:

Original: The Bayesian marginal likelihood: L Bayes ( D| M ) :
Revised: The Bayesian marginal likelihood: L Bayes ( D| M ) :

Original: = − log (cid:82) θ p ( D| θ, M ) p ( θ ) dθ and its variational upper bound L ELBO ( D| M ) :=
Revised: = − log (cid:82) θ p ( D| θ, M ) p ( θ ) dθ and its variational upper bound L ELBO ( D| M ) :=

Original: KL ( q ( θ ) | p ( θ )) − E θ ∼ q log p ( D| θ, M ) .
Revised: KL ( q ( θ ) | p ( θ )) − E θ ∼ q log p ( D| θ, M ) .

Original: Both depend on the chosen prior p ( θ ) and require access to the posterior distribution over parameters given data; or an approximation thereof.
Revised: Both depend on the chosen prior p ( θ ) and require access to the posterior distribution over parameters given data; or an approximation thereof.

Original: Normalized Maximum Likelihood (NML): L NML ( D | M ) :
Revised: Normalized Maximum Likelihood (NML): L NML ( D| M ) :

Original: = − log [ p ( D | ˆ θ ML

Original: ( D ) , M ) /

Original:  (cid:82) z p ( D| ˆ θ ML ( z ) , M ) dz ] only relies on the maximum likelihood estimator ˆ θ ML ( · ) , but normalizes over all potentially observable data z ∈ Y N, which is often intractable and even undefined for many model families of interest.
Revised: = − log [ p ( D| ˆ θ ML ( D ) , M ) / (cid:82) z p ( D| ˆ θ ML ( z ) , M ) dz ] only relies on the maximum likelihood estimator ˆ θ ML ( · ) , but normalizes over all potentially observable data z ∈ Y N which is often intractable and even undefined for many model families of interest.

Original: The log-numerator is also called the complexity of M and measures how well the model could fit all possible data.
Revised: The log-numerator is also called the complexity of M and measures how well the model could fit all possible data

Original: Under NML, a model M achieves a short description length only when it fits the given data well (high log p ( D| ˆ θ ML ( D) , M) ) as well as not fit well many different data (low denominator).
Revised: Under NML, a model M achieves a short description length only when it fits the given data well (high log p ( D| ˆ θ ML ( D , M

Revised: as well

Revised: as not fit

Original:  The prequential approach: L plugin ( D| M ) := − (cid:80)
Revised: well many different data (low denominator The prequential approach: L plugin ( D| M ) := − (cid:80)

Original: Nt =1 log p ( y t | ˆ θ ( D < t ) , M ) decomposes the description length over datapoints and relies on the choice of a suitable plug-in parameter estimator ˆ θ ( D <t ).
Revised: Nt =1 log p ( y t | ˆ θ ( D <t ) , M ) decomposes the description length over datapoints and relies on the choice of a suitable plug-in parameter estimator ˆ θ ( D <t )

Original: It emphasizes that in order to achieve a short codelength, a model M must not only be a good predictor given all training data D, but already given only parts of the data D <t for allt , i.e. be a sample efficient predictor.
Revised: It emphasizes that in order to achieve a short codelength , a model M must not only be a good predictor given all training data D but already given only parts of the data D <t for all

Revised: t

Revised: i

Revised: .e.

Original:  Model complexity and sample efficiency canthus be considered
Revised: be a sample efficient predictor Model complexity and sample efficiency can

Original:  two sides of the same coin.
Revised: thus be considered two sides of the same coin.

Original: Approaches to computing description lengths for some data D under a model family p ( D| θ, M ) are called universal codes when they result in description lengths that stay close to the (generally unachievable) maximum likelihood codelength for that family: L universal ( D| M ) ≤ − log p ( D| ˆ θ ML ( D ) , M ) + O (log N ) .
Revised: Approaches to computing description lengths for some data D under a model family p ( D| θ, M ) are called universal codes when they result in description lengths that stay close to the (generally unachievable) maximum likelihood codelength for that family: L universal ( D| M ) ≤ − log p ( D| ˆ θ ML ( D ) , M ) + O (log N ) .

Original: L NML ( D| M ) is by definition universal; L Bayes ( D| M ) and
Revised: L NML ( D| M ) is by definition universal; L Bayes ( D| M ) and

Original: L plugin ( D| M ) are universal for many reasonable choices of the prior p ( θ ) and the estimator ˆ θ ( D <t ) respectively
Revised: L plugin ( D| M ) are universal for many reasonable choices of the prior p ( θ ) and the estimator ˆ θ ( D <t ) respectively

Original: (Gr ¨ unwald, 2007b).
Revised: (Gr ¨ unwald, 2007b).

Original: Methods for Representation Evaluation.
Revised: Methods for Representation Evaluation.

Original: Linear probing, using a linear layer for readout, is the most prevalent evaluation protocol for representations, the performance of which is reported and compared in almost all of the recent representation learning works (van den Oord et al., 2018;
Revised: Linear probing, using a linear layer for readout, is the most prevalent evaluation protocol for representations, the performance of which is reported and compared in almost all of the recent representation learning works (van den Oord et al., 2018;

Original: Chen et al., 2020; Grill et al., 2020;
Revised: Chen et al., 2020; Grill et al., 2020;

Original: Li et al., 2021; Caron et al., 2021;
Revised: Li et al., 2021; Caron et al., 2021;

Original: He et al., 2021).
Revised: He et al., 2021).

Original: Apart from linear probing, clustering algorithms such as Hebbian linear classifier (Bengio, 2012) and K-Nearest
Revised: Apart from linear probing, clustering algorithms such as Hebbian linear classifier (Bengio, 2012) and K-Nearest

Original: Neighbor (Caron et al., 2021) are also popular choices.
Revised: Neighbor (Caron et al., 2021) are also popular choices.

Original: However, concerns have been raised for model comparison using these simple probes.
Revised: However, concerns have been raised for model comparison using these simple probes.

Original: Resnick et al.
Revised: Resnick et al.

Original: (2019) shows that the performance of linear probing can be misleading, since models performing weakly with linear probing can perform better under alternative protocols.
Revised: (2019) shows that the performance of linear probing can be misleading, since models performing weakly with linear probing can perform better under alternative protocols.

Original: He et al.
Revised: He et al.

Original: (2021) also argue that the use of linear probes limits the development of methods that induce non-linear representations.
Revised: (2021) also argue that the use of linear probes limits the development of methods that induce non-linear representations.

Original: Hewitt & Liang (2019) question the practice of using downstream accuracy as the evaluation metric and propose to design probes based on their selectivity, which puts the task accuracy in context with the probe’s capacity to memorize.
Revised: Hewitt & Liang (2019) question the practice of using downstream accuracy as the evaluation metric and propose to design probes based on their selectivity, which puts the task accuracy in context with the probe’s capacity to memorize.

Original: Yogatama et al.
Revised: Yogatama et al.

Original: and Voita & Titov (2020) propose to use MDL, either variational or prequential code, as the evaluation metric, which takes into account the complexity in the final measurement.
Revised: and Voita & Titov (2020) propose to use MDL, either variational or prequential code, as the evaluation metric, which takes into account the complexity in the final measurement.

Original: H´enaff et al.
Revised: H´enaff et al.

Original: (2019) bring to light the benefit of pre-training for downstream data efficiency.
Revised: (2019) bring to light the benefit of pre-training for downstream data efficiency.

Original: Whitney et al.
Revised: Whitney et al.

Original: (2020) propose a modified version of MDL that introduces a baseline corresponding to solving the downstream task.
Revised: (2020) propose a modified version of MDL that introduces a baseline corresponding to solving the downstream task.

Original: While previous works proposing MDL as an evaluation metric use only a single readout protocol, our method can be applied for combining different readout protocols regardless of the backbone being frozen or fine-tuned.
Revised: While previous works proposing MDL as an evaluation metric use only a single readout protocol, our method can be applied for combining different readout protocols regardless of the backbone being frozen or fine-tuned.

Original: Our use of an online framework to compute prequential MDL scores which has not been used previously to evaluate representations leads to shorter description lengths (Bornschein et al., 2022) and crucially enables the model switching between readout-methods at each timestep.
Revised: Our use of an online framework to compute prequential MDL scores which has not been used previously to evaluate representations leads to shorter description lengths (Bornschein et al., 2022) and crucially enables the model switching between readout-methods at each timestep.

Original: As a result, our framework gives insights into properties of the representations that can guide us in choosing the best way to use the representation.
Revised: As a result, our framework gives insights into properties of the representations that can guide us in choosing the best way to use the representation.

Original:  E VALUATING R EPRESENTATIONS WITH R EADOUT M ODEL S WITCHING
Revised:  E VALUATING R EPRESENTATIONS WITH R EADOUT M ODEL S WITCHING

Original: We formally define the model selection problem for representation evaluation under the MDL principle as follows: the objective is to compare the encoder ϕ ( . ) that minimizes the code- length L ( D| ϕ )
Revised: We formally define the model selection problem for representation evaluation under the MDL principle as follows: the objective is to compare the encoder ϕ ( . ) that minimizes the codelength L ( D| ϕ )

Original: = − log p ( D| ϕ ) , where D = { ( x t , y t ) } N 1 is the inputs x t and associated prediction targets y t .
Revised: = − log p ( D| ϕ ) , where D = { ( x t , y t ) } N 1 is the inputs x t and associated prediction targets y t .

Original: Prequential MDL decomposes the codelength into the cumulative prediction loss
Revised: Prequential MDL decomposes the codelength into the cumulative prediction loss

Original: L ( D| ϕ )
Revised: L ( D| ϕ )

Original: = − (cid:80)
Revised: = − (cid:80)

Original: Nt =1 log p ( y t | ϕ ≤ t , y <t ) , where ϕ t := ϕ ( x t ) is the encoded feature of input x t .
Revised: Nt =1 log p ( y t | ϕ ≤ t , y <t ) , where ϕ t := ϕ ( x t ) is the encoded feature of input x t .

Original: In order to perform the prediction task from the features, previous works have to fix the probe to be a linear layer or a shallow MLP.
Revised: In order to perform the prediction task from the features, previous works have to fix the probe to be a linear layer or a shallow MLP.

Original: In our study, instead of picking a fixed model, we design a hybrid of a continuous and discrete-valued ( k, θ k ) model space of the readout models, where “expert” k ∈ [1 , . . .
Revised: In our study, instead of picking a fixed model, we design a hybrid of a continuous and discrete-valued ( k, θ k ) model space of the readout models, where “expert” k ∈ [1 , . . .

Original: , K ] is the model class and θ k are the continuous-valued parameters corresponding to the model class k .
Revised: , K ] is the model class and θ k are the continuous-valued parameters corresponding to the model class k .

Original: Each readout model can be seen as an expert forecasting system that learns to predict the data independently in an online fashion.
Revised: Each readout model can be seen as an expert forecasting system that learns to predict the data independently in an online fashion.

Original: At each datapoint x t , the prediction ˆ y t is a combination of the experts’ predictions.
Revised: At each datapoint x t , the prediction ˆ y t is a combination of the experts’ predictions.

Original: The final MDL score is the cumulative predictive performance of the board of K experts.
Revised: The final MDL score is the cumulative predictive performance of the board of K experts.

Original: By constructing the probe using multiple readout models and switching between them, the final
Revised: By constructing the probe using multiple readout models and switching between them, the final

Original: MDL score is marginalized over the choice of the readout architectures.
Revised: MDL score is marginalized over the choice of the readout architectures.

Original: Since MDL automatically takes into account the complexity of the model, i.e. any extra complexity used is going to cost in the
Revised: Since MDL automatically takes into account the complexity of the model, i.e. any extra complexity used is going to cost in the

Original: MDL score, our framework allows the representation to select the most suitable readout mechanism at any data size while still maintaining a comparable final score for the representations.
Revised: MDL score, our framework allows the representation to select the most suitable readout mechanism at any data size while still maintaining a comparable final score for the representations.

Original: In addition, after computing the score, we can inspect the posterior over the readout models at any given data size, as well as data efficiency along the trajectory, providing valuable insight into the characteristics of the representation.
Revised: In addition, after computing the score, we can inspect the posterior over the readout models at any given data size, as well as data efficiency along the trajectory, providing valuable insight into the characteristics of the representation.

Original: Readout Model Switching.
Revised: Readout Model Switching.

Original: Given K readout models, we define random variables ξ t ∈ [1 , . . .
Revised: Given K readout models, we define random variables ξ t ∈ [1 , . . .

Original: , K ] , ∀ t ∈ [1 , . . .
Revised: , K ] , ∀ t ∈ [1 , . . .

Original: , N ] the readout model to use at timestep t .
Revised: , N ] the readout model to use at timestep t .

Original: The Expert Sequence (ES) prior (Koolen & de Rooij, 2008) uses a hidden Markov model (HMM) to define the joint distribution of the data y 1: N and random variables ξ 1: N as p ( y 1: N , ξ 1: N | ϕ )
Revised: The Expert Sequence (ES) prior (Koolen & de Rooij, 2008) uses a hidden Markov model (HMM) to define the joint distribution of the data y 1: N and random variables ξ 1: N as p ( y 1: N , ξ 1: N | ϕ )

Original: = (cid:81)
Revised: = (cid:81)

Original: Nt =1 p ξ t ( y t | ϕ ≤ t , y <t ) p ( ξ t |
Revised: Nt =1 p ξ t ( y [Equation] | ϕ ≤ [Equation]

Revised: , y <

Original: [Equation]
Revised: [Equation]

Original: t
Revised: ) p ( ξ [Equation] | ξ

Original: [Equation]
Revised: [Equation]

Original: 1 ) , where p ( ξ 1 | ξ 0 )= p ( ξ 1 ) is the initial distribution and p k ( · ) is the prediction of the k th readout model.
Revised: − [Equation] ) , where p ( ξ 1 | ξ 0 )= p ( ξ 1 ) is the initial distribution and p k ( · ) is the prediction of the k th readout model.

Original: At each timestep t , we use the previously observed data to estimate ˆ θ k
Revised: At each timestep t , we use the previously observed data to estimate ˆ θ k

Original: ( ϕ <t , y <t ) for the parameters of the readout models.
Revised: ( ϕ <t , y <t ) for the parameters of the readout models.

Original: Therefore, the k th readout model prediction p k ( y t | ϕ ≤ t , y <t ) is the plugin distribution p ( y t | ϕ t , ˆ θ k ( ϕ <t , y <t )) .
Revised: Therefore, the k th readout model prediction p k ( y t | ϕ ≤ t , y <t ) is the plugin distribution p ( y t | ϕ t , ˆ θ k ( ϕ <t , y <t )) .

Original: The final readout model switching codelength function has the form:
Revised: The final readout model switching codelength function has the form:

Original: [Equation]
Revised: [Equation]

Original: The readout model switching defined in Equation 1 represents a family of codes that combines prequential and Bayesian MDL.
Revised: The readout model switching defined in Equation 1 represents a family of codes that combines prequential and Bayesian MDL.

Original: From the perspective of the HMM, p ( ξ t | ξ t − 1 ) is the transition matrix and, in our context, it is equivalent to a switching strategy between the readout models.
Revised: From the perspective of the HMM, p ( ξ t | ξ t − 1 ) is the transition matrix and, in our context, it is equivalent to a switching strategy between the readout models.

Original: Every switching strategy corresponds to a specific code in the family.
Revised: Every switching strategy corresponds to a specific code in the family.

Original: For example, if p ( ξ t | ξ t − 1 ) = 1 iff ξ t = ξ t − 1 (zero otherwise), i.e when we believe a single readout model is responsible for modeling the whole data sequence, then Equation 1 simplifies to a Bayesian mixture code L BM ( y 1: N | ϕ )
Revised: For example, if p ( ξ t | ξ t − 1 ) = 1 iff ξ t = ξ t − 1 (zero otherwise), i.e when we believe a single readout model is responsible for modeling the whole data sequence, then Equation 1 simplifies to a Bayesian mixture code L BM ( y 1: N | ϕ )

Original: = − log (cid:80) k p ( ξ 1 = k ) (cid:81) N t =1 p ( y t | ϕ t , ˆ θ k ( ϕ <t , y <t )) .
Revised: = − log (cid:80) k p ( ξ 1 = k ) (cid:81) Nt =1 p ( y t | ϕ t , ˆ θ k ( ϕ <t , y <t )) .

Original: For the rest of the paper, we are going to use Fixed Share (FS) (Herbster & Warmuth, 1998; 2004) as the switching strategy.
Revised: For the rest of the paper, we are going to use Fixed Share (FS) (Herbster & Warmuth, 1998; 2004) as the switching strategy.

Original: The prior defined in fixed share can be interpreted as follows: at timestep t , with a probability 1 - α t , the expert is the same as in the previous timestep; with probability α t , it switches to another model according to the probability w ( k ) .
Revised: The prior defined in fixed share can be interpreted as follows: at timestep t , with a probability 1 - α t , the expert is the same as in the previous timestep; with probability α t , it switches to another model according to the probability w ( k ) .

Original: We are going to use the uniform distribution w ( k ) =
Revised: We are going to use the uniform distribution w ( k ) =

Original: 1 /K through out the paper.
Revised: 1 /K through out the paper.

Original: The switching strategy can be simplified and expressed as
Revised: The switching strategy can be simplified and expressed as

Original: [Equation]
Revised: [Equation]

Original: where

Original: [Equation]

Original: 2) is decreasing in timestep t , and m is a hyperparameter.
Revised: is decreasing in timestep t , and m is a hyperparameter.

Original: Note that there are other switching strategies that could be used, e.g. elementwise mixture, fixed share with constant switching rate, switch distribution and run-length model (Koolen & de Rooij, 2008).
Revised: Note that there are other switching strategies that could be used, e.g. elementwise mixture, fixed share with constant switching rate, switch distribution and run-length model (Koolen & de Rooij, 2008).

Original: We provide a comparison of these strategies in Appendix A.6.
Revised: We provide a comparison of these strategies in Appendix A.6.

Original: In general, we find that Bayesian mixture and elementwise mixture perform significantly worse than fixed share and other more sophisticated switching strategies.
Revised: In general, we find that Bayesian mixture and elementwise mixture perform significantly worse than fixed share and other more sophisticated switching strategies.

Original: The poor codelength of the Bayesian mixture is due to the catch-up phenomena (van Erven et al., 2012): it assumes that a single model is responsible for modeling the whole data sequence, while evidently the best model changes with data set size.
Revised: The poor codelength of the Bayesian mixture is due to the catch-up phenomena (van Erven et al., 2012): it assumes that a single model is responsible for modeling the whole data sequence, while evidently the best model changes with data set size.

Original: Regret Bound.
Revised: Regret Bound.

Original: To characterize the gap between the codelength for our readout model switching and the shortest codelength that can be achieved in hindsight, we derive a regret bound in Theorem 1.
Revised: To characterize the gap between the codelength for our readout model switching and the shortest codelength that can be achieved in hindsight, we derive a regret bound in Theorem 1.

Original: This bound tells us whether the readout model switching converges to the optimal solution and, if so, how fast it converges.
Revised: This bound tells us whether the readout model switching converges to the optimal solution and, if so, how fast it converges.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let a sequence x N = ( x i )
Revised: Let a sequence x N = ( x i )

Original: Ni =1 be sampled i.i.d.
Revised: Ni =1 be sampled i.i.d.

Original: from distribution P .
Revised: from distribution P .

Original: Let all readout models M 1 , . . .
Revised: Let all readout models M 1 , . . .

Original: M K be in the exponential family.
Revised: M K be in the exponential family.

Original: Let ˆ θ k ( x t ) be the regularized Maximum Likelihood (ML) estimator of M k after the first t examples x t = ( x i ) ti =1 .
Revised: Let ˆ θ k ( x t ) be the regularized Maximum Likelihood (ML) estimator of M k after the first t examples x t = ( x i ) ti =1 .

Original: The expected regret of readout model switching w.r.t.
Revised: The expected regret of readout model switching w.r.t.

Original: the best sequence of readout model ξ ∗ 1: N and their best fit parameters ˆ θ ξ ∗ t ( x N ) is
Revised: the best sequence of readout models ξ ∗ 1: N and their best fit parameters ˆ θ ξ ∗ t ( x N ) is

Original: [Equation]
Revised: [Equation]

Original: where µ ∗ k = E P [ T k ( x )] , and T k ( . ) is the sufficient statistic and µ k is the mean value parameter of
Revised: where µ ∗ k = E P [ T k ( x )] , and T k ( . ) is the sufficient statistic and µ k is the mean value parameter of

Original: M k .
Revised: M k .

Original: The proof of Theorem 1 can be found in Appendix A.1.
Revised: The proof of Theorem 1 can be found in Appendix A.1.

Original: The expected regret consists of two terms: the regret of the switching strategy and the regret from the plugin distribution.
Revised: The expected regret consists of two terms: the regret of the switching strategy and the regret from the plugin distribution.

Original: The latter is bounded by C log N with C being a constant determined by the model family and the true distribution.
Revised: The latter is bounded by C log N with C being a constant determined by the model family and the true distribution.

Original: For
Revised: For

Original: Fixed Share with decreasing switching rate, − E P [log p ( ξ ∗ 1: N )] ≤ m log K + ( m − 1) log N/ ( m − 1) , where m − 1 is the number of switch points (Veness et al., 2012; Koolen & de Rooij, 2013).
Revised: Fixed Share with decreasing switching rate, − E P [log p ( ξ ∗ 1: N )] ≤ m log K + ( m − 1) log N/ ( m − 1) , where m − 1 is the number of switch points (Veness et al., 2012; Koolen & de Rooij, 2013).

Original: Therefore, the expected regret is of the order of log N =
Revised: Therefore, the expected regret is of the order of log N =

Original: o ( N ) , making the corresponding code a universal code.
Revised: o ( N ) , making the corresponding code a universal code.

Original: Many other switching strategies have the same property, such as Bayesian mixture and run-length code (Koolen & de Rooij, 2013).
Revised: Many other switching strategies have the same property, such as Bayesian mixture and run-length code (Koolen & de Rooij, 2013).

Original: Like many theories and algorithms applied to DNN, some of the assumptions of Theorem 1 do not hold in practice: DNNs are not in the exponential family and the solutions found by Stochastic
Revised: Like many theories and algorithms applied to DNN, some of the assumptions of Theorem 1 do not hold in practice: DNNs are not in the exponential family and the solutions found by Stochastic

Original: Gradient Descent (SGD) training are not ML solutions.
Revised: Gradient Descent (SGD) training are not ML solutions.

Original: Although the exact nature of the final codelength heavily relies on the training algorithm and much of the understanding of SGD is left for future works, as long as the estimator converges asymptotically to the optimal solution, we can use the code for model selection.
Revised: Although the exact nature of the final codelength heavily relies on the training algorithm and much of the understanding of SGD is left for future works, as long as the estimator converges asymptotically to the optimal solution, we can use the code for model selection.

Original: In fact, there exist estimators that outperform the ML estimator such that the resulting prequential code better approximates the Normalized Maximum Likelihood (NML) code which is the optimal universal code (Gr ¨ unwald & Kotlowski, 2010).
Revised: In fact, there exist estimators that outperform the ML estimator such that the resulting prequential code better approximates the Normalized Maximum Likelihood (NML) code which is the optimal universal code (Gr ¨ unwald & Kotlowski, 2010).

Original: Computing MDL with Online Learning.
Revised: Computing MDL with Online Learning.

Original: Algorithm 1 describes the online implementation for computing the description length and for performing inference over the preferred readout models 
Revised: Algorithm 1 describes the online implementation for computing the description length and for performing inference over the preferred readout models p ( ξ t |D <t ) .

Revised: For each example in the sequence we first obtain a marginal prediction by weighting the predictions from the readout models according to our current belief over the mixture.

Original: Require:
Revised: After

Revised: Algorithm 1 MDL with Readout Model Switching

Original:  data D = ( x t , y t )
Revised: Require: data D = ( x t , y t )

Original: Nt =1 ; K readout models; initial model probability p ( ξ 1 ) and switching strategy p (  t  ) ; subroutine  updates the parameters given a dataset UpdateParameters 
Revised: Nt =1 ; K readout models; initial model probability p ( ξ 1 ) and switching strategy p ( ξ t | ξ t − 1 ) ; subroutine that updates the parameters given a dataset UpdateParameters 1: Initialize

Original: : Initialize

Original: , θ K empty store posterior 
Revised: Q store posterior p

Original:  < do 3: Compute
Revised: D < do 3: Compute

Original: t , y t − 1 | x t − 1 ) s ← log(cid:80) ξ t − 1 exp (log p ( ξ t | ξ t − 1 ) + s ) 4: Compute posterior p
Revised: Compute store in Q : log p ( [Equation]

Original: t |D < 
Revised: D < t

Revised: [Equation]

Original: t ) and store in Q : log p ( ξ t |D < t )
Revised: = log p [Equation] [Equation] [Equation]

Revised: log

Original: = log p ( ξ t , y t − 1 | x t − 1 ) − log (cid:80) ξ t p ( ξ t , y t − 1 | x t − 1 ) 5:
Revised: [Equation] t p ( [Equation] t , y t − 1 | x t − 1 ) 5:

Original: Compute next-step loss of K readout models: L kt := − log p k ( y t | x t )
Revised: Compute next-step loss of K readout models: L kt := − log p k ( y t | x t )

Original: = − log p ( y t | x t ; θ k ) 6:
Revised: = − log p ( y t | x t ; θ k ) 6:

Original: Combine K models to update log p ( ξ t , y t | x t ) : s ← −L ξ t t + s 7: for k = 1 to K do 8:
Revised: Combine K models to update log p ( ξ t , y t | x t ) : s ← −L ξ t t + s 7: for k = 1 to K do 8:

Original: Update parameters: θ k ← UpdateParameters ( θ k , D ≤ t ) 9: end for 10: end for 11: Compute total codelength L switch ← − log (cid:80) ξ N exp( s ) 12: return L switch and Qp ( ξ t |D <t ) . For each example in the sequence we first obtain a marginal prediction by weighting the predictions from the readout models according to our current belief over the mixture. After receiving the label y t , we compute the cross-entropy loss for the marginal prediction and add it to the cumulative loss.
Revised: Update parameters: θ k ← UpdateParameters ( θ k , D ≤ t ) 9: end for 10: end for 11: Compute total codelength L switch ← − log (cid:80) ξ N exp( s ) 12: return L switch and Q receiving the label y t , we compute the cross-entropy loss for the marginal prediction and add it to the cumulative loss.

Original: Then, the belief over the mixture is updated and the new example is added to the training data for readout model parameter estimation.
Revised: Then, the belief over the mixture is updated and the new example is added to the training data for readout model parameter estimation.

Original: The latter is done by online SGD with replay.
Revised: The latter is done by online SGD with replay.

Original: For the posterior belief over the model switches p ( ξ t |D <t ) the above procedure amounts to the well known, efficient, and exact forward pass HMM inference algorithm.
Revised: For the posterior belief over the model switches p ( ξ t |D <t ) the above procedure amounts to the well known, efficient, and exact forward pass HMM inference algorithm.

Original: For the neural network parameters θ k on the other hand we use SGD as the plugin estimator for prequential MDL.
Revised: For the neural network parameters θ k on the other hand we use SGD as the plugin estimator for prequential MDL.

Original: Prior work has shown that SGD is an effective and reliable plug-in estimator for modern neural networks (Blier & Ollivier, 2018).
Revised: Prior work has shown that SGD is an effective and reliable plug-in estimator for modern neural networks (Blier & Ollivier, 2018).

Original: Note that most prior works on computing prequential MDL scores with neural networks use a block-wise approximation; i.e. they partition the data into a small number of segments and evaluate each segment independently by training on all the previous segments to convergence (Voita & Titov, 2020; Bornschein et al., 2020; Whitney et al., 2020).
Revised: Note that most prior works on computing prequential MDL scores with neural networks use a block-wise approximation; i.e. they partition the data into a small number of segments and evaluate each segment independently by training on all the previous segments to convergence (Voita & Titov, 2020; Bornschein et al., 2020; Whitney et al., 2020).

Original: Since we need a prediction at each step t to update the mixture weights, we here instead use the online mini-batch incremental training with replay described in Bornschein et al.
Revised: Since we need a prediction at each step t to update the mixture weights, we here instead use the online mini-batch incremental training with replay described in Bornschein et al.

Original: This approach updates the parameters continually with online mini-batch gradient descent training on the new and previously seen examples.
Revised: This approach updates the parameters continually with online mini-batch gradient descent training on the new and previously seen examples.

Original: Instead of using a replay buffer to store examples in-memory, we use replay streams to iterate through the examples in their original order from permanent storage.
Revised: Instead of using a replay buffer to store examples in-memory, we use replay streams to iterate through the examples in their original order from permanent storage.

Original: By tuning the number of streams and the reset probability, we can adjust the amount of replay training and change from uniform sampling to a sampling strategy that is biased towards more recent samples.
Revised: By tuning the number of streams and the reset probability, we can adjust the amount of replay training and change from uniform sampling to a sampling strategy that is biased towards more recent samples.

Original: However, in this work, we use uniform random replay samples for all our experiments.
Revised: However, in this work, we use uniform random replay samples for all our experiments.

Original: As suggested in Bornschein et al.
Revised: As suggested in Bornschein et al.

Original: , we use Exponential Moving Average (EMA) and label smoothing to improve performance of the online training, but not weight standardization as we don’t want to alter the architectures.
Revised: , we use Exponential Moving Average (EMA) and label smoothing to improve performance of the online training, but not weight standardization as we don’t want to alter the architectures.

Original: A strictly equivalent alternative to the fully online implementation is a two-stage implementation (see Appendix A.3):
Revised: A strictly equivalent alternative to the fully online implementation is a two-stage implementation (see Appendix A.3):

Original: First, the prequential online parameter estimation is computed for each of the readout models independently while the per-step predictions and log-losses are recorded for later analysis.
Revised: First, the prequential online parameter estimation is computed for each of the readout models independently while the per-step predictions and log-losses are recorded for later analysis.

Original: In a second stage, the predictions and losses can be used to compute the mixtures, switching posterior p ( ξ t |D <t ) and overall description lengths.
Revised: In a second stage, the predictions and losses can be used to compute the mixtures, switching posterior p ( ξ t |D <t ) and overall description lengths.

Original: To facilitate experimenting with different switching strategies we use this 2-stage implementation for all our experiments.
Revised: To facilitate experimenting with different switching strategies we use this 2-stage implementation for all our experiments.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: First, we compare our evaluation framework with linear and a combination of MLP evaluations using vision encoders pre-trained on ImageNet (Deng et al., 2009) with different objectives.
Revised: First, we compare our evaluation framework with linear and a combination of MLP evaluations using vision encoders pre-trained on ImageNet (Deng et al., 2009) with different objectives.

Original: For this purpose, we use the VTAB benchmark (Zhai et al., 2019) as downstream tasks.
Revised: For this purpose, we use the VTAB benchmark (Zhai et al., 2019) as downstream tasks.

Original: Then, we use
Revised: Then, we use

Original: ImageNet classification as downstream task to showcase the insights, such as data efficiency and preferred readout model, and for comparing in-domain and out-of-domain transfer, revealed by our evaluation method.
Revised: ImageNet classification as downstream task to showcase the insights, such as data efficiency and preferred readout model, and for comparing in-domain and out-of-domain transfer, revealed by our evaluation method.

Original: The VTAB benchmark consists of 19 datasets that can be grouped into three categories: natural, specialized and structured.
Revised: The VTAB benchmark consists of 19 datasets that can be grouped into three categories: natural, specialized and structured.

Original: For some VTAB datasets, notably KittiDistance, SmallNORBAzimuth,
Revised: For some VTAB datasets, notably KittiDistance, SmallNORBAzimuth,

Original: PatchCamelyon, we discover that there is a distribution shift between train and test subsets (see details in Appendix A.2).
Revised: PatchCamelyon, we discover that there is a distribution shift between train and test subsets (see details in Appendix A.2).

Original: Therefore, in order to make sure that data are truly i.i.d, we reshuffle the train and test data and use the reshuffled datasets for all our experiments.
Revised: Therefore, in order to make sure that data are truly i.i.d, we reshuffle the train and test data and use the reshuffled datasets for all our experiments.

Original: To summarize the performance over multiple datasets, we follow the procedure recommended by
Revised: To summarize the performance over multiple datasets, we follow the procedure recommended by

Original: Demˇsar (2006) and report the average rank of the representations: for each dataset, we rank the representations by the evaluation metric (accuracy or MDL score); then, we compute the average rank of a representation over the 19 datasets.
Revised: Demˇsar (2006) and report the average rank of the representations: for each dataset, we rank the representations by the evaluation metric (accuracy or MDL score); then, we compute the average rank of a representation over the 19 datasets.

Original: For any pair of representations A and B , the difference of the average rank can serve as a statistic for a hypothesis test with H0:
Revised: For any pair of representations A and B , the difference of the average rank can serve as a statistic for a hypothesis test with H0:

Original: A performs the same as
Revised: A performs the same as

Original: B .
Revised: B .

Original: If the hypothesis is rejected, we conclude the representations perform significantly different.
Revised: If the hypothesis is rejected, we conclude the representations perform significantly different.

Original: We adopt the Nemenyi test (Nemenyi, 1963) which suggests that the null-hypothesis is rejected if the average ranks differ by at least a critical value.
Revised: We adopt the Nemenyi test (Nemenyi, 1963) which suggests that the null-hypothesis is rejected if the average ranks differ by at least a critical value.

Original: This critical value is computed as q γ (cid:112) r ( r + 1) / 6 N , where q γ is a constant depending on the confidence level γ , r is the number of representations to compare and N is the number of datasets.
Revised: This critical value is computed as q γ (cid:112) r ( r + 1) / 6 N , where q γ is a constant depending on the confidence level γ , r is the number of representations to compare and N is the number of datasets.

Original: In the following comparisons, we also report the critical value with a confidence level of γ = 0 . 1 .
Revised: In the following comparisons, we also report the critical value with a confidence level of γ = 0 . 1 .

Original: We use the online learning framework as described in Section 3 and compute the Fixed Share code length with decreasing switching rate α t defined in (2).
Revised: We use the online learning framework as described in Section 3 and compute the Fixed Share code length with decreasing switching rate α t defined in (2).

Original: The raw MDL scores per dataset are reported in Appendix A.7.
Revised: The raw MDL scores per dataset are reported in Appendix A.7.

Original: For ImageNet, we use readout model switching between 4 model architectures:
Revised: For ImageNet, we use readout model switching between 4 model architectures:

Original: Linear and 1-3 layer MLPs.
Revised: Linear and 1-3 layer MLPs.

Original: For VTAB, we include up to 7-layer MLP’s as readout architectures.
Revised: For VTAB, we include up to 7-layer MLP’s as readout architectures.

Original: We furthermore instantiate each of these architectures with all hyperparameter settings from the hyperparameter grid outlined in Appendix A.5.
Revised: We furthermore instantiate each of these architectures with all hyperparameter settings from the hyperparameter grid outlined in Appendix A.5.

Original: The model switching is therefore performed between
Revised: The model switching is therefore performed between

Original: K =
Revised: K =

Original: # architectures × # hyperparameters many independent models.
Revised: # architectures × # hyperparameters many independent models.

Original: We report results using a switching rate m = 2 for ImageNet and m = 11 for VTAB.
Revised: We report results using a switching rate m = 2 for ImageNet and m = 11 for VTAB.

Original: The switching rates were chosen to optimize the overall description length.
Revised: The switching rates were chosen to optimize the overall description length.

Original: Pre-training Objectives.
Revised: Pre-training Objectives.

Original: We compare downstream performance of different pre-training objectives on the VTAB benchmark.
Revised: We compare downstream performance of different pre-training objectives on the VTAB benchmark.

Original: To make the comparison, we choose two backbone architectures,
Revised: To make the comparison, we choose two backbone architectures,

Original: ResNet-50 v1 (He et al., 2015) and ViT/
Revised: ResNet-50 v1 (He et al., 2015) and ViT/

Original: B16 (Dosovitskiy et al., 2020), and only change the pretraining objectives.
Revised: B16 (Dosovitskiy et al., 2020), and only change the pretraining objectives.

Original: For ResNet-50, we pre-train on supervised SimCLR and BYOL losses; for
Revised: For ResNet-50, we pre-train on supervised SimCLR and BYOL losses; for

Original: ViT/
Revised: ViT/

Original: B16, we pre-train on supervised (Touvron et al., 2020; He et al., 2021) DINO and MAE losses.
Revised: B16, we pre-train on supervised (Touvron et al., 2020; He et al., 2021) DINO and MAE losses.

Original: SimCLR (Chen et al., 2020) is a self-supervised method based on contrastive learning.
Revised: SimCLR (Chen et al., 2020) is a self-supervised method based on contrastive learning.

Original: BYOL (Grill et al., 2020) and DINO (Caron et al., 2021)
Revised: BYOL (Grill et al., 2020) and DINO (Caron et al., 2021)

Original: both use a teacher network.
Revised: both use a teacher network.

Original: BYOL is a negative-free method for self-supervised representation, while DINO uses the ViT architecture and learns through self-distillation.
Revised: BYOL is a negative-free method for self-supervised representation, while DINO uses the ViT architecture and learns through self-distillation.

Original: MAE (He et al., 2021) uses masked reconstruction tasks to learn representation.
Revised: MAE (He et al., 2021) uses masked reconstruction tasks to learn representation.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: Table 1a reports the average rank of the pre-training objectives comparing model selection with different evaluation methods.
Revised: Table 1a reports the average rank of the pre-training objectives comparing model selection with different evaluation methods.

Original: As baselines, we report linear evaluation and a combination of MLPs up to 7-layers.
Revised: As baselines, we report linear evaluation and a combination of MLPs up to 7-layers.

Original: To combine results from MLPs, we take the best performance achieved by MLPs.
Revised: To combine results from MLPs, we take the best performance achieved by MLPs.

Original: Given a confidence interval γ = 0 .
Revised: Given a confidence interval γ = 0 .

Original: 1 , the performance of the representations are significantly different if the difference is above q 0 .
Revised: 1 , the performance of the representations are significantly different if the difference is above q 0 .

Original: 1 (cid:112) r
Revised: 1 (cid:112) r

Original: ( r + 1) / 6 N = 1 . 894 , where q 0 . 1 = 3 .
Revised: ( r + 1) / 6 N = 1 . 894 , where q 0 . 1 = 3 .

Original: 12 is the critical value for the two-tailed Nemenyi test, and r = 6 is the number of representations to compare and N = 19 is the number of datasets.
Revised: 12 is the critical value for the two-tailed Nemenyi test, and r = 6 is the number of representations to compare and N = 19 is the number of datasets.

Original: Despite fixing the backbone architecture, the choice of readout model has an impact on model selection under accuracy-based evaluations (first two methods in Table 1a).
Revised: Despite fixing the backbone architecture, the choice of readout model has an impact on model selection under accuracy-based evaluations (first two methods in Table 1a).

Original: This is consistent with previous observations (Resnick et al., 2019;
Revised: This is consistent with previous observations (Resnick et al., 2019;

Original: He et al., 2021) and demonstrates the importance of incorporating a multitude of readout methods into the evaluation framework.
Revised: He et al., 2021) and demonstrates the importance of incorporating a multitude of readout methods into the evaluation framework.

Original: Our method solves this issue by combining probing protocols and taking into account their complexity.
Revised: Our method solves this issue by combining probing protocols and taking into account their complexity.

Original: Furthermore, under our evaluation framework the performance in the small data regime is also an important factor and representations performing worse with little data are penalized in the final rank.
Revised: Furthermore, under our evaluation framework the performance in the small data regime is also an important factor and representations performing worse with little data are penalized in the final rank.

Original: We observe that for ResNet50 backbone, BYOL features outperforms both SUPervised and SimCLR features, while
Revised: We observe that for ResNet50 backbone, BYOL features outperforms both SUPervised and SimCLR features, while

Original: SimCLR and SUPervised features have no significant difference.
Revised: SimCLR and SUPervised features have no significant difference.

Original: For ViT/
Revised: For ViT/

Original: B16 backbone, DINO features outperform both supervised and MAE, while supervised and MAE have no significant difference.
Revised: B16 backbone, DINO features outperform both supervised and MAE, while supervised and MAE have no significant difference.

Original: Therefore, it appears that with the same encoder capacity, distillation-based objectives have an advantage in terms of transfer.
Revised: Therefore, it appears that with the same encoder capacity, distillation-based objectives have an advantage in terms of transfer.

Original: DINO seems slightly better than BYOL, but the rank difference is not statistically significant for a clear conclusion.
Revised: DINO seems slightly better than BYOL, but the rank difference is not statistically significant for a clear conclusion.

Original: In Figure 7, we investigate the transfer performance under distribution shift by reporting statistics under each VTAB group.
Revised: In Figure 7, we investigate the transfer performance under distribution shift by reporting statistics under each VTAB group.

Original: We observe that self-supervised representations have a better performance than the supervised counterpart on structured datasets, despite the latter perform decently on natural datasets.
Revised: We observe that self-supervised representations have a better performance than the supervised counterpart on structured datasets, despite the latter perform decently on natural datasets.

Original: MAE generalizes well with dataset shift, but the performance on natural datasets suffers compared to other methods.
Revised: MAE generalizes well with dataset shift, but the performance on natural datasets suffers compared to other methods.

Original: [Table]
Revised: [Table]

Original: Model Scaling.
Revised: Model Scaling.

Original: Another interesting aspect about architecture choice is the model scaling: do larger models result in better performance?
Revised: Another interesting aspect about architecture choice is the model scaling: do larger models result in better performance?

Original: We again use ResNet and ViT backbones pre-trained on ImageNet and evaluate on VTAB, but with increasing model sizes.
Revised: We again use ResNet and ViT backbones pre-trained on ImageNet and evaluate on VTAB, but with increasing model sizes.

Original: For ResNet, we select ResNet-50, ResNet-101 and ResNet152, and pre-train with either SUPervised or BYOL objective.
Revised: For ResNet, we select ResNet-50, ResNet-101 and ResNet152, and pre-train with either SUPervised or BYOL objective.

Original: For ViT, we use
Revised: For ViT, we use

Original: ViT/S16, ViT/B16 and ViT/L16, and pretrain with either SUPervised or MAE objective.
Revised: ViT/S16, ViT/B16 and ViT/L16, and pretrain with either SUPervised or MAE objective.

Original: We rank the performance of the 12 representations and report the average ranks in Table 1b.
Revised: We rank the performance of the 12 representations and report the average ranks in Table 1b.

Original: With confidence interval of 0 . 1 , we obtain a threshold of 3 .
Revised: With confidence interval of 0 . 1 , we obtain a threshold of 3 .

Original: 65 for the difference in average rank to make a statistical claim.
Revised: 65 for the difference in average rank to make a statistical claim.

Original: The scaling experiment more compellingly demonstrates the difference of multiple readout models and the emphasis on performance in the small data regime with our method (Table 1b).
Revised: The scaling experiment more compellingly demonstrates the difference of multiple readout models and the emphasis on performance in the small data regime with our method (Table 1b).

Original: Larger models, such as ResNet-152 and ViT/L16, which perform better with MLP head need to demonstrate their performance in the small data regime to justify the additional capacity used for the readout.
Revised: Larger models, such as ResNet-152 and ViT/L16, which perform better with MLP head need to demonstrate their performance in the small data regime to justify the additional capacity used for the readout.

Original: We can see that scaling up the ResNet architecture, in particular under supervised pre-training, doesn’t result in any improvement in the transfer performance.
Revised: We can see that scaling up the ResNet architecture, in particular under supervised pre-training, doesn’t result in any improvement in the transfer performance.

Original: There is a weak positive scaling effect for
Revised: There is a weak positive scaling effect for

Original: BYOL when scaling to ResNet-101, but it is not statistically significant.
Revised: BYOL when scaling to ResNet-101, but it is not statistically significant.

Original: Although ViT/S16 has approximately the same parameter counts as ResNet-50, its predictive performance is much worse, suggesting that ViT’s capacity is used less efficiently than ResNet’s.
Revised: Although ViT/S16 has approximately the same parameter counts as ResNet-50, its predictive performance is much worse, suggesting that ViT’s capacity is used less efficiently than ResNet’s.

Original: Scaling up ViT from S16 to B leads to a significant performance improvement.
Revised: Scaling up ViT from S16 to B leads to a significant performance improvement.

Original: However, further scaling up for supervised loss is not useful.
Revised: However, further scaling up for supervised loss is not useful.

Original: Interestingly, the reconstruction-based MAE objective scales well to larger architectures and has not saturated at ViT/L16.
Revised: Interestingly, the reconstruction-based MAE objective scales well to larger architectures and has not saturated at ViT/L16.

Original: Natural
Revised: Natural

Original: Special
Revised: Special

Original: Structured
Revised: Structured

Original: A r c h it ec t u r e
Revised: A r c h it ec t u r e

Original: O b j ec ti v e
Revised: O b j ec ti v e

Original: C a lt ec h
Revised: C a lt ec h

Original: C i f a r
Revised: C i f a r

Original: D T D F l o w e r s
Revised: D T D F l o w e r s

Original: P e t s S VHN S
Revised: P e t s S VHN S

Original: un397 E u r o S
Revised: un397 E u r o S

Original: A T
Revised: A T

Original: P a t c h C a m e l yon
Revised: P a t c h C a m e l yon

Original: R e s i s c
Revised: R e s i s c

Original: R e ti nop a t hy
Revised: R e ti nop a t hy

Original: C l e v r C oun t
Revised: C l e v r C oun t

Original: C l e v r D
Revised: C l e v r D

Original: i s t a n ce
Revised: i s t a n ce

Original: D M L a b
Revised: D M L a b

Original: D S p r it e s L
Revised: D S p r it e s L

Original: o ca ti on
Revised: o ca ti on

Original: D S p r it e s O r
Revised: D S p r it e s O r

Original: i e n t a ti on
Revised: i e n t a ti on

Original: K itti D i s t a n ce
Revised: K itti D i s t a n ce

Original: S m a ll
Revised: S m a ll

Original: NO RB
Revised: NO RB

Original: A z i m u t h
Revised: A z i m u t h

Original: S 
Revised: S m

Original: [Table]
Revised: [Table]

Original: Readout Models.
Revised: Readout Models.

Original: One strength of our evaluation framework is that it can compare readout models of different capacity and dynamically choose the most suitable one based on downstream task and training data size.
Revised: One strength of our evaluation framework is that it can compare readout models of different capacity and dynamically choose the most suitable one based on downstream task and training data size.

Original: We compute the posterior over the readout models to inspect which readout model the representation prefers.
Revised: We compute the posterior over the readout models to inspect which readout model the representation prefers.

Original: We observe that model switching is more frequent for out-of-domain datasets than in-domain datasets.
Revised: We observe that model switching is more frequent for out-of-domain datasets than in-domain datasets.

Original: In Figure 2, we report the readout model that has most often the highest probability arg max k 1 T (cid:80)
Revised: In Figure 2, we report the readout model that has most often the highest probability arg max k 1 T (cid:80)

Original: [Equation]
Revised: Tt =1 p ( ξ t = k |D <t ) .

Original: The result suggests that the most important factor for preferring deeper models is the dataset type - structured datasets need deeper models.
Revised: The result suggests that the most important factor for preferring deeper models is the dataset type - structured datasets need deeper models.

Original: The influence of the pre-training objective doesn’t seem to be significant.
Revised: The influence of the pre-training objective doesn’t seem to be significant.

Original: Data Efficiency.
Revised: Data Efficiency.

Original: Data efficiency is an integral part of MDL.
Revised: Data efficiency is an integral part of MDL.

Original: With the prequential perspective on MDL, we can visualize whether an algorithm is more data efficient by inspecting the predictive loss as a function of the data size.
Revised: With the prequential perspective on MDL, we can visualize whether an algorithm is more data efficient by inspecting the predictive loss as a function of the data size.

Original: As a baseline we train a ResNet-50 model from scratch within our evaluation framework, and record the cumulative next-step losses as a reference  baseline (comparator).
Revised: As a baseline we train a ResNet-50 model from scratch within our evaluation framework, and record the cumulative next-step losses as a reference [Figure] baseline (comparator).

Original: Then, we evaluate BYOL pretrained ResNet-50 representations with 3 different read- readout but with fine-tuning the whole encoder.
Revised: Then, we evaluate BYOL pretrained ResNet-50 representations with 3 different readout models: 1. linear probing; 2. deep MLP readout models; 3. linear readout but with fine-tuning the whole encoder.

Original: Because MDL already takes into account the model complexity, we can directly compare their MDL scores.
Revised: Because MDL already takes into account the model complexity, we can directly compare their MDL scores.

Original: We plot the “regret” (in this case it is more of an advantage) of different fine-tuning strategies relative to the training-from-scratch model as the baseline.
Revised: We plot the “regret” (in this case it is more of an advantage) of different fine-tuning strategies relative to the training-from-scratch model as the baseline.

Original: Figure 3 illustrates the “regret” along the data size for ImageNet, an in-domain dataset, and PatchCamelyon, an out-ofdomain dataset for the BYOL representations.
Revised: Figureillustrates the “regret” along the data size for ImageNet, an in-domain dataset, and PatchCamelyon, an out-of-domain dataset for the BYOL representations.

Original: Learning ImageNet classification with a pre-trained representation is much more efficient than training from scratch.
Revised: Learning ImageNet classification with a pre-trained representation is much more efficient than training from scratch.

Original: Shallow architectures, which are based on a frozen encoder, are sufficient.
Revised: Shallow architectures, which are based on a frozen encoder, are sufficient.

Original: Only when the dataset is large enough (800k in case of ImageNet), we can finally see the benefit from fine-tuning the encoder.
Revised: Only when the dataset is large enough (800k in case of ImageNet), we can finally see the benefit from fine-tuning the encoder.

Original: The picture is different for out-of-domain datasets.
Revised: The picture is different for out-of-domain datasets.

Original: In general, pre-training still has benefits if deeper, more flexible readout models are used.
Revised: In general, pre-training still has benefits if deeper, more flexible readout models are used.

Original: The amount of capacity needed depends on the nature of the downstream dataset.
Revised: The amount of capacity needed depends on the nature of the downstream dataset.

Original: In Figure 3, on PatchCamelyon, training from scratch surpasses linear probing after seeing only 150k examples.
Revised: In Figure 3, on PatchCamelyon, training from scratch surpasses linear probing after seeing only 150k examples.

Original: We show the regret plots for all 19 VTAB tasks in Appendix A.9.
Revised: We show the regret plots for all 19 VTAB tasks in Appendix A.9.

Original:  C ONCLUSIONS & F UTURE WORKS
Revised:  C ONCLUSIONS & F UTURE WORKS

Original: [Figure] We presented readout model switching, an approach derived from the MDL principle, to evaluate representations on given downstream tasks.
Revised: We presented readout model switching, an approach derived from the MDL principle, to evaluate representations on given downstream tasks.

Original: Our approach supports model switching to choose the most appropriate readout model for each representation, downstream task, and data size.
Revised: Our approach supports model switching to choose the most appropriate readout model for each representation, downstream task, and data size.

Original: Our metric takes model complexity and data efficiency into account, making it a unified measure for model comparison.
Revised: Our metric takes model complexity and data efficiency into account, making it a unified measure for model comparison.

Original: We evaluated vision representations and compared our approach to accuracy-based methods.
Revised: We evaluated vision representations and compared our approach to accuracy-based methods.

Original: Using our evaluation framework, we were also able to provide insights into model scaling properties and preferred readout models, as well as compare data efficiency.
Revised: Using our evaluation framework, we were also able to provide insights into model scaling properties and preferred readout models, as well as compare data efficiency.

Original: In theory, different permutations of the data sequence result in different codelengths.
Revised: In theory, different permutations of the data sequence result in different codelengths.

Original: Although this does not appear to cause too much trouble in practice (see Appendix A.7), we reckon that this is an area for future work.
Revised: Although this does not appear to cause too much trouble in practice (see Appendix A.7), we reckon that this is an area for future work.

Original: With the MDL formulation the performance metric under consideration is the log-loss (cross-entropy), which directly corresponds to compression.
Revised: With the MDL formulation the performance metric under consideration is the log-loss (cross-entropy), which directly corresponds to compression.

Original: Extensions to other losses such as 0/1-loss etc. will need careful consideration.
Revised: Extensions to other losses such as 0/1-loss etc. will need careful consideration.

Original: We here report a single metric per downstreamtask and representation.
Revised: We here report a single metric per downstream-task and representation.

Original: MDL supports non-stationary sequences and future work could aim for reporting a single metric per sequence-of-tasks by concatenating them.
Revised: MDL supports non-stationary sequences and future work could aim for reporting a single metric per sequence-of-tasks by concatenating them.

Original: However this poses questions about the order of downstream tasks, and how to deal with multiple output spaces Y , and whether forward transfer between these tasks is permissible.
Revised: However this poses questions about the order of downstream tasks, and how to deal with multiple output spaces Y , and whether forward transfer between these tasks is permissible.

Original: A A PPENDIX A. P ROOF OF R EGRET B OUND
Revised: A A PPENDIX A. P ROOF OF R EGRET B OUND

Original: Assume that x ∼ P and a sequence of examples x N = { x 1 , . . .
Revised: Assume that x ∼ P and a sequence of examples x N = { x 1 , . . .

Original: , x N } are i.i.d samples from P .
Revised: , x N } are i.i.d samples from P .

Original: Let ˆ θ ( x N ) be the parameters of a readout model M that best fit the sequence x N , i.e. ˆ θ ( x N ) = inf θ M ( x N ; θ ) .
Revised: Let ˆ θ ( x N ) be the parameters of a readout model M that best fit the sequence x N , i.e. ˆ θ ( x N ) = inf θ M ( x N ; θ ) .

Original: Since x N are i.i.d samples from P , ˆ θ ( x N ) is the ML estimator of x N .
Revised: Since x N are i.i.d samples from P , ˆ θ ( x N ) is the ML estimator of x N .

Original: We assume that all readout models are from exponential family.
Revised: We assume that all readout models are from exponential family.

Original: But we do not assume P ∈ M , i.e. the model can be misspecifed.
Revised: But we do not assume P ∈ M , i.e. the model can be misspecifed.

Original: Define the expected regret as the regret w.r.t the best sequence of readout head ξ ∗ 1: N and their best fit parameters ˆ θ ξ ∗ 1: N ( x N ) for x N :
Revised: Define the expected regret as the regret w.r.t the best sequence of readout head ξ ∗ 1: N and their best fit parameters ˆ θ [Equation] 1: N ( x N ) for x N :

Original: [Equation]
Revised: [Equation]

Original: Since the readout models are from exponential family, they can be parameterized instead by the mean parameters.
Revised: Since the readout models are from exponential family, they can be parameterized instead by the mean parameters.

Original: Recall that the exponential family is the probability distribution that can be written in the form of P ( x ; η )
Revised: Recall that the exponential family is the probability distribution that can be written in the form of P ( x ; η )

Original: = h ( x ) exp (cid:0) η ⊤ T ( x ) − A
Revised: = h ( x ) exp (cid:0) η ⊤ T ( x ) − A

Original: ( η ) (cid:1) where η is the natural parameters, T ( x ) is the sufficient statistics, A ( η ) is the cumulant function.
Revised: ( η ) (cid:1) where η is the natural parameters, T ( x ) is the sufficient statistics, A ( η ) is the cumulant function.

Original: The mean parameters are µ :=
Revised: The mean parameters are µ :=

Original: E
Revised: E

Original: [ T ( x )] and have a one-to-one mapping to natural parameters η (we use η ( µ ) to denote the mapping).
Revised: [ T ( x )] and have a one-to-one mapping to natural parameters η (we use η ( µ ) to denote the mapping).

Original: Therefore, the exponential family can also be parameterized by the mean parameters
Revised: Therefore, the exponential family can also be parameterized by the mean parameters

Original: P ( x ; µ ) = h ( x ) exp (cid:0) η ( µ ) ⊤ T ( x ) − A ( η ( µ )) (cid:1) .
Revised: P ( x ; µ ) = h ( x ) exp (cid:0) η ( µ ) ⊤ T ( x ) − A ( η ( µ )) (cid:1) .

Original: We also know that the sufficient statistics is also an exponential family in the form of P ( T ; µ )
Revised: We also know that the sufficient statistics is also an exponential family in the form of P ( T ; µ )

Original: =
Revised: =

Original: h ( T ) exp (cid:0) η ( µ ) ⊤ T − A ( η ( µ )) (cid:1) .
Revised: h ( T ) exp (cid:0) η ( µ ) ⊤ T − A ( η ( µ )) (cid:1) .

Original: Because of the assumption that M is exponential family model, E x ∼ P [log M ( x ; θ )]
Revised: Because of the assumption that M is exponential family model, E x ∼ P [log M ( x ; θ )]

Original: =
Revised: =

Original: E T ∼ P [log M ( T ; µ )] .
Revised: E T ∼ P [log M ( T ; µ )] .

Original: Here, we use the same P as the distribution induced on T .
Revised: Here, we use the same P as the distribution induced on T .

Original: The maximum likelihood estimator of the mean parameters is ˆ µ ( x N ) = 1 N (cid:80)
Revised: The maximum likelihood estimator of the mean parameters is ˆ µ ( x N ) = 1 N (cid:80)

Original: Nt =1 T ( x t ) .
Revised: Nt =1 T ( x t ) .

Original: Given any estimator ˆ µ , we can Taylor expand the likelihood at µ = E P [ T ( x )] : 
Revised: Given any estimator ˆ µ , we can Taylor expand the likelihood at µ = E P [ T ( x )] : [Equation] ˆ µ

Original: [Equation]
Revised: [Equation]

Original: Where I ( µ ) is the Fisher information.
Revised: Where I ( µ ) is the Fisher information.

Original: We first use E P [ ∇ ˆ µ log M ( x, ˆ µ )] | µ = ∇ ˆ µ E P [log M ( x, ˆ µ )] | µ = 0 .
Revised: We first use E P [ ∇ ˆ µ log M ( x, ˆ µ )] | µ = ∇ ˆ µ E P [log M ( x, ˆ µ )] | µ = 0 .

Original: Then we use I ( µ ) =
Revised: Then we use I ( µ ) =

Original: Cov − 1 M µ [ T ( x )] , RHS is the covariance matrix under the distribution M µ .
Revised: Cov − 1 M µ [ T ( x )] , RHS is the covariance matrix under the distribution M µ .

Original: For a readout model M , after seeing x t examples, we have
Revised: For a readout model M , after seeing x t examples, we have

Original: [Equation]
Revised: [Equation]

Original: We use the fact that trace is a linear operator, therefore we could exchange the order with expectation.
Revised: We use the fact that trace is a linear operator, therefore we could exchange the order with expectation.

Original: The last step comes from ˆ µ ( x t − 1 ) =
Revised: The last step comes from ˆ µ ( x t − 1 ) =

Original: 1 t − 1 (cid:80) i T ( x i ) ; denote T i
Revised: 1 t − 1 (cid:80) i T ( x i ) ; denote T i

Original: = T ( x i ) , we have
Revised: = T ( x i ) , we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: [ T [Equation] x )] .Note here the covariance is under the true data distribution P .
Revised: ] .Note here the covariance is under the true data distribution P .

Original: For each term inside the sum over timestep t in Equation (4), we have
Revised: For each term inside the sum over timestep t in Equation (4), we have

Original: [Equation]
Revised: [Equation]

Original: We first change it from parameterized by natural parameters to mean parameters.
Revised: We first change it from parameterized by natural parameters to mean parameters.

Original: Then for each term, we can compare it to µ = E P [ T ( x )] .
Revised: Then for each term, we can compare it to µ = E P [ T ( x )] .

Original: The total sum of t = 1 , . .
Revised: The total sum of t = 1 , . .

Original: , T in Equation (4) is
Revised: , T in Equation (4) is

Original: [Equation]
Revised: [Equation]

Original: Inequality in line 2 uses that When t = 1 , we use the prior and incur a constant regret.
Revised: Inequality in line 2 uses that When t = 1 , we use the prior and incur a constant regret.

Original: Inequality in line 5 uses harmonic series (cid:80) Ni =1 1 i
Revised: Inequality in line 5 uses harmonic series (cid:80) Ni =1 1 i

Original: = log N + const .
Revised: = log N + const .

Original: Note that if there is no model misspecification in any of the readout models, we have M µ ∗ k = P .
Revised: Note that if there is no model misspecification in any of the readout models, we have M µ ∗ k = P .

Original: In this case, Cov − 1 M µ ∗ k
Revised: In this case, Cov − 1 M µ ∗ k

Original: [ T k ( x )]
Revised: [ T k ( x )]

Original: Cov P [ T k ( x )]
Revised: Cov P [ T k ( x )]

Original: =
Revised: =

Original: I and Tr ( I ) =
Revised: I and Tr ( I ) =

Original: k .
Revised: k .

Original: We find the classical result for regret of prequential code k 2 log N .
Revised: We find the classical result for regret of prequential code k 2 log N .

Original: Combining with Equation (4), the expected regret is then
Revised: Combining with Equation (4), the expected regret is then

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The expected regret for our readout model switching is composed of 2 terms: one is the regret from expert switching algorithm and the other is the regret of the readout models.
Revised: The expected regret for our readout model switching is composed of 2 terms: one is the regret from expert switching algorithm and the other is the regret of the readout models.

Original: The growth of second term is well behaved and bounded by C log N where C is a constant that relates to the model family chosen.
Revised: The growth of second term is well behaved and bounded by C log N where C is a constant that relates to the model family chosen.

Original: The first term is not necessarily bounded by log N , but relates to the switching strategy chosen.
Revised: The first term is not necessarily bounded by log N , but relates to the switching strategy chosen.

Original: However, for many switching algorithms, such as Bayesian mixture, fixed share with decreasing switching rate and run-length code, − E P [log p ( ξ ∗ 1: N )] is in the order of log N .
Revised: However, for many switching algorithms, such as Bayesian mixture, fixed share with decreasing switching rate and run-length code, − E P [log p ( ξ ∗ 1: N )] is in the order of log N .

Original: In particular, for fixed share with decreasing switching rate, we have the result of − E P [log p ( ξ ∗ 1: N )] ≤ m log K + ( m − 1) log Nm − 1 where m − 1 is the number of switch points (i.e. m is the number of blocks in which the same expert is used).
Revised: In particular, for fixed share with decreasing switching rate, we have the result of − E P [log p ( ξ ∗ 1: N )] ≤ m log K + ( m − 1) log Nm − 1 where m − 1 is the number of switch points (i.e. m is the number of blocks in which the same expert is used).

Original: See Koolen & de Rooij (2013) for proof of the fixed share and more details of the regret bound of other switching strategy.
Revised: See Koolen & de Rooij (2013) for proof of the fixed share and more details of the regret bound of other switching strategy.

Original: A similar proof is given by Veness et al.
Revised: A similar proof is given by Veness et al.

Original: (2012) for the particular case of m = 2 .
Revised: (2012) for the particular case of m = 2 .

Original: A. D ISTRIBUTION S HIFT BETWEEN T RAIN AND T EST IN VTAB
Revised: A. D ISTRIBUTION S HIFT BETWEEN T RAIN AND T EST IN VTAB

Original: Figure 4 shows the top-1 accuracy on train and test subsets with the original VTAB and reshuffled
Revised: Figure 4 shows the top-1 accuracy on train and test subsets with the original VTAB and reshuffled

Original: VTAB datasets.
Revised: VTAB datasets.

Original: For the same dataset, a difference in test set performance between the original and reshuffled datasets suggests data shift between the two versions.
Revised: For the same dataset, a difference in test set performance between the original and reshuffled datasets suggests data shift between the two versions.

Original: KittiDistance have a significant data shift.
Revised: KittiDistance have a significant data shift.

Original: DMLab, PatchCamelyon and SmallNorbAzimuth also have distribution shift but to a much less degree compared to KittiDistance.
Revised: DMLab, PatchCamelyon and SmallNorbAzimuth also have distribution shift but to a much less degree compared to KittiDistance.

Original: To make sure that the dataset is i.i.d, we combine all its subsets, randomly shuffle then partition into the same number of train/test examples.
Revised: To make sure that the dataset is i.i.d, we combine all its subsets, randomly shuffle then partition into the same number of train/test examples.

Original: We only take the training subset for online learning, which results in sequence lengths showed in Table 3.
Revised: We only take the training subset for online learning, which results in sequence lengths showed in Table 3.

Original: Our experiments  are run with the reshuffled version of the VTAB dataset. 
Revised: To ensure the distribution shift doesn’t affect the conclusion, our experiments in the main paper are run with the reshuffled version of the VTAB dataset. However, we provide additional results on the original VTAB datasets in Appendix A.7. [Figure]

Original: A. 2-S TAGE O NLINE AND O FFLINE I MPLEMENTATION
Revised: O NLINE AND O FFLINE I MPLEMENTATION

Original: The 2-stage implementation is straight forward: first stage, we compute the cross entropy losses for all models at each timestep and store the results; second stage, we use model switching to compute the final codelength (see Algorithm 2).
Revised: The 2-stage implementation is straight forward: first stage, we compute the cross entropy losses for all models at each timestep and store the results; second stage, we use model switching to compute the final codelength (see Algorithm 2).

Original: [Table]
Revised: [Table]

Original: Stage1:
Revised: Stage

Original: Require: data D= ( x t , y t )
Revised: Require: data D

Original: Tt =1 ; K readout models; subroutinethat updates the parametersgiven a dataset UpdateParameters1: Initialize: parameters θ θ K an empty list L to store loss per step and per model 2: for do 3: Compute p k y t t p t t
Revised: K readout models; subroutine updates the parameters a dataset UpdateParameters do end

Original: 5: for to do 6:
Revised: 9: return L

Original: Update parameters: θ k ← UpdateParameters
Revised: Require: entropy results from probability p ( ξ strat egy Initialize p

Original: D
Revised: D

Original: 7: end for 8: end for 9: return L Stage 2: Require: Cross entropy results from stage 1 L ; initial model probability p ( ξ 1 ) and switching strategy p (ξ t | ξ t − 1 ) 1: Initialize: s = log p ( ξ 1 ) ; an empty list Q to store posterior p ( ξ t |D <t ) 2: for t 
Revised: Compute log p ( t y t − 1 | x t −

Original: 1 to T do 3: Compute log p ( ξ t , y t − 1 | x t − 1 ) : s← log (cid:80) ξ t − 1 exp (log p 
Revised: : s log (cid:80) ξ t − 1 exp (log p (

Original: A. A CCURACY - BASED E VALUATIONS
Revised: A. A CCURACY - BASED E VALUATIONS

Original: For both linear and a combination of MLP evaluations, we perform a hyperparameter search on learning rate, weight decay and EMA with a batch size 1024 (see Table 4 for details).
Revised: For both linear and a combination of MLP evaluations, we perform a hyperparameter search on learning rate, weight decay and EMA with a batch size 1024 (see Table 4 for details).

Original: We split the dataset into training and validation where the validation set contains 10% of the data.
Revised: We split the dataset into training and validation where the validation set contains 10% of the data.

Original: We choose the best performing hyperparameter on validation set and report top-1 accuracy of the test set in
Revised: We choose the best performing hyperparameter on validation set and report top-1 accuracy of the test set in

Original: Table 1a and Table 1b.
Revised: Table 1a and Table 1b

Original: We treat the readout models type as one of the hyperparameters, choose the one with the lowest loss on validation set and report the test set accuracy.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Average accuracy v.s. Ranking on multiple datasets.
Revised: Average accuracy v.s. Ranking on multiple datasets.

Original: Table 5 shows the difference on model selection when using average accuracy and average ranking for aggregating performances on multiple datasets.
Revised: Table 5 shows the difference on model selection when using average accuracy and average ranking for aggregating performances on multiple datasets.

Original: We see that the two summarization methods don’t have the same results for model selection.
Revised: We see that the two summarization methods don’t have the same results for model selection.

Original: The reason why it’s problematic to average accuracy over multiple datasets is discussed in Dem ˇ sar (2006).
Revised: The reason why it’s problematic to average accuracy over multiple datasets is discussed in Dem ˇ sar (2006).

Original: In brief, averaging over accuracy is only meaningful when the results on different datasets are comparable and averages are susceptible to outliers.
Revised: In brief, averaging over accuracy is only meaningful when the results on different datasets are comparable and averages are susceptible to outliers.

Original: Since the VTAB benchmark consists of 19 datasets for classification tasks of very different nature, averaging over the accuracy is an ill-conceived practice and we choose to use average ranking for presenting our results.
Revised: Since the VTAB benchmark consists of 19 datasets for classification tasks of very different nature, averaging over the accuracy is an ill-conceived practice and we choose to use average ranking for presenting our results.

Original: A. I MPLEMENTATION D ETAILS A.5. VTAB
Revised: A. I MPLEMENTATION D ETAILS A.5. VTAB

Original: Frozen Encoders For frozen encoders, we use the readout models: linear layer and MLP of 17 hidden layers.
Revised: Frozen Encoders For frozen encoders, we use the readout models: linear layer and MLP of 17 hidden layers.

Original: We use AdamW as the optimizer and hyperparameters listed in Table 6.
Revised: We use AdamW as the optimizer and hyperparameters listed in Table 6.

Original: The switching probability for fix share if α t = 10 /t .
Revised: The switching probability for fix share if α t = 10 /t .

Original: Images are center cropped and resized to 224 × 224 .
Revised: Images are center cropped and resized to 224 × 224 .

Original: [Table]
Revised: [Table]

Original: Train from Scratch
Revised: Train from Scratch

Original: This section correspond to the training from scratch experiment on VTAB presented in Section 4.
Revised: This section correspond to the training from scratch experiment on VTAB presented in Section 4.

Original: We use SGD with nesterov momentum for training from scratch.
Revised: We use SGD with nesterov momentum for training from scratch.

Original: We take the center crop and resize the image to 224 × 224 for preprocessing.
Revised: We take the center crop and resize the image to 224 × 224 for preprocessing.

Original: No readout models is needed as the encoder is trained on the relevant task.
Revised: No readout models is needed as the encoder is trained on the relevant task.

Original: For each VTAB task, we uniformly randomly select 300 configurations from the hyperparameters listed in Table
Revised: For each VTAB task, we uniformly randomly select 300 configurations from the hyperparameters listed in Table

Original: [Table]
Revised: [Table]

Original: -tuning This section correspond to experiment on VTAB in

Original: We use experiment

Original: The encoder parameters are initialized with the pre-trained parameters of BYOL, but updated on the downstream task.

Original: We use SGD with nesterov

Original: Images are center cropped and resized to for preprocessing.

Original: For each

Original: , we uniform randomly select

Original: [Table]
Revised: [Table]

Original: fine-tuning pre-trained encoders on VTAB.
Revised: -tuning pre-trained encoders on VTAB.

Original: A.5. I MAGE N ET
Revised: A.5. I MAGE N ET

Original: Frozen Encoders Images are preprocessed with spatial augmentation during training: randomly cropped, randomly flipped and resized to 224 × 224 .
Revised: Frozen Encoders Images are preprocessed with spatial augmentation during training: randomly cropped, randomly flipped and resized to 224 × 224 .

Original: We use the readout models: linear layer and
Revised: We use the readout models: linear layer and

Original: MLP of 1-3 hidden layers.
Revised: MLP of 1-3 hidden layers.

Original: We use AdamW as the optimizer and hyperparameters listed in Table 9.
Revised: We use AdamW as the optimizer and hyperparameters listed in Table 9.

Original: The switching probability for fix share if α t = 1 /t .
Revised: The switching probability for fix share if α t = 1 /t .

Original: [Table]
Revised: [Table]

Original: Fine-tuning This section correspond to the fine-tuning experiment on ImageNet presented in Section 4.
Revised: Fine-tuning This section correspond to the fine-tuning experiment on ImageNet presented in Section 4.

Original: We use a ResNet-50 encoder for the fine-tuning experiment.
Revised: We use a ResNet-50 encoder for the fine-tuning experiment.

Original: Readout model is a linear layer.
Revised: Readout model is a linear layer.

Original: The encoder parameters are initialized with the pre-trained parameters from BYOL, but updated on the downstream task.
Revised: The encoder parameters are initialized with the pre-trained parameters from BYOL, but updated on the downstream task.

Original: We use AdamW and spatial augmentation for preprocessing.
Revised: We use AdamW and spatial augmentation for preprocessing.

Original: Images are randomly cropped and resized to 224 × 224 .
Revised: Images are randomly cropped and resized to 224 × 224 .

Original: We use the same set of hyperparameters as the frozen encoder experiments listed in Table 9.
Revised: We use the same set of hyperparameters as the frozen encoder experiments listed in Table 9.

Original: Train from Scratch We train a standard ResNet-50 architecture from scratch on ImageNet and follow the experimental setting described in (Bornschein et al., 2022).
Revised: Train from Scratch We train a standard ResNet-50 architecture from scratch on ImageNet and follow the experimental setting described in (Bornschein et al., 2022).

Original: In particular we use replaystreams training with uniform replay, AdamW, use randaugment for data augmentation, and forward calibration.
Revised: In particular we use replaystreams training with uniform replay, AdamW, use randaugment for data augmentation, and forward calibration.

Original: We run a random hyperparameter search by sampling 50 hyperparameter configurations from the intervals listed in appendix A.5.2.
Revised: We run a random hyperparameter search by sampling 50 hyperparameter configurations from the intervals listed in appendix A.5.2.

Original: [Table]
Revised: [Table]

Original: A. C OMPARISON OF S WITCHING S TRATEGIES
Revised: A. C OMPARISON OF S WITCHING S TRATEGIES

Original: We describe briefly other switching strategy we experimented with our evaluation framework:
Revised: We describe briefly other switching strategy we experimented with our evaluation framework:

Original: • Bayesian mixture: switching probability is 1 after the first step.
Revised: • Bayesian mixture: switching probability is 1 after the first step.

Original: [Equation]
Revised: [Equation]

Original: • Elementwise mixture: at each step, it is allowed to switch to another model with a fixed probability which doesn’t depend on the current model used.
Revised: • Elementwise mixture: at each step, it is allowed to switch to another model with a fixed probability which doesn’t depend on the current model used.

Original: [Equation]
Revised: [Equation]

Original: Where w ( ξ t ) is a fixed mixture weight which doesn’t depend on ξ t − 1 .
Revised: Where w ( ξ t ) is a fixed mixture weight which doesn’t depend on ξ t − 1 .

Original: • Switch distribution: A more sophisticated switching strategy which augments the transition with two latent variables u
Revised: • Switch distribution: A more sophisticated switching strategy which augments the transition with two latent variables u

Original: and s for unstable and stable chain respectively.
Revised: and s for unstable and stable chain respectively.

Original: Once switching to the stable chain, it stay with the same model for the rest of the sequence.
Revised: Once switching to the stable chain, it stay with the same model for the rest of the sequence.

Original: For 
Revised: For more

Original: more details, see Koolen & De Rooij (2008).
Revised: details, see Koolen & De Rooij (2008).

Original: Figure 5 compares the above switching strategies with the fixed share with decreasing switching rate on ImageNet using a ResNet50 pre-trained with BYOL.
Revised: Figure 5 compares the above switching strategies with the fixed share with decreasing switching rate on ImageNet using a ResNet50 pre-trained with BYOL.

Original: Bayesian mixture and elementwise mixture perform worse than fixed share by a large margin.
Revised: Bayesian mixture and elementwise mixture perform worse than fixed share by a large margin.

Original: Switch distribution is very close to fixed share performance.
Revised: Switch distribution is very close to fixed share performance.

Original: A. MDL ON VTAB D ATASETS
Revised: A. MDL ON VTAB D ATASETS

Original: We report the final MDL score using fixed share switching strategy on all the VTAB datasets in
Revised: We report the final MDL score using fixed share switching strategy on all the VTAB datasets in

Original: Table 10.
Revised: Table 10.

Original: To better compare the numbers, we average the MDL score by the number of examples in the datasets.
Revised: To better compare the numbers, we average the MDL score by the number of examples in the datasets.

Original: For cumulative MDL score, we should multiple by the number of examples which is reported in Table 3.
Revised: For cumulative MDL score, we should multiple by the number of examples which is reported in Table 3.

Original: [Table]
Revised: [Table]

Original: A r c h it ec t u r e
Revised: A r c h it ec t u r e

Original: O b j ec ti v e
Revised: O b j ec ti v e

Original: C a lt ec h
Revised: C a lt ec h

Original: C i f a r
Revised: C i f a r

Original: D T D
Revised: D T D

Original: F l o w e r s
Revised: F l o w e r s

Original: P e t s
Revised: P e t s

Original: S VHN
Revised: S VHN

Original: S un
Revised: S un

Original: E u r o S
Revised: E u r [Table] S

Original: A T
Revised: A T

Original: P a t c h C a me l yon
Revised: P a t [Table] a m

Original: R e s i s c
Revised: [Table]

Original:  Re ti nop a t hy
Revised: l yon R

Original: C l e v r C oun t C l e v r D
Revised: [Table] s i s [Table]

Original: i s t a n ce
Revised: nop

Original: [Table]
Revised: [Table]

Original: M L a b
Revised: v r

Original: [Table]
Revised: [Table]

Original: S p r it e s L o ca ti
Revised: oun t C l e v r

Original: [Table]
Revised: [Table]

Original: D S p r it e s O r

Original: i e n t a ti
Revised: i s t a n

Original: [Table]
Revised: [Table]

Original: K itti
Revised: M

Original: [Table]
Revised: [Table]

Original: i s t a n
Revised: b

Original: [Table]
Revised: [Table]

Revised: p r it e

Revised: [Table]

Revised: r i e

Revised: [Table]

Revised: s

Revised: [Table]

Original: S m a ll
Revised: S m a ll

Original: NO RB
Revised: NO RB

Original: A z i m u t h
Revised: A z i m u t h

Original: S m a ll
Revised: S m a ll

Original: NO RB
Revised: NO RB

Original: E l e v a ti on
Revised: [Table] l [Table] ti on

Original: [Table]
Revised: [Table]

Original: In VTAB, datasets in the same group are similar to each other, but there is a significant distribution shift between groups.
Revised: In VTAB, datasets in the same group are similar to each other, but there is a significant distribution shift between groups.

Original: And natrual datasets are closer to the ImageNet which is used during pretraining.
Revised: And natrual datasets are closer to the ImageNet which is used during pretraining.

Original: To investigate if the transfer performance differs with input distribution shift, we report the average rankings by dataset group in Figure 7.
Revised: To investigate if the transfer performance differs with input distribution shift, we report the average rankings by dataset group in Figure 7.

Original: Although the number of datasets in each group is limited and, therefore, the result from hypothesis test is not conclusive, we can still get some idea of the transfer performance from the ranking difference.
Revised: Although the number of datasets in each group is limited and, therefore, the result from hypothesis test is not conclusive, we can still get some idea of the transfer performance from the ranking difference.

Original: We can see that self-supervised representations have a better performance than the supervised counterpart on structured datasets, despite the latter perform decently on natural datasets.
Revised: We can see that self-supervised representations have a better performance than the supervised counterpart on structured datasets, despite the latter perform decently on natural datasets.

Original: MAE generalizes well with dataset shift, but the performance on natural datasets suffers compared to other methods.
Revised: MAE generalizes well with dataset shift, but the performance on natural datasets suffers compared to other methods.

Original: [Figure]
Revised: [Figure]

Original: Data ordering and Variance of MDL score.
Revised: Data ordering and Variance of MDL score.

Original: Theoretically, different permutations of the data sequence results in different codelengths.
Revised: Theoretically, different permutations of the data sequence results in different codelengths.

Original: We reckon that this is an area for future work.
Revised: We reckon that this is an area for future work.

Original: Rissanen (1986) suggests taking the permutation with the shortest codelength, but it is computationally too expensive.
Revised: Rissanen (1986) suggests taking the permutation with the shortest codelength, but it is computationally too expensive.

Original: Hansen & Yu (2001) notes that the codelength is invariant over orderings when a
Revised: Hansen & Yu (2001) notes that the codelength is invariant over orderings when a

Original: Bayesian predictor is used in prequential MDL.
Revised: Bayesian predictor is used in prequential MDL.

Original: De Rooij & Gr ¨ unwald (2011) points out that while the ordering of data affects the codelength, it does not appear to cause too much trouble in practice.
Revised: De Rooij & Gr ¨ unwald (2011) points out that while the ordering of data affects the codelength, it does not appear to cause too much trouble in practice.

Original: Here, we provide an empirical evaluation of the variance of our metric.
Revised: Here, we provide an empirical evaluation of the variance of our metric.

Original: We evaluate ResNet50-SUP representation on Cifar100 with 5 permutations and obtain the MDL score of 55906 with a standard deviation of 134.
Revised: We evaluate ResNet50-SUP representation on Cifar100 with 5 permutations and obtain the MDL score of 55906 with a standard deviation of 134.

Original: To give an idea, the representations that are ranked immediately above and below achieve the MDL score of 50969 and 60549 respectively.
Revised: To give an idea, the representations that are ranked immediately above and below achieve the MDL score of 50969 and 60549 respectively.

Original: Therefore, we believe our evaluation method is stable and reliable.
Revised: Therefore, we believe our evaluation method is stable and reliable.

Original: We would also like to point out that the current practice of comparing the test accuracy is not immune to randomness, e.g. initialization, dataset split and randomization training
Revised: We would also like to point out that the current practice of comparing the test accuracy is not immune to randomness, e.g. initialization, dataset split and randomization training.

Revised: MDL Results on original VTAB Datasets Table 11 reports the MDL score on the original VTAB datasets, as well as the average rank.

Revised: The distribution shift results in changes of the MDL score (colored in blue and red in Table 11).

Revised: The datasets where the distribution shift occur are consistent with our previous assessment in Appendix A.2.

Revised: However, the overall rank of the pre-training method doesn’t seem to be affected.

Revised: This is because the number of datasets affected by the shift is relatively small and the effect of the shift is similar to all pre-training methods.

Original:  A. P OSTERIOR OF THE READOUT MODELS
Revised: [Table] ViT/ [Table] A. P OSTERIOR OF THE READOUT MODELS

Original: Figure 9 shows the posterior of readout models p ( ξ t = k |D <t ) where k ∈ [1 , . .
Revised: Figure 9 shows the posterior of readout models p ( ξ t = k |D <t ) where k ∈ [1 , . .

Original: , K ] on ImageNet for 4 different architecture and pre-training methods: ResNet50 + SimCLR, ResNet50 + BYOL,
Revised: , K ] on ImageNet for 4 different architecture and pre-training methods: ResNet50 + SimCLR, ResNet50 + BYOL,

Original: ViT/B16 + DINO and ViT/B16 + MAE.
Revised: ViT/B16 + DINO and ViT/B16 + MAE

Original: We use 4 readout models: linear layer and MLP with 1hidden layers.
Revised: We [Figure]

Original: D ATA E FFICIENCY AND F INE - TUNING
Revised: D ATA E FFICIENCY AND F INE - TUNING

Original: Figure 10 shows the regret plots, relative to training from scratch, for training a linear layer on top of the pre-trained and frozen BYOL backbone, training MLPs (1-7 hidden layers) on top of the frozen BYOL backbone and for a linear readout but while fine-tuning all the encoder parameters.
Revised: Figure 10 shows the regret plots, relative to training from scratch, for training a linear layer on top of the pre-trained and frozen BYOL backbone, training MLPs (1-7 hidden layers) on top of the frozen BYOL backbone and for a linear readout but while fine-tuning all the encoder parameters.

Original: The network architecture is a ResNet-50 in all cases.
Revised: The network architecture is a ResNet-50 in all cases.