Original: R ANK M E : A SSESSING THE D OWNSTREAM P ERFORMANCE OF P RETRAINED S ELF -S UPERVISED R EPRESENTATIONS BY T HEIR R ANK
Revised: R ANK M E : A SSESSING THE D OWNSTREAM P ERFORMANCE OF P RETRAINED S ELF -S UPERVISED R EPRESENTATIONS BY T HEIR R ANK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations and few principled guidelines that would help practitioners to successfully deploy those methods.
Revised: Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations and few principled guidelines that would help practitioners to successfully deploy those methods.

Original: The main reason for that pitfall actually comes from JE-SSL’s core principle of not employing any input reconstruction.
Revised: The main reason for that pitfall actually comes from JE-SSL’s core principle of not employing any input reconstruction.

Original: Without any visual clue, it becomes extremely cryptic to judge the quality of a learned representation without having access to a labelled dataset.
Revised: Without any visual clue, it becomes extremely cryptic to judge the quality of a learned representation without having access to a labelled dataset.

Original: We hope to correct those limitations by providing a single –theoretically motivated– criterion that reflects the quality of learned JE-SSL representations: their effective rank.
Revised: We hope to correct those limitations by providing a single –theoretically motivated– criterion that reflects the quality of learned JE-SSL representations: their effective rank.

Original: Albeit simple and computationally friendly, this method —coined RankMe — allows one to assess the performance of JE-SSL representations, even on different downstream datasets, without requiring any labels, training or parameters to tune.
Revised: Albeit simple and computationally friendly, this method —coined RankMe — allows one to assess the performance of JE-SSL representations, even on different downstream datasets, without requiring any labels, training or parameters to tune.

Original: Through thorough empirical experiments involving hundreds of repeated training episodes, we demonstrate how RankMe can be used for hyperparameter selection with nearly no loss in final performance compared to the current selection method that involve dataset labels.
Revised: Through thorough empirical experiments involving hundreds of repeated training episodes, we demonstrate how RankMe can be used for hyperparameter selection with nearly no loss in final performance compared to the current selection method that involves dataset labels.

Original: We hope that RankMe will facilitate the use of JE-SSL in domains with little or no labeled data.
Revised: We hope that RankMe will facilitate the use of JE-SSL in domains with little or no labeled data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning (SSL) has shown great progress to learn informative data representations in recent years (Chen et al., 2020a;
Revised: Self-supervised learning (SSL) has shown great progress to learn informative data representations in recent years (Chen et al., 2020a;

Original: He et al., 2020; Chen et al., 2020b; Grill et al., 2020; Lee et al., 2021; Caron et al., 2020; Zbontar et al., 2021; Bardes et al., 2021; Tomasev et al., 2022; Caron et al., 2021; Chen et al., 2021; Li et al., 2022a;
Revised: He et al., 2020; Chen et al., 2020b; Grill et al., 2020; Lee et al., 2021; Caron et al., 2020; Zbontar et al., 2021; Bardes et al., 2021; Tomasev et al., 2022; Caron et al., 2021; Chen et al., 2021; Li et al., 2022a;

Original: Zhou et al., 2022a;b; HaoChen et al., 2021; He et al., 2022), catching up to supervised baselines and even surpassing them in few-shot learning, i.e. when evaluating the SSL model from only a few labeled examples.
Revised: Zhou et al., 2022a;b; HaoChen et al., 2021; He et al., 2022), catching up to supervised baselines and even surpassing them in few-shot learning, i.e., when evaluating the SSL model from only a few labeled examples.

Original: Although various SSL families of losses have emerged, most are variants of the joint-embedding (JE) framework with a siamese network architecture (Bromley et al., 1994), denoted as JE-SSL for short.
Revised: Although various SSL families of losses have emerged, most are variants of the joint-embedding (JE) framework with a siamese network architecture (Bromley et al., 1994), denoted as JE-SSL for short.

Original: The only technicality we ought to introduce to make our study precise is the fact that JE-SSL has introduced some different notations to denote an input’s representation.
Revised: The only technicality we ought to introduce to make our study precise is the fact that JE-SSL has introduced some different notations to denote an input’s representation.

Original: In short, JE-SSL often composes a backbone or encoder network e.g. a ResNet50 and a projector network e.g. a multilayer perceptron.
Revised: In short, JE-SSL often composes a backbone or encoder network e.g., a ResNet50 and a projector network e.g., a multilayer perceptron.

Original: This projector is only employed during training, and we refer to its outputs as embeddings , while the actual inputs’ representation employed for downstream tasks are obtained at the encoder’s output.
Revised: This projector is only employed during training, and we refer to its outputs as embeddings , while the actual inputs’ representation employed for downstream tasks are obtained at the encoder’s output.

Original: Although downstream tasks performance of JE-SSL representations might seem impressive, one pondering fact should be noted: all existing methods, hyper-parameters, models — and thus performance — of JE-SSL are obtained by ad-hoc manual search involving the labels of the training samples .
Revised: Although downstream tasks performance of JE-SSL representations might seem impressive, one pondering fact should be noted: all existing methods, hyperparameters, models — and thus performance — of JE-SSL are obtained by ad-hoc manual search involving the labels of the training samples .

Original: In words, JE-SSL is tuned by monitoring the supervised performance of the model at hand.
Revised: In words, JE-SSL is tuned by monitoring the supervised performance of the model at hand.

Original: Hence, although labels are not directly employed to compute the weight updates, they are used as a proxy to signal the JE-SSL designer on how to refine their method.
Revised: Hence, although labels are not directly employed to compute the weight updates, they are used as a proxy to signal the JE-SSL designer on how to refine their method.

Original: This single limitation prevents the deployment of JE-SSL in challenging domains where the number of available labelled examples is limited and such search can not be performed.
Revised: This single limitation prevents the deployment of JE-SSL in challenging domains where the number of available labelled examples is limited and such search can not be performed.

Original: Adding to the challenge, one milestone of JE-SSL is to move away from reconstruction based learning; hence without labels and without visual cues, tuning JE-SSL methods on unlabeled datasets remain challenging.
Revised: Adding to the challenge, one milestone of JE-SSL is to move away from reconstruction based learning; hence without labels and without visual cues, tuning JE-SSL methods on unlabeled datasets remains challenging.

Original: This led to the application of feature inversion methods e.g. Deep Image
Revised: This led to the application of feature inversion methods e.g. Deep Image

Original: Prior (Ulyanov et al., 2018) or conditional diffusion model (Bordes et al., 2021) to be deployed onto learned JE-SSL representation to try to visualize the learned features.
Revised: Prior (Ulyanov et al., 2018) or conditional diffusion models (Bordes et al., 2021) to be deployed onto learned JE-SSL representation to try to visualize the learned features.

Original: This first step towards removing the need for labels has seen some success but is doomed by the computational complexity of the methods, and their biases towards natural images i.e. it is not clear how such methods would perform in different data modalities.
Revised: This first step towards removing the need for labels has seen some success but is doomed by the computational complexity of the methods, and their biases towards natural images i.e. it is not clear how such methods would perform on different data modalities.

Original: In this study, we propose a first attempt at making JE-SSL truly unsupervised by providing the first method —coined RankMe— able to assess a model’s performance without having access to any label and without requiring any training or tuning.
Revised: In this study we propose RankMe, which is able to assess a model’s performance without having access to any labels and without requiring any training or tuning.

Original: RankMe accurately predicts a model’s performance both on In-Distribution (ID), i.e. same data distribution as used during theJE-SSL training, and on
Revised: RankMe accurately predicts a model’s performance both on In-Distribution (ID), i.e., same data distribution as used during the

Original:  Out-Of-Distribution (OOD), i.e. different data distribution scenarios.
Revised: JE-SSL training, and on Out-Of-Distribution (OOD), i.e., different data distribution scenarios.

Original: We highlight this property at the top of fig.
Revised: We highlight this property at the top of fig.

Original: The strength of RankMe lies in the fact that it is solely based on the singular values distribution of the learned representation, and thus does not rely on any parameters that need training, nor requires any ID/OOD labels.
Revised: The strength of RankMe lies in the fact that it is solely based on the singular values distribution of the learned embeddings, and thus does not rely on any parameters that need training, nor requires any ID/OOD labels.

Original: In fact, RankMe’s construction hinges on Cover’s theorem (Cover, 1965) that states how increasing the rank of a linear classifier’s input increases its training performance, and three simple hypotheses that we summarize below and thoroughly validate empirically.
Revised: In fact, RankMe’s motivation hinges on Cover’s theorem (Cover, 1965) that states how increasing the rank of a linear classifier’s input increases its training performance, and three simple hypotheses that we summarize below and thoroughly validate empirically.

Original: As such, RankMe provides asignificant step towards (unlabeled) JE-SSL by allowing practitioners to cross-validate hyper-parameters and select models without resorting to labels or feature inversion methods.
Revised: As such, RankMe provides a step towards (unlabeled) JE-SSL by allowing practitioners to cross-validate hyperparameters and select models without resorting to labels or feature inversion methods.

Original: We hope that RankMe will enable JE-SSL to be deployed even in challenging domains that possess no or little labelled data; we summarize our contributions below:
Revised: We hope that RankMe will enable JE-SSL to be deployed even in challenging domains that possess no or little labelled data; we summarize our contributions below:

Original: We introduce (eq.
Revised: We introduce (eq.

Original: (1)) andtheoretically motivate RankMe which combines Cover’s theorem with the following three key hypotheses: (H1) increasing training performance increases testing performance on both representations and embeddings i.e. no over-fitting is observed from the (non)linear probe, validated  in  bottom left of fig.(H2) embeddings’ rank scale linearly between datasets. 
Revised: and motivate RankMe which combines Cover’s theorem with the following three key hypotheses: (H1) increasing training performance increases testing performance on both representations and embeddings i.e. no over-fitting is observed from the (non)linear probe, validated empirically in the bottom left of fig.(H2) embeddings’ rank scale linearly between datasets. Assuming a pretraining on the same dataset, if a set of embeddings has a greater rank than another on a dataset, it also holds on another one

Revised: validated empirically in the top row

Revised: of fig.

Original: Assuming a pretraining on the same dataset, if a set of embeddings have a greater rank than another on one dataset, it also holds on another one, validated in top row of fig.(H3) increasing embeddings performance increases representations performance , validated  in  bottom right of fig.concluding that embeddings with greater rank will have greater train performance (Cover’s theorem) and test performance (H1) on ID and OOD cases (H2) and even before the projector (H3).
Revised: H3) increasing embeddings performance increases representations performance , validated empirically in the bottom right of fig. concluding that embeddings with greater rank will have greater train performance (Cover’s theorem) and test performance (H1) on ID and OOD cases (H2), even before the projector (H3).

Original: We demonstrate that RankMe’s ability to assess JE-SSL downstream performance is robust across methods e.g. VICReg, SimCLR and their variants, and is also robust to architecture changes e g. using a projector network and/or a nonlinear evaluation method (see fig.
Revised: We demonstrate that RankMe’s ability to assess JE-SSL downstream performance is robust across methods, e.g. VICReg, SimCLR, and their variants, and is also robust to architecture changes, e.g. using a projector network and/or a nonlinear evaluation method (see fig.

Original: 3 and section 3.3).
Revised: 3 and section 3.3).

Original: We demonstrate that RankMe enables hyper-parameter cross-validation for any given JE-SSL method; RankMe is able to retrieve and sometimes surpass most of the performance previously found by manual search using labels, on both in domain and out of domain datasets; see bottom of fig.
Revised: We demonstrate that RankMe enables hyperparameter cross-validation for any given JE-SSL method; RankMe is able to retrieve and sometimes surpass most of the performance previously found by manual search using labels, on both in domain and out of domain datasets; see bottom of fig.

Original: 1 and table 1.
Revised: 1 and table 1.

Original: We provide an hyper-parameter free numerically stable implementation of RankMe in section 3. and pseudo-code for cross-validation in fig.
Revised: We provide a hyperparameter free numerically stable implementation of RankMe in section 3.1 and pseudo-code for cross-validation in fig.

Original: Through extensive experiments involving 6 different datasets and more than 85 trained models over 4 methods, we demonstrate that in the linear and nonlinear probing regime, RankMe is able to tell apart high and low performing models, even on different downstream tasks without having access to labels or downstream task data samples.
Revised: Through extensive experiments involving 11 different datasets and more than 85 trained models over 4 methods, we demonstrate that in the linear and nonlinear probing regime, RankMe is able to tell apart high and low performing models, even on different downstream tasks without having access to labels or downstream task data samples.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Joint embedding self-supervised learning (JE-SSL).
Revised: Joint embedding self-supervised learning (JE-SSL).

Original: In JE-SSL, two main families of method can be distinguished: contrastive and non-contrastive.
Revised: In JE-SSL, two main families of method can be distinguished: contrastive and non-contrastive.

Original: Contrastive methods (Chen et al., 2020a;
Revised: Contrastive methods (Chen et al., 2020a;

Original: He et al., 2020; Chen et al., 2020b; 2021; Yeh et al., 2021) mostly rely on the InfoNCE criterion (Oord et al., 2018) except for HaoChen et al.
Revised: He et al., 2020; Chen et al., 2020b; 2021; Yeh et al., 2021) mostly rely on the InfoNCE criterion (Oord et al., 2018) except for HaoChen et al.

Original: (2021) which uses squared similarities between the embedding.
Revised: (2021) which uses squared similarities between the embedding.

Original: A clustering variant of contrastive learning has also emerged (Caron et al., 2018; 2020; 2021) and can be thought of as contrastive methods but between cluster centroids instead of samples.
Revised: A clustering variant of contrastive learning has also emerged (Caron et al., 2018; 2020; 2021) and can be thought of as contrastive methods, but between cluster centroids instead of samples.

Original: Non-contrastive methods (Grill et al., 2020; Chen & He, 2020; Caron et al., 2021; Bardes et al., 2021; Zbontar et al., 2021; Ermolov et al., 2021; Li et al., 2022b) aim at bringing together embeddings of positive samples, similar to contrastive learning.
Revised: Non-contrastive methods (Grill et al., 2020; Chen & He, 2020; Caron et al., 2021; Bardes et al., 2021; Zbontar et al., 2021; Ermolov et al., 2021; Li et al., 2022b) aim at bringing together embeddings of positive samples, similar to contrastive learning.

Original: However, a key difference with contrastive learning lies in how those methods prevent a representational collapse.
Revised: However, a key difference with contrastive learning lies in how those methods prevent a representational collapse.

Original: In the former, the criterion explicitly pushes away negative samples i.e., all samples that are not positive, from each other.
Revised: In the former, the criterion explicitly pushes away negative samples, i.e., all samples that are not positive, from each other.

Original: In the latter, the criterion 
Revised: In the latter, the criterion does not prevent collapse by distinguishing positive and negative samples, but instead considers the embeddings as a whole and encourages information content maximization e.g., by regularizing

Original: Dataset
Revised: Dataset

Original: Method
Revised: Method

Revised: Labels

Original: VICReg
Revised: VICReg

Original: SimCLR cov.
Revised: SimCLR cov.

Original: inv.
Revised: inv.

Original: LR
Revised: LR

Original: WD temp.
Revised: WD temp.

Original: LR.
Revised: LR.

Original: WD.
Revised: WD.

Original: ImageNet
Revised: ImageNet

Original: ImageNet Oracle [-68 .68 .
-]
Revised: ImageNet Oracle [-68 .68 .
-]✓ 68. α -ReQ X 67.

Original: RankMe[-
-]
Revised: RankMe[-
-]

Original: Difference − 0 .− 0 .− 0 .− 0 .− 1 .− 0 .=
Revised: X 67.

Original: OOD
Revised: OOD

Original: ImageNet Oracle 62 .62 .62 .62 .62 .62 .62 .
Revised: ImageNet Oracle ✓ 67. α -ReQ X 66.

Original: RankMe[-
-]
Revised: RankMe[-
-]

Original: [-Difference − 1 .− 0 .− 
 .= − 0 .− 
-]
Revised: [-Difference − 1 .− 0 .− 
 .= − 0 .− 
-]X 65.

Original: .= does not prevent collapse by distinguishing positive and negative samples, but instead considers the embeddings as a whole and encourage information content maximization e.g. by regularizing the empirical covariance matrix of the embeddings.
Revised: the empirical covariance matrix of the embeddings.

Original: Such a categorization is not needed for our development and we thus refer to any of the above method as JE-SSL.
Revised: Such a categorization is not needed for our development, and we thus refer to any of the above method as JE-SSL.

Original: For completeness, we provide in appendix A the precise definitions of some JE-SSL methods that are VICReg, SimCLR, and their variants which we focus on.

Original: Dimensional collapse in JE-SSL.
Revised: Dimensional collapse in JE-SSL.

Original: The phenomenon of learning rank-deficient embeddings, or dimensional collapse, in JE-SSL has recently been studied from both a theoretical and empirical point of view.
Revised: The phenomenon of learning rank-deficient embeddings, or dimensional collapse, in JE-SSL has recently been studied from both a theoretical and empirical point of view.

Original: The emprivial emergence of dimensional collapse was studied in Hua et al.
Revised: The empirical emergence of dimensional collapse was studied in Hua et al.

Original: (2021) where they proposed the use of a whitening batch normalization layer to help alleviate it.
Revised: (2021) where they proposed the use of a whitening batch normalization layer to help alleviate it.

Original: In Jing et al.
Revised: In Jing et al.

Original: (2022), a focus on contrastive approaches in a linear setting enabled a better understanding of dimensional collapse and the role of augmentations in its emergence.
Revised: (2022), a focus on contrastive approaches in a linear setting enabled a better understanding of dimensional collapse and the role of augmentations in its emergence.

Original: Performance in a low label regime of a partially collapsed encoder can also be improved by forcing the whitening of its output, as shown in He & Ozay (2022).
Revised: Performance in a low label regime of a partially collapsed encoder can also be improved by forcing the whitening of its output, as shown in He & Ozay (2022).

Original: Furthermore, it was shown in Balestriero & LeCun (2022) how dimensional collapse is a phenomenon that should not necessarily happen in theory and how its emergence is mostly due to practical concerns.
Revised: Furthermore, it was shown in Balestriero & LeCun (2022) how dimensional collapse is a phenomenon that should not necessarily happen in theory and how its emergence is mostly due to practical concerns.

Original: Interestingly, we will see through the lens of
Revised: Interestingly, we will see through the lens of

Original: RankMe that while reducing dimensional collapse is often beneficial, doing so “at all cost” can lead to degenerate solutions.
Revised: RankMe that while reducing dimensional collapse is often beneficial, doing so “at all cost” can lead to degenerate solutions.

Original: The collapse induced from training using a softmax layer was also studied in Ganea et al.
Revised: The collapse induced from training using a softmax layer was also studied in Ganea et al.

Original: (2019), where they show that high rank embeddings are desirable.
Revised: (2019), where they show that high rank embeddings are desirable.

Original: Evaluation of JE-SSL representations.
Revised: Evaluation of JE-SSL representations.

Original: Evaluating the representations learned by JE-SSL methods is fundamental to enable the optimal selection of those methods’ hyper-parameters, which are numerous.
Revised: Evaluating the representations learned by JE-SSL methods is fundamental to enable the optimal selection of those methods’ hyperparameters, which are numerous.

Original: Yet, due to the imprecise nature of what makes a good representation, multiple strategies have emerged which evaluate different properties of representations.
Revised: Yet, due to the imprecise nature of what makes a good representation, multiple strategies have emerged which evaluate different properties of representations.

Original: The most common approach relies on the strong assumption of having labels on the dataset where the JE-SSL method is trained on.
Revised: The most common approach relies on the strong assumption of having labels on the dataset where the JE-SSL method is trained on.

Original: In this case, on trains a linear classifier on the JE-SSL representations (Misra & Maaten, 2020) and directly use the test accuracy to compare models.
Revised: In this case, on trains a linear classifier on the JE-SSL representations (Misra & Maaten, 2020) and directly use the test accuracy to compare models.

Original: This method was extended to the use of nonlinear classifiers e.g. a k -nn classifier (Wu et al., 2018; Zhuang et al., 2019).
Revised: This method was extended to the use of nonlinear classifiers, e.g., a k -nn classifier (Wu et al., 2018; Zhuang et al., 2019).

Original: Performance evaluation without labels can also be done using a pretext-task, such as rotation prediction.
Revised: Performance evaluation without labels can also be done using a pretext-task, such as rotation prediction.

Original: This technique helped in selecting data augmentation policies in Reed et al.
Revised: This technique helped in selecting data augmentation policies in Reed et al.

Original: One limitation lies in the need to select and train the classifier of the pretext-task, and on the strong assumption that rotation were not part of the transformations one aimed to be invariant to.
Revised: One limitation lies in the need to select and train the classifier of the pretext-task, and on the strong assumption that rotation were not part of the transformations one aimed to be invariant to.

Original: Since (supervised) linear evaluation is the most widely used evaluation method, we will focus on showing how RankMe compares with it.
Revised: Since (supervised) linear evaluation is the most widely used evaluation method, we will focus on showing how RankMe compares with it.

Revised: Most related to us is Ghosh et al.

Revised: (2022) where representations are evaluated by their eigenspectrum decay, giving a baseline for unsupervised hyperparameter selection.

Original:  R EPRESENTATIONS ’ R ANK C ORRELATE WITH D OWNSTREAM PERFORMANCE A CROSS T ASKS AND M ODELS
Revised:  R EPRESENTATIONS ’ R ANK C ORRELATE WITH D OWNSTREAM PERFORMANCE A CROSS T ASKS AND M ODELS

Original: The goal of this section is to formally introduce and motivate RankMe while providing a numerically stable implementation (section 3.1).
Revised: The goal of this section is to formally introduce and motivate RankMe while providing a numerically stable implementation (section 3.1).

Original: The construction of RankMe hinges on three hypotheses that we validate empirically throughout this section.
Revised: The construction of RankMe hinges on three hypotheses that we validate empirically throughout this section.

Original: 3. R ANK M E : F ROM T HEORY TO I MPLEMENTATION
Revised: 3. R ANK M E : F ROM T HEORY TO I MPLEMENTATION

Original: We first want  build notations and intuition into the construction of RankMe.
Revised: We first want to build notations and intuition into the construction of RankMe.

Original: To that hand, we first quantify approximation and classification errors of learned embeddings as a function of their rank, and then motivate how embeddings’ rank is indeed sufficient to compare test performance of JE-SSL models’s representations.
Revised: To that hand, we first quantify approximation and classification errors of learned embeddings as a function of their rank, and then motivate how embeddings’ rank can be sufficient to compare test performance of JE-SSL models’s representations.

Original: This criterion should however only be used to compare different runs of a given method, since embedding rank is not the only factor that affects performance.
Revised: This criterion should however only be used to compare different runs of a given method, since the embeddings’ rank is not the only factor that affects performance.

Original: To ease notations, we refer to the (train) dataset used to obtain the JE-SSL model as the source dataset , and the test set on the same dataset or a different OOD dataset as the target dataset .
Revised: To ease notations, we refer to the (train) dataset used to obtain the JE-SSL model as the source dataset , and the test set on the same dataset or a different OOD dataset as the target dataset .

Original: From Source Embeddings’s Rank to Target Representations performance.
Revised: From Source Embeddings’s Rank to Target Representations performance.

Original: We first build some intuition in the regression settings.
Revised: We first build some intuition in the regression settings.

Original: In this case, a common linear algebra result ties the best-case and worst-case approximation error of any target matrix Y ∈ R N × C from a rank- R matrix P ∈ R N × C to the singular values of Y that run from R to the rank of Y when ordered in decreasing order.
Revised: In this case, a common linear algebra result ties the best-case and worst-case approximation error of any target matrix Y ∈ R N × C from a rank- R matrix P ∈ R N × C to the singular values of Y that run from R to the rank of Y when ordered in decreasing order.

Original: Without loss of generality, we only consider the case C > N in this study i.e. we have more samples than dimensions.
Revised: Without loss of generality, we only consider the case C > N in this study, i.e., we have more samples than dimensions.

Original: Formally, this provides a lower bound on
Revised: Formally, this provides a lower bound on

Original: [Equation]
Revised: [Equation]

Original:  which is tight for P of rank R , and with σ k the operator returning the k th singular value of its argument, ordered in decreasing order.
Revised: ) , which is tight for P of rank R , and with σ k the operator returning the k th singular value of its argument, ordered in decreasing order.

Original: This result, on which RankMe relies on, demonstrates that a necessary (but not sufficient) condition for an approximation P to well approximate Y is to have at least the same rank as Y .
Revised: This result, on which RankMe relies on, demonstrates that a necessary (but not sufficient) condition for an approximation P to well approximate Y is to have at least the same rank as Y .

Original: A similar result can be obtained in classification by considering multiple one-vs-all classifiers.
Revised: A similar result can be obtained in classification by considering multiple one-vs-all classifiers.

Original: In practice however, we commonly employ a linear probe network on top of given embeddings Z to best adapt them to the target Y  i.e., P = ZW + 1 b T .
Revised: In practice, however, we commonly employ a linear probe network on top of given embeddings Z to best adapt them to the target Y , i.e., P = ZW + 1 b T .

Original: However, a linear transformation is not able to increase the rank of the input matrix since
Revised: However, a linear transformation is not able to increase the rank of the input matrix since

Original: [Equation]
Revised: [Equation]

Original: We directly obtain that min W , b ∥ Y − ZW − [Equation] b T ∥ 2 F ≥ [Equation]
Revised: We directly obtain that min W , b ∥ Y − ZW − 1 b T ∥ 2 F ≥ (cid:80)

Original: Cr
Revised: Cr

Original: [Equation]
Revised: =

Original: R +1{++
+} σ 2 r ( Y ) .
Revised: R {++
+} σ 2 r ( Y ) .

Original: In short, the approximation lower bound is not improved by allowing linear transformation of the embeddings.
Revised: In short, the approximation lower bound is not improved by allowing linear transformation of the embeddings.

Original: Further supporting the above, we ought to recall Cover’s theorem (Cover, 1965) stating that the probability of a randomly labeled set of points being linearly separable only increases if N is reduced or R is increased.
Revised: Further supporting the above, we ought to recall Cover’s theorem (Cover, 1965) stating that the probability of a randomly labeled set of points being linearly separable only increases if N is reduced or R is increased.

Original: We formalize those results below.
Revised: We formalize those results below.

Original: Proposition 1.
Revised: Proposition 1.

Original: The maximum training accuracy of given embeddings in linear regression or classification increases with their rank.
Revised: The maximum training accuracy of given embeddings in linear regression or classification increases with their rank.

Original: For classification, it plateaus when the rank surpasses the number of classes.
Revised: For classification, it plateaus when the rank surpasses the number of classes.

Original: We thus introduce RankMe formally as the following smooth rank measure, originally introduced in Roy & Vetterli (2007),
Revised: We thus introduce RankMe formally as the following smooth rank measure, originally introduced in Roy & Vetterli (2007),

Original: [Equation]
Revised: [Equation]

Original: where Z is the source dataset’s embeddings.
Revised: where Z is the source dataset’s embeddings.

Original: By noticing that RankMe provides a smooth measure of the embeddings’ rank (more details in the implementation section) we can lean on proposition 1 to see that given two models, the one with greater RankMe value will have greater training performance.
Revised: By noticing that RankMe provides a smooth measure of the embeddings’ rank (more details in the implementation section) we can lean on proposition 1 to see that given two models, the one with greater RankMe value will have greater training performance.

Original: This is only guaranteed for different models of the same method, since embedding rank is not necessarily the only factor that affects performance.
Revised: This is only guaranteed for different models of the same method, since embedding rank is not necessarily the only factor that affects performance.

Original: The above result is however not too practical yet since what we are truly interested in are (i) performance on unseen samples i.e. on the test set and out-of-distribution tasks, and (ii) performance on the representations and not the embeddings since it is common to ablate the projector network of JE-SSL models.
Revised: The above result is however not too practical yet since what we are truly interested in are (i) performance on unseen samples, i.e., on the test set and out-of-distribution tasks, and (ii) performance on the representations and not the embeddings since it is common to ablate the projector network of JE-SSL models.

Original: Below, we validate three key hypotheses which, when verified, imply that we can extend the impact of RankMe such that (OOD) test performance of JE-SSL representations are increased when RankMe’s value on their train set embeddings is increased .
Revised: Below, we validate three key hypotheses which, when verified, imply that we can extend the impact of RankMe such that (OOD) test performance of JE-SSL representations are increased when RankMe’s value on their train set embeddings is increased .

Original: Validating RankMe’s
Revised: Validating RankMe’s

Original: Hypotheses The development of RankMe is theoretically grounded when it comes to guaranteeing improved source dataset embeddings performance.
Revised: Hypotheses The development of RankMe is theoretically grounded when it comes to guaranteeing improved source dataset embeddings performance.

Original: To  extend it to target dataset representations performance we need to verify three hypotheses: (i) linear probes do not overfit, (ii) embeddings and representations performance are monotonically linked, and (iii) source and (OOD) target embeddings ranks are monotinically linked.
Revised: To empirically extend it to target dataset representations performance we need to verify three hypotheses: (i) linear probes do not overfit, (ii) embeddings and representations performance are monotonically linked, and (iii) source and (OOD) target embeddings ranks are monotonically linked.

Original: Due to the different nature of datasets used for downstream tasks, there is no inherent reason for the rank of embeddings to transfer in a monotonic way to them.
Revised: Due to the different nature of datasets used for downstream tasks, there is no inherent reason for the rank of embeddings to transfer in a monotonic way to them.

Original: However if the source dataset is diverse enough and target datasets have some semantic overlap with the source dataset, then we have
Revised: However, if the source dataset is diverse enough and target datasets have some semantic overlap with the source dataset, then we have

Original: [Equation]
Revised: [Equation]

Original: We observe in section 3.2 and fig.
Revised: We observe in section 3.2 and fig.

Original: 2 that the rank of JE-SSL representations scales linearly between different input distributions e.g. going from a source task such as Imagenet (Deng et al., 2009) to a target task such as iNaturalist.
Revised: 2 that the rank of JE-SSL representations scales linearly between different input distributions e.g. going from a source task such as Imagenet (Deng et al., 2009) to a target task such as iNaturalist.

Original: This is further confirmed by Pearson correlation coefficients greater than 0 . 99 , except for StanfordCard where it is 0 . 88 .
Revised: This is further confirmed by Pearson correlation coefficients greater than 0 . 99 , except for StanfordCars where it is 0 . 88 .

Original: Interestingly, we observe that the StanfordCars dataset suffers from a less distinctive linear scaling due to the dataset distribution having a small overlap with ImageNet.
Revised: Interestingly, we observe that the StanfordCars dataset suffers from a less distinctive linear scaling due to the dataset distribution having a small overlap with ImageNet.

Original: This indicates that as long as the source dataset is relatively diverse then using RankMe to select a model with greater embeddings’ rank will also correspond to selecting a model with greater embeddings’ rank on the target dataset.
Revised: This indicates that as long as the source dataset is relatively diverse, then using RankMe to select a model with greater embeddings’ rank will also correspond to selecting a model with greater embeddings’ rank on the target dataset.

Original: Furthermore, as the train performance increases, so does the test performance.
Revised: Furthermore, as the train performance increases, so does the test performance.

Original: We validate this in fig.
Revised: We validate this in fig.

Original: As a result, using RankMe to select a model with greater train performance is enough to also select a model with greater test performance.
Revised: As a result, using RankMe to select a model with greater train performance is enough to also select a model with greater test performance.

Original: Finally, we report in fig.
Revised: Finally, we report in fig.

Original: 2 that the performance of embeddings and representations scales almost monotonically.
Revised: 2 that the performance of embeddings and representations scales almost monotonically.

Original: These results are supported by visualization of representations and embeddings from feature inversion models (Bordes et al., 2021).
Revised: These results are supported by visualization of representations and embeddings from feature inversion models (Bordes et al., 2021).

Original: Hence, using RankMe to select the model maximizing the performance on the former also selects a model maximizing performance on the latter.
Revised: Hence, using RankMe to select the model maximizing the performance on the former also selects a model maximizing performance on the latter.

Original: With these three hypotheses validated empirically, we can confidently say that RankMe computed on the embeddings of the source dataset is a predictor of representations’ performance on target datasets.
Revised: With these three hypotheses validated empirically, we can confidently say that RankMe computed on the embeddings of the source dataset is a predictor of representations’ performance on target datasets.

Original: Robust RankMe
Revised: Robust RankMe

Original: Implementation.
Revised: Implementation.

Original: One of the most crucial step of RankMe is the estimation of the embeddings’s rank.
Revised: One of the most crucial step of RankMe is the estimation of the embeddings’ rank.

Original: A trivial solution could be to check at the number of nonzero singular values.
Revised: A trivial solution could be to check at the number of nonzero singular values.

Original: Denoting by σ k the k th singular value of the ( N × K ) embedding matrix Z , this would lead to rank( Z ) =
Revised: Denoting by σ k the k th singular value of the ( N × K ) embedding matrix Z , this would lead to rank( Z ) =

Original: (cid:80) min( N,K ) k
Revised: (cid:80) min( N,K ) k

Original: = 1 { σ k > 0 } .
Revised: = 1 { σ k > 0 } .

Original: However, such a definition is too rigid for practical scenarios.
Revised: However, such a definition is too rigid for practical scenarios.

Original: For example, round-off error alone could have a dramatic impact on the rank estimate.
Revised: For example, round-off error alone could have a dramatic impact on the rank estimate.

Original: Instead, alternative and robust rank definitions have emerged (Press et al., 2007) such as rank( Z ) =
Revised: Instead, alternative and robust rank definitions have emerged (Press et al., 2007) such as rank( Z ) =

Original: (cid:80) min( N,K ) k
Revised: (cid:80) min( N,K ) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: M,N ) × ϵ } , where ϵ is a small constant dependant on the data type, typically 10 − 7 for float32 .
Revised: } , where ϵ is a small constant dependent on the data type, typically 10 − 7 for float32 .

Original: An alternative measure of rank comes from a probability viewpoint where the singular values are normalized to sum to 1 and the Shannon Entropy (Shannon, 1948) is used which corresponds to our definition of RankMe from eq.
Revised: An alternative measure of rank comes from a probability viewpoint where the singular values are normalized to sum to 1 and the Shannon Entropy (Shannon, 1948) is used, which corresponds to our definition of RankMe from eq.

Original: As opposed to section 3.1, the chosen eq.
Revised: As opposed to the classical rank, the chosen eq.

Original: (1) does not rely on specifying the exact threshold at which the singular value is treated as nonzero.
Revised: (1) does not rely on specifying the exact threshold at which the singular value is treated as nonzero.

Original: Throughout our study webe employ eq.
Revised: Throughout our study, we employ eq.

Original: (1) and provide the matching analysis with section 3.1 in the appendix.
Revised: (1), and provide the matching analysis with the classical rank in the appendix.

Original: Another benefit of RankMe’s eq.
Revised: Another benefit of RankMe’s eq.

Original: (1) is in its quantification of the whitening of the embeddings in addition to their rank, which is known to simplify optimization of (non)linear probes put on top of them (Santurkar et al., 2018).
Revised: (1) is in its quantification of the whitening of the embeddings in addition to their rank, which is known to simplify optimization of (non)linear probes put on top of them (Santurkar et al., 2018).

Original: Lastly, although eq.
Revised: Lastly, although eq.

Original: (1) is defined with the full embedding matrix Z , we observe that not all the samples need to be used to have an accurate estimate of RankMe.
Revised: (1) is defined with the full embedding matrix Z , we observe that not all the samples need to be used to have an accurate estimate of RankMe.

Original: In practice we use 25600 samples as ablation studies provided in supplementary section C and fig.
Revised: In practice, we use 25600 samples as ablation studies provided in appendix G and fig.

Original: S indicate that this provides an highly accurate estimate.
Revised: S11 indicate that this provides a highly accurate estimate.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: m

Original: [Figure]
Revised: [Figure]

Original:  dd i
Revised: m b e dd i

Original: [Figure]
Revised: [Figure]

Original: g s
Revised: g s

Original: [Figure]
Revised: [Figure]

Original: 3. R ANK M E P REDICTS L INEAR P ROBINGD OWNSTREAM PERFORMANCE E VEN ON U NSEEN D ATASETS
Revised: 3. R ANK M E P REDICTS L INEAR P ROBING PERFORMANCE E VEN ON U NSEEN D ATASETS

Original: In order to empirically validate RankMe, we compare it to linear evaluation, which is the default evaluation method of JE-SSL methods.
Revised: In order to empirically validate RankMe, we compare it to linear evaluation, which is the default evaluation method of JE-SSL methods.

Original: Finteuning has gained in popularity with Masked Image
Revised: Finetuning has gained in popularity with Masked Image

Original: Modeling methods but this can have a significant impact on the properties of the embeddings and alters what was learned during the pretraining.
Revised: Modeling methods, but this can have a significant impact on the properties of the embeddings and alters what was learned during the pretraining.

Original: As such we do not focus on this evaluation.
Revised: As such, we do not focus on this evaluation.

Original: Experimental Methods and Datasets Considered
Revised: Experimental Methods and Datasets Considered

Original: In order to provide a meaningful assessment of the embeddings rank’s impact on performance we focus on 4 JE-SSL methods.
Revised: In order to provide a meaningful assessment of the embeddings rank’s impact on performance, we focus on 4 JE-SSL methods.

Original: We use SimCLR as a representative contrastive method, VICReg as a representative covariance based method, and
Revised: We use SimCLR as a representative contrastive method, VICReg as a representative covariance based method, and

Original: VICReg-exp and VICReg-ctr which were introduced in Garrido et al.
Revised: VICReg-exp and VICReg-ctr which were introduced in Garrido et al.

Original: To make our work self-contained, we present the methods in supplementary section A.
Revised: To make our work self-contained, we present the methods in appendix A.

Original: We chose to use VICReg-exp and VICReg-ctr as they provide small modifications to VICReg and SimCLR while producing embeddings with different rank properties.
Revised: We chose to use VICReg-exp and VICReg-ctr as they provide small modifications to VICReg and SimCLR while producing embeddings with different rank properties.

Original: For each method we vary parameters that directly influence the rank of the embeddings, whether it is the temperature use in softmax based methods, which directly impacts the hardness of the softmax, or the loss weights to give more or less importance to the regularizing aspect of loss functions.
Revised: For each method we vary parameters that directly influence the rank of the embeddings, whether it is the temperature use in softmax based methods, which directly impacts the hardness of the softmax, or the loss weights to give more or less importance to the regularizing aspect of loss functions.

Original: We also vary optimization parameters such as the learning rate and weight decay to provide a more complete analysis.
Revised: We also vary optimization parameters such as the learning rate and weight decay to provide a more complete analysis.

Original: We provide the hyperparameters used for all experiments in supplementary section G. All approaches were trained in the same experimental setting with a ResNet-50 (He et al., 2016) backbone with a MLP projector having intermediate layers of size 8192 , 8192 , 2048 , which avoids any architectural rank constraints.
Revised: We provide the hyperparameters used for all experiments in appendix K. All approaches were trained in the same experimental setting with a ResNet-50 (He et al., 2016) backbone with a MLP projector having intermediate layers of size 8192 , 8192 , 2048 , which avoids any architectural rank constraints.

Original: The models were trained forepochs on ImageNet with the LARS (You et al., 2017; Goyal et al., 2017) optimizer.
Revised: The models were trained for 100 epochs on ImageNet with the LARS (You et al., 2017; Goyal et al., 2017) optimizer.

Original: In order to evaluate the methods, we used ImageNet (our source dataset), as well as INaturalist18 (Horn et al., 2018), Places205 (Zhou et al., 2014), EuroSat (Helber et al., 2019), SUN397 (Xiao et al., 2010) and StanfordCars (Krause et al., 2013) to evaluate the trained models on unseen datasets.
Revised: In order to evaluate the methods, we used ImageNet (our source dataset), as well as iNaturalist18 (Horn et al., 2018), Places205 (Zhou et al., 2014), EuroSat (Helber et al., 2019), SUN397 (Xiao et al., 2010) and StanfordCars (Krause et al., 2013) to evaluate the trained models on unseen datasets.

Original: These commonly used datasets provide a wide range of scenarios that differ from ImageNet and provide meaningful ways to test the robustness of RankMe.
Revised: These commonly used datasets provide a wide range of scenarios that differ from ImageNet and provide meaningful ways to test the robustness of RankMe.

Original: For example, iNaturalist18 consists ofclasses focused on fauna and flora which requires more granularity than similar classes on ImageNet, SUN397 focuses on scene understanding, deviating from the single object and object-centric images of ImageNet, and EuroSat consists of satellite images which again differ from ImageNet.
Revised: For example, iNaturalist18 consists of 8412 classes focused on fauna and flora which requires more granularity than similar classes on ImageNet, SUN397 focuses on scene understanding, deviating from the single object and object-centric images of ImageNet, and EuroSat consists of satellite images which again differ from ImageNet.

Original: Datasets such as iNaturalist can allow theoretical limitations to manifest themselves more clearly due to the number of classes being significantly higher than the rank of learned representations.
Revised: Datasets such as iNaturalist can allow theoretical limitations to manifest themselves more clearly due to the number of classes being significantly higher than the rank of learned representations.

Revised: While we focus on these datasets for our visualizations, we also include CIFAR10, CIFAR100 Krizhevsky et al.

Revised: (2009), Food101 Bossard et al.

Revised: (2014), VOC07 Everingham et al.

Revised: and CLVR-count Johnson et al.

Revised: (2017) for our hyperparameter selection results, and provide visualizations in appendix D.

Original: In order to evaluate on those datasets, we relied on the VISSL library (Goyal et al., 2021).
Revised: In order to evaluate on those datasets, we relied on the VISSL library (Goyal et al., 2021).

Original: We provide complete details on the pretraining and evaluations in supplementary section E.
Revised: We provide complete details on the pretrainings and evaluations in appendix I.

Original: RankMe as a prediction of linear classification accuracy.
Revised: RankMe as a prediction of linear classification accuracy.

Original: As can be seen in fig.
Revised: As can be seen in fig.

Original: 3, for a given method the performance on the embedding is improved by with a higher embedding rank, whether we look on ImageNet on which the models were pretrained or on downstream datasets.
Revised: 3, for a given method the performance on the embedding is improved by with a higher embedding rank, whether we look on ImageNet on which the models were pretrained or on downstream datasets.

Original: Nonetheless, there are some visible outliers but they are mostly on SimCLR in settings with very high error rates compared to before the projector, such as in iNaturalist or StanfordCars.
Revised: Nonetheless, there are some visible outliers, but they are mostly on SimCLR in settings with very high error rates compared to before the projector, such as in iNaturalist or StanfordCars.

Original: The conceptual closeness between VICReg-ctr and SimCLR pointed out in Garrido et al.
Revised: The conceptual closeness between VICReg-ctr and SimCLR pointed out in Garrido et al.

Original: (2022) would also suggest that these results need to be interpreted carefully, but they do reinforce the fact that a high rank is a necessary and not sufficient condition for improved performance on downstream tasks.
Revised: (2022) would also suggest that these results need to be interpreted carefully, but they do reinforce the fact that a high rank is a necessary and not sufficient condition for improved performance on downstream tasks.

Original: It is also very tempting to draw conclusions when comparing different approaches, especially when looking at the ImageNet performance, however since dimensional collapse is not the only performance deciding factor one should refrain from doing so.
Revised: It is also very tempting to draw conclusions when comparing different approaches, especially when looking at the ImageNet performance, however since dimensional collapse is not the only performance deciding factor one should refrain from doing so.

Original: The link between embedding rank and performance is even clearer when evaluating on the representations, as is usually done.
Revised: The link between embedding rank and performance is even clearer when evaluating on the representations, as is usually done.

Original: In this scenario the link iseven more consistent across datasets where we observe again than a higher rank is necessary for improved performance.
Revised: In this scenario the link is more consistent across datasets, where we observe again that a higher rank is necessary for improved performance.

Original: This solidifies the use of RankMe as a performance metric that can be used in practical scenarios.
Revised: This solidifies the use of RankMe as a performance metric that can be used in practice.

Original: 3. G OING F URTHER : R ANK M E A LSO H OLDS FOR N ONLINEAR P ROBING AND FOR D IFFERENT A RCHITECTURES
Revised: 3. G OING F URTHER : R ANK M E A LSO H OLDS FOR N ONLINEAR P ROBING AND FOR D IFFERENT A RCHITECTURES

Original: Non-linear evaluation.
Revised: Non-linear evaluation.

Original: While we have been focusing on linear evaluation, one can wonder if the behaviour changes when using a more complex task-related head.
Revised: While we have been focusing on linear evaluation, one can wonder if the behaviour changes when using a more complex task-related head.

Original: We thus give some evidence that the previously observed behaviours are similar with a non-linear classification head.
Revised: We thus give some evidence that the previously observed behaviours are similar with a non-linear classification head.

Original: We used a simple 3 layer MLP with intermediate dimensions 2048 , where each layer is followed by a ReLU activation.
Revised: We used a simple 3 layer MLP with intermediate dimensions 2048 , where each layer is followed by a ReLU activation.

Original: This choice of dimensions ensures that there are no architectural rank constraints on the embeddings.
Revised: This choice of dimensions ensures that there are no architectural rank constraints on the embeddings.

Original: We focused on SUN397 and StanfordCars for this study due to their conceptual differences to ImageNet.
Revised: We focused on SUN397 and StanfordCars for this study due to their conceptual differences to ImageNet.

Original: The low rank of embeddings produced by SimCLR on these datasets would suggest that a non-linear classifier might help improve performance since it is not as theoretically limited by the embeddings’ rank as it is in the linear setting.
Revised: The low rank of embeddings produced by SimCLR on these datasets would suggest that a non-linear classifier might help improve performance, since it is not as theoretically limited by the embeddings’ rank as it is in the linear setting.

Original: However, as we can see in fig.
Revised: However, as we can see in fig.

Original: 4, the behaviours for all methods is the same as in the linear regime.
Revised: 4, the behaviors for all methods is the same as in the linear regime.

Original: This would suggest that RankMe is also a suitable metric to evaluate downstream performance in a non-linear setting.
Revised: This would suggest that RankMe is also a suitable metric to evaluate downstream performance in a non-linear setting.

Revised: Dimensional collapse on different architectures.

Revised: Our results so far have only focused on ResNet50s, and a concern could be that the architecture played a significant role in the introduction of

Original: [Figure]
Revised: [Figure]

Original: Algorithm

Original: Require:
Revised: Require:

Original: Models f 1 , . .
Revised: Models f 1 , . .

Original: , f N to compare, in increasing value of the hyperparameter
Revised: , f N to compare, in increasing value of the hyperparameter

Original: Require:
Revised: Require:

Original: Corresponding ranks r 1 , . . .
Revised: Corresponding ranks r 1 , . . .

Original: , r N 1: f best ← f 1 , r best ← r2: for i = 2 to N do 3: if r i > r best then 4: f best ← f i , r best ← r i 5: else if r i = r best and ( r i > r i − 1 or r i > r i +1 )
Revised: , r N 1: f best ← f 1 , r best ← r2: for i = 2 to N do 3: if r i > r best then 4: f best ← f i , r best ← r i 5: else if r i = r best and ( r i > r i − 1 or r i > r i +1 )

Original: then 6: f best ← f i , r best ← r i 7: return f best
Revised: then 6: f best ← f i , r best ← r i 7: return f best

Original: [Figure]
Revised: [Figure]

Original: Dimensional collapse on different architectures. Our results so far have only focused on ResNet50s, and a concern could be that the architecture played a significant role in the introduction of collapse.
Revised: collapse.

Original: As such we trained VICReg in the same setting as before but using ConvNext-T (Liu et al., 2022) as the backbone architecture.
Revised: As such, we trained VICReg in the same setting as before but using ConvNext-T (Liu et al., 2022) as the backbone architecture.

Original: As we can see in fig.
Revised: As we can see in fig.

Original: 4, collapse still appears in this case, with an even stronger impact on performance on ImageNet.
Revised: 4, collapse still appears in this case, with an even stronger impact on performance on ImageNet.

Original: This reinforces the findings of Jing et al.
Revised: This reinforces the findings of Jing et al.

Original: He & Ozay (2022), which study collapse through the used loss function independently of the architecture of the backbone.
Revised: He & Ozay (2022), which study collapse through the used loss function independently of the architecture of the backbone.

Revised: RankMe in more diverse settings.

Revised: While our focus has been on contrastive methods, we further study in appendix C how RankMe can be applied to clustering methods such as DINO, where it shows great effectiveness.

Revised: We also take a look at the effectiveness of RankMe when pretraining on other source datasets in appendix B, validating RankMe on iNaturalist18 pretraining.

Original:  R ANK M E FOR L ABEL -F REE HYPERPARAMETER SELECTION IN SSL
Revised:  R ANK M E FOR L ABEL -F REE HYPERPARAMETER SELECTION IN SSL

Original: We previously focused on validating RankMe by comparing overall performance compared to linear evaluation.
Revised: We previously focused on validating RankMe by comparing overall performance compared to linear evaluation.

Original: In this section we focus on the evolution of rank and performance when varying one hyperparameter at a time in order to demonstrate how RankMe can be used for hyperparameter selection.
Revised: In this section we focus on the evolution of rank and performance when varying one hyperparameter at a time in order to demonstrate how RankMe can be used for hyperparameter selection.

Original: We focus on loss specific hyperparameters such as the loss weights or temperature as well as hyperparameters related to optimization, such as the learning rate and weight decay.
Revised: We focus on loss specific hyperparameters such as the loss weights or temperature as well as hyperparameters related to optimization, such as the learning rate and weight decay

Original:  4. U SING R ANK M E TO CHOOSE THE CORRECT HYPERPARAMETER VALUE
Revised: . 4. U SING R ANK M E TO CHOOSE THE CORRECT HYPERPARAMETER VALUE

Original: As we have shown before, having a higher rank leads to better performance, and using RankMe to find the correct value of an hyperparameter is as simple as choosing the value that leads to the highest rank, as illustrated in fig.
Revised: As we have shown before, having a higher rank is necessary for better performance, and using RankMe to find the best value of an hyperparameter is as simple as choosing the value that leads to the highest rank, as illustrated in fig.

Original: Certain hyperparemeters will lead to plateaus of equal rank, and in those the value that first achieves the maximal value should be selected.
Revised: Certain hyperparameters will lead to plateaus of equal rank, and in those the value that first achieves the maximal value should be selected.

Original: This second part is  only applicable when hyperparameter values can be ordered, and may be omitted for categorical values. This is analogous to the elbow method that is used in clustering. [Table]
Revised: This second part is however only applicable when hyperparameter values can be ordered.

Original: Even in cases where the values cannot be compared and equal ranks are found in a different setting
Revised: Even in cases where the values cannot be compared, and equal ranks are found in a different setting, this still makes it possible to discard some runs and only focus on the one that achieve the maximal

Original: [Table]
Revised: [Table]

Original: this still makes it possible to discard some runs and only focus on the one that achieve the maximal rank.
Revised: rank.

Original: This further highlight how maximal rank is only a necessary condition for good performance, however when the hyperparameters are ordered we can go one step further and use the rank alone to find a good hyperparameter value.
Revised: This further highlight how maximal rank is only a necessary condition for good performance. Nonetheless, when the hyperparameters are ordered we can go one step further and use the rank alone to find a good hyperparameter value.

Original: 4. E XPERIMENTS
Revised: 4. E XPERIMENTS

Original: In order to demonstrate the effectiveness of RankMe for hyperparameter selection
Revised: In order to demonstrate the effectiveness of RankMe for hyperparameter selection

Original: [Table]
Revised: [Table]

Original: we apply the algorithm presented in fig.
Revised: we apply the algorithm presented in fig.

Original: 5 to find the best values for a given set of hyperparameters for VICReg and SimCLR.
Revised: 5 to find the best values for a given set of hyperparameters for VICReg and SimCLR.

Original: Our focus is on the covariance and invariance weights in VICReg, the temperature in
Revised: Our focus is on the covariance and invariance weights in VICReg, the temperature in

Original: SimCLR
Revised: SimCLR

Original: [Table]
Revised: [Table]

Original: and on learning rate and weight decay for both.
Revised: and on learning rate and weight decay for both.

Original: We compare the performance on ImageNet as well as the average performance on the previously discussed OOD datasets to models selected by their ImageNet top-1 accuracy on its validation set.
Revised: We compare the performance on ImageNet as well as the average performance on the previously discussed OOD datasets to models selected by their ImageNet top-1 accuracy on its validation set.

Original: For detailed performance on every dataset, confer supplementary section F.
Revised: For per dataset performance, confer appendix J.

Original: As can be seen in table 1, using RankMe we are able to retrieve most of the performance on
Revised: As can be seen in table 1, using RankMe we are able to retrieve most of the performance on

Original: ImageNet, with gaps being lower than half a point.
Revised: ImageNet, with gaps being lower than half a point.

Original: It is not possible to beat the selection using
Revised: It is not possible to beat the selection using

Original: ImageNet’s validation since this is the metric we are evaluating on.
Revised: ImageNet’s validation, since this is the metric we are evaluating on.

Original: However, on OOD datasets we are able to improve the performance in certain settings, and match it in the others.
Revised: However, on OOD datasets we are able to improve the performance in certain settings, and match it in the others.

Original: Thus, when comparing performance after the projector, RankMe is the better approach of the two to select the hyperparameters that will generalize best to unseen datasets.
Revised: Thus, when comparing performance after the projector, RankMe is the better approach of the two to select the hyperparameters that will generalize best to unseen datasets.

Revised: When comparing to α -Req, RankMe achieves better in domain performance, but on OOD datasets α -ReQ performs slightly better, though with bigger worst case performance drops.

Revised: We provide an in-depth analysis of α -ReQ in appendix E, where we find that the power law prior of α -ReQ fails on the embeddings and as such those results must be interpreted with care.

Original: As pointed out in Girish et al.
Revised: As pointed out in Girish et al.

Original: (2022), using ImageNet performance to select models can lead to suboptimal performance in downstream tasks, which our results further confirm and reinforces the need for a new way of selecting hyperparameters.
Revised: (2022), using ImageNet performance to select models can lead to suboptimal performance in downstream tasks, which our results further confirm and reinforces the need for a new way of selecting hyperparameters.

Original: When looking at performance before the projector in fig.
Revised: When looking at performance before the projector in fig.

Original: 1, we can see that here RankMe does not beat the models selected with ImageNet’s validation set, even on OOD datasets.
Revised: 1, we can see that here RankMe does not beat the models selected with ImageNet’s validation set, even on OOD datasets.

Revised: However, RankMe performs better than α -ReQ in most settings, while not suffering from severe drops in the worst cases.

Original: Nonetheless the gaps  are on average of less than half a point, which shows how competitive RankMe can be for hyperparameter selection, despite using no labeled data, having no parameters to tune, and being able to be computed in a couple of minutes.
Revised: Nevertheless, the gaps between RankMe and the ImageNet oracle are on average of less than half a point, which shows how competitive RankMe can be for hyperparameter selection, despite using no labeled data, having no parameters to tune, and being able to be computed in a couple of minutes.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown how the phenomenon of dimensional collapse in self-supervised learning can be used as a powerful metric to evaluate models.
Revised: We have shown how the phenomenon of dimensional collapse in self-supervised learning can be used as a powerful metric to evaluate models.

Original: By using a theoretically motivated analogue of the rank of embeddings, we show that the performance on downstream datasets can easily be assessed by only looking at the training dataset, without any labels, training or parameters.
Revised: By using a theoretically motivated analogue of the rank of embeddings, we show that the performance on downstream datasets can easily be assessed by only looking at the training dataset, without any labels, training, or parameters.

Original: While our work focuses on linear classification, we show promising results in non-linear classification that raise the question of how general this simple metric can be.
Revised: While our work focuses on linear classification, we show promising results in non-linear classification that raise the question of how general this simple metric can be.

Original: Furthermore, its competitiveness with traditional oracle based hyperparameter selection methods makes it a promising tool in settings where labels are scarce, such as in the case of large uncurated datasets.
Revised: Furthermore, its competitiveness with traditional oracle based hyperparameter selection methods makes it a promising tool in settings where labels are scarce, such as in the case of large uncurated datasets.

Original: As such, this work makes a step towards completely label-less self-supervised learning, as most existing approaches’ hyperparameters are tuned with the help of ImageNet’s validation set.
Revised: As such, this work makes a step towards completely label-less self-supervised learning, as most existing approaches’ hyperparameters are tuned with the help of ImageNet’s validation set.

Original: Further work will explore the use of RankMe in more varied scenarios, to further legitimize its use in designing better self-supervised approaches.
Revised: Further work will explore the use of RankMe in more varied scenarios, to further legitimize its use in designing better self-supervised approaches.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: While reproducing the pretrainings is prohibitively expensive since each training takes around a day on 8 V100 GPUs, we provide all of the hyperparameters used in supplementary section G. We also provide all of the pretraining details in supplementary section E, along with the hyperparameters used for the linear evaluations in the same section.
Revised: While reproducing the pretrainings is prohibitively expensive since each training takes around a day on 8 V100 GPUs, we provide all of the hyperparameters used in appendix K. We also provide all of the pretraining details in appendix I, along with the hyperparameters used for the linear evaluations in the same section.

Original: We also provide all the performance and rank values to reproduce our main figures in supplementary section G. While the implementation of RankMe is straightforward, we prove an example algorithm to use it in fig.
Revised: We also provide all the performance and rank values to reproduce our main figures in appendix K. While the implementation of RankMe is straightforward, we provide an example algorithm to use it in fig.

Original: All of these efforts should make our results reproducible and verifiable.
Revised: All of these efforts should make our results reproducible and verifiable.

Original: A B ACKGROUND
Revised: A B ACKGROUND

Original: In order to make our work as self-contained as possible, we recall the loss functions of the methods we study.
Revised: In order to make our work as self-contained as possible, we recall the loss functions of the methods we study.

Original: For conciseness we refer to the outputs of the encoder as representations and the outputs of the projection head as embeddings , which we denote by z i ∈ R d .
Revised: For conciseness, we refer to the outputs of the encoder as representations and the outputs of the projection head as embeddings , which we denote by z i ∈ R d .

Original: We first briefly recall that the
Revised: We first briefly recall that the

Original: SimCLR loss is given by
Revised: SimCLR loss is given by

Original: [Equation]
Revised: [Equation]

Original: with P the set of all positive pairs in the current mini-batch or dataset that comprise N exemplars.
Revised: with P the set of all positive pairs in the current mini-batch or dataset that comprise N exemplars.

Original: VICReg’s loss is defined with three components.
Revised: VICReg’s loss is defined with three components.

Original: The variances loss v acts as a norm regularizer for the dimensions, and the covariance loss aims at decorrelating dimensions in the embeddings.
Revised: The variances loss v acts as a norm regularizer for the dimensions, and the covariance loss aims at decorrelating dimensions in the embeddings.

Original: They are respectively defined as
Revised: They are respectively defined as

Original: [Equation]
Revised: [Equation]

Original: Both of these loss are combined with an invariance loss that matches positives pairs, giving a final loss of
Revised: Both of these loss are combined with an invariance loss that matches positives pairs, giving a final loss of

Original: [Equation]
Revised: [Equation]

Original: VICReg-exp is definied similarly, but with the exponential covariance loss defined as
Revised: VICReg-exp is defined similarly, but with the exponential covariance loss defined as

Original: [Equation]
Revised: [Equation]

Original: VICReg-ctr is then VICReg-exp but applied to Z T , making it a contrastive approach and conceptually similar to SimCLR.
Revised: VICReg-ctr is then VICReg-exp but applied to Z T , making it a contrastive approach and conceptually similar to SimCLR.

Original: These methods give us different scenarios of collapse and allow us to make a more general study of the rank of representations 
Revised: These methods give us different scenarios of collapse and allow us to make a more general study of the rank of representations as a powerful metric. B I N ATURALIST 18 PRETRAINING [Figure] Dataset

Revised: Method

Original: B C OMPARISON OF THE RANK ESTIMATORS{+SimCLR VICReg temp. cov. iNat iNat
+}
Revised: {+SimCLR VICReg temp. cov. iNat iNat
+}

Revised: Oracle 28 .37 .

Revised: RankMe 27 .36 .

Revised: Difference − 1 .− 0 .

Original: {+ImageNet iNat18 Oracle 
+}
Revised: {+ImageNet iNat18 Oracle 
+}

Revised: RankMe 52 .55 .

Revised: Difference − 1 .+0 .

Revised: While our experiments have previously focused on ImageNet pretraining due to its wide use in the community, one can wonder if RankMe is still applicable when trained on another source dataset.

Revised: To verify this, we pretrained VICReg and SimCLR on iNaturalist18, and respectively varied the covariance loss’ weight and the temperature to study the influence on the rank of embeddings.

Revised: We used the same protocol as ImageNet pretraining but trained for 300 epochs instead of 100 to obtain a similar number of iterations.

Revised: We then evaluated the performance on iNaturalist18 and ImageNet.

Revised: We use 8192 dimensional embeddings due to the high number of classes of iNaturalist, but since the representations are 2048 dimensional, the rank cannot intrinsically go higher so we consider that all highers ranks are effectively 2048.

Revised: As we can see in fig.

Revised: S1, RankMe provides a similar level of performance as on ImageNet pretrainings, validating it on a different source dataset.

Revised: RankMe is even able to improve performance on ImageNet compared to the iNaturalist18 oracle, further showing the limitations of such oracles on downstream tasks.

Revised: C A PPLICABILITY TO CLUSTER BASED METHODS

Revised: While we have studied the applicability of RankMe on contrastive methods, cluster based methods such as DINO have become extremely popular

Original: [Figure]
Revised: [Figure]

Revised: and since the definition of embeddings is not as clear cut in them

Revised: [Figure]

Revised: a thorough analysis is required.

Revised: We will proceed in two steps:

Revised: • Show that dimensional collapse happens right before the clustering layer, and not on the prototypes • Show that RankMe is a good measure of performance on DINO

Revised: [Figure]

Revised: As we can see in figure S

Revised: [Figure]

Revised: DINO’s projector can be interpreted as both a classical projector and a clustering layer, whose weights are clustering prototypes.

Revised: This interpretation comes from the softmax that is applied on the output of the projection head which can be interpreted as an InfoNCE between the embeddings and the clustering prototypes that make up the clustering layer.

Revised: While prototypes themselves are not particularly collapsed, the embeddings that are obtained before the clustering present dimensional collapse.

Revised: [Figure]

Revised: Method

Revised: Temperature

Revised: Student

Revised: Teacher

Revised: ImageNet Oracle 72 .72 .

Revised: RankMe 72 .72 .

Revised: Difference = − 0 .

Revised: As we can see in fig.

Original: Since we do not rely on  the classical threshold-based rank estimator
Revised: S3, the phenomenon of dimensional collapse is highly visible in DINO, which enables the use RankMe to find optimal hyperparameter values. Validating this on ImageNet, we see that RankMe is able to match the performance of the oracle, or leads to slightly lower performance, further validating RankMe on another popular method. D R ESULTS ON SUPPLEMENTARY DATASETS While we previously focused on certain datasets for their interesting natures

Original: [Figure]
Revised: [Figure]

Revised: we will provide additional visualizations for the remaining datasets.

Revised: [Figure]

Revised: [Figure]

Revised: As we can see in figs.

Revised: S4 and S5, we find similar behaviours as before, apart from Food101 where performance are almost identical for all methods.

Original: it is important to verify how well our entropy based one correlates with it.
Revised: This reinforces the previous validation of RankMe. The relative simplicity of the datasets targeted here make the theoretical limitations of rank-deficient embeddings harder to see, even though we still see that a high rank helps generalization. E D ETAILED RESULTS FOR α -R E Q In order to further study the performance of α -ReQ, we reproduce our plots for RankMe using α -ReQ instead of the rank of embeddings.

Revised: We compare both the intended use of α -ReQ in fig.

Revised: S6, as well as applying it on the embeddings to measure performance on the representations, which we foudn was necessary for RankMe in fig.

Revised: S7.

Revised: As we can see in fig.

Revised: S6, there are no clear link visible between the value of α -ReQ and downstream performance.

Revised: Especially we are unable to see the tendency of performance to increase as α tends to one.

Revised: Nonetheless α -ReQ was still able to lead to good performance when used for hyperparameter selection.

Revised: When applying α -ReQ as we would RankMe, we can see in fig.

Revised: S7 that there is again no trend of performance increase when α tends to one.

Revised: On the contrary we even find that performance tends to get better with a lower α , as is most visible in StanfordCars, iNaturalist18 or ImageNet for example.

Revised: α going towards one means that the singular values of the embeddings tends to a uniform distribution, in line with the goal of RankMe.

Revised: [Figure]

Revised: T o pr e p r e s e n t a t i o n s

Revised: [Figure]

Revised: As we can see in figs. S8 and S

Revised: [Figure]

Revised: the power-law prior of α -ReQ holds well in the case of non-collapsed embeddings

Revised: [Figure]

Revised: but when we apply it on collapsed ones

Revised: [Figure]

Revised: this assumptions fails.

Revised: It even provides a poor approximation of the main rank "plateau" with the highest singular values as can be seen on the right

Revised: [Figure]

Original: As we can see in fig.
Revised: This further confirms the findings of He & Ozay (2022), and shows that when applying α -ReQ directly on the embeddings one must be careful since the core assumptions of the method is violated. [Figure] [Figure] F C OMPARISON OF THE RANK ESTIMATORS

Original: S1, both estimates discussed previously correlate extremely well, showing that using one or the other should not lead to significant differences, as validated in supplementary section
Revised: [Figure] [Figure]

Original: D. Nonetheless the entropic estimatortakes into account the degree of whitening of the embeddings, which links better to theoretical results
Revised: Since we do not rely on the classical threshold-based rank estimator

Original: [Figure]
Revised: [Figure]

Revised: it is important to verify how well our entropy based one correlates with it.

Revised: As we can see in fig.

Revised: S10, both estimates discussed previously correlate extremely well, showing that using one or the other should not lead to significant differences, as validated in appendix H. Nonetheless, the entropic estimator takes into account the degree of whitening of the embeddings, which links better to theoretical results.

Original: C C ONVERGENCE OF THE RANK ESTIMATORS
Revised: G C ONVERGENCE OF THE RANK ESTIMATORS

Original: [Figure]
Revised: [Figure]

Original: As we can see in fig.
Revised: As we can see in fig.

Original: S2, the rank estimates converge extremely quickly, especially for VICReg.
Revised: S11, the rank estimates converge extremely quickly, especially for VICReg.

Original: For both VICReg and SimCLR, 10000 samples are enough to obtain more than 95% of the final rank.
Revised: For both VICReg and SimCLR, 10000 samples are enough to obtain more than 95% of the final rank.

Original: It is worth noting that the entropic rank estimator converges more slowly that the classical rank estimator as it is more sensitive to the singular values.
Revised: It is worth noting that the entropic rank estimator converges more slowly than the classical rank estimator, as it is more sensitive to the singular values.

Original: The fact that the rank can be approximated with few samples is encouraging for its use during training and not only as a measure of performance after pretraining.
Revised: The fact that the rank can be approximated with few samples is encouraging for its use during training and not only as a measure of performance after pretraining.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D R EPRODUCTION OF FIGURES WITH THE CLASSICAL RANK ESTIMATOR
Revised: H R EPRODUCTION OF FIGURES WITH THE CLASSICAL RANK ESTIMATOR

Original: As can be seen in figs.
Revised: As can be seen in figs.

Original: S3 and S4, the results that we obtain using the classical threshold-based rank estimator are extremely similar to the ones with the entropic estimator.
Revised: S12 and S13, the results that we obtain using the classical threshold-based rank estimator are extremely similar to the ones with the entropic estimator.

Original: The exact values do differ, but the behaviours stay the same.
Revised: The exact values do differ, but the behaviors stay the same.

Original: One of the main differences is illustrated in fig.
Revised: One of the main differences is illustrated in fig.

Original: S
Revised: S

Original: [Figure]
Revised: [Figure]

Original: where we can see that the target rank is almost identical to the source one when we previously saw a drop of around 50% .
Revised: where we can see that the target rank is almost identical to the source one when we previously saw a drop of around 50% .

Original: This can be explained by the fact that some features may be less present in the target dataset, reducing the associated singular values, and thus the entropic rank.
Revised: This can be explained by the fact that some features may be less present in the target dataset, reducing the associated singular values [Figure] and thus the entropic rank.

Original: All of this shows that using one or the other will lead to similar results in practical scenarios
Revised: All of this shows that using one or the other will lead to similar results in practical scenarios

Original: [Figure]
Revised: [Figure]

Original: E D ETAILED TRAINING AND EVALUATION PROCEDURES E. P RETRAINING
Revised: I D ETAILED TRAINING AND EVALUATION PROCEDURES I. P RETRAINING

Original: All pretraining were done with ResNet-50 backbones.
Revised: All pretrainings were done with ResNet-50 backbones.

Original: The projector used is a MLP with intermediate dimensions 8192 , 8192 , 2048 .
Revised: The projector used is a MLP with intermediate dimensions 8192 , 8192 , 2048 .

Original: They were trained with the LARS optimizer using a momentum of 0 . 9 , weight decay 10 − 6 and varying learning rates depending on the method.
Revised: They were trained with the LARS optimizer using a momentum of 0 . 9 , weight decay 10 − 6 and varying learning rates depending on the method.

Original: VICReg used 0 .
Revised: VICReg used 0 .

Original: 3 base learning rate, SimCLR 0 .
Revised: 3 base learning rate, SimCLR 0 .

Original: 3 or 0 .
Revised: 5 or 0 .

Original: 5 depending on the experiment, VICReg-exp 0 .
Revised: 6 depending on the experiment, VICReg-exp 0 .

Original: 6 and VICReg-ctr 0 .
Revised: 6 and VICReg-ctr 0 .

Original:  The learning rate is then computed as lr = base _ lr ∗ batch _ size/ 256 .
Revised: [Table] The learning rate is then computed as lr = base _ lr ∗ batch _ size/ 256 .

Original: We do a 10 -epochs linear[Table] warmup and then use cosine annealing.
Revised: We do a 10 -epochs linear warmup and then use cosine annealing.

Original: We used batch sizes of 2048 for SimCLR and 1024 for other methods.
Revised: We used batch sizes of 2048 for SimCLR and 1024 for other methods.

Original: SimCLR and VICReg-ctr also use a default temperature of 0 . 15 , and 0 .
Revised: SimCLR and VICReg-ctr also use a default temperature of 0 . 15 , and 0 .

Original: 1 for VICReg-exp.
Revised: 1 for VICReg-exp.

Original: We used the image augmentation strategy from Grill et al.
Revised: We used the image augmentation strategy from Grill et al.

Original: (2020) illustrated in table S
Revised: (2020) illustrated in table S

Original: [Table]
Revised: [Table]

Original:  E 
Revised: I. E VALUATION

Original: E VALUATION

Original: [Table]
Revised: [Table]

Original: For all datasets except StanfordCars, we use the standard protocol in VISSL.
Revised: For all datasets except StanfordCars, we use the standard protocol in VISSL.

Original: On StanfordCars we mostly tuned the learning rate.
Revised: On StanfordCars we mostly tuned the learning rate.

Original: The parameters that we used are described in table S2.
Revised: The parameters that we used are described in table S2.

Original: For data augmentation, we used random resized crops and random horizontal flips during training, and center crop for evaluation.
Revised: For data augmentation, we used random resized crops and random horizontal flips during training, and center crop for evaluation.

Revised: For VOC07, we follow the common protocol using SVMs, as used in Bardes et al.

Revised: We use the default VISSL settings for this evaluation.

Original: F D ETAILED TABLES FOR HYPERPARAMETER SELECTION
Revised: J D ETAILED TABLES FOR HYPERPARAMETER SELECTION

Original: [Table]
Revised: [Table]

Original: {+RankMe 
+}
Revised: {+RankMe 
+}

Revised: .68 .69 .α -ReQ

Revised: Cars

Original: {+ImageNet Oracle 55 .55 .
+}
Revised: {+ImageNet Oracle 55 .55 .
+}

Original: {+RankMe 
+}
Revised: {+RankMe 
+}

Revised: .54 .α -ReQ

Revised: FOOD

Original: {+ImageNet Oracle 75 .75 .75 .75 .75 .
+}
Revised: {+ImageNet Oracle 75 .75 .75 .75 .75 .
+}

Original: {+RankMe 
+}
Revised: {+RankMe 
+}

Revised: .75 .α -ReQ

Revised: VOC

Revised: ImageNet Oracle 84 .84 .84 .84 .84 .84 .83 .

Original: {+RankMe 
+}
Revised: {+RankMe 
+}

Revised: .83 .83 .α -ReQ

Revised: CLEVR-Count

Revised: ImageNet Oracle 55 .55 .56 .56 .51 .51 .53 .

Original: {+RankMe 
+}
Revised: {+RankMe 
+}

Revised: .53 .α -ReQ

Revised: CIFAR

Original: {+ImageNet Oracle 90 .
+}
Revised: {+ImageNet Oracle 90 .
+}

Original: {+RankMe 89 .89 .90 .
+}
Revised: {+RankMe 89 .89 .90 .
+}

Revised: .90 .α -ReQ

Revised: [Table]

Revised: [Table]

Original: {+RankMe 71 .72 .
+}
Revised: {+RankMe 71 .72 .
+}

Revised: [Table]

Original: {+-ReQ 
+}
Revised: {+-ReQ 
+}

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: Difference +

Original: [Table]
Revised: [Table]

Original: [-− 0 .− 
 .+0 .+0 .− 0 .=-]
Revised: [-− 0 .− 
 .+0 .+0 .− 0 .=-][Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: − 4 .− 1 .− 0 .− 0 .− 2 .− 0 .=
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [-Difference − 1 .− 0 .− 
 .= − 0 .− 
 .=-]
Revised: [-Difference − 1 .− 0 .− 
 .= − 0 .− 
 .=-][Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Difference +0 .={+RankMe 57 .57 .
+}
Revised: {+RankMe 57 .57 .
+}

Original: = − 0 .+2 .−{+.55 .56 .α -ReQ 57 .
+}
Revised: {+.55 .56 .α -ReQ 57 .
+}

Original: 1 .=
Revised: Cars

Original: [Table] ImageNet [Table]{+Oracle 12 .12 .
+}
Revised: ImageNet {+Oracle 12 .12 .
+}

Original: RankMe 11 .

Revised: RankMe 11 .

Original: Difference − 0 .={+.13 .α -ReQ FOOD ImageNet Oracle 75 .75 .75 .75 .75 .
+}
Revised: {+.13 .α -ReQ FOOD ImageNet Oracle 75 .75 .75 .75 .75 .
+}

Original: ={+RankMe 
 .75 .α -ReQ VOC ImageNet Oracle 79 .79 .79 .79 .79 .79 .79 .+}
Revised: {+RankMe 
 .75 .α -ReQ VOC ImageNet Oracle 79 .79 .79 .79 .79 .79 .79 .+}

Original: [-− 0 .+3 .− 
-]{+RankMe 79 .
 .79 .α -ReQ+}
Revised: [-− 0 .+3 .− 
-]{+RankMe 79 .
 .79 .α -ReQ+}

Original: .={+CLEVR-Count ImageNet Oracle 43 .44 .43 .46 .
 RankMe 43 .44 .
 .46 .α+}
Revised: {+CLEVR-Count ImageNet Oracle 43 .44 .43 .46 .
 RankMe 43 .44 .
 .46 .α+}

Original: [Table]
Revised: -ReQ CIFAR

Original: ImageNet {+Oracle 80 .81 .80 .
+}
Revised: ImageNet {+Oracle 80 .81 .80 .
+}

Original: [Table]
Revised: [Table]

Original: [-RankMe 46 .
-]
Revised: [-RankMe 46 .
-]

Original: Difference +0 .=
Revised: [Table] [Table] [Table] [Table]

Original: =
Revised: [Table] [Table] [Table] [Table]

Revised: ReQ

Original: [-− 0 .+2 .− 
-]
Revised: [-− 0 .+2 .− 
-][Table] [Table] [Table] [Table]

Original: .=
Revised: [Table] [Table] [Table] [Table]

Original: G C OMPLETE PERFORMANCE TABLES
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [-5 , µ : 25 , ν :64 .31 .
 .42 .λ : 10 , µ : 25 , ν : .47 .λ : 15 , µ : 25 , ν :67 .
 .67 .51 .λ : 20 , µ : 25 , ν :67 .
-]
Revised: [-5 , µ : 25 , ν :64 .31 .
 .42 .λ : 10 , µ : 25 , ν : .47 .λ : 15 , µ : 25 , ν :67 .
 .67 .51 .λ : 20 , µ : 25 , ν :67 .
-][Table]

Original: [-.68 .54 .λ : 30 , µ : 25 , ν :67 .38 .
 .68 .56 .λ : 35 , µ : 25 , ν :67 .38 .50 .
-]
Revised: [-.68 .54 .λ : 30 , µ : 25 , ν :67 .38 .
 .68 .56 .λ : 35 , µ : 25 , ν :67 .38 .50 .
-][Table] [Table]

Original: .λ :
Revised: [Table]

Original: [-40 , µ : 25 , ν :67 .38 .50 .
 .68 .56 .λ : 45 , µ : 25 , ν :66 .
 .67 .57 .λ : 50 , µ : 25 , ν :66 .
-]
Revised: [-40 , µ : 25 , ν :67 .38 .50 .
 .68 .56 .λ : 45 , µ : 25 , ν :66 .
 .67 .57 .λ : 50 , µ : 25 , ν :66 .
-]K C OMPLETE PERFORMANCE TABLES

Original: [-.67 .57 .λ : 25 , µ : 25 , ν :59 .28 .
 .43 .λ : 25 , µ : 25 , ν :66 .35 .50 .
 .67 .51 .λ : 25 , µ : 25 , ν :68 .38 .51 .
 .68 .55 .λ : 25 , µ : 25 , ν :68 .38 .51 .
-]
Revised: [-.67 .57 .λ : 25 , µ : 25 , ν :59 .28 .
 .43 .λ : 25 , µ : 25 , ν :66 .35 .50 .
 .67 .51 .λ : 25 , µ : 25 , ν :68 .38 .51 .
 .68 .55 .λ : 25 , µ : 25 , ν :68 .38 .51 .
-][Table]

Original: [-.68 .55 .λ : 25 , µ : 25 , ν :62 .31 .
 .43 .λ : 25 , µ : 25 , ν :67 .38 .
 .68 .55 .λ : 25 , µ : 25 , ν :67 .38 .
 .68 .55 .λ : 25 , µ : 25 , ν :67 .38 .
-]
Revised: [-.68 .55 .λ : 25 , µ : 25 , ν :62 .31 .
 .43 .λ : 25 , µ : 25 , ν :67 .38 .
 .68 .55 .λ : 25 , µ : 25 , ν :67 .38 .
 .68 .55 .λ : 25 , µ : 25 , ν :67 .38 .
-]VICReg

Original: [-.68 .54 .λ : 25 , µ : 25 , ν :63 .38 .51 .
 .68 .55 .λ : 25 , µ : 25 , ν :54 .37 .51 .
 .67 .54 .λ : 25 , µ : 25 , ν :-]
Revised: [-.68 .54 .λ : 25 , µ : 25 , ν :63 .38 .51 .
 .68 .55 .λ : 25 , µ : 25 , ν :54 .37 .51 .
 .67 .54 .λ : 25 , µ : 25 , ν :-][Table]

Original: [-VICReg-exp λ : 1 , µ : 1 , ν : 2 , τ : 0 .67 .
 .68 .52 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .67 .38 .
 .68 .54 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .67 .38 .
 .68 .55 .λ : 1 , µ : 1 , ν : 2 , τ : 
-]
Revised: [-VICReg-exp λ : 1 , µ : 1 , ν : 2 , τ : 0 .67 .
 .68 .52 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .67 .38 .
 .68 .54 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .67 .38 .
 .68 .55 .λ : 1 , µ : 1 , ν : 2 , τ : 
-]exp

Original: .56 .λ : 1 , µ : 1 , ν : 2 , τ : [-0 .
-]
Revised: [Table] 1 , ν : 2 , τ : [-0 .
-][Table]

Original: .46 .λ
Revised: [Table]

Original:  : 1 , µ : 1 , ν [-: 2 , τ : 0 .
-]
Revised: [Table] λ : 1 , µ : 1 , ν [-: 2 , τ : 0 .
-][Table]

Original: .41 .λ : 1 , µ : 1 ,ν : 0 .
Revised: 2 , τ [Table] [Table] λ : 1 ,

Original:  1 , τ [-: 0 .
-]
Revised: [Table] [Table] 1 , τ [-: 0 .
-][Table]

Original: .39 .λ : 1 , µ : 1 , ν : 0 .
Revised: [Table] [Table] [Table]

Original: 5 , τ [-: 0 .
-]
Revised: 5 , τ [-: 0 .
-][Table]

Original: .55 .λ : 1 , µ : 1 , ν : 1 , τ : [-0 .
-]
Revised: [Table] : 1 , µ : 1 , ν : 1 , τ : [-0 .
-][Table]

Original: .λ
Revised: [Table]

Original:  1 , µ : 1 , ν : 4 , τ : [-0 .68 .
-]
Revised: λ : 1 , µ : 1 , ν : 4 , τ : [-0 .68 .
-][Table]

Original: .96 .69 .53 .λ : 1 , µ : 1 , ν : 8 , τ : [-0 .67 .37 .51 .
-]
Revised: [Table] [Table] λ : 1 , µ : 1 , ν : 8 , τ : [-0 .67 .37 .51 .
-][Table]

Original: .68 .52 .λ : 1 , µ : 1 ,[-ν : 16 , τ : 0 .
-]
Revised: [Table] [Table] : 1 , µ : 1 ,[-ν : 16 , τ : 0 .
-]

Original: [Table]
Revised: [Table]

Original: SimCLR d : 512 , τ[-: 0 .
-]
Revised: : 16 , τ[-: 0 .
-]

Original: [-.42 .d : 512 , τ : 0 .
-]
Revised: [-.42 .d : 512 , τ : 0 .
-][Table]

Original: .43 .d : 512 , τ : 0 [-.
-]
Revised: [Table] [Table] , τ : 0 [-.
-].  0 .

Revised: [Table]

Original: .49 .d : 512 , τ : 0 [-.
-]
Revised: d : 512 , τ : 0 [-.
-]. [Table]

Original: .68 .49 .d : 512 , τ [-: 
-]
Revised: [Table] d : 2048 , τ [-: 
-][Table]

Original: [-.67 .48 .d : 512 , τ : 0 .
-]
Revised: [-.67 .48 .d : 512 , τ : 0 .
-][Table] [Table]

Original: .67 .48 .d : 2048 , τ [-: 0 .
-]
Revised: [Table] d : 2048 , τ [-: 0 .
-][Table] [Table]

Original: .41 .d : 2048 , τ [-: 0 .
-]
Revised: [Table] d : 2048 , τ [-: 0 .
-][Table]

Original: [-.66 .44 .d : 2048 , τ : 0 .66 .36 .51 .
-]
Revised: [-.66 .44 .d : 2048 , τ : 0 .66 .36 .51 .
-][Table]

Original: .48 .d : 2048 , τ : 0 [-.66 .36 .
-]
Revised: [Table] d : 2048 , τ : 0 [-.66 .36 .
-].  0 .

Revised: [Table]

Original: .68 .50 .d : 2048 , τ : [-
-]
Revised: d : 2048 , τ : [-
-]0 . [Table]

Original: .49 .d : 2048 , τ[-: 0 .
-]
Revised: [Table] d : 2048 , τ[-: 0 .
-]

Original: .66 .48 .d : 2048 , τ[-: 
-]
Revised: [Table] [Table] [Table] d : 2048 , τ[-: 
-]

Revised: [Table]

Original: .42 .d : 2048 , τ[-: 0 .
-]
Revised: [Table] [Table] [Table] d : 2048 , τ[-: 0 .
-]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: d : 2048 , τ

Revised: [Table]

Revised: [Table]

Original: .66 .45 .d : 2048 , τ [-: 0 .67 .
-]
Revised: [Table] d : 2048 , τ [-: 0 .67 .
-][Table] [Table]

Original: .69 .51 .d : 2048 , τ [-: 0 .
-]
Revised: d : 2048 , τ [-: 0 .
-][Table] [Table]

Original: .68 .54 .d : 2048 , τ : 0 [-.
-]
Revised: [Table] d : 2048 , τ : 0 [-.
-].

Original: [-.69 .55 .d : 2048 , τ : 0 .67 .
-]
Revised: [-.69 .55 .d : 2048 , τ : 0 .67 .
-][Table]

Original: .68 .51 .d : 2048 , τ : 0 [-.
-]
Revised: [Table] d : 2048 , τ : 0 [-.
-].  0 .

Original: [-.68 .49 .d : 2048 , τ : 0 .
 .69 .54 .d : 2048 , τ : 0 .
-]
Revised: [-.68 .49 .d : 2048 , τ : 0 .
 .69 .54 .d : 2048 , τ : 0 .
-][Table]

Original: .69 .54 .d : 2048 , τ : 0 [-.
-]
Revised: d : 2048 , τ : 0 [-.
-].  0 .

Original: [-.69 .55 .d : 2048 , τ : 0 .
 .69 .55 .d : 2048 , τ : 0 .
-]
Revised: [-.69 .55 .d : 2048 , τ : 0 .
 .69 .55 .d : 2048 , τ : 0 .
-][Table]

Original: .68 .54 .d : 2048 , τ : 0 [-.
-]
Revised: d : 2048 , τ : 0 [-.
-].  0 .

Original: [-.68 .54 .d : 2048 , τ : 0 .66 .
 .68 .52 .d : 2048 , τ : 0 .67 .38 .
-]
Revised: [-.68 .54 .d : 2048 , τ : 0 .66 .
 .68 .52 .d : 2048 , τ : 0 .67 .38 .
-][Table]

Original: .68 .53 .d : 2048 , τ : 0 [-.
-]
Revised: d : 2048 , τ : 0 [-.
-].  0 .

Original: [-.69 .54 .d : 2048 , τ : 0 .
 .68 .54 .d : 2048 , τ : 0 .
-]
Revised: [-.69 .54 .d : 2048 , τ : 0 .
 .68 .54 .d : 2048 , τ : 0 .
-][Table]

Original: .68 .55 .d : 2048 , τ : 0 [-.
-]
Revised: d : 2048 , τ : 0 [-.
-].  0 .

Original: [Table]
Revised: [Table]

Original: [-ν :53 .
-]
Revised: [-ν :53 .
-]d : 2048 , τ : 0 .  0 . [Table] d : 2048 , τ : 0 .  0 .10 −d : 2048 , τ : 0 .

Revised: [Table]

Revised: [Table]

Original: [-.10 .λ : 10 , µ : 25 , ν :57 .14 .
-]
Revised: [-.10 .λ : 10 , µ : 25 , ν :57 .14 .
-][Table] [Table] [Table] Method

Original: [-.11 .λ : 15 , µ : 25 , ν :59 .14 .43 .91 .
-]
Revised: [-.11 .λ : 15 , µ : 25 , ν :59 .14 .43 .91 .
-]Run [Table] VICReg [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: .12 .λ : 20 , µ : 25 , ν :59 .14 .43 .91 .
 
Revised: [Table] [Table] VICReg-exp 67 .37 .51 .
 Method

Original: [-.11 .λ : 30 , µ : 25 , ν :59 .
-]
Revised: [-.11 .λ : 30 , µ : 25 , ν :59 .
-]Run [Table] Places [Table]

Revised: Cars

Original: [-.λ : 35 , µ : 25 , ν :58 .11 .41 .
-]
Revised: [-.λ : 35 , µ : 25 , ν :58 .11 .41 .
-][Table] [Table] [Table] [Table]

Original: [-.54 .11 .λ : 40 , µ : 25 , ν :56 .10 .40 .
-]
Revised: [-.54 .11 .λ : 40 , µ : 25 , ν :56 .10 .40 .
-][Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: .10 .λ : 45 , µ : 25 , ν :55 .

Revised: [Table] [Table] [Table] SimCLR 57 .

Original: .10 .λ : 50 , µ : 25 , ν :53 .

Revised: Method Run [Table] VICReg 63 .88 .69 .

Original: {+VICReg-exp 67 .
+}
Revised: {+VICReg-exp 67 .
+}

Original: .10 .λ : 25 , µ : 25 , ν :
Revised: Method [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [-.91 .54 .13 .λ : 25 , µ : 25 , ν :57 .
-]
Revised: [-.91 .54 .13 .λ : 25 , µ : 25 , ν :57 .
-][Table] [Table] [Table] SimCLR

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [-.12 .λ : 25 , µ : 25 , ν :59 .
-]
Revised: [-.12 .λ : 25 , µ : 25 , ν :59 .
-][Table] [Table] Method [Table]

Revised: VICReg

Original: [-.11 .λ : 25 , µ : 25 , ν :56 .
-]
Revised: [-.11 .λ : 25 , µ : 25 , ν :56 .
-][Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: .88 .53 .8 .λ : 25 , µ : 25 , ν :36 .
 
Revised: [Table] [Table] VICReg-exp 58 .12 .
 Method

Revised: Run

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [-.λ : 25 , µ : 25 , ν :59 .
-]
Revised: [-.λ : 25 , µ : 25 , ν :59 .
-][Table] [Table] [Table] SimCLR 43 .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [-.11 .λ : 25 , µ : 25 , ν :59 .
-]
Revised: [-.11 .λ : 25 , µ : 25 , ν :59 .
-][Table] [Table] [Table] [Table]

Original: [-.56 .11 .λ : 25 , µ : 25 , ν :59 .13 .
 .11 .λ : 25 , µ : 25 , ν :59 .14 .
-]
Revised: [-.56 .11 .λ : 25 , µ : 25 , ν :59 .13 .
 .11 .λ : 25 , µ : 25 , ν :59 .14 .
-]Method Run

Original: [-.12 .λ : 25 , µ : 25 , ν :56 .
-]
Revised: [-.12 .λ : 25 , µ : 25 , ν :56 .
-][Table]

Original: .12 .λ : 25 , µ : 25 , ν :49 .

Revised: count VICReg 26 .59 .25 .

Original: VICReg-exp [-λ : 1 , µ : 1 , ν : 2 , τ : 0 .
-]
Revised: VICReg-exp [-λ : 1 , µ : 1 , ν : 2 , τ : 0 .
-]58 .80 .52 .75 .79 .45 . Method

Original: [-.10 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .58 .
-]
Revised: [-.10 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .58 .
-][Table] [Table] [Table] [Table]

Original: [-.10 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .
-]
Revised: [-.10 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .
-][Table] [Table] [Table] [Table]

Original: [-.10 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .47 .
-]
Revised: [-.10 .λ : 1 , µ : 1 , ν : 2 , τ : 0 .47 .
-][Table] [Table] [Table] [Table]

Original: .λ
Revised: [Table] [Table] [Table] [Table]

Original: [-1 , µ : 1 , ν : 2 , τ : 0 .25 .
-]
Revised: [-1 , µ : 1 , ν : 2 , τ : 0 .25 .
-][Table] [Table] [Table] [Table]

Original: [-.λ : 1 , µ : 1 , ν : 2 , τ : 0 .19 .
-]
Revised: [-.λ : 1 , µ : 1 , ν : 2 , τ : 0 .19 .
-][Table] [Table] [Table] [Table]

Original: .λ : 1 , µ : 1 , ν : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: [-1 , τ : 0 .12 .
-]
Revised: [-1 , τ : 0 .12 .
-][Table]

Original: .λ : 1 , µ : 1 , ν : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: [-5 , τ : 0 .47 .
-]
Revised: [-5 , τ : 0 .47 .
-][Table]

Original: .87 .46 .8 .λ : 1 , µ : 1 , ν : 1 , τ : 0 .53 .7 .38 .89 .52 .9 .λ
Revised: [Table] [Table] [Table] [Table]

Original: [-: 1 , µ : 1 , ν : 4 , τ : 0 .58 .
-]
Revised: [-: 1 , µ : 1 , ν : 4 , τ : 0 .58 .
-][Table] [Table] [Table] [Table]

Original: [-.10 .λ : 1 , µ : 1 , ν : 8 , τ : 0 .58 .12 .
-]
Revised: [-.10 .λ : 1 , µ : 1 , ν : 8 , τ : 0 .58 .12 .
-][Table] [Table] [Table] [Table]

Original: [-.10 .λ : 1 , µ : 1 , ν : 16 , τ : 0 .
-]
Revised: [-.10 .λ : 1 , µ : 1 , ν : 16 , τ : 0 .
-][Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: SimCLR[-d : 512 , τ : 0 .
-]
Revised: SimCLR[-d : 512 , τ : 0 .
-]

Original: [-.22 .d : 512 , τ : 
 .58 .21 .d : 512 , τ : 0 .54 .
-]
Revised: [-.22 .d : 512 , τ : 
 .58 .21 .d : 512 , τ : 0 .54 .
-][Table] [Table] [Table]

Original: [-.16 .d : 512 , τ : 0 .50 .
-]
Revised: [-.16 .d : 512 , τ : 0 .50 .
-][Table]

Original: [-.88 .51 .11 .d : 512 , τ : 0 .45 .6 .36 .87 .48 .10 .d : 512 , τ : 0 .41 .
-]
Revised: [-.88 .51 .11 .d : 512 , τ : 0 .45 .6 .36 .87 .48 .10 .d : 512 , τ : 0 .41 .
-][Table] [Table] [Table] [Table]

Original: .d
Revised: [Table]

Original: [-2048 , τ : 0 .
-]
Revised: [-2048 , τ : 0 .
-][Table]

Original: [-.55 .23 .d : 2048 , τ : 
 .59 .22 .d : 2048 , τ : 0 .54 .
-]
Revised: [-.55 .23 .d : 2048 , τ : 
 .59 .22 .d : 2048 , τ : 0 .54 .
-][Table] [Table] [Table]

Original: [-.18 .d : 2048 , τ : 0 .50 .
 .54 .12 .d : 2048 , τ : 0 .43 .
-]
Revised: [-.18 .d : 2048 , τ : 0 .50 .
 .54 .12 .d : 2048 , τ : 0 .43 .
-][Table] [Table] [Table]

Original: .d
Revised: [Table]

Original: [-2048 , τ : 0 .41 .
-]
Revised: [-2048 , τ : 0 .41 .
-][Table]

Original: [-.34 .85 .48 .10 .d : 2048 , τ : 0 .44 .21 .
 .94 .56 .22 .d : 2048 , τ : 0 .51 .
-]
Revised: [-.34 .85 .48 .10 .d : 2048 , τ : 0 .44 .21 .
 .94 .56 .22 .d : 2048 , τ : 0 .51 .
-][Table] Method [Table]

Original: [-.59 .21 .d : 2048 , τ : 
 .59 .17 .d : 2048 , τ : 0 .56 .10 .41 .
-]
Revised: [-.59 .21 .d : 2048 , τ : 
 .59 .17 .d : 2048 , τ : 0 .56 .10 .41 .
-][Table] [Table] [Table]

Original: [-.56 .14 .d : 2048 , τ : 0 .54 .7 .
 .87 .54 .11 .d : 2048 , τ : 0 .49 .
-]
Revised: [-.56 .14 .d : 2048 , τ : 0 .54 .7 .
 .87 .54 .11 .d : 2048 , τ : 0 .49 .
-][Table] [Table] [Table]

Original: .10 .d : 2048 , τ : 0 .44 .3 .35 .86 .48 .9 .d
Revised: [Table] [Table] [Table] [Table]

Original: [-2048 , τ : 0 .
-]
Revised: [-2048 , τ : 0 .
-][Table]

Original: [-.10 .41 .89 .56 .13 .d : 2048 , τ : 0 .56 .10 .
 .90 .56 .13 .d : 2048 , τ : 0 .56 .10 .41 .
-]
Revised: [-.10 .41 .89 .56 .13 .d : 2048 , τ : 0 .56 .10 .
 .90 .56 .13 .d : 2048 , τ : 0 .56 .10 .41 .
-][Table] [Table] [Table]

Original: [-.56 .13 .d : 2048 , τ : 
 .89 .56 .13 .d : 2048 , τ : 0 .35 .10 .
-]
Revised: [-.56 .13 .d : 2048 , τ : 
 .89 .56 .13 .d : 2048 , τ : 0 .35 .10 .
-][Table] [Table] [Table]

Original: [-.89 .56 .13 .d : 2048 , τ : 0 .28 .
-]
Revised: [-.89 .56 .13 .d : 2048 , τ : 0 .28 .
-][Table]

Original: [-.56 .13 .d : 2048 , τ : 0 .52 .
-]
Revised: [-.56 .13 .d : 2048 , τ : 0 .52 .
-][Table]

Original: [-.90 .55 .13 .d : 2048 , τ : 0 .55 .
-]
Revised: [-.90 .55 .13 .d : 2048 , τ : 0 .55 .
-][Table] [Table] [Table]

Original: [-.89 .55 .13 .d : 2048 , τ : 0 .56 .10 .41 .
 .56 .13 .d : 2048 , τ : 0 .56 .10 .
 .14 .d : 2048 , τ : 0 .56 .10 .
 .13 .d : 2048 , τ : 0 .
-]
Revised: [-.89 .55 .13 .d : 2048 , τ : 0 .56 .10 .41 .
 .56 .13 .d : 2048 , τ : 0 .56 .10 .
 .14 .d : 2048 , τ : 0 .56 .10 .
 .13 .d : 2048 , τ : 0 .
-]VICReg-exp 1006 .530 .637 .501 .551 .142 .

Original: [Table]
Revised: [Table]

Original: : 25 , ν :      λ : 35 , µ : 25 , ν :      λ : 40 , µ : 25 , ν :      λ : 45 , µ : 25 , ν :      λ : 50 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :      λ : 25 , µ : 25 , ν :
Revised: Run [Table] [Table] [Table]

Original: VICReg-exp λ : 1 , µ : 1 , ν : 2 , τ : 0 .
Revised: [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 2 , τ : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 2 , τ : 0 .
Revised: [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 2 , τ : 0 .
Revised: [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 2 , τ : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 2 , τ : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 0 .
Revised: [Table] [Table] [Table]

Original: 1 , τ : 0 .
Revised: [Table]

Original: λ : 1 , µ : 1 , ν : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: 5 , τ : 0 .
Revised: [Table]

Original: λ : 1 , µ : 1 , ν : 1 , τ : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 4 , τ : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 8 , τ : 0 .
Revised: [Table] [Table] [Table] [Table]

Original: λ : 1 , µ : 1 , ν : 16 , τ : 0 .
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: SimCLRd : 512 , τ :
Revised: SimCLR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .
Revised: [Table]

Original: d : 2048 , τ : 0 .      d : 2048 , τ : 0 .
Revised: [Table] [Table] [Table]

Original: d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .
Revised: [Table] [Table]

Original: d : 2048 , τ : 0 .
Revised: [Table]

Original: d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .
Revised: [Table]

Original: d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .
Revised: [Table]

Original: d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .
Revised: [Table]

Original: d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .      d : 2048 , τ : 0 .
Revised: [Table]