Original: N O R EASON FOR N O S UPERVISION : I MPROVED G ENERALIZATION IN S UPERVISED M ODELS
Revised: N O R EASON FOR N O S UPERVISION : I MPROVED G ENERALIZATION IN S UPERVISED M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at both the training task as well as at other (future) transfer tasks.
Revised: We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at both the training task as well as at other (future) transfer tasks.

Original: These two seemingly contradictory properties impose a trade-off between improving the model’s generalization while maintaining its performance on the original task.
Revised: These two seemingly contradictory properties impose a trade-off between improving the model’s generalization while maintaining its performance on the original task.

Original: Models trained with self-supervised learning tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds.
Revised: Models trained with self-supervised learning tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds.

Original: We extensively analyse supervised training using multi-scale crops for data augmentation and an expendable projector head, and reveal that the design of the projector allows us to control the trade-off between performance on the training task and transferability.
Revised: We extensively analyse supervised training using multi-scale crops for data augmentation and an expendable projector head, and reveal that the design of the projector allows us to control the trade-off between performance on the training task and transferability.

Original: We further replace the last layer of class weights with class prototypes computed on the fly using a memory bank and derive two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A model on IN1K while performing better on transfer tasks.
Revised: We further replace the last layer of class weights with class prototypes computed on the fly using a memory bank and derive two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A model on IN1K while performing better on transfer tasks.

Original: Finally, we perform several analyses of the features and class weights to present insights on how each component of our setup affects the training and learned representations.
Revised: Finally, we perform several analyses of the features and class weights to present insights on how each component of our setup affects the training and learned representations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Previous SotA
Revised: PreviousSotA

Original: New SotA (this paper)
Revised: New SotA (this paper)

Original: RSB-A
Revised: [Figure]

Original: SupCon
Revised: SupCon

Revised: LOOK

Original: [Figure]
Revised: [Figure]

Original: * , two ResNet50 models trained with an improved supervised learning setup on ImageNet (IN1K), with strong performance on both transfer learning (y-axis, averaged over 13 tasks) and IN1K (x-axis), respectively outperforming DINO and RSB-A1 the state of the art (SotA) along each dimension.
Revised: * , two ResNet50 models trained with an improved supervised learning setup on ImageNet (IN1K), with strong performance on both transfer learning (y-axis, averaged over 13 tasks) and IN1K (x-axis), respectively outperforming DINO and RSB-A1 the state of the art (SotA) along each dimension.

Original: Deep convolutional neural networks trained on large annotated image sets like ImageNet-1K (IN1K) (Russakovsky et al., 2015) have shown strong generalization properties.
Revised: Deep convolutional neural networks trained on large annotated image sets like ImageNet-1K (IN1K) (Russakovsky et al., 2015) have shown strong generalization properties.

Original: This motivated their application to a broad range of transfer tasks including the recognition of concepts that are not encountered during training (Donahue et al., 2014; Razavian et al., 2014).
Revised: This motivated their application to a broad range of transfer tasks including the recognition of concepts that are not encountered during training (Donahue et al., 2014; Razavian et al., 2014).

Original: Recently, models trained in a self-supervised learning (SSL) framework have become popular due to their ability to learn without manual annotations, as well as their capacity to surpass supervised models in the context of transferable visual representations.
Revised: Recently, models trained in a self-supervised learning (SSL) framework have become popular due to their ability to learn without manual annotations, as well as their capacity to surpass supervised models in the context of transferable visual representations.

Original: SSL models like MoCo (He et al., 2020), SwAV (Caron et al., 2020), BYOL (Grill et al., 2020) or DINO (Caron et al., 2021) exhibit stronger transfer learning performance than models (Wightman et al., 2021) trained on the same data with annotations (Sariyildiz et al., 2021).
Revised: SSL models like MoCo (He et al., 2020), SwAV (Caron et al., 2020), BYOL (Grill et al., 2020) or DINO (Caron et al., 2021) exhibit stronger transfer learning performance than models (Wightman et al., 2021) trained on the same data with annotations (Sariyildiz et al., 2021).

Original: This achievement is on the one hand exciting, as SSL approaches do not require an expensive and errorprone annotation process, but also seemingly counter-intuitive (Wang et al., 2022b) as it suggests that access to additional information, i.e., image labels, actually hinders the generalization properties of a model.
Revised: This achievement is on the one hand exciting, as SSL approaches do not require an expensive and errorprone annotation process, but also seemingly counter-intuitive (Wang et al., 2022b) as it suggests that access to additional information, i.e., image labels, actually hinders the generalization properties of a model.

Original: Models learned via SSL are however not able to match their supervised counterparts on IN1K classification, i.e., on the concepts seen during training.
Revised: Models learned via SSL are however not able to match their supervised counterparts on IN1K classification, i.e., on the concepts seen during training.

Original: Top-performing SSL and semi-supervised methods like DINO (Caron et al., 2021) or PAWS (Assran et al., 2021) still result in 3-5% lower top-1 accuracy compared to optimized supervised models such as RSB-A1 (Wightman et al., 2021).
Revised: Top-performing SSL and semi-supervised methods like DINO (Caron et al., 2021) or PAWS (Assran et al., 2021) still result in 3-5% lower topaccuracy compared to optimized supervised models such as RSB-A1 (Wightman et al., 2021).

Original: In this paper, we argue that access to more information (in the form of manual annotations) should not hurt generalization, and we seek to improve the transferability of encoders learned in a supervised manner, while retaining their state-of-the-art performance on the supervised training task.
Revised: In this paper, we argue that access to more information (in the form of manual annotations) should not hurt generalization, and we seek to improve the transferability of encoders learned in a supervised manner, while retaining their state-of-the-art performance on the supervised training task.

Original: The mismatch observed between IN1K and transfer performance suggests that this goal is not trivial.
Revised: The mismatch observed between IN1K and transfer performance suggests that this goal is not trivial.

Original: It has been shown, for example, that popular regularization techniques such as Label Smoothing (Szegedy et al., 2016), Dropout (Srivastava et al., 2014) or CutMix (Yun et al., 2019), which improve IN1K performance, actually lead to less transferable representations (Kornblith et al., 2021;
Revised: It has been shown, for example, that popular regularization techniques such as Label Smoothing (Szegedy et al., 2016), Dropout (Srivastava et al., 2014) or CutMix (Yun et al., 2019), which improve IN1K performance, actually lead to less transferable representations (Kornblith et al., 2021;

Original: Sariyildiz et al., 2021).
Revised: Sariyildiz et al., 2021).

Original: We identify two key training components from the most successful SSL approaches that may lead to more transferable representations: multi-crop data augmentation (Caron et al., 2020) and the use of an expendable projector head, i.e., auxiliary modules added after the encoder during training and discarded at test time (Chen et al., 2020a).
Revised: We identify two key training components from the most successful SSL approaches that may lead to more transferable representations: multi-crop data augmentation (Caron et al., 2020) and the use of an expendable projector head, i.e., auxiliary modules added after the encoder during training and discarded at test time (Chen et al., 2020a).

Original: We study the impact of these two components on the transfer performance together with the performance on the training task, and present novel insights on role of the projector design in this context.
Revised: We study the impact of these two components on the transfer performance together with the performance on the training task, and present novel insights on role of the projector design in this context.

Original: Furthermore, inspired by recent work on supervised learning (Feng et al., 2022; Khosla et al., 2020), we introduce Online Class Means , a memory-efficient variant of the Nearest Class Means classifier (Mensink et al., 2012) that computes class prototypes in an “online” manner with the help of a memory queue.
Revised: Furthermore, inspired by recent work on supervised learning (Feng et al., 2022; Khosla et al., 2020), we introduce Online Class Means , a memory-efficient variant of the Nearest Class Means classifier (Mensink et al., 2012) that computes class prototypes in an “online” manner with the help of a memory queue.

Original: This further increases performance.
Revised: This further increases performance.

Original: We perform an extensive analysis on how each component affects the learned representations.
Revised: We perform an extensive analysis on how each component affects the learned representations.

Original: We look at feature sparsity and redundancy as well as intra-class distance.
Revised: We look at feature sparsity and redundancy as well as intra-class distance.

Original: We also study the training dynamics and show that class prototypes and classifier weights change in very different ways across iterations.
Revised: We also study the training dynamics and show that class prototypes and classifier weights change in very different ways across iterations.

Original: These observations motivate the design of a supervised learning setup that produces highly competitive models effective for both IN1K and transfer learning.
Revised: These observations motivate the design of a supervised learning setup that produces highly competitive models effective for both IN1K and transfer learning.

Original: We single out the two ResNet instantiations that perform best at one of these two dimensions, denoted as t-ReX and t-ReX* .
Revised: We single out the two ResNet instantiations that perform best at one of these two dimensions, denoted as t-ReX and t-ReX* .

Original: t-ReX exceeds the state-of-the-art transfer learning performance of DINO (Caron et al., 2021) or
Revised: t-ReX exceeds the state-of-the-art transfer learning performance of DINO (Caron et al., 2021) or

Original: PAWS (Assran et al., 2021) and still performs much better than these two on IN1K classification.
Revised: PAWS (Assran et al., 2021) and still performs much better than these two on IN1K classification.

Original: t-ReX* outperforms the state-of-the-art results of RSB-A1 (Wightman et al., 2021) on IN1K while generalizing better to transfer tasks.
Revised: t-ReX* outperforms the state-of-the-art results of RSB-A1 (Wightman et al., 2021) on IN1K while generalizing better to transfer tasks.

Original: We visualize the performance of these two selected models, together with those of other top-performing configurations from our setup in Fig.
Revised: We visualize the performance of these two selected models, together with those of other top-performing configurations from our setup in Fig.

Original: 1, and compare it to state-of-the-art supervised, semi-supervised and self-supervised learning methods, across two dimensions: IN1K accuracy and mean transfer accuracy across 13 transfer tasks.
Revised: 1, and compare it to state-of-the-art supervised, semi-supervised and self-supervised learning methods, across two dimensions: IN1K accuracy and mean transfer accuracy across 13 transfer tasks.

Original: This intuitively conveys how the proposed training setup pushes the envelope of the training-versus-transfer performance trade-off (from the “Previous SotA” region, to the “New SotA” one) and offers strong pretrained encoders that future approaches could build on.
Revised: This intuitively conveys how the proposed training setup pushes the envelope of the training-versus-transfer performance trade-off (from the “Previous SotA” region, to the “New SotA” one) and offers strong pretrained encoders that future approaches could build on.

Original: Contributions.
Revised: Contributions.

Original: We propose a supervised training setup that incorporates multi-crop data augmentation and an expendable projector and can produce models with favorable performance both on the training task of IN1K and on diverse transfer tasks.
Revised: We propose a supervised training setup that incorporates multi-crop data augmentation and an expendable projector and can produce models with favorable performance both on the training task of IN1K and on diverse transfer tasks.

Original: We thoroughly ablate this setup and reveal that the design of the projector allows to control the performance trade-off between these two dimensions, while a number of analyses of the features and class weights give insights on how each component of our setup affects the training and learned representations.
Revised: We thoroughly ablate this setup and reveal that the design of the projector allows to control the performance trade-off between these two dimensions, while a number of analyses of the features and class weights give insights on how each component of our setup affects the training and learned representations.

Original: We also introduce
Revised: We also introduce

Original: Online Class Means , a prototype-based training objective that increases performance even further and gives state-of-the-art models for transfer learning ( t-ReX ) and IN1K ( t-ReX* ).
Revised: Online Class Means , a prototype-based training objective that increases performance even further and gives state-of-the-art models for transfer learning ( t-ReX ) and IN1K ( t-ReX* ).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Visual representations learned by deep networks trained for IN1K classification can transfer to other tasks and datasets (Donahue et al., 2014; Razavian et al., 2014).
Revised: Visual representations learned by deep networks trained for IN1K classification can transfer to other tasks and datasets (Donahue et al., 2014; Razavian et al., 2014).

Original: This generalization capability of networks has motivated researchers to propose practical approaches for measuring transfer learning (Goyal et al., 2019;
Revised: This generalization capability of networks has motivated researchers to propose practical approaches for measuring transfer learning (Goyal et al., 2019;

Original: Pándy et al., 2022; Zhai et al., 2019) or contribute to a formal understanding of generalization properties (Kornblith et al., 2019; Tripuraneni et al., 2020; Yosinski et al., 2014).
Revised: Pándy et al., 2022; Zhai et al., 2019) or contribute to a formal understanding of generalization properties (Kornblith et al., 2019; Tripuraneni et al., 2020; Yosinski et al., 2014).

Original: Recent work in this context (Kornblith et al., 2021; Sariyildiz et al., 2021) shows that the best representations for IN1K are not necessarily the ones transferring best.
Revised: Recent work in this context (Kornblith et al., 2021; Sariyildiz et al., 2021) shows that the best representations for IN1K are not necessarily the ones transferring best.

Original: For instance, some regularization techniques or loss functions improving IN1K classification lead to underwhelming transfer results.
Revised: For instance, some regularization techniques or loss functions improving IN1K classification lead to underwhelming transfer results.

Original: A parallel line of work based on self-supervised learning (Caron et al., 2020; Chen et al., 2020a; Grill et al., 2020) focuses on training models without manual labels, and demonstrates their strong generalization capabilities to many transfer datasets, clearly surpassing their supervised 
Revised: A parallel line of work based on self-supervised learning (Caron et al., 2020; Chen et al., 2020a; Grill et al., 2020) focuses on training models without manual labels, and demonstrates their strong generalization capabilities to many transfer datasets, clearly surpassing their supervised counterparts (Sariyildiz et al., 2021).

Revised: Yet, as expected, SSL models are no match to the supervised models on the IN1K classification task itself.

Original: [Figure]
Revised: [Figure]

Original: counterparts A few approaches tackle the task of training supervised models that also transfer well and share motivation with our work.
Revised: A few approaches tackle the task of training supervised models that also transfer well and share motivation with our work.

Original: SupCon (Khosla et al., 2020) extends SimCLR (Chen et al., 2020a) using image labels to build positive pairs.
Revised: SupCon (Khosla et al., 2020) extends SimCLR (Chen et al., 2020a) using image labels to build positive pairs.

Original: As such, its formulation is close to neighborhood component analysis (NCA) (Goldberger et al., 2004).
Revised: As such, its formulation is close to neighborhood component analysis (NCA) (Goldberger et al., 2004).

Original: It circumvents the need for large batches by adding a momentum and a memory similar to MoCo (He et al., 2020).
Revised: It circumvents the need for large batches by adding a momentum and a memory similar to MoCo (He et al., 2020).

Original: Supervised-MoCo (Zhao et al., 2021) filters out false negatives in the memory bank of MoCo using image labels, while LOOK (Feng et al., 2022) modifies the NCA objective to only consider the closest neighbors of each query image.
Revised: Supervised-MoCo (Zhao et al., 2021) filters out false negatives in the memory bank of MoCo using image labels, while LOOK (Feng et al., 2022) modifies the NCA objective to only consider the closest neighbors of each query image.

Original: We experimentally observe that our model design leads to better transfer than all these works.
Revised: We experimentally observe that our model design leads to better transfer than all these works.

Original: In this work, we propose an effective training setup, which leverages multi-crop augmentation (Caron et al., 2020) and an expendable projector head (Chen et al., 2020a), two key components in many successful SSL approaches.
Revised: In this work, we propose an effective training setup, which leverages multi-crop augmentation (Caron et al., 2020) and an expendable projector head (Chen et al., 2020a), two key components in many successful SSL approaches.

Original: Creating multiple augmented versions (a.k.a. crops) of images in a batch was first proposed by Hoffer et al.
Revised: Creating multiple augmented versions (a.k.a. crops) of images in a batch was first proposed by Hoffer et al.

Original: (2020), while Caron et al.
Revised: (2020), while Caron et al.

Original: (2020) further consider crops with different scales and resolutions in a self-supervised learning setting, creating challenging views of an image for which the model is encouraged to learn consistent representations (Assran et al., 2021; Caron et al., 2021).
Revised: (2020) further consider crops with different scales and resolutions in a self-supervised learning setting, creating challenging views of an image for which the model is encouraged to learn consistent representations (Assran et al., 2021; Caron et al., 2021).

Original: A recent work (Wang et al., 2022a) argues that multi-crop increases representation variance and is useful for online self-distillation, while it has recently been shown to also improve vision and language pretraining (Ko & Gu, 2022).
Revised: A recent work (Wang et al., 2022a) argues that multi-crop increases representation variance and is useful for online self-distillation, while it has recently been shown to also improve vision and language pretraining (Ko & Gu, 2022).

Original: We show that multi-crop over different resolutions works out-of-the-box also for supervised training on IN1K.
Revised: We show that multi-crop over different resolutions works out-of-the-box also for supervised training on IN1K.

Original: Following the observation that the last layer of a model is not always the one producing features that transfer best, early SSL approaches (Zhang et al., 2016; Gidaris et al., 2018) would evaluate representations from multiple layers. 
Revised: Using features from intermediate layers of networks has been considered before, e.g., for training object detectors (Lin et al., 2017) or image classification (Lee et al., 2015) models, or evaluating the transferability of individual (Zhang et al., 2016; Gidaris et al., 2018) or group (Evci et al., 2022) of layers. However, selecting which layers are optimal for the problem is computationally infeasible due to its combinatorial nature.

Original: SimCLR (Chen et al., 2020a) proposed instead to rely on an expendable projector, a design that is now common practice in SSL (Zhou et al., 2022; Zbontar et al., 2021) and starts being adopted by supervised approaches like SupCon (Khosla et al., 2020) and LOOK (Feng et al., 2022).
Revised: SimCLR (Chen et al., 2020a) proposed instead to rely on an expendable projector, a design that is now common practice in SSL (Zhou et al., 2022; Zbontar et al., 2021) and starts being adopted by supervised approaches like SupCon (Khosla et al., 2020) and LOOK (Feng et al., 2022).

Original: The impact of those projectors on the representation quality has only seldomly been studied.
Revised: The impact of those projectors on the representation quality has only seldomly been studied.

Original: Wang et al.
Revised: Wang et al.

Original: (2022b) have looked at the impact of projectors, but only for transfer and in isolation.
Revised: (2022b) have looked at the impact of projectors, but only for transfer and in isolation.

Original: Our work goes one step further and studies how projectors affect performance both on the training task and for transfer.
Revised: Our work goes one step further and studies how projectors affect performance both on the training task and for transfer.

Original: We ablate many projector designs and study them jointly with multi-crop.
Revised: We ablate many projector designs and study them jointly with multi-crop.

Original: Through our study, we uncover how useful projectors are at navigating the trade-off between training and transfer performance, leading to state-of-the-art results on both dimensions.
Revised: Through our study, we uncover how useful projectors are at navigating the trade-off between training and transfer performance, leading to state-of-the-art results on both dimensions.

Original:  A N IMPROVED TRAINING SETUP FOR SUPERVISED LEARNING
Revised:  A N IMPROVED TRAINING SETUP FOR SUPERVISED LEARNING

Original: In this section we present an improved training setup for learning supervised models that achieve high performance on both IN1K classification and a diverse set of transfer tasks.
Revised: In this section we present an improved training setup for learning supervised models that achieve high performance on both IN1K classification and a diverse set of transfer tasks.

Original: Our setup trains a model (or encoder )
Revised: Our setup trains a model (or encoder )

Original: f θ , parameterized by θ .
Revised: f θ , parameterized by θ .

Original: This model encodes an image I into a transferable representation x ∈ R d .
Revised: This model encodes an image I into a transferable representation x ∈ R d .

Original: We follow the common protocol (Feng et al., 2022; Kornblith et al., 2021) and train all variants of our model on IN1K using a ResNet50 (He et al., 2016) encoder.
Revised: We follow the common protocol (Feng et al., 2022; Kornblith et al., 2021) and train all variants of our model on IN1K using a ResNet50 (He et al., 2016) encoder.

Original: This choice of encoder is influenced by recent observations (Wightman et al., 2021) that carefully optimized ResNet50 models perform on par with the best Vision Transformers (ViTs, Beyer et al.
Revised: This choice of encoder is influenced by recent observations (Wightman et al., 2021) that carefully optimized ResNet50 models perform on par with the best Vision Transformers (ViTs, Beyer et al.

Original: (2022)) of comparable size on IN1K. After training our models, we perform transfer learning.
Revised: (2022)) of comparable size on IN1K. After training our models, we perform transfer learning.

Original: We freeze the model’s parameters so they are only used to produce transferable representations ( x ) , to be appended with a linear classifier for each transfer task (e.g., IN1K or any other dataset, see Fig. 2b).
Revised: We freeze the model’s parameters so they are only used to produce transferable representations ( x ) , to be appended with a linear classifier for each transfer task (e.g., IN1K or any other dataset, see Fig. 2b).

Original: Our improved training setup enriches the standard supervised learning paradigm with multi-crop augmentation and an expendable projector head (see Fig.
Revised: Our improved training setup enriches the standard supervised learning paradigm with multi-crop augmentation and an expendable projector head (see Fig.

Original: 2a).
Revised: 2a).

Original: We train our models with one of the two following training objectives: the standard softmax cross entropy loss that learns class weights, or an online variant of nearest class means that is based on class prototypes computed on-the-fly from a memory bank.
Revised: We train our models with one of the two following training objectives: the standard softmax cross entropy loss that learns class weights, or an online variant of nearest class means that is based on class prototypes computed on-the-fly from a memory bank.

Original: We detail all the proposed improvements below.
Revised: We detail all the proposed improvements below.

Original: Multi-crop data augmentation.
Revised: Multi-crop data augmentation.

Original: Caron et al.
Revised: Caron et al.

Original: (2020) leveraged many image crops of multiple scales and different resolutions when learning invariance to data augmentation in the context of SSL.
Revised: (2020) leveraged many image crops of multiple scales and different resolutions when learning invariance to data augmentation in the context of SSL.

Original: Their data augmentation pipeline, termed multi-crop , is defined over two sets of global and local crops that respectively retain larger and smaller portions of an image.
Revised: Their data augmentation pipeline, termed multi-crop , is defined over two sets of global and local crops that respectively retain larger and smaller portions of an image.

Original: These crops are processed at different resolutions.
Revised: These crops are processed at different resolutions.

Original: We adapt this component to our supervised setup.
Revised: We adapt this component to our supervised setup.

Original: Given an input image I , we define two scale parameters, for global and local crops, which determine the size ratio between random crops and the image I .
Revised: Given an input image I , we define two scale parameters, for global and local crops, which determine the size ratio between random crops and the image I .

Original: We follow Caron et al.
Revised: We follow Caron et al.

Original: and resize global and local crops to 224 × 224 and 96 × 96 , respectively.
Revised: and resize global and local crops to 224 × 224 and 96 × 96 , respectively.

Original: We extract multiple global and local crops, respectively M g and M l .
Revised: We extract multiple global and local crops, respectively M g and M l .

Original: Fig.
Revised: Fig.

Original: 2a illustrates one global M g = 1 and four local M l = 4 crops.
Revised: 2a illustrates one global M g = 1 and four local M l = 4 crops.

Original: In Sec. 4, we explore the use of multi-crop for supervised learning, and study the effect of different hyper-parameters under that setting.
Revised: In Sec. 4, we explore the use of multi-crop for supervised learning, and study the effect of different hyper-parameters under that setting.

Original: Expendable projector head.
Revised: Expendable projector head.

Original: To countervail the lack of annotations, SSL approaches tackle proxy tasks, such as learning augmentation invariance.
Revised: To countervail the lack of annotations, SSL approaches tackle proxy tasks, such as learning augmentation invariance.

Original: In order to prevent the encoder from learning representations that overfit to a potentially unimportant pretext task, SSL architectures often introduce an expendable projector between the encoder and the loss function.
Revised: In order to prevent the encoder from learning representations that overfit to a potentially unimportant pretext task, SSL architectures often introduce an expendable projector between the encoder and the loss function.

Original: On the contrary, for supervised learning, performance on the training task is a major goal in its own right.
Revised: On the contrary, for supervised learning, performance on the training task is a major goal in its own right.

Original: Here, we aim to learn supervised models that perform well on the training and on transfer tasks.
Revised: Here, we aim to learn supervised models that perform well on the training and on transfer tasks.

Original: These two requirements are not aligned and it is necessary to find a trade-off (Kornblith et al., 2021).
Revised: These two requirements are not aligned and it is necessary to find a trade-off (Kornblith et al., 2021).

Original: BatchNorm + GELU
Revised: BatchNorm + GELU

Original: Fully-connected ( ) -normalization -normalization
Revised: Fully-connected ( ) -normalization -normalization

Original: L
Revised: L

Original: Fully-connected ( )
Revised: Fully-connected ( )

Original: We argue that one can control this trade-off using an additional projector in the context of supervised learning.
Revised: We argue that one can control this trade-off using an additional projector in the context of supervised learning.

Original: Similar to SSL methods (Chen et al., 2020a;b; Chen & He, 2021) and to the recent SL-MLP (Wang et al., 2022b)
Revised: Similar to SSL methods (Chen et al., 2020a;b; Chen & He, 2021) and to the recent SL-MLP (Wang et al., 2022b)

Original: we introduce a Multi Layer Perceptron (MLP) projector as part of our supervised training pipeline.
Revised: we introduce a Multi Layer Perceptron (MLP) projector as part of our supervised training pipeline.

Original: Let g ϕ : R d → R d b denote this projector, parameterized by ϕ .
Revised: Let g ϕ : R d → R d b denote this projector, parameterized by ϕ .

Original: g ϕ is composed of an MLP with L hidden layers followed by a linear projection to a bottleneck of d b dimensions.
Revised: g ϕ is composed of an MLP with L hidden layers followed by a linear projection to a bottleneck of d b dimensions.

Original: Each hidden layer is composed of a sequence of a linear fully-connected layer, batch-normalization (Ioffe & Szegedy, 2015) and a GeLU (Hendrycks & Gimpel, 2016) non-linearity.
Revised: Each hidden layer is composed of a sequence of a linear fully-connected layer, batch-normalization (Ioffe & Szegedy, 2015) and a GeLU (Hendrycks & Gimpel, 2016) non-linearity.

Original: We further apply ℓ 2 -normalization to the output of g ϕ and optionally also to the input.
Revised: We further apply ℓ 2 -normalization to the output of g ϕ and optionally also to the input.

Original: We illustrate this architecture in Fig.
Revised: We illustrate this architecture in Fig.

Original: Note that SL-MLP (Wang et al., 2022b) uses a similar head but with only one hidden layer and no input or output ℓ 2 -normalization, so
Revised: Note that SL-MLP (Wang et al., 2022b) uses a similar head but with only one hidden layer and no input or output ℓ 2 -normalization, so

Original: SL-MLP can be seen as a special case of our projector architecture.
Revised: SL-MLP can be seen as a special case of our projector architecture.

Original: We compare to their design in Sec. 4 and investigate how the number and dimension of hidden layers among other design choices affect the transfer performance of the learned models, verifying and extending the findings of Wang et al.
Revised: We compare to their design in Sec. 4 and investigate how the number and dimension of hidden layers among other design choices affect the transfer performance of the learned models, verifying and extending the findings of Wang et al.

Original: (2022b).
Revised: (2022b).

Original: On top of this, we study transfer performance in juxtaposition to performance on the training task , and derive the novel insight that projector design allows to control the trade-off between performance on the training task and transferability.
Revised: On top of this, we study transfer performance in juxtaposition to performance on the training task , and derive the novel insight that projector design allows to control the trade-off between performance on the training task and transferability.

Original: Cosine softmax cross-entropy loss.
Revised: Cosine softmax cross-entropy loss.

Original: Incorporating both components described above in a standard supervised learning paradigm, we can train with the standard softmax cross entropy loss using class labels.
Revised: Incorporating both components described above in a standard supervised learning paradigm, we can train with the standard softmax cross entropy loss using class labels.

Original: The training pipeline is illustrated in Fig.
Revised: The training pipeline is illustrated in Fig.

Original: 2a.
Revised: 2a.

Original: It uses multi-crop data augmentation on each input image I to produce M = M g + M l crops I j , j = 1 , . . .
Revised: It uses multi-crop data augmentation on each input image I to produce M = M g + M l crops I j , j = 1 , . . .

Original: , M .
Revised: , M .

Original: Each crop is individually input to the network composed of the encoder followed by the projector, and produces an embedding z j = g ϕ
Revised: Each crop is individually input to the network composed of the encoder followed by the projector, and produces an embedding z j = g ϕ

Original: ( f θ ( I j )) .
Revised: ( f θ ( I j )) .

Original: To predict class labels, we multiply embeddings with trainable class weights
Revised: To predict class labels, we multiply embeddings with trainable class weights

Original: W = { w c ∈ R d b } C c =1 , where C is the number of classes.
Revised: W = { w c ∈ R d b } Cc =1 , where C is the number of classes.

Original: We train the whole pipeline using the cosine softmax loss as it was shown to improve IN1K performance (Kornblith et al., 2021):
Revised: We train the whole pipeline using the cosine softmax loss as it was shown to improve IN1K performance (Kornblith et al., 2021):

Original: [Equation]
Revised: [Equation]

Original: where y ∈ { 0 , 1 } C is the C -dim one-hot label vector corresponding to image I , τ is a temperature hyper-parameter and ¯ w c = w c / ∥ w c ∥ .
Revised: where y ∈ { 0 , 1 } C is the C -dim one-hot label vector corresponding to image I , τ is a temperature hyper-parameter and ¯ w c = w c / ∥ w c ∥ .

Original: Note that projector outputs z are already ℓ 2 -normalized.
Revised: Note that projector outputs z are already ℓ 2 -normalized.

Original: Online Class Means.
Revised: Online Class Means.

Original: Motivated by the recent success of momentum encoders as a way of maintaining online memory banks for large-scale training (He et al., 2020), we revisit the prototype-based Nearest
Revised: Motivated by the recent success of momentum encoders as a way of maintaining online memory banks for large-scale training (He et al., 2020), we revisit the prototype-based Nearest

Original: Class Means (NCM) approach of Mensink et al. (2012) and introduce a memory-efficient variant that computes class prototypes in an “online” manner with the help of a memory queue.
Revised: Class Means (NCM) approach of Mensink et al. (2012) and introduce a memory-efficient variant that computes class prototypes in an “online” manner with the help of a memory queue.

Original: Concretely, following Mensink et al.
Revised: Concretely, following Mensink et al.

Original: , we define u c to be the class prototype or class mean for class c , i.e., the mean of all embeddings from that class, and define U = { u c } Cc =1 .
Revised: , we define u c to be the class prototype or class mean for class c , i.e., the mean of all embeddings from that class, and define U = { u c } Cc =1 .

Original: Given that we jointly learn class means and the embeddings, computing the exact mean at each iteration is computationally prohibitive.
Revised: Given that we jointly learn class means and the embeddings, computing the exact mean at each iteration is computationally prohibitive.

Original: Instead, we formulate an online version of NCM that uses a memory bank Q which stores ℓ 2 -normalized embeddings z output by the projector, similar to the memory bank from MoCo (He et al., 2020).
Revised: Instead, we formulate an online version of NCM that uses a memory bank Q which stores ℓ 2 -normalized embeddings z output by the projector, similar to the memory bank from MoCo (He et al., 2020).

Original: Given the memory Q , we do not learn class weights, but instead compute a prototype for each class, on-the-fly, as the average of the embeddings in the memory which belong to that class.
Revised: Given the memory Q , we do not learn class weights, but instead compute a prototype for each class, on-the-fly, as the average of the embeddings in the memory which belong to that class.

Original: Formally, if Q c denotes samples in memory that belong to class c , and
Revised: Formally, if Q c denotes samples in memory that belong to class c , and

Original: N c
Revised: N c

Original: =
Revised: =

Original: |Q c | , the loss function becomes:
Revised: |Q c | , the loss function becomes:

Original: [Equation]
Revised: [Equation]

Original: We refer to the above training objective as Online Class Means or OCM .
Revised: We refer to the above training objective as Online Class Means or OCM .

Original: To make sure the embeddings stored in the memory remain relevant as the encoder is updated during training, we follow MoCo (He et al., 2020) and store in memory embeddings from an exponential moving average (EMA) model trailing f θ and g ϕ .
Revised: To make sure the embeddings stored in the memory remain relevant as the encoder is updated during training, we follow MoCo (He et al., 2020) and store in memory embeddings from an exponential moving average (EMA) model trailing f θ and g ϕ .

Original: As we show in our analysis in Sec. 4.2, estimating class prototypes using only the relatively small subset of samples in the memory bank leads to class prototypes that drift more across iterations compared to SGD-optimized class weights that converge faster.
Revised: As we show in our analysis in Sec. 4.2, estimating class prototypes using only the relatively small subset of samples in the memory bank leads to class prototypes that drift more across iterations compared to SGD-optimized class weights that converge faster.

Original: Further details as well as other variants are discussed in Appendix A.
Revised: Further details as well as other variants are discussed in Appendix A.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our analysis begins with a summary in Sec.
Revised: Our analysis begins with a summary in Sec.

Original: 4.1 of the main findings from an exhaustive study on the design of the main components of our setup, i.e., multi-crop augmentation, projectors, and OCM.
Revised: 4.1 of the main findings from an exhaustive study on the design of the main components of our setup, i.e., multi-crop augmentation, projectors, and OCM.

Original: We then analyze the learned representations, class weights, and prototypes in Sec. 4.2.
Revised: We then analyze the learned representations, class weights, and prototypes in Sec. 4.2.

Original: There, we explore how each component affects several facets like feature sparsity and redundancy, average coding length, as well as intra-class distance.
Revised: There, we explore how each component affects several facets like feature sparsity and redundancy, average coding length, as well as intra-class distance.

Original: We also study training dynamics like gradient similarity for multi-crop or how prototypes and classifier weights change across iterations for OCM.
Revised: We also study training dynamics like gradient similarity for multi-crop or how prototypes and classifier weights change across iterations for OCM.

Original: Finally, in Sec. 4.3 we plot the performance of multiple variants of the proposed training setup on the training-versus-transfer performance plane, empirically verifying its superiority over the current state of the art.
Revised: Finally, in Sec. 4.3 we plot the performance of multiple variants of the proposed training setup on the training-versus-transfer performance plane, empirically verifying its superiority over the current state of the art.

Original: Protocol.
Revised: Protocol.

Original: All our models are trained on the training set of ImageNet-1K (IN1K)
Revised: All our models are trained on the training set of ImageNet-1K (IN1K)

Original: (Russakovsky et al., 2015).
Revised: (Russakovsky et al., 2015).

Original: Due to the computational cost of training models on IN1K, each configuration is trained only once.
Revised: Due to the computational cost of training models on IN1K, each configuration is trained only once.

Original: Given an IN1K-trained model, we discard all the training-specific modules (e.g., the projector g ϕ , the class weights W ), and use the encoder f θ as a feature extractor, similar to Kornblith et al.
Revised: Given an IN1K-trained model, we discard all the training-specific modules (e.g., the projector g ϕ , the class weights W ), and use the encoder f θ as a feature extractor, similar to Kornblith et al.

Original: ; Sariyildiz et al.
Revised: ; Sariyildiz et al.

Original: For each dataset we evaluate on, we learn a linear logistic regression classifier with the pre-extracted features and independently optimize each classifier’s hyper-parameters for every model and every evaluation dataset using Scikit-learn (Pedregosa et al., 2011) and Optuna (Akiba et al., 2019)
Revised: For each dataset we evaluate on, we learn a linear logistic regression classifier with the pre-extracted features and independently optimize each classifier’s hyper-parameters for every model and every evaluation dataset using Scikit-learn (Pedregosa et al., 2011) and Optuna (Akiba et al., 2019)

Original: (see details in Appendix B).
Revised: (see details in Appendix B).

Original: We repeat this process 5 times with different random seeds and report the average accuracy (variance is negligible).
Revised: We repeat this process 5 times with different random seeds and report the average accuracy (variance is negligible).

Original: Note that the feature extractor is never fine-tuned, and, because we start from pre-extracted features, no additional data augmentation is used when learning the linear classifiers.
Revised: Note that the feature extractor is never fine-tuned, and, because we start from pre-extracted features, no additional data augmentation is used when learning the linear classifiers.

Original: This protocol is illustrated in Fig. 2b.
Revised: This protocol is illustrated in Fig. 2b.

Original: Evaluation datasets and measures.
Revised: Evaluation datasets and measures.

Original: We measure performance on the training task by evaluating classification accuracy on the IN1K validation set.
Revised: We measure performance on the training task by evaluating classification accuracy on the IN1K validation set.

Original: To evaluate transfer learning, we measure 
Revised: To evaluate transfer learning, we measure classification performance on 13 datasets: the 5 ImageNet-CoG datasets (Sariyildiz et al., 2021) that were recently proposed to measure concept generalization, and 8 commonly used smaller-scale datasets: Aircraft (Maji et al., 2013), Cars196 (Krause et al., 2013), DTD (Cimpoi et al., 2014),

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: classification performance on 13 datasets: the 5 ImageNet-CoG datasets (Sariyildiz et al., 2021) that were recently proposed to measure concept generalization, and 8 commonly used smaller-scale datasets: Aircraft (Maji et al., 2013), Cars196 (Krause et al., 2013), DTD (Cimpoi et al., 2014),
Revised: [Figure]

Original: EuroSAT (Helber et al., 2019), Flowers (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012),
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: EuroSAT (Helber et al., 2019), Flowers (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012),

Original: Food101 (Bossard et al., 2014) and SUN397 (Xiao et al., 2010).
Revised: Food101 (Bossard et al., 2014) and SUN397 (Xiao et al., 2010).

Original: We report two metrics: Topaccuracy on IN1K and transfer accuracy via log-odds (Kornblith et al., 2019) averaged over thetransfer datasets.
Revised: We report two metrics: Topaccuracy on IN1K and transfer accuracy via log-odds (Kornblith et al., 2019) averaged over thetransfer datasets.

Original: Note that we provide details on the datasets, the exact log-odds formulation, and per dataset results in the Appendix, respectively in Tab. 4, Appendix B.3, and Tab. 7.
Revised: Note that we provide details on the datasets, the exact log-odds formulation, and per dataset results in the Appendix, respectively in Tab. 4, Appendix B.3, and Tab. 7.

Original: Appendix C. also presents evaluations on IN1K-Sketch (Wang et al., 2019) and IN1K-v2 (Recht et al., 2019).
Revised: Appendix C. also presents evaluations on IN1K-Sketch (Wang et al., 2019) and IN1K-v2 (Recht et al., 2019).

Original: Implementation details.
Revised: Implementation details.

Original: f θ is a ResNet50 (He et al., 2016) encoder, trained for 100 epochs with mixed precision in PyTorch (Paszke et al., 2019) using 4 GPUs where batch norm layers are synchronized.
Revised: f θ is a ResNet50 (He et al., 2016) encoder, trained for 100 epochs with mixed precision in PyTorch (Paszke et al., 2019) using 4 GPUs where batch norm layers are synchronized.

Original: We use an SGD optimizer with 0.9 momentum, a batch size of 256, 1e-4 weight decay and a learning rate of 0 .
Revised: We use an SGD optimizer with 0.9 momentum, a batch size of 256, 1e-4 weight decay and a learning rate of 0 .

Original: 1 × batch size / 256 , which is linearly increased during the first 10 epochs and then decayed with a cosine schedule.
Revised: 1 × batch size / 256 , which is linearly increased during the first 10 epochs and then decayed with a cosine schedule.

Original: We set τ = 0 .
Revised: We set τ = 0 .

Original: 1 and, unless otherwise stated, we use the data augmentation pipeline from DINO (Caron et al., 2021) with 1 global and 8 local crops ( M g = 1 and
Revised: 1 and, unless otherwise stated, we use the data augmentation pipeline from DINO (Caron et al., 2021) with 1 global and 8 local crops ( M g = 1 and

Original: M l = 8).
Revised: M l = 8).

Original: A detailed list of the training hyper-parameters is given in the Appendix (Tab. 3).
Revised: A detailed list of the training hyper-parameters is given in the Appendix (Tab. 3).

Original: Training one of our models takes up to 3 days with 4 V100 GPUs depending on its projector configuration.
Revised: Training one of our models takes up to 3 days with 4 V100 GPUs depending on its projector configuration.

Original: 4. A NALYSIS OF COMPONENT DESIGN AND HYPER - PARAMETERS
Revised: 4. A NALYSIS OF COMPONENT DESIGN AND HYPER - PARAMETERS

Original: Multi-crop data augmentation.
Revised: Multi-crop data augmentation.

Original: We start by studying the effect that the number of local crops has on IN1K and transfer performance.
Revised: We start by studying the effect that the number of local crops has on IN1K and transfer performance.

Original: We train supervised models using Eq.
Revised: We train supervised models using Eq.

Original: (1) with 1 global and 2, 4, 6 or 8 local crops, and projectors composed of 1, 2 or 3 hidden layers, and report results in Fig.
Revised: (1) with 1 global and 2, 4, 6 or 8 local crops, and projectors composed of 1, 2 or 3 hidden layers, and report results in Fig.

Original: Our main observations are: a) training with local crops improves the performance on both IN1K and transfer tasks, and b) although increasing the number of local crops generally helps, performance saturates with 8 local crops.
Revised: Our main observations are: a) training with local crops improves the performance on both IN1K and transfer tasks, and b) although increasing the number of local crops generally helps, performance saturates with 8 local crops.

Original: We set M g = 1 and M l = 8 for all subsequent evaluations.
Revised: We set M g = 1 and M l = 8 for all subsequent evaluations.

Original: Further ablations using different scale and resolution parameters are presented in Appendix C.3.
Revised: Further ablations using different scale and resolution parameters are presented in Appendix C.3.

Original: Note that using local crops increases the effective batch size, which, in turn, increases training time.
Revised: Note that using local crops increases the effective batch size, which, in turn, increases training time.

Original: We therefore conduct two experiments to see if a longer training or a larger batch size would lead to similar gains.
Revised: We therefore conduct two experiments to see if a longer training or a larger batch size would lead to similar gains.

Original: We train two models using a single crop sampled from a wide scale range (i.e., able to focus on both large and small image regions), one with 9 × larger batch size, the other for 800 epochs.
Revised: We train two models using a single crop sampled from a wide scale range (i.e., able to focus on both large and small image regions), one with 9 × larger batch size, the other for 800 epochs.

Original: Unlike multi-crop, these models bring no significant gain (see Tab. 8 in Appendix for details).
Revised: Unlike multi-crop, these models bring no significant gain (see Tab. 8 in Appendix for details).

Original: Expendable projector head.
Revised: Expendable projector head.

Original: We study the impact of different architectural choices and hyperparameters for the projector.
Revised: We study the impact of different architectural choices and hyperparameters for the projector.

Original: We vary the number of hidden layers ( L ), the dimension of the hidden ( d h ) and bottleneck ( d b ) layers, and whether or not to ℓ 2 -normalize the projector input ( ℓ 2 ).
Revised: We vary the number of hidden layers ( L ), the dimension of the hidden ( d h ) and bottleneck ( d b ) layers, and whether or not to ℓ 2 -normalize the projector input ( ℓ 2 ).

Original: We start from a default configuration: L = 1 , d h = 2048 , d b = 256 and with ℓ 2 -normalized inputs.
Revised: We start from a default configuration: L = 1 , d h = 2048 , d b = 256 and with ℓ 2 -normalized inputs.

Original: We ablate each parameter separately by training models optimizing Eq.
Revised: We ablate each parameter separately by training models optimizing Eq.

Original: We use multi-crop in all cases.
Revised: We use multi-crop in all cases.

Original: The most interesting results from this analysis are presented in Tab. 1.
Revised: The most interesting results from this analysis are presented in Tab. 1.

Original: We see that the number of hidden layers ( L ) is an important hyper-parameter that controls the trade-off between IN1K and transfer performance .
Revised: We see that the number of hidden layers ( L ) is an important hyper-parameter that controls the trade-off between IN1K and transfer performance .

Original: Adding a projector head with a single hidden layer not only improves the already strong IN1K performance of multi-crop (Base+Mc in Tab. 1), but also significantly boosts its average transfer performance.
Revised: Adding a projector head with a single hidden layer not only improves the already strong IN1K performance of multi-crop (Base+Mc in Tab. 1), but also significantly boosts its average transfer performance.

Original: More hidden layers seem to increase transfer performance, at the cost of a decrease in IN1K accuracy.
Revised: More hidden layers seem to increase transfer performance, at the cost of a decrease in IN1K accuracy.

Original: The same can be said about the dimension of the hidden layer, yet[Figure] [Figure] we further see that a larger d h significantly increases transfer performance, and moderately decreases
Revised: The same can be said about the dimension of the hidden layer, yet we further see that a larger d h significantly increases transfer performance, and moderately decreases

Original: IN1K accuracy.
Revised: IN1K accuracy.

Original: On the contrary, we observe the bottleneck dimension d b and input ℓ 2 -normalization only have a small influence on IN1K and transfer performance (see Appendix C.4).
Revised: On the contrary, we observe the bottleneck dimension d b and input ℓ 2 -normalization only have a small influence on IN1K and transfer performance (see Appendix C.4).

Original: Overall, our  observations verify and significantly extend the ones recently presented by Wang et al.
Revised: Overall, our [Figure] [Figure] observations verify and significantly extend the ones recently presented by Wang et al.

Original: (2022b).
Revised: (2022b).

Original: We not only study the design of projectors jointly with multi-crop, but also, more importantly, analyse transfer performance jointly with performance on IN1K, revealing a trade-off between the two, that is fully controlled through the design of the project head.
Revised: We not only study the design of projectors jointly with multi-crop, but also, more importantly, analyse transfer performance jointly with performance on IN1K, revealing a trade-off between the two, that is fully controlled through the design of the project head.

Original: Online class means.
Revised: Online class means.

Original: There are two main hyper-parameters in OCM: the size of the memory bank and the momentum of the EMA models that populate the memory bank and provide the embeddings for class prototypes.
Revised: There are two main hyper-parameters in OCM: the size of the memory bank and the momentum of the EMA models that populate the memory bank and provide the embeddings for class prototypes.

Original: We explored momentum values 0 and 0 .
Revised: We explored momentum values 0 and 0 .

Original: 999 (in the former, we directly use f θ and g ϕ to compute prototypes) and see that trailing EMA is essential for maintaining high performance, i.e., momentum = 0 performs poorly, aligning with the observations for MoCo (He et al., 2020).
Revised: 999 (in the former, we directly use f θ and g ϕ to compute prototypes) and see that trailing EMA is essential for maintaining high performance, i.e., momentum = 0 performs poorly, aligning with the observations for MoCo (He et al., 2020).

Original: However, unlike MoCo or other recent methods such as LOOK (Feng et al., 2022), OCM does not require a large memory bank to achieve the highest performance.
Revised: However, unlike MoCo or other recent methods such as LOOK (Feng et al., 2022), OCM does not require a large memory bank to achieve the highest performance.

Original: We experimented with memory sizes between 2048 and 65546, and found that 8192 works best (see Fig. 12b in Appendix).
Revised: We experimented with memory sizes between 2048 and 65546, and found that 8192 works best (see Fig. 12b in Appendix).

Original: 4. A NALYSIS OF LEARNED FEATURES , CLASS WEIGHTS AND PROTOTYPES
Revised: 4. A NALYSIS OF LEARNED FEATURES , CLASS WEIGHTS AND PROTOTYPES

Original: We now investigate how different components of our setup affect training or the learned representations.
Revised: We now investigate how different components of our setup affect training or the learned representations.

Original: We analyse the features produced, class weights and prototypes from the following models: a)
Revised: We analyse the features produced, class weights and prototypes from the following models: a)

Original: Base : a model trained using cosine softmax loss without multi-crop and projector, b) Base (BS=2K) :
Revised: Base : a model trained using cosine softmax loss without multi-crop and projector, b) Base (BS=2K) :

Original: Base but with 9 × larger batch size, c) Base+Mc : Base with multi-crop, d) Base+Pr : Base with a projector, e) Base+Mc+Pr , and f) OCM : a model trained using Eq.
Revised: Base but with 9 × larger batch size, c) Base+Mc : Base with multi-crop, d) Base+Pr : Base with a projector, e) Base+Mc+Pr , and f) OCM : a model trained using Eq.

Original: (2); details in Appendix C.6.
Revised: (2); details in Appendix C.6.

Original: Intra-class distance.
Revised: Intra-class distance.

Original: We start by analysing the ℓ 2 -normalized features for the four models, Base,
Revised: We start by analysing the ℓ 2 -normalized features for the four models, Base,

Original: Base+Mc, Base+Pr and Base+Mc+Pr, by computing the average ℓ 2 -distance between samples from the same class (i.e., intra-class distance).
Revised: Base+Mc, Base+Pr and Base+Mc+Pr, by computing the average ℓ 2 -distance between samples from the same class (i.e., intra-class distance).

Original: We see in Fig.
Revised: We see in Fig.

Original: 5 (left) that multi-crop reduces intra-class distance on IN1K, while projectors increase it.
Revised: 5 (left) that multi-crop reduces intra-class distance on IN1K, while projectors increase it.

Original: Not surprisingly, this correlates with training task performance, i.e., lower intra-class distance translates to better performance on IN1K. On the transfer datasets, however, we found no strong correlation between the two, i.e., transfer performance does not necessarily depend on intra-class distance.
Revised: Not surprisingly, this correlates with training task performance, i.e., lower intra-class distance translates to better performance on IN1K. On the transfer datasets, however, we found no strong correlation between the two, i.e., transfer performance does not necessarily depend on intra-class distance.

Original: Sparsity.
Revised: Sparsity.

Original: In Fig.
Revised: In Fig.

Original: 5 (right)
Revised: 5 (right)

Original: we report feature sparsity ratio , i.e., the percentage of feature dimensions close to zero for ℓ 2 -normalized features from the four models.
Revised: we report feature sparsity ratio , i.e., the percentage of feature dimensions close to zero for ℓ 2 -normalized features from the four models.

Original: We see that: a) the average sparsity 
Revised: We see that: a) the average sparsity ratio on the transfer datasets is inversely correlated with performance, i.e., linear classifiers trained on less sparse features achieve better transfer performance , and b) projectors dramatically reduce sparsity . We find this last observation intuitive: features from the layer right before the cross-entropy

Original: [Figure]
Revised: [Figure]

Original: M
Revised: n

Original: [Figure]
Revised: [Figure]

Original: n T ra
Revised: r

Original: [Figure]
Revised: [Figure]

Original: s f e r
Revised: s f e r

Original: A cc .
Revised: A cc .

Original: ( L og

Original: [Figure]
Revised: [Figure]

Original: ratio on the transfer datasets is inversely correlated with performance, i e., linear classifiers trained on less sparse features achieve better transfer performance and b) projectors dramatically reduce sparsity .

Original: We find this last observation intuitive: features from the layer right before the cross-entropy loss are encouraged to be as close to a one-hot vector as possible and therefore sparse.
Revised: loss are encouraged to be as close to a one-hot vector as possible and therefore sparse.

Original: Introducing projectors in between allows the encoder to output less sparse features, which improves transfer.
Revised: Introducing projectors in between allows the encoder to output less sparse features, which improves transfer.

Original: Coding length.
Revised: Coding length.

Original: To further investigate our observations on sparsity, we follow Yu et al.
Revised: To further investigate our observations on sparsity, we follow Yu et al.

Original: and compute the average coding length per sample on the transfer datasets (see Fig.
Revised: and compute the average coding length per sample on the transfer datasets (see Fig.

Original: 6 (left)).
Revised: 6 (left)).

Original: We see that projectors largely increase the “information content” of representations.
Revised: We see that projectors largely increase the “information content” of representations.

Original: This was also verified by analysing singular values per dimension for models with and without projectors.
Revised: This was also verified by analysing singular values per dimension for models with and without projectors.

Original: We observed that the feature variance is more uniformly distributed over dimensions when a projector is used.
Revised: We observed that the feature variance is more uniformly distributed over dimensions when a projector is used.

Original: (see Appendix C.6).
Revised: (see Appendix C.6).

Original: These observations might explain why projectors reduce overfitting to IN1K concepts.
Revised: These observations might explain why projectors reduce overfitting to IN1K concepts.

Original: Gradient similarity.
Revised: Gradient similarity.

Original: To understand why using multi-crop increases performance for the same batch size, we examine  class weightsgradients ∇ W L CE for two models that have the same effective batch size, with and without multi-crop.
Revised: To understand why using multi-crop increases performance for the same batch size, we examine the gradients of class weights ∇ W L CE for two models that have the same effective batch size, with and without multi-crop.

Original: We see that the averageabsolute cosine similarity between  gradients of individual class weights ∇ w c  L CE increases substantially with multi-crop (see Fig.
Revised: At each iteration of training, we compute the average cosine similarity between individual gradients of every pair of class weights ∇ W c i L CE and ∇ W c j L CE for any c i ̸

Original:  6 (middle)).
Revised: = c j . As we see from Fig. 6 (middle), cosine similarity increases substantially with multi-crop. In other words, on average, classifier gradients (and therefore the class weights themselves) are more entangled.

Original: This suggests that, for a given batch size, local crops bring more diversity which lead to more challenging samples, and  in turn result in an encoder that provides more discriminative features.
Revised: We attribute this to the fact that some of the local crops (e.g., the ones that mostly cover background and hence are not really discriminative for the class at hand) are harder to classify. This leads to a harder task and gradients of higher variance (shown also in Fig. 16 in Appendix).

Original: Change in class weights and prototypes.
Revised: Change in class weights and prototypes.

Original: To understand the differences between the training objectives in Eq.
Revised: To understand the differences between the training objectives in Eq.

Original: (1) and Eq.
Revised: (1) and Eq.

Original: (2), we measure how much class weights W and prototypes U change during the training phase.
Revised: (2), we measure how much class weights W and prototypes U change during the training phase.

Original: In Fig.
Revised: In Fig.

Original: 6 (right), we plot the average change over all classes by computing the Frobenius norm between before and after each iteration,
Revised: 6 (right), we plot the average change over all classes by computing the Frobenius norm between before and after each iteration,

Original: i.e., ∆ W = ∥
Revised: i.e., ∆ W = ∥

Original: ¯ W t − ¯ W t − 1 ∥ 2 and ∆ U =
Revised: ¯ W t − ¯ W t − 1 ∥ 2 and ∆ U =

Original: ∥
Revised: ∥

Original: ¯ U t − ¯ U t − 1 ∥ 2 , where t is the training iteration, and ¯ W and ¯ U are the class weight w c and prototype u c ℓ 2 -normalized per class and concatenated, respectively.
Revised: ¯ U t − ¯ U t − 1 ∥ 2 , where t is the training iteration, and ¯ W and ¯ U are the class weight w c and prototype u c ℓ 2 -normalized per class and concatenated, respectively.

Original: Interestingly, we observe that prototypes U change orders of magnitude more than class weights W throughout training .
Revised: Interestingly, we observe that prototypes U change orders of magnitude more than class weights W throughout training .

Original: We believe this is because we compute class prototypes using only the small subset of images from our memory bank.
Revised: We believe this is because we compute class prototypes using only the small subset of images from our memory bank.

Original: The average number of samples per class on IN1K is 1281, whereas, on average we have only 8 per class in the memory bank.
Revised: The average number of samples per class on IN1K is 1281, whereas, on average we have only 8 per class in the memory bank.

Original: We argue that prevents OCM from overfitting, leadingto higher IN1K performance, as we show next.
Revised: We argue that prevents OCM from overfitting, leading

Original: 4. P USHING THE ENVELOPE OF TRAINING - VERSUS - TRANSFER PERFORMANCE
Revised: 4. P USHING THE ENVELOPE OF TRAINING - VERSUS - TRANSFER PERFORMANCE

Original: In this section we report and analyse results from more than 100 different models trained on IN1K, all different instantiations of the proposed training setup.
Revised: In this section we report and analyse results from more than 100 different models trained on IN1K, all different instantiations of the proposed training setup.

Original: We varied hyper-parameters such as the number of hidden layers in the expandable projector head or the training objective.
Revised: We varied hyper-parameters such as the number of hidden layers in the expandable projector head or the training objective.

Original: The most important results are depicted in Fig.
Revised: The most important results are depicted in Fig.

Original: 7, while an extended version of this figure is presented in Fig.
Revised: 7, while an extended version of this figure is presented in Fig.

Original: 10 in Appendix.
Revised: 10 in Appendix.

Original: Previous state of the art.
Revised: Previous state of the art.

Original: RSB-A1 (Wightman et al., 2021) is a highly optimized supervised
Revised: RSB-A1 (Wightman et al., 2021) is a highly optimized supervised

Original: ResNet50 model with top performance on IN1K. The self-supervised DINO (Caron et al., 2021) model has shown top transfer learning performance, while also performing well on IN1K. The semi-supervised PAWS (Assran et al., 2021) model matches DINO in transfer performance, with improved IN1K accuracy.
Revised: ResNet50 model with top performance on IN1K. The self-supervised DINO (Caron et al., 2021) model has shown top transfer learning performance, while also performing well on IN1K. The semi-supervised PAWS (Assran et al., 2021) model matches DINO in transfer performance, with improved IN1K accuracy.

Original: To our knowledge, RSB-A1 and DINO/PAWS are the current state-of-the-art ResNet50 models for IN1K classification and transfer learning respectively.
Revised: To our knowledge, RSB-A1 and DINO/PAWS are the current state-of-the-art ResNet50 models for IN1K classification and transfer learning respectively.

Original: We also compare to three recent supervised models:
Revised: We also compare to three recent supervised models:

Original: SupCon (Khosla et al., 2020), LOOK (Feng et al., 2022) and SL-MLP (Wang et al., 2022a).
Revised: SupCon (Khosla et al., 2020), LOOK (Feng et al., 2022) and SL-MLP (Wang et al., 2022a).

Original: For all models except LOOK and SL-MLP, we evaluate the models provided by the authors.
Revised: For all models except LOOK and SL-MLP, we evaluate the models provided by the authors.

Original: Due to the absence of official code we reproduced LOOK and
Revised: Due to the absence of official code we reproduced LOOK and

Original: SL-MLP ourselves, enhancing them with multi-crop.
Revised: SL-MLP ourselves, enhancing them with multi-crop.

Original: Our reproductions achieve higher performance than the one reported in the original papers.
Revised: Our reproductions achieve higher performance than the one reported in the original papers.

Original: In both cases, we use a projector with 1 hidden layer.
Revised: In both cases, we use a projector with 1 hidden layer.

Original: Notations.
Revised: Notations.

Original: Models trained with the basic version of the proposed training setup, i.e., using multi-crop, a projector with L hidden layers and a cosine softmax cross-entropy loss are reported as t-ReX L .
Revised: Models trained with the basic version of the proposed training setup, i.e., using multi-crop, a projector with L hidden layers and a cosine softmax cross-entropy loss are reported as t-ReX L .

Original: For models using the OCM training objective we append -OCM .
Revised: For models using the OCM training objective we append -OCM .

Original: Models on the “envelope” (i.e., the convex hull) of Fig.
Revised: Models on the “envelope” (i.e., the convex hull) of Fig.

Original: 7 are highlighted with a star (exact configurations in Tabs.
Revised: 7 are highlighted with a star (exact configurations in Tabs.

Original: 3 and 7 in Appendix).
Revised: 3 and 7 in Appendix).

Original: Main results.
Revised: Main results.

Original: Our main observations from results presented in Fig.
Revised: Our main observations from results presented in Fig.

Original: 7 can be summarized as follows.
Revised: 7 can be summarized as follows.

Original: • Pushing the envelope.
Revised: • Pushing the envelope.

Original: Many variants from our supervised training setup “push” the envelope beyond the previous state of the art, across both axes.
Revised: Many variants from our supervised training setup “push” the envelope beyond the previous state of the art, across both axes.

Original: Several of these models improve over the state of the art on one or the other axis, but no single model outperforms all the others on both dimensions.
Revised: Several of these models improve over the state of the art on one or the other axis, but no single model outperforms all the others on both dimensions.

Original: As the number of hidden layers of the projector increases, models gradually move from the lower right to the upper left corner of the plane.
Revised: As the number of hidden layers of the projector increases, models gradually move from the lower right to the upper left corner of the plane.

Original: This shows again that increasing the projector complexity improves transfer performance at the cost of IN1K (training task) performance.
Revised: This shows again that increasing the projector complexity improves transfer performance at the cost of IN1K (training task) performance.

Original: • No reason for no supervision.
Revised: • No reason for no supervision.

Original: A large number of supervised variants outperform the DINO method with respect to transfer learning, while also being significantly better on IN1K. We therefore show that training with label supervision does not necessarily require to sacrifice transfer learning performance and one should use label information if available.
Revised: A large number of supervised variants outperform the DINO method with respect to transfer learning, while also being significantly better on IN1K. We therefore show that training with label supervision does not necessarily require to sacrifice transfer learning performance and one should use label information if available.

Original: • State-of-the-art IN1K performance with three simple modifications.
Revised: • State-of-the-art IN1K performance with three simple modifications.

Original: A number of t-ReX 1 models outperform the highly optimized RSB-A1 on IN1K, essentially by using only three components over the “vanilla” supervised learning process that is considered standard practice: a) cosine softmax with temperature, b) multi-crop data augmentation, and c) an expendable projector.
Revised: A number of t-ReX 1 models outperform the highly optimized RSB-A1 on IN1K, essentially by using only three components over the “vanilla” supervised learning process that is considered standard practice: a) cosine softmax with temperature, b) multi-crop data augmentation, and c) an expendable projector.

Original: • Training with class prototypes brings further gain.
Revised: • Training with class prototypes brings further gain.

Original: Given the same projector configuration, training models with the OCM objective (Eq.
Revised: Given the same projector configuration, training models with the OCM objective (Eq.

Original: (2)) has a small advantage over training with cosine softmax (Eq.
Revised: (2)) has a small advantage over training with cosine softmax (Eq.

Original: We see that 4 of the 6 points on the convex hull in Fig.
Revised: We see that 4 of the 6 points on the convex hull in Fig.

Original: 7 are t-ReX-OCM models.
Revised: 7 are t-ReX-OCM models.

Original: This suggests that using class prototypes is a viable alternative to learning class weights end-to-end.
Revised: This suggests that using class prototypes is a viable alternative to learning class weights end-to-end.

Original: Introducing t-ReX and t-ReX*.
Revised: Introducing t-ReX and t-ReX*.

Original: We single out the two instantiations that respectively excel on the transfer learning and IN1K axes, i.e., t-ReX 3 -OCM and t-ReX 1 -OCM .
Revised: We single out the two instantiations that respectively excel on the transfer learning and IN1K axes, i.e., t-ReX 3 -OCM and t-ReX 1 -OCM .

Original: We rename them t-ReX and t-ReX* , respectively.
Revised: We rename them t-ReX and t-ReX* , respectively.

Original: We envision these two t ransferable Re sNet50 models and their corresponding training setups to serve as strong supervised baselines for future research on transfer learning and IN1K. All hyper-parameters for these two models can be found in Tab. 3 in
Revised: We envision these two t ransferable Re sNet50 models and their corresponding training setups to serve as strong supervised baselines for future research on transfer learning and IN1K. All hyper-parameters for these two models can be found in Tab. 3 in

Original: Appendix.
Revised: Appendix.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a supervised training setup that leverages components from self-supervised learning, and improves generalization without conceding on the performance of the original task, i.e., IN1K classification.
Revised: We present a supervised training setup that leverages components from self-supervised learning, and improves generalization without conceding on the performance of the original task, i.e., IN1K classification.

Original: We also show that substituting class weights with prototypes used an online class mean classifier over a small memory bank boosts performance even further.
Revised: We also show that substituting class weights with prototypes used an online class mean classifier over a small memory bank boosts performance even further.

Original: We extensively analyse the design choices and parameters of those models, and show that many variants push the envelope on the IN1K-transfer performance plane.
Revised: We extensively analyse the design choices and parameters of those models, and show that many variants push the envelope on the IN1K-transfer performance plane.

Original: This validates our intuition that image-level supervision, if available, can be beneficial to both IN1K classification and transfer tasks.
Revised: This validates our intuition that image-level supervision, if available, can be beneficial to both IN1K classification and transfer tasks.

Original: A PPENDIX C ONTENTSA Further details
Revised: A PPENDIX C ONTENTS

Original:  on the improved supervised training setup
Revised: A Further details on the improved supervised training setup

Original: A.1 Data augmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Data augmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Online Class Means (OCM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2 Online Class Means (OCM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.3 Variant: Random orthogonal classifiers (t-ReX L - orth ) . . . . . . . . . . . . . . . .
Revised: A.3 Variant: Random orthogonal classifiers (t-ReX L - orth ) . . . . . . . . . . . . . . . .

Original: A.4 Variant: Online Component Analysis (t-ReX-OCA) . . . . . . . . . . . . . . . . .
Revised: A.4 Variant: Online Component Analysis (t-ReX-OCA) . . . . . . . . . . . . . . . . .

Original: B Further details on the evaluation process
Revised: B Further details on the evaluation process

Original: B.1 Hyper-parameters for IN1K training . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.1 Hyper-parameters for IN1K training . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2 Evaluation datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.2 Evaluation datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.3 Evaluation metrics: the average log odds transferability score . . . . . . . . . . . .
Revised: B.3 Evaluation metrics: the average log odds transferability score . . . . . . . . . . . .

Original: B.4 Evaluation protocols . .
Revised: B.4 Evaluation protocols . .

Original: B B
Revised: B B

Original: C Extended results and evaluations
Revised: C Extended results and evaluations

Original: C.1 Results per dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Results per dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.2 Extended version of Fig.
Revised: C.2 Extended version of Fig.

Revised: [Table]

Original: D Limitations
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: D Limitations

Original: A F URTHER DETAILS ON THE IMPROVED SUPERVISED TRAINING SETUP A. D ATA AUGMENTATION
Revised: A F URTHER DETAILS ON THE IMPROVED SUPERVISED TRAINING SETUP A. D ATA AUGMENTATION

Original: Our training setup for improving the generalization performance of supervised models (described in Sec.
Revised: Our training setup for improving the generalization performance of supervised models (described in Sec.

Original: 3 of the main paper) includes the multi-crop augmentation initially proposed in SwAV (Caron et al., 2020).
Revised: 3 of the main paper) includes the multi-crop augmentation initially proposed in SwAV (Caron et al., 2020).

Original: In our experiments, we use the multi-crop implementation from DINO (Caron et al., 2021), which consists of three augmentation branches (two for global crops and one for local crops).
Revised: In our experiments, we use the multi-crop implementation from DINO (Caron et al., 2021), which consists of three augmentation branches (two for global crops and one for local crops).

Original: The pipeline of this multi-crop augmentation is illustrated in Fig.
Revised: The pipeline of this multi-crop augmentation is illustrated in Fig.

Original: 8a.
Revised: 8a.

Original: We also illustrate the pipelines of the “vanilla” PyTorch and SimSiam augmentations that are used in some of our ablation studies in Fig.
Revised: We also illustrate the pipelines of the “vanilla” PyTorch and SimSiam augmentations that are used in some of our ablation studies in Fig.

Original: 8b and Fig.
Revised: 8b and Fig.

Original: 8c respectively.
Revised: 8c respectively.

Original: For our experiments with two global crops, i.e., when M g =in Fig. 11, we use all three branches.
Revised: For our experiments with two global crops, i.e., when M g =in Fig. 11, we use all three branches.

Original: In all the other experiments, which have only one global crop
Revised: In all the other experiments, which have only one global crop

Original: M g = 1 , we use two of the branches: the second global crop and the one for local crops.
Revised: M g = 1 , we use two of the branches: the second global crop and the one for local crops.

Original: Tab.summarizes the parameters of the augmentation operations used in our experiments.
Revised: Tab.summarizes the parameters of the augmentation operations used in our experiments.

Original: RandomResizedCropsize=(224x224)scale=(0.4, 1.00)
Revised: RandomResizedCropsize=(224x224)scale=(0.4, 1.00)

Original: RandomHorizontalFlip
Revised: RandomHorizontalFlip

Original: RandomColorJitter
Revised: RandomColorJitter

Original: RandomGrayscale
Revised: RandomGrayscale

Original: GaussianBlur
Revised: GaussianBlur

Original: Normalize with mean and std
Revised: Normalize with mean and std

Original: RandomResizedCropsize=(224x224)scale=(0.4, 1.00)
Revised: RandomResizedCropsize=(224x224)scale=(0.4, 1.00)

Original: RandomHorizontalFlip
Revised: RandomHorizontalFlip

Original: RandomColorJitter
Revised: RandomColorJitter

Original: RandomGrayscale
Revised: RandomGrayscale

Original: RandomGaussianBlur
Revised: RandomGaussianBlur

Original: RandomSolarization
Revised: RandomSolarization

Original: Normalize with mean and std
Revised: Normalize with mean and std

Original: RandomResizedCropsize=(96x96)scale=(0.05, 0.4)
Revised: RandomResizedCropsize=(96x96)scale=(0.05, 0.4)

Original: RandomHorizontalFlip
Revised: RandomHorizontalFlip

Original: RandomColorJitter
Revised: RandomColorJitter

Original: RandomGrayscale
Revised: RandomGrayscale

Original: RandomGaussianBlur
Revised: RandomGaussianBlur

Original: Normalize with mean and std 1st
Revised: Normalize with mean and std 1st

Original: Global 2nd
Revised: Global 2nd

Original: Global
Revised: Global

Original: Locals (
Revised: Locals (

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: RandomResizedCropsize=(
Revised: RandomResizedCropsize=(

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: RandomHorizontalFlip
Revised: RandomHorizontalFlip

Original: Normalize
Revised: Normalize

Original: [Figure]
Revised: [Figure]

Original: (PyTorch)
Revised: (PyTorch)

Original: [Figure]
Revised: [Figure]

Original: PyTorch
Revised: PyTorch

Original: RandomResizedCropsize=(224x224)
Revised: RandomResizedCropsize=(224x224)

Original: [Figure]
Revised: [Figure]

Original: =(
Revised: =(

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: RandomHorizontalFlip
Revised: RandomHorizontalFlip

Original: ColorJitter
Revised: ColorJitter

Original: RandomGrayscale
Revised: RandomGrayscale

Original: RandomGaussianBlur
Revised: RandomGaussianBlur

Original: Normalize with mean and std (SimSiam)
Revised: Normalize with mean and std (SimSiam)

Original: [Figure]
Revised: [Figure]

Original: Library.
Revised: Library.

Original: Accordingly, names these
Revised: Accordingly, names these

Original: [Table]
Revised: [Table]

Original: RandomSolarization
Revised: RandomSolarization

Original: [Table]
Revised: [Table]

Original: A. O NLINE C LASS M EANS (OCM)
Revised: A. O NLINE C LASS M EANS (OCM)

Original: In Sec.
Revised: In Sec.

Original: 3 of the main paper, we introduce the “online” version of the Nearest Class Mean classifier (Mensink et al., 2012), referred to as Online Class Mean (OCM).
Revised: 3 of the main paper, we introduce the “online” version of the Nearest Class Mean classifier (Mensink et al., 2012), referred to as Online Class Mean (OCM).

Original: In this section, we give a schematic illustration of this model and describe its implementation details.
Revised: In this section, we give a schematic illustration of this model and describe its implementation details.

Original: Illustrations of the different loss functions.
Revised: Illustrations of the different loss functions.

Original: In Fig.
Revised: In Fig.

Original: 9 we visualize the supervised models.
Revised: 9 we visualize the supervised models.

Original: We train it using the two loss functions defined in Sec.
Revised: We train it using the two loss functions defined in Sec.

Original: 3 of the main paper.
Revised: 3 of the main paper.

Original: Fig.
Revised: Fig.

Original: 9a and Fig.
Revised: 9a and Fig.

Original: 9b correspond to the models for Eq.
Revised: 9b correspond to the models for Eq.

Original: (1) and Eq.
Revised: (1) and Eq.

Original: (2) of the main paper, respectively.
Revised: (2) of the main paper, respectively.

Original: As seen from Fig.
Revised: As seen from Fig.

Original: 9b, and as explained in Sec.
Revised: 9b, and as explained in Sec.

Original: 3 of the main paper, OCM follows SupCon (Khosla et al., 2020) and LOOK (Feng et al., 2022) and uses the momentum network and memory queue proposed in He et al.
Revised: 3 of the main paper, OCM follows SupCon (Khosla et al., 2020) and LOOK (Feng et al., 2022) and uses the momentum network and memory queue proposed in He et al.

Original: We explain them in detail below.
Revised: We explain them in detail below.

Original: Momentum encoder f ξ and projector g ζ .
Revised: Momentum encoder f ξ and projector g ζ .

Original: To keep a memory bank of slowly evolving features, we keep an exponential moving average (EMA) of the encoder f θ and projector g ϕ parameters.
Revised: To keep a memory bank of slowly evolving features, we keep an exponential moving average (EMA) of the encoder f θ and projector g ϕ parameters.

Original: [Figure]
Revised: [Figure]

Original: Concretely, momentum encoder f ξ and momentum projector g ζ , whose parameters are respectively
Revised: Concretely, momentum encoder f ξ and momentum projector g ζ , whose parameters are respectively

Original: EMA of θ and ϕ , are defined as: ξ ← m × ξ + (1 − m ) × θ and ζ ← m × ζ + (1 − m ) × ϕ , where m = 0 .
Revised: EMA of θ and ϕ , are defined as: ξ ← m × ξ + (1 − m ) × θ and ζ ← m × ζ + (1 − m ) × ϕ , where m = 0 .

Original: 999 is the momentum parameter.
Revised: 999 is the momentum parameter.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 9b, we only feed global crops I g through these momentum networks f ξ and g ζ during training.
Revised: 9b, we only feed global crops I g through these momentum networks f ξ and g ζ during training.

Original: Both global and local crops I g,l are passed through the encoder f θ and projector g ϕ .
Revised: Both global and local crops I g,l are passed through the encoder f θ and projector g ϕ .

Original: Expendable predictor h ψ after projector g ϕ .
Revised: Expendable predictor h ψ after projector g ϕ .

Original: In several SSL methods with dual-network architectures, e.g., BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), breaking the architectural symmetry, by adding a multi-layer perceptron to one of the branches, was shown to improve the generalization of representations.
Revised: In several SSL methods with dual-network architectures, e.g., BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), breaking the architectural symmetry, by adding a multi-layer perceptron to one of the branches, was shown to improve the generalization of representations.

Original: Following this practice, we also experimented with training OCM models optionally using an expendable predictor head h ψ : R d b → R d b with parameters ψ , added after the projector head g ϕ (as shown in Fig. 9b).
Revised: Following this practice, we also experimented with training OCM models optionally using an expendable predictor head h ψ : R d b → R d b with parameters ψ , added after the projector head g ϕ (as shown in Fig. 9b).

Original: These predictor heads contain fully-connected, batch-normalization (Ioffe & Szegedy, 2015) and GeLU (Hendrycks & Gimpel, 2016) layers, followed by another fully-connected layer and ℓ 2 -normalization.
Revised: These predictor heads contain fully-connected, batch-normalization (Ioffe & Szegedy, 2015) and GeLU (Hendrycks & Gimpel, 2016) layers, followed by another fully-connected layer and ℓ 2 -normalization.

Original: The first (resp.
Revised: The first (resp.

Original: second) fully-connected layer maps from R d b to R d p (resp.
Revised: second) fully-connected layer maps from R d b to R d p (resp.

Original: R d p to R d b ).
Revised: R d p to R d b ).

Original: In our experiments d p is generally 2048 dimensions.
Revised: In our experiments d p is generally 2048 dimensions.

Original: See Appendix C.5 for a discussion on the impact of predictors on performance; Note that neither t-ReX nor t-ReX* is trained with an expendable predictor head h ψ .
Revised: See Appendix C.5 for a discussion on the impact of predictors on performance; Note that neither t-ReX nor t-ReX* is trained with an expendable predictor head h ψ .

Original: Memory bank.
Revised: Memory bank.

Original: The original NCM (Mensink et al., 2012) formulation require access to the entire dataset at each SGD training iteration, which is not possible in our case.
Revised: The original NCM (Mensink et al., 2012) formulation require access to the entire dataset at each SGD training iteration, which is not possible in our case.

Original: To circumvent this, we use a memory queue Q which stores ℓ 2 -normalized momentum projector outputs Q = { g ζ ( f ξ ( I g )) } for global crops.
Revised: To circumvent this, we use a memory queue Q which stores ℓ 2 -normalized momentum projector outputs Q = { g ζ ( f ξ ( I g )) } for global crops.

Original: In OCM, we compute a “prototype” for each class c , as the mean over all memory points from that class u c = 1 / N c (cid:80)
Revised: In OCM, we compute a “prototype” for each class c , as the mean over all memory points from that class u c = 1 / N c (cid:80)

Original: z
Revised: z

Original: ∈Q c z , where Q c denotes samples in the queue that belong to class c and N c
Revised: ∈Q c z , where Q c denotes samples in the queue that belong to class c and N c

Original: = |Q c | .
Revised: = |Q c | .

Original: Then, for a given training crop I j (can be either a global or local crop), we compute its predictor outputs h ψ ( g ϕ ( f θ ( I j ))) and obtain class prediction scores by taking the inner product between this predictor output and the set of all class prototypes U
Revised: Then, for a given training crop I j (can be either a global or local crop), we compute its predictor outputs h ψ ( g ϕ ( f θ ( I j ))) and obtain class prediction scores by taking the inner product between this predictor output and the set of all class prototypes U

Original: = { u c } 1000 c =1 as defined in Eq.
Revised: = { u c } 1000 c =1 as defined in Eq.

Original: (2) of the main paper.
Revised: (2) of the main paper.

Original: In Appendix C.5 we study the impact that the size of the memory bank has on IN1K and transfer performance.
Revised: In Appendix C.5 we study the impact that the size of the memory bank has on IN1K and transfer performance.

Original: Class prototype neighbors.
Revised: Class prototype neighbors.

Original: In Tab.
Revised: In Tab.

Original: 11 we provide a visualization of the top-5 nearest prototypes for a random set of classes for t-ReX* .
Revised: 12 we provide a visualization of the top-5 nearest prototypes for a random set of classes for t-ReX* .

Original: A. V ARIANT : R ANDOM ORTHOGONAL CLASSIFIERS ( T -R E X L - orth )
Revised: A. V ARIANT : R ANDOM ORTHOGONAL CLASSIFIERS ( T -R E X L - orth )

Original: In Sec.
Revised: In Sec.

Original: 3 of the main paper, we propose the OCM variant where class weights W are replaced by class means U
Revised: 3 of the main paper, we propose the OCM variant where class weights W are replaced by class means U

Original: = { u c } C c =1 which are obtained in an online manner using a memory bank
Revised: = { u c } C c =1 which are obtained in an online manner using a memory bank

Original: [Figure]
Revised: [Figure]

Original: t-
Revised: t-

Original: [Figure]
Revised: [Figure]

Original: M ea n T r a n s f e r
Revised: M ea n T r a n s f e r

Original: A cc .
Revised: A cc .

Original: ( L og odd
Revised: ( L og odd

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: we explore another variant of our setup where class weights W = { w c } 1000 c
Revised: we explore another variant of our setup where class weights W = { w c } 1000 c

Original: =1 are initialized with random orthogonal vectors and kept frozen while the encoder f θ and projector g ϕ networks are trained.
Revised: =1 are initialized with random orthogonal vectors and kept frozen while the encoder f θ and projector g ϕ networks are trained.

Original: Our motivation is that such vectors may lead to higher class separation obtaining higher accuracy on the training task, as also noted by Kornblith et al.
Revised: Our motivation is that such vectors may lead to higher class separation obtaining higher accuracy on the training task, as also noted by Kornblith et al.

Original: To this end, without using momentum networks, a predictor head or a memory bank, we simply optimize Eq.
Revised: To this end, without using momentum networks, a predictor head or a memory bank, we simply optimize Eq.

Original: (1) in the main paper with “fixed” class weights, and denote these models with the t-ReX L - orth notation.
Revised: (1) in the main paper with “fixed” class weights, and denote these models with the t-ReX L - orth notation.

Original: Due to our limited computational budget, we train 6 t-ReX L - orth variants with and without projector input ℓ 2 -norm, L = [1 , 2 , 3] , d h = 2048 and d b = 256 , and plot their IN1K and average transfer accuracies along with the others from the main paper in Fig.
Revised: Due to our limited computational budget, we train 6 t-ReX L - orth variants with and without projector input ℓ 2 -norm, L = [1 , 2 , 3] , d h = 2048 and d b = 256 , and plot their IN1K and average transfer accuracies along with the others from the main paper in Fig.

Original: We observe that none of those six models were able to expand the envelope of training-versus-transfer performance.
Revised: We observe that none of those six models were able to expand the envelope of training-versus-transfer performance.

Original: A. V ARIANT : O NLINE C OMPONENT A NALYSIS ( T -R E X-OCA)
Revised: A. V ARIANT : O NLINE C OMPONENT A NALYSIS ( T -R E X-OCA)

Original: Recent supervised learning methods like SupCon (Khosla et al., 2020) or LOOK (Feng et al., 2022) are variants of the soft k -NN loss introduced in Neighborhood Component Analysis (NCA)
Revised: Recent supervised learning methods like SupCon (Khosla et al., 2020) or LOOK (Feng et al., 2022) are variants of the soft k -NN loss introduced in Neighborhood Component Analysis (NCA)

Original: (Goldberger et al., 2004).
Revised: (Goldberger et al., 2004).

Original: We therefore explore a variant of our training setup where the loss directly minimizes the log NCA probabilities.
Revised: We therefore explore a variant of our training setup where the loss directly minimizes the log NCA probabilities.

Original: Specifically, as in the OCM variant, we use a memory bank Q which stores ℓ 2 -normalized embeddings z output by the projector instead of using the full dataset to compute the soft k -NN, as the latter would be intractable.
Revised: Specifically, as in the OCM variant, we use a memory bank Q which stores ℓ 2 -normalized embeddings z output by the projector instead of using the full dataset to compute the soft k -NN, as the latter would be intractable.

Original: This variant then optimizes the following loss:
Revised: This variant then optimizes the following loss:

Original: [Equation]
Revised: [Equation]

Original: (3) where M is the number of crops, C is the number of classes, y ∈ { 0 , 1 } C is the C -dim one-hot label vector, τ is temperature for NCA probabilities and Q c denotes samples in the memory bank that belong to class c .
Revised: (3) where M is the number of crops, C is the number of classes, y ∈ { 0 , 1 } C is the C -dim one-hot label vector, τ is temperature for NCA probabilities and Q c denotes samples in the memory bank that belong to class c .

Original: Since this is an “online” variant of the NCA objective, we refer to this training objective as Online Component Analysis or OCA.
Revised: Since this is an “online” variant of the NCA objective, we refer to this training objective as Online Component Analysis or OCA.

Original: Similar to OCM, we use momentum networks in
Revised: Similar to OCM, we use momentum networks in

Original: OCA, i.e., momentum encoder f ξ and projector g ζ .
Revised: OCA, i.e., momentum encoder f ξ and projector g ζ .

Original: For more details, see Appendix A.2.
Revised: For more details, see Appendix A.2.

Original: We experimentally show in Fig.
Revised: We experimentally show in Fig.

Original: 10 that our t-ReX-OCA models achieve a balance between IN1K and transfer performances, i.e., they are towards the middle of the envelope in Fig.
Revised: 10 that our t-ReX-OCA models achieve a balance between IN1K and transfer performances, i.e., they are towards the middle of the envelope in Fig.

Original: Moreover, in our evaluations we see that this variant is better overall than LOOK+ multi-crop .
Revised: Moreover, in our evaluations we see that this variant is better overall than LOOK+ multi-crop .

Original: Note that the main difference between LOOK+ multi-crop and the t-ReX-OCA variants is that the former restricts the soft k-NN loss (Goldberger et al., 2004) to the close neighborhood of each training sample in the memory bank.
Revised: Note that the main difference between LOOK+ multi-crop and the t-ReX-OCA variants is that the former restricts the soft k-NN loss (Goldberger et al., 2004) to the close neighborhood of each training sample in the memory bank.

Original: Our observation suggests that relying on more points in the memory bank is beneficial for learning better representations.
Revised: Our observation suggests that relying on more points in the memory bank is beneficial for learning better representations.

Original: We further observed that for this variant the size of the memory bank |Q| does not severely affect the performance of learned representations, but the smoothness of NCA probabilities computed in Eq.
Revised: We further observed that for this variant the size of the memory bank |Q| does not severely affect the performance of learned representations, but the smoothness of NCA probabilities computed in Eq.

Original: (3) does.
Revised: (3) does.

Original: Therefore, it is important to tune the temperature parameter τ for a given memory bank size to carefully control this smoothness.
Revised: Therefore, it is important to tune the temperature parameter τ for a given memory bank size to carefully control this smoothness.

Original: B F URTHER DETAILS ON THE EVALUATION PROCESS B. H YPER - PARAMETERS FOR IN1K TRAINING
Revised: B F URTHER DETAILS ON THE EVALUATION PROCESS B. H YPER - PARAMETERS FOR IN1K TRAINING

Original: As we discuss in the previous section, we train supervised models on IN1K with different objectives and architecture configurations.
Revised: As we discuss in the previous section, we train supervised models on IN1K with different objectives and architecture configurations.

Original: We observe that the hyper-parameters for IN1K training that we share in Tab.
Revised: We observe that the hyper-parameters for IN1K training that we share in Tab.

Original: 3 work well for the most effective models we studied.
Revised: 3 work well for the most effective models we studied.

Original: B. E VALUATION DATASETS
Revised: B. E VALUATION DATASETS

Original: Once we train our models on IN1K, we evaluate their encoder representations f θ ( I ) by training linear logistic regression (Log.
Revised: Once we train our models on IN1K, we evaluate their encoder representations f θ ( I ) by training linear logistic regression (Log.

Original: Reg.)
Revised: Reg.)

Original: classifiers on 13 transfer datasets which include 5 ImageNetCoG levels (Sariyildiz et al., 2021) and 8 commonly used small-scale datasets: Aircraft (Maji et al., 2013), Cars196 (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), Flowers (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012), Food101 (Bossard et al., 2014) and SUN397 (Xiao et al., 2010).
Revised: classifiers on 13 transfer datasets which include 5 ImageNet-CoG levels (Sariyildiz et al., 2021) and 8 commonly used small-scale datasets: Aircraft (Maji et al., 2013), Cars196 (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), Flowers (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012), Food101 (Bossard et al., 2014) and SUN397 (Xiao et al., 2010).

Original: Additionally, we test the generalization of models to IN1K concepts using the three test sets of IN1K-v2 (Recht et al., 2019) and out of domain images of IN1K-Sketch (Wang et al., 2019). 
Revised: To test the generalization of models to IN1K concepts, we use the three test sets of IN1K-v2 (Recht et al., 2019) and out of domain images of IN1K-Sketch (Wang et al., 2019). Finally, to test performance on long-tail classification tasks, we use iNaturalist 2018 and 2019 (Van Horn et al., 2018).

Original: Statistics of all these datasets are provided in Tab. 4.
Revised: Statistics of all these datasets are provided in Tab. 4.

Original: When a val split is not provided for a dataset, we randomly split its train set into two, following the size of train and val splits from either Feng et al.
Revised: When a val split is not provided for a dataset, we randomly split its train set into two, following the size of train and val splits from either Feng et al.

Original: (2022) or Grill et al.
Revised: (2022) or Grill et al.

Original: We also created different train / val splits when tuning hyper-parameters with different seeds, thus further increasing the robustness of our scores.
Revised: We also created different train / val splits when tuning hyper-parameters with different seeds, thus further increasing the robustness of our scores.

Original: Other notes on the datasets are as follows:
Revised: Other notes on the datasets are as follows:

Original: (i) For DTD (Cimpoi et al., 2014)
Revised: (i) For DTD (Cimpoi et al., 2014)

Original: (resp.
Revised: (resp.

Original: EuroSAT (Helber et al., 2019)), there are 10 official train / val / test (resp.
Revised: EuroSAT (Helber et al., 2019)), there are 10 official train / val / test (resp.

Original: train / test ) splits.
Revised: train / test ) splits.

Original: Following Feng et al.
Revised: Following Feng et al.

Original: ; Grill et al.
Revised: ; Grill et al.

Original: (2020), we use the first split.
Revised: (2020), we use the first split.

Original: (ii)
Revised: (ii)

Original: For EuroSAT (Helber et al., 2019), we are not aware of either an official dataset split or the exact splits used in prior work, e.g., by Feng et al.
Revised: For EuroSAT (Helber et al., 2019), we are not aware of either an official dataset split or the exact splits used in prior work, e.g., by Feng et al.

Original: So, we create random train / val / test splits following the number of samples in each split from Feng et al.
Revised: So, we create random train / val / test splits following the number of samples in each split from Feng et al.

Original: (2022), ensuring that the val and test splits are balanced to contain the same number of samples for each class.
Revised: (2022), ensuring that the val and test splits are balanced to contain the same number of samples for each class.

Original: (iii)
Revised: (iii)

Revised: For iNaturalist 2018 and 2019 (Van Horn et al., 2018), we use the official val split as the test split and create a random val split each time with a different seed.

Original:  We use the Log.
Revised: (iv) We use the Log.

Original: Reg.
Revised: Reg.

Original: classifier trained on IN1K for predicting image labels in the three test sets of IN1K-v2 or the test set of IN1K-Sketch.
Revised: classifier trained on IN1K for predicting image labels in the three test sets of IN1K-v2 or the test set of IN1K-Sketch.

Original: This is because IN1K-v2 and IN1K-Sketch are only composed of test sets and no training data is provided for them.
Revised: This is because IN1K-v2 and IN1K-Sketch are only composed of test sets and no training data is provided for them.

Original: B. E VALUATION METRICS : THE AVERAGE LOG ODDS TRANSFERABILITY SCORE 
Revised: B. E VALUATION METRICS : THE AVERAGE LOG ODDS TRANSFERABILITY SCORE Following Kornblith

Original: Following Kornblith et al.
Revised: et al.

Original: (2019), we compute log odds over all transfer datasets and use their average as a transferability score .
Revised: (2019), we compute log odds over all transfer datasets and use their average as a transferability score .

Original: This is the main metric we report in the different plots of the main paper.
Revised: This is the main metric we report in the different plots of the main paper.

Original: Denoting n correct and n incorrect as the number of correct and incorrect predictions for a dataset, we compute the accuracy p and log odds score as follows:
Revised: Denoting n correct and n incorrect as the number of correct and incorrect predictions for a dataset, we compute the accuracy p and log odds score as follows:

Original: [Equation]
Revised: [Equation]

Original: Then we report log odds averaged over all transfer datasets.
Revised: Then we report log odds averaged over all transfer datasets.

Original: See Appendix C.2 for per-dataset topaccuracies and average log odd scores for the models we compare in the main paper.
Revised: See Appendix C.2 for per-dataset topaccuracies and average log odd scores for the models we compare in the main paper.

Original: [Table]
Revised: [Table]

Original: B. E VALUATION PROTOCOLS
Revised: B. E VALUATION PROTOCOLS

Original: We perform image classification on each evaluation dataset with logistic regression (Log.
Revised: We perform image classification on each evaluation dataset with logistic regression (Log.

Original: Reg.)
Revised: Reg.)

Original: classifiers following one of the two protocols proposed by Sariyildiz et al.
Revised: classifiers following one of the two protocols proposed by Sariyildiz et al.

Original: (2021) (for the 5 CoG levels) or by Kornblith et al.
Revised: (2021) (for the 5 CoG levels) or by Kornblith et al.

Original: (2019) (for the 8 small-scale datasets).
Revised: (2019) (for the 8 small-scale datasets).

Original: In all cases, we first extract and store a (single) feature vector for each image and then learn the Log.
Revised: In all cases, we first extract and store a (single) feature vector for each image and then learn the Log.

Original: Reg.
Revised: Reg.

Original: classifiers on top of those features.
Revised: classifiers on top of those features.

Original: Our classifiers are therefore trained without data augmentation , and this is why we report lower performance for the RSB model than the one presented by Wightman et al.
Revised: Our classifiers are therefore trained without data augmentation , and this is why we report lower performance for the RSB model than the one presented by Wightman et al.

Original: We extract image representations from the encoders f θ by resizing an image with bicubic interpolation such that its shortest side is 224 pixels and then taking a central crop of size 224 × 224 pixels.
Revised: We extract image representations from the encoders f θ by resizing an image with bicubic interpolation such that its shortest side is 224 pixels and then taking a central crop of size 224 × 224 pixels.

Original: The protocols for training Log.
Revised: The protocols for training Log.

Original: Reg.
Revised: Reg.

Original: classifiers on the 5 CoG levels and on the other 8 datasets are different and detailed below.
Revised: classifiers on the 5 CoG levels and on the other 8 datasets are different and detailed below.

Original: Log.
Revised: Log.

Original: Reg.
Revised: Reg.

Original: on the ImageNet-CoG levels.
Revised: on the ImageNet-CoG levels.

Original: We apply ℓ 2 -normalization to the pre-extracted features using the publicly-available source code of Sariyildiz et al.
Revised: We apply ℓ 2 -normalization to the pre-extracted features using the publicly-available source code of Sariyildiz et al.

Original: and then train Log.
Revised: and then train Log.

Original: Reg.
Revised: Reg.

Original: classifiers
Revised: classifiers

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Hyper-parameters for evaluating our t-ReX and t-ReX* models using Log.

Original: Reg. classifiers on transfer datasets found by Optuna (Akiba et al., 2019). “Learning rate” and “Weight decay” are the parameters of the SGD optimizer used for training Log. Reg.
Revised: [Table]

Original: classifiers as in the ImageNet-CoG protocol Sariyildiz et al.

Original: C is the inverse regularization coefficient used when training

Original: Log.

Original: Reg.

Original: classifiers with L-BFGS implemented in Scikit-learn (Pedregosa et al., 2011).

Original: [Table]
Revised: [Table]

Original: using SGD with momentum = 0.9 and batch size = 1024 for 100 epochs.
Revised: using SGD with momentum = 0.9 and batch size = 1024 for 100 epochs.

Original: To treat each model as fairly as possible, we set the learning rate and weight decay hyper-parameters using train / val splits ( val splits are randomly sampled using 20% of the original train splits).
Revised: To treat each model as fairly as possible, we set the learning rate and weight decay hyper-parameters using train / val splits ( val splits are randomly sampled using 20% of the original train splits).

Original: We use Optuna (Akiba et al., 2019) and sample 30 different pairs.
Revised: We use Optuna (Akiba et al., 2019) and sample 30 different pairs.

Original: We train the final classifiers with these hyper-parameters on the union of the train and val splits, and report top-1 accuracy on the test splits.
Revised: We train the final classifiers with these hyper-parameters on the union of the train and val splits, and report top-1 accuracy on the test splits.

Original: We repeat this process 5 times with different seeds and report averaged results.
Revised: We repeat this process 5 times with different seeds and report averaged results.

Original: Log.
Revised: Log.

Original: Reg.
Revised: Reg.

Original: on the smaller-scale transfer datasets.
Revised: on the smaller-scale transfer datasets.

Original: Following Kornblith et al.
Revised: Following Kornblith et al.

Original: (2019), we train
Revised: (2019), we train

Original: Log.
Revised: Log.

Original: Reg.
Revised: Reg.

Original: classifiers with pre-extracted features using L-BFGS (Liu & Nocedal, 1989).
Revised: classifiers with pre-extracted features using L-BFGS (Liu & Nocedal, 1989).

Original: To this end, we use the implementation in Scikit-learn (Pedregosa et al., 2011).
Revised: To this end, we use the implementation in Scikit-learn (Pedregosa et al., 2011).

Original: We set the inverse regularization coefficient (“C”) on each dataset using Optuna and their train / val splits over 25 trials.
Revised: We set the inverse regularization coefficient (“C”) on each dataset using Optuna and their train / val splits over 25 trials.

Original: If a dataset does not have a fixed validation set, then we repeat hyper-parameter selection 5 times with different seeds and report the average result.
Revised: If a dataset does not have a fixed validation set, then we repeat hyper-parameter selection 5 times with different seeds and report the average result.

Original: B. H YPER - PARAMETERS FOR T -R E X AND T -R E X* ON TRANSFER DATASETS
Revised: B. H YPER - PARAMETERS FOR T -R E X AND T -R E X* ON TRANSFER DATASETS

Original: In Tab. 5 we present the hyper-parameters found by Optuna (Akiba et al., 2019) for the Log.
Revised: In Tab. 5 we present the hyper-parameters found by Optuna (Akiba et al., 2019) for the Log.

Original: Reg.
Revised: Reg.

Original: classifiers trained on the transfer datasets for t-ReX and t-ReX* .[Table] [Table] [Table]
Revised: classifiers trained on the transfer datasets for t-ReX and t-ReX* .

Original: B. L IST OF PUBLICLY AVAILABLE PRETRAINED MODELS USED FOR COMPARISONS
Revised: B. L IST OF PUBLICLY AVAILABLE PRETRAINED MODELS USED FOR COMPARISONS

Original: In Sec.
Revised: In Sec.

Original: 4.3 of the main paper, we compare our models to several prior works which are state-of-the-art either for IN1K classification or for transfer learning.
Revised: 4.3 of the main paper, we compare our models to several prior works which are state-of-the-art either for IN1K classification or for transfer learning.

Original: These include self-supervised DINO (Caron et al., 2021), semi-supervised PAWS (Assran et al., 2021) and supervised SupCon (Khosla et al., 2020),
Revised: These include self-supervised DINO (Caron et al., 2021), semi-supervised PAWS (Assran et al., 2021) and supervised SupCon (Khosla et al., 2020),

Original: RSB-A1 (Wightman et al., 2021), LOOK (Feng et al., 2022) and SL-MLP (Wang et al., 2022b).
Revised: RSB-A1 (Wightman et al., 2021), LOOK (Feng et al., 2022) and SL-MLP (Wang et al., 2022b).

Original: Additionally, we evaluated another self-supervised model Barlow Twins (Zbontar et al., 2021) but found it to be inferior to DINO on both IN1K and transfer datasets.
Revised: Additionally, we evaluated another self-supervised model Barlow Twins (Zbontar et al., 2021) but found it to be inferior to DINO on both IN1K and transfer datasets.

Original: For all the models except
Revised: For all the models except

Original: LOOK and SL-MLP, we evaluate the best ResNet50 encoders trained on IN1K by their respective authors.
Revised: LOOK and SL-MLP, we evaluate the best ResNet50 encoders trained on IN1K by their respective authors.

Original: Since there was neither publicly-available model nor source code for these two models, we reproduced the methods ourselves and found that they can perform significantly better when  combined with multi-crop.
Revised: Since there was neither publicly-available model nor source code for these two models, we reproduced the methods ourselves and found that they can perform significantly better when [Table] [Table] combined with multi-crop.

Original: We compare to these enhanced version which we call LOOK+ multi-crop and SL-MLP+ multi-crop .
Revised: We compare to these enhanced version which we call LOOK+ multi-crop and SL-MLP+ multi-crop .

Original: In Tab. 6, we give a list of the compared models with public encoder weights.
Revised: In Tab. [Table] we give a list of the compared models with public encoder weights [Table]

Original: C E XTENDED RESULTS AND EVALUATIONS C. R ESULTS PER DATASET
Revised: C E XTENDED RESULTS AND EVALUATIONS C. R ESULTS PER DATASET

Original: In Tab. 7, we report top-1 accuracy on each transfer dataset and on IN1K. Results are obtained by
Revised: In Tab. [Table] we report top-1 accuracy on each transfer dataset and on IN1K. Results are obtained by

Original: Log.
Revised: Log.

Original: Reg.
Revised: Reg.

Original: classifiers, for all the models listed in Tab. 6 and for the ones that belong to the “convex hull” or envelope, denoted by stars in Fig.
Revised: classifiers [Table] for all the models listed in Tab. 6 and for the ones that belong to the “convex hull” or envelope, denoted by stars in Fig.

Original: 7 of the main paper.
Revised: 7 of the main paper.

Original: [Figure]
Revised: [Table]

Revised: I m a g e

Original: [Figure]
Revised: N [Table]

Revised: tK

Original: [Figure]
Revised: [Table]

Revised: cc

Original: [Figure]
Revised: [Figure]

Original:  M is the number of local crops
Revised: [Table] M [Table] softmax,

Original: We use 1 global crop, i.e., M g = 1 , for each experiment.
Revised: [Figure] [Table] for each experiment.

Original: PyTorch,
Revised: PyTorch,

Original: Simsiam and DINO augmentation pipelines are from Paszke et al.
Revised: Simsiam and DINO augmentation pipelines are from Paszke et al.

Original: , Chen & He (2021) and
Revised: , Chen & He (2021) and

Original: Caron et al.
Revised: Caron et al.

Original: (2021), respectively.
Revised: (2021), respectively.

Original: Note that for the experiments presented in this table we train models using vanilla softmax, see Appendix C.3 for a discussion.
Revised: Note that for the experiments presented in this table we train models using vanilla softmax, see Appendix C.3 for a discussion.

Original: [Table]
Revised: [Table]

Original: C. E XTENDED VERSION OF F IG .
Revised: C. E XTENDED VERSION OF F IG .

Original: In Fig.
Revised: In Fig.

Original: 10 we present an extended version of Fig.
Revised: 10 we present an extended version of Fig.

Original: 7 in the main paper that further includes results for the variants presented in Appendices A.3 and A.4.
Revised: 7 in the main paper that further includes results for the variants presented in Appendices A.3 and A.4.

Original: C. E XTENDED ABLATIONS ON MULTI - CROP
Revised: C. E XTENDED ABLATIONS ON MULTI - CROP

Original: In Sec.
Revised: In Sec.

Original: 4.1 of the main paper, we ablate the hyper-parameters of multi-crop using a projector head.
Revised: 4.1 of the main paper, we ablate the hyper-parameters of multi-crop using a projector head.

Original: This is because cosine softmax loss, when used without a projector head, suffers from overfitting to IN1K, and yields much worse transfer performance, as we show in Fig.
Revised: This is because cosine softmax loss, when used without a projector head, suffers from overfitting to IN1K, and yields much worse transfer performance, as we show in Fig.

Original: Note that this phenomenon has also been observed by Kornblith et al.
Revised: Note that this phenomenon has also been observed by Kornblith et al.

Original: and explained by the fact that cosine softmax increases class separability of seen concepts in the feature space which reduces transferability.
Revised: and explained by the fact that cosine softmax increases class separability of seen concepts in the feature space which reduces transferability.

Original: Therefore, we believe that using cosine softmax loss when training “multi-crop-only” models is clearly sub-optimal to set multi-crop hyper-parameters.
Revised: Therefore, we believe that using cosine softmax loss when training “multi-crop-only” models is clearly sub-optimal to set multi-crop hyper-parameters.

Original: However, we note that, as shown in Sec.
Revised: However, we note that, as shown in Sec.

Original: 4 of the main paper, this overfitting of cosine softmax is alleviated by projector heads.
Revised: 4 of the main paper, this overfitting of cosine softmax is alleviated by projector heads.

Original: In the remaining of  this section, we train “multi-crop-only” models using vanilla softmax, i.e., encoder features x and class weights { w c } 1000 c =1 are not ℓ 2 -normalized and τ = 1 . 0 in Eq.
Revised: In the remaining of [Table] this section, we train “multi-crop-only” models using vanilla softmax, i.e., encoder features x and class weights { w c } 1000 c =1 are not ℓ 2 -normalized and τ = 1 . 0 in Eq.

Original: (1) of the main paper.
Revised: (1) of the main paper.

Original: In the main paper, we use (0.05, 0.4) and (0.4, 1.0) as the scale range for local and global crops, i.e., we sample scale values from these intervals.
Revised: In the main paper, we use (0.05, 0.4) and (0.4, 1.0) as the scale range for local and global crops, i.e., we sample scale values from these intervals.

Original: Tab. 8 provides an ablation on the maximum and minimum[Table] scales for local and global crops, respectively (see rows 6-8 in Tab. 8).
Revised: Tab. 8 provides an ablation on the maximum and minimum scales for local and global crops, respectively (see rows 6-8 in Tab. 8).

Original: We evaluate 3 different values (0.15, 0.25 and 0.4) for the maximum scale of local crops, which is also set as the minimum scale for global crops.
Revised: We evaluate 3 different values (0.15, 0.25 and 0.4) for the maximum scale of local crops, which is also set as the minimum scale for global crops.

Original: Although the results are comparable, we see that 0.40 produces slightly better results.
Revised: Although the results are comparable, we see that 0.40 produces slightly better results.

Original: We further verify if the improvements from local crops are due to the fact that a) models are trained using more variants for each image, or b) the effective batch size is increased during training (as we use one batch of images for each crop).
Revised: We further verify if the improvements from local crops are due to the fact that a) models are trained using more variants for each image, or b) the effective batch size is increased during training (as we use one batch of images for each crop).

Original: To test a), we train a model for 800 epochs using single crops with scale range (0.05, 1.0), and observe that (see row 5 in Tab.
Revised: To test a), we train a model for 800 epochs using single crops with scale range (0.05, 1.0), and observe that (see row 5 in Tab.

Original: 8) it performs comparably to training models without multi-crop, i [Table] e., it barely improves over training models with PyTorch and SimSiam augmentations (rows 1 and 2) and performs similar when DINO augmentations are used without multi-crop (row 3).
Revised: 8) it performs comparably to training models without multi-crop, i.e., it barely improves over training models with PyTorch and SimSiam augmentations (rows 1 and 2) and performs similar when DINO augmentations are used without multi-crop (row 3).

Original: To test b), we train a model for 100 epochs using single crops with scale range (0.05, 1.0) but increase batch size by 9 × (see row 4).
Revised: To test b), we train a model for 100 epochs using single crops with scale range (0.05, 1.0) but increase batch size by 9 × (see row 4).

Original: Yet, we again observe no improvements over training models without local crops.
Revised: Yet, we again observe no improvements over training models without local crops.

Original: Finally, we ablate the resolution of local crops (rows 9-15 in Tab. 8).
Revised: Finally, we ablate the resolution of local crops (rows 9-15 in Tab. 8).

Original: Using two local crops
Revised: Using two local crops

Original: M l = 2 and a single global crop M g = 0 , we try several values as for the resolution of local crops [224 × 224 , 192 × 192 , 160 × 160 , 128 × 128 , 96 × 96 , 64 × 64 , 32 × 32] and find that 96 × 96 works best on both IN1K and transfer datasets.
Revised: M l = 2 and a single global crop M g = 0 , we try several values as for the resolution of local crops [224 × 224 , 192 × 192 , 160 × 160 , 128 × 128 , 96 × 96 , 64 × 64 , 32 × 32] and find that 96 × 96 works best on both IN1K and transfer datasets.

Original: C. E XTENDED ABLATIONS ON THE PROJECTOR DESIGN
Revised: C. E XTENDED ABLATIONS ON THE PROJECTOR DESIGN

Original: We present additional ablations on the projector design, complementing our discussion in Sec. 4.
Revised: We present additional ablations on the projector design, complementing our discussion in Sec. 4.

Original: of the main paper.
Revised: of the main paper.

Original: We analyse the effect of the bottleneck layer dimension ( d b ), and whether or not to ℓ 2 -normalize the projector input ( ℓ 2 ) to the performance on IN1K and transfer tasks.
Revised: We analyse the effect of the bottleneck layer dimension ( d b ), and whether or not to ℓ 2 -normalize the projector input ( ℓ 2 ) to the performance on IN1K and transfer tasks.

Original: As we see in Tab. 9, they have only a small influence on the performance.
Revised: As we see in Tab. 9, they have only a small influence on the performance.

Original: C. E XTENDED ABLATIONS FOR O NLINE C LASS M EANS (OCM)
Revised: C. E XTENDED ABLATIONS FOR O NLINE C LASS M EANS (OCM)

Revised: Varying the projector and predictor head architecture.

Revised: In Tab. 1 of the main paper, we study the impact of the projector head parameters for the models we train with Eq.

Revised: Here, we replicate this study for the OCM variant.

Revised: Specifically, we ablate the L and d h parameters for the OCM model defined by Eq.

Original: ImageNet-1K Accuracy (%)
Revised: We show the results in Fig. 12a, and see that our observation still holds, i.e., increasing the complexity of projector improves transfer performance at the cost of IN1K performance. We also tested the impact of predictors in OCM models, and observed a similar behavior: Small predictors improve the transferability of encoder representations by sacrificing IN1K performance; allowing these variants to move along the envelope. Consequently, training OCM models with no predictor, i.e., when h ψ is an identity mapping in Fig. 9b, improves IN1K performance.

Revised: Indeed, our best model on IN1K, i.e., t-ReX* , is a t-ReX 1 -OCM variant with no predictor head.

Revised: [Figure]

Original: M ea n T r a n s f e r
Revised: M ea n T r a n s f e r

Original: A cc .
Revised: A cc .

Original: ( L og odd s )
Revised: ( L og odd s )

Original: L =1, d h = L =1, d h =
Revised: L =1, d h = L =1, d h =

Original: L =2, d h = L =2, d h =
Revised: L =2, d h = L =2, d h =

Original: L =3, d h = L =3, d h = 
Revised: L =3, d h = L =3, d h = (

Revised: [Figure]

Original: (a) Projector parameters L and d h  1 .1 .1 .
Revised: parameters [Figure] and [Figure]

Original: ImageNet-1K Accuracy (%)
Revised: ImageNet-1K Accuracy (%)

Original: M ea n T r a n s f e r
Revised: M ea n T r a n s f e r

Original: A cc .
Revised: A cc .

Original: ( L og odd s ) |Q| = |Q| = |Q| = |Q| = |Q| = |Q|=
Revised: ( L og odd s ) |Q| = |Q| = |Q| = |Q| = |Q| = |Q|

Original: (b
Revised: =

Original: [Figure]
Revised: [Figure]

Original: Varying the projector and predictor head architecture.

Original: In Tab. 1 of the main paper, we study the impact of the projector head parameters for the models we train with Eq. Here, we replicate this study for the OCM variant. Specifically, we ablate the L and d h parameters for the OCM model defined by Eq. We show the results in Fig. 12a, and see that our observation still holds, i.e., increasing the complexity of projector improves transfer performance at the cost of IN1K performance.
Revised: Memory

Original: We also tested the impact of predictors in OCM models, and observed a similar behavior: Small predictors improve the transferability of encoder representations by sacrificing IN1K performance; allowing these variants to move along the envelope. Consequently, training OCM models with no predictor, i.e., when h ψ is an identity mapping in Fig. 9b, improves IN1K performance. Indeed, our best model on IN1K, i.e., t-ReX* , is a t-ReX 1 -OCM variant with no predictor head.
Revised: [Figure]

Original: Varying the memory bank size.
Revised: Varying the memory bank size.

Original: Fig.
Revised: Fig.

Original: 12b shows how the size of the memory bank impacts performance for the OCM variant, where we see that even a moderately-sized memory bank of |Q| = 8192 , i.e., containing only 8 points per class on average is sufficient to obtain high performance on both axes.
Revised: 12b shows how the size of the memory bank impacts performance for the OCM variant, where we see that even a moderately-sized memory bank of |Q| = 8192 , i.e., containing only 8 points per class on average is sufficient to obtain high performance on both axes.

Original: C. E XTENDED ANALYSES OF LEARNED FEATURES , CLASS WEIGHTS AND PROTOTYPES
Revised: C. E XTENDED ANALYSES OF LEARNED FEATURES , CLASS WEIGHTS AND PROTOTYPES

Original: In Figs.
Revised: In Figs.

Original: 5 and 6 of the main paper, we compare six models: a) Base : a model trained using cosine softmax loss without multi-crop and projector, b) Base (BS=2K) :
Revised: 5 and 6 of the main paper, we compare six models: a) Base : a model trained using cosine softmax loss without multi-crop and projector, b) Base (BS=2K) :

Original: Base but with 9 × larger batch size, c) Base+Mc : Base plus multi-crop, d) Base+Pr : Base plus a projector, e) Base+Mc+Pr , and f)
Revised: Base but with 9 × larger batch size, c) Base+Mc : Base plus multi-crop, d) Base+Pr : Base plus a projector, e) Base+Mc+Pr , and f)

Original: OCM : a model trained using Eq.
Revised: OCM : a model trained using Eq.

Original: (2), from multiple aspects to get more insights on how our proposed training setup improves performance.
Revised: (2), from multiple aspects to get more insights on how our proposed training setup improves performance.

Original: We extend these analyses below.
Revised: We extend these analyses below.

Original: When comparing models, we use either features extracted using these models on each dataset, their class weights, or their prototypes.
Revised: When comparing models, we use either features extracted using these models on each dataset, their class weights, or their prototypes.

Original: For the analyses with features, we use only the samples in the test splits of IN1K and CoG levels, but all the samples in the others (as these datasets are small-scale).
Revised: For the analyses with features, we use only the samples in the test splits of IN1K and CoG levels, but all the samples in the others (as these datasets are small-scale).

Original: Pairwise ℓ 2 -distance.
Revised: Pairwise ℓ 2 -distance.

Original: We ℓ 2 -normalize features and compute average pairwise ℓ 2 -distance either between samples from the same class (intra-class) or between all samples in a dataset (all-sample).
Revised: We ℓ 2 -normalize features and compute average pairwise ℓ 2 -distance either between samples from the same class (intra-class) or between all samples in a dataset (all-sample).

Original: In Fig.
Revised: In Fig.

Original: 5 (left) of the main paper, we show intra-class distance for the four models (a, c, d and e) and for completeness, in Fig.
Revised: 5 (left) of the main paper, we show intra-class distance for the four models (a, c, d and e) and for completeness, in Fig.

Original: 13 (left), we provide all-sample ℓ 2 -distances for them.
Revised: 13 (left), we provide all-sample ℓ 2 -distances for them.

Original: Sparsity.
Revised: Sparsity.

Original: We compute sparsity as the percentage of feature dimensions close to zero.
Revised: We compute sparsity as the percentage of feature dimensions close to zero.

Original: To compute this metric, we apply a threshold ϵ to individual dimensions of ℓ 2 -normalized features.
Revised: To compute this metric, we apply a threshold ϵ to individual dimensions of ℓ 2 -normalized features.

Original: In Fig.
Revised: In Fig.

Original: 5 (right) of the main paper, we show the results obtained with ϵ = 10 − 5 , but we found that the conclusions obtained with a range of ϵ values logarithmically sampled between 10 − 3 and 10 − 8 were consistent.
Revised: 5 (right) of the main paper, we show the results obtained with ϵ = 10 − 5 , but we found that the conclusions obtained with a range of ϵ values logarithmically sampled between 10 − 3 and 10 − 8 were consistent.

Original: Coding length.
Revised: Coding length.

Original: We compute average coding length (Ma et al., 2007; Yu et al., 2020) of samples in a dataset as R ( X , ϵ ) = 1 / 2 log det( I d + d / Nϵ 2 X ⊤ X ) , where I d is a d -by- d identity matrix, ϵ 2 is the precision parameter set to 0 .
Revised: We compute average coding length (Ma et al., 2007; Yu et al., 2020) of samples in a dataset as R ( X , ϵ ) = 1 / 2 log det( I d + d / Nϵ 2 X ⊤ X ) , where I d is a d -by- d identity matrix, ϵ 2 is the precision parameter set to 0 .

Original: 5 and X ∈ R N × d is a feature matrix containing N samples each encoded into a d -dimensional representation (2048 in our case).
Revised: 5 and X ∈ R N × d is a feature matrix containing N samples each encoded into a d -dimensional representation (2048 in our case).

Original: In Fig.
Revised: In Fig.

Original: 6 (left) of the main paper, we show average coding length for the four models (a, c, d and e) obtained on transfer datasets (i.e., coding length values are further averaged over transfer datasets).
Revised: 6 (left) of the main paper, we show average coding length for the four models (a, c, d and e) obtained on transfer datasets (i.e., coding length values are further averaged over transfer datasets).

Original: In Fig.
Revised: In Fig.

Original: 14 we show coding length for the same four models plus DINO Caron et al.
Revised: 14 we show coding length for the same four models plus DINO Caron et al.

Original: (2021) obtained on each dataset, separately.
Revised: (2021) obtained on each dataset, separately.

Original: Moreover, we compute singular values per dimension for models with and without projectors (Base+Pr and
Revised: Moreover, we compute singular values per dimension for models with and without projectors (Base+Pr and

Original: Base+Mc).
Revised: Base+Mc).

Original: For each model, we compute singular values on each transfer dataset which are normalized by their sum so that they sum to 1.
Revised: For each model, we compute singular values on each transfer dataset which are normalized by their sum so that they sum to 1.

Original: We then sort these normalized singular values by decreasing order, and average them over transfer datasets.
Revised: We then sort these normalized singular values by decreasing order, and average them over transfer datasets.

Original: As can be seen in Fig.
Revised: As can be seen in Fig.

Original: 13 (right), feature variance is more uniformly distributed over dimensions when a projector is used.
Revised: 13 (right), feature variance is more uniformly distributed over dimensions when a projector is used.

Original: Gradient similarity and variance.
Revised: Gradient similarity and variance.

Original: To get a better understanding on why multi-crop improves performance significantly for the same effective batch size, we compare the models Base+Mc and
Revised: To get a better understanding on why multi-crop improves performance significantly for the same effective batch size, we compare the models Base+Mc and

Original: Base (BS=2K) by analyzing gradients of their class weights ∇ W L CE at each SGD update.
Revised: Base (BS=2K) by analyzing gradients of their class weights ∇ W L CE at each SGD update.

Original: We compute absolute cosine similarity between “class gradients” for a pair of classes
Revised: We compute absolute cosine similarity between “class gradients” for a pair of classes

Original: c i and c j as  | sim ( ∇ w ci L CE , ∇ w cj L CE ) | , where sim ( · , · ) denotes cosine similarity.
Revised: c i and c j as [Figure] [Figure] | sim ( ∇ w ci L CE , ∇ w cj L CE ) | , of the main paper.

Original: Then we average them over all class pairs ( c i , c j ) ∼ P in a dataset, which is reported in Fig.

Original: 6 (middle) of the main paper.

Original: Additionally, in Fig.
Revised: Additionally, in Fig.

Original: 16 we show Frobenius norm and standard deviation computed over ∇ W L CE .
Revised: 16 we show Frobenius norm and standard deviation computed over ∇ W L CE .

Original: We observe that norm and variance of gradients increase with multi-crop, which we believe is due to increasing variance in embeddings as noted by Wang et al.
Revised: We observe that norm and variance of gradients increase with multi-crop, which we believe is due to increasing variance in embeddings as noted by Wang et al.

Original: (2022a).
Revised: (2022a).

Original: [Figure]

Original: [Figure]

Original: Feature redundancy.
Revised: Feature redundancy.

Original: In addition to the metrics described above, following Wang et al.
Revised: In addition to the metrics described above, following Wang et al.

Original: (2022b)
Revised: (2022b)

Original: , we compute “redundancy” of features X ∈ R N × d (defined as in the “coding length” paragraph above) as R = 1 / d 2 (cid:80) i (cid:80) j | ρ ( X : ,i , X : ,j ) | , where , ρ ( X : ,i , X : ,j ) is the Pearson correlation between a pair of feature dimensions i and j .
Revised: , we compute “redundancy” of features X ∈ R N × d (defined as in the “coding length” paragraph above) as R = 1 / d 2 (cid:80) i (cid:80) j | ρ ( X : ,i , X : ,j ) | , where , ρ ( X : ,i , X : ,j ) is the Pearson correlation between a pair of feature dimensions i and j .

Original: In Fig.
Revised: In Fig.

Original: 15 we show redundancy score for the four models (a, c, d and e) plus DINO Caron et al.
Revised: 15 we show redundancy score for the four models (a, c, d and e) plus DINO Caron et al.

Original: (2021) obtained on each dataset, separately.
Revised: (2021) obtained on each dataset, separately.

Original: C. R ESULTS ON IN1K- V 2 AND IN1K-S KETCH
Revised: C. R ESULTS ON IN1K- V 2 AND IN1K-S KETCH

Original: We compare t-ReX and t-ReX* to the previous state of the art on IN1K-v2 and IN1K-Sketch.
Revised: We compare t-ReX and t-ReX* to the previous state of the art on IN1K-v2 and IN1K-Sketch.

Original: As before, for each model, we use the trained encoder as a feature extractor, and we reuse the linear classifier trained on IN1K and apply it directly to the test images of IN1K-v2 and IN1K-Sketch.
Revised: As before, for each model, we use the trained encoder as a feature extractor, and we reuse the linear classifier trained on IN1K and apply it directly to the test images of IN1K-v2 and IN1K-Sketch.

Original: Note that there are 3 test sets for IN1K-v2, and we evaluate over all of them.
Revised: Note that there are 3 test sets for IN1K-v2, and we evaluate over all of them.

Original: Tab.
Revised: Tab.

Original: 10 presents our results.
Revised: 10 presents our results.

Original: Looking at the mean top-1 accuracy over the three test sets of IN1K-v2, we observe that t-ReX* also matches the performance of RSB-A1 on IN1K-v2 outperforming all others, showing strong domain generalization capabilities.
Revised: Looking at the mean top-1 accuracy over the three test sets of IN1K-v2, we observe that t-ReX* also matches the performance of RSB-A1 on IN1K-v2 outperforming all others, showing strong domain generalization capabilities.

Original: On the other hand, SupCon performs the best on IN1K-Sketch, where t-ReX* is the second best.
Revised: On the other hand, SupCon performs the best on IN1K-Sketch, where t-ReX* is the second best.

Original: We think that the contrastive loss used in SupCon might have improved its out-of-distribution robustness for the training concepts.
Revised: We think that the contrastive loss used in SupCon might have improved its out-of-distribution robustness for the training concepts.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: C. R ESULTS ON CLASS - IMBALANCED TRANSFER DATASETS

Revised: We evaluate the long-tail transfer classification performance of DINO, PAWS, RSB-A1, t-ReX and t-ReX* on two class-imbalanced datasets, iNaturalist 2018 and iNaturalist 2019 (Van Horn et al., 2018).

Revised: For these evaluations, we follow the Log.

Revised: Reg.

Revised: protocol from the ImageNet-CoG benchmark (see Appendix B.4).

Revised: Results are reported in Tab. 11.

Revised: We see that our t-ReX and t-ReX* models still outperform RSB-A1 and DINO respectively, despite a challenging long-tail class distribution.

Original: D L IMITATIONS
Revised: D L IMITATIONS

Original: Requirement for annotations.
Revised: Requirement for annotations.

Original: Firstly, our training setup is tailored for supervised learning, and therefore, its performance on both training and transfer tasks depends on the availability of highquality and diverse annotations for the training images.
Revised: Firstly, our training setup is tailored for supervised learning, and therefore, its performance on both training and transfer tasks depends on the availability of highquality and diverse annotations for the training images.

Original: Image-level annotation, when involving a large number of potentially fine-grained classes is an expensive and error-prone process.
Revised: Image-level annotation, when involving a large number of potentially fine-grained classes is an expensive and error-prone process.

Original: In this work, we show that given a large-scale curated and annotated dataset, more precisely given IN1K, which is composed of 1.28M images annotated for 1000 different concepts, it is possible to learn more generic representations than self- or semi-supervised models.
Revised: In this work, we show that given a large-scale curated and annotated dataset, more precisely given IN1K, which is composed of 1.28M images annotated for 1000 different concepts, it is possible to learn more generic representations than self- or semi-supervised models.

Original: When only a handful of concepts is annotated, or when annotations contain too much noise, these conclusions might not be accurate anymore, and both the pretraining classification task and the transfer tasks results might be degraded.
Revised: When only a handful of concepts is annotated, or when annotations contain too much noise, these conclusions might not be accurate anymore, and both the pretraining classification task and the transfer tasks results might be degraded.

Original: In this case, self- or semi-supervised approaches might become more relevant.
Revised: In this case, self- or semi-supervised approaches might become more relevant.

Original: Yet, those scenarios are out of the scope of our study.
Revised: Yet, those scenarios are out of the scope of our study.

Original: Specific encoder architecture.
Revised: Specific encoder architecture.

Original: Secondly, we develop our training setup based on a single encoder architecture, ResNet50, and do not test it on other architectures.
Revised: Secondly, we develop our training setup based on a single encoder architecture, ResNet50, and do not test it on other architectures.

Original: This was motivated by the fact that ResNet50 is still a very commonly used architecture.
Revised: This was motivated by the fact that ResNet50 is still a very commonly used architecture.

Original: Also, we note that our training setup components, multi-crop, a data-augmentation operator, and the expendable projector, added after the encoder, are architecture-agnostic so those contributions can be seamlessly applied to any other architecture of choice.
Revised: Also, we note that our training setup components, multi-crop, a data-augmentation operator, and the expendable projector, added after the encoder, are architecture-agnostic so those contributions can be seamlessly applied to any other architecture of choice.

Original: Therefore, it is reasonable to expect that our setup would consistently improve other architecture families, such as Vision Transformers (ViTs) Dosovitskiy et al.
Revised: Therefore, it is reasonable to expect that our setup would consistently improve other architecture families, such as Vision Transformers (ViTs) Dosovitskiy et al.

Original: In fact, both components were previously successfully used with ViTs for self-supervised learning Caron et al.
Revised: In fact, both components were previously successfully used with ViTs for self-supervised learning Caron et al.

Original: We leave studying the applicability of our training setup to other encoder architectures as future work.
Revised: We leave studying the applicability of our training setup to other encoder architectures as future work.

Original: [Table]
Revised: [Table]