L EARNING USEFUL REPRESENTATIONS FOR SHIFTING TASKS AND DISTRIBUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Does the dominant approach to learn representations (as a side effect of optimizing an expected cost for a single training distribution) remain a good approach when we are dealing with multiple distributions. Our thesis is that such scenarios are better served by representations that are “richer” than those obtained with a single optimization episode.
This is supported by a collection of empirical results obtained with an apparently naïve ensembling technique: concatenating the representations obtained with multiple training episodes using the same data, model, algorithm, and hyper-parameters, but different random seeds. These independently trained networks perform similarly. Yet, in a number of scenarios involving new distributions, the concatenated representation performs substantially better than an equivalently sized network trained from scratch.
This proves that the representations constructed by multiple training episodes are in fact different. Although their concatenation carries little additional information about the training task under the training distribution, it becomes substantially more informative when tasks or distributions change. Meanwhile, a single training episode is unlikely to yield such a redundant representation because the optimization process has no reason to accumulate features that do not incrementally improve the training performance.
I NTRODUCTION
Although the importance of features in machine learning systems was already clear when the
Perceptron was invented (Rosenblatt, 1957), learning features from examples was often considered a hopeless task (Minsky and Papert, 1969).
Some researchers hoped that random features were good enough, as illustrated by the Perceptron.
Other researchers preferred to manually design features using substantive knowledge of the problem (Simon, 1989).
This changed when Rumelhart et al.
(1986) showed that the possibility of feature learning as a side effect of the risk optimization.
Despite reasonable concerns about the optimization of nonconvex cost functions, feature discovery through optimization has driven the success of deep learning methods.
There are however many cues suggesting that learning features no longer can be solely understood through the optimization of the expected error for a single data distribution.
First, adversarial examples (Szegedy et al., 2014) and shortcut learning (Geirhos et al., 2020) illustrate the need to make learning systems that are robust to certain changes of the underlying data distribution and therefore involve multiple expected errors.
Second, the practice of transferring features across related tasks (Bottou, 2011; Collobert et al., 2011; Oquab et al., 2014) is now viewed as foundational (Bommasani et al., 2021) and intrinsically involves multiple data distributions and cost functions.
It is therefore timely to question whether the optimization of a single cost function creates and accumulates features in ways that make the most sense in this broader context.
This contribution reports on experiments showing how the out-of-distribution performance of a deep learning model can benefit from internal representations that are richer and more diverse than those computed through the usual optimization process.
For instance, in a variety of changing distribution scenarios, merely concatenating the penultimate layer representations obtained with several training episodes can outperform networks of equivalent size trained from scratch.
Limitations
This work only considers the representations extracted by the penultimate layer of a deep network with a linear elementary classifier, or, in a single case, a distance-based classifier.
Our experimental results also focus image recognition problems. Whether the same findings hold for inner representations or other applicative domains is left to future work.
Despite this limited focus, it is worth noticing that our experiments cover a variety of model sizes, ranging from a couple millions to a couple billions of parameters.
Finally, we only consider two very simple ways to create rich representations, leaving considerable room for improved variants.
Organization of the manuscript Section 3 argues that a vast subspace of the penultimate layer representation of a deep network can contain information that has no bearing on the optimization cost and yet can have a substantial impact on the transfer learning performance of the network.
Sectionpresents simple techniques that can be used to produce networks with slightly different representations and to accumulate them in order to construct a richer and internally redundant representation vector.
Sections 5, 6, 7, and 8, present experimental results pertaining respectively to supervised transfer learning, self-supervised transfer learning, meta-learning, and out-of-distribution learning.
R ELATED WORK
Several authors (e.g. Huang et al., 2020; Teney et al., 2022) propose to work around the shortcut learning problem (Geirhos et al., 2020) by shaping the last-layer classifier or introducing penalty terms in the optimization process in a manner that favors richer representations.
Zhang et al.
(2022) shows that many of these additions make the optimization challenging but more manageable if one initializes the networks with rich features constructed by alternate methods.
The methods discussed in our work resemble the RFC algorithm of Zhang et al.
(2022) because they also rely on multiple training episodes.
However, we find that merely training with different random seeds provides sufficient diversity to achieve excellent performances for a variety of problems involving changes in data distribution, revealing the limitations of representations constructed with a single training run.
Our experimental approach is related to deep ensembles (e.g. Lakshminarayanan et al., 2017) which combine the predictions of separately trained networks in order to achieve superior in-distribution performance. Ensembling techniques work best when one makes sure that the individual models are diverse and have weakly correlated errors. Engineering diversity is often an expression of prior knowledge about the problem at hand. In contrast, we purposely refrain from engineering diversity (other than changing the seed) and we focus on the performance of our ensembles for new tasks and data distributions.
The idea of rich representation is also related to recent work (Wang et al., 2022; Dvornik et al., 2020; Bilen and Vedaldi, 2017; Gontijo-Lopes et al., 2021;
Li et al., 2021; 2022) on “universal representations” that combine the representations obtained with different tasks, datasets, network architectures, hyper-parameters, etc. Because we purposely refrain from engineering such a diversity and still observe a substantial effect, we cast a new light on both the desirable properties of a robust representation and the limitations of the usual training process.
T HE REPRESENTATION NULLSPACE
For the purposes of this work, we view a deep learning network as the composition of a feature extractor Φ that maps a pattern x into a representation Φ( x ) ∈ R d and a linear layer that produces an output of dimension k that we assume is substantially smaller than d (i.e. k < d ).
The network output is then expressed as w ◦ Φ( x ) where w is the k × d weight matrix of the linear output layer.
Since we assume that k < d , the matrix w has a substantial nullspace N of dimension greater or equal to d − k .
Therefore the function implemented by the network remains unchanged when one replaces the feature extractor Φ by any Φ ′ such that Φ( x ) − Φ ′ ( x ) ∈ N with high probability.
This implies that there is a considerable choice of potential feature extractors that produce different representations and yet have no impact on the network output.
Since such equivalent representations do not affect the empirical risk, they can only be differentiated by regularization, either explicit or implicit.
In order to improve the in-distribution generalization performance of the network, common regularization techniques, such as the ubiquitous weight decay, aim at zeroing the nullspace component of the representations Φ( x ) .
For instance, Papyan et al.
(2020) show how the penultimate layer representation of various deep network collapses to a “simplex equi-angular tight frame” of dimension k − 1 when trained for a very long time using a cross-entropy loss and a slight weight decay.
When this happens, the representation carries very little information other than a noise-tolerant encoding of the network output.
Although this situation can be beneficial for in-distribution generalization, we argue that it is often problematic when the task or the data distribution changes.
• In scenarios such as transfer learning and meta-learning, one is allowed to adapt the network after the task or distribution change.
Because the gradient back-propagated through the linear layer is orthogonal to the nullspace N , the only pressure to change the nullspace component of the representation Φ( x ) comes from fluctuations of the nullspace itself.
Such fluctuations occur when the linear layer weights w are adapted to exploit a temporary correlation between the desired output and potentially interesting features presented in the nullspace.
This is easier when the initial training leaves more information in the nullspace.
• In other scenarios, one seeks to construct networks that are robust with respect to targeted data distribution changes.
This is usually achieved by special optimization objectives such as distributional robust optimization (Sagawa et al., 2020) or such as invariant learning (Arjovsky et al., 2020).
In order to tame these more challenging optimization problems, it is common to pretrain the networks with empirical risk minimization.
However, Zhang et al.
(2022) show that constructing rich representations gives far better results than pretraining with empirical risk minimization.
Table 1 reports on a simple experiment consisting of pre-training a RESNET 18 on the C IFARtask and transferring its learned features to a C IFAR 100 task by simply retraining the last linear layer (see setups in Appendix A).
Although the best in-distribution performance, 94.9%, is achieved using a slight weight decay, the transfer learning performance of the features learned without weight decay is far superior (49.6% versus 29.1%).
The same observation holds when one reverses the role of the C IFAR 10 and C IFAR 100 datasets.
Because we are far from a full network collapse, this effect disappears when one fine-tunes the transferred feature extractor, effectively recovering the performance of a network directly trained on the target task.
Sections 5 and 6 offer a more thorough discussion of fine-tuning.
[Table]
Things become of course more complicated when we abandon the assumptions that allowed us to define the representation nullspace.
For instance, the final layer might still be linear but with a number of outputs or classes k that exceeds the dimension d of the penultimate layer.
Alternatively the final layer might not be linear at all, as, for instance, in the siamese networks Bromley et al.
(1993) increasingly used in self-supervised scenarios.
Yet, the work of Papyan et al.
(2020), the information bottleneck approach of Shwartz-Ziv and Tishby (2017), and the gradient starvation observations (Pezeshki et al., 2021) of, still suggest that in the long run, the regularized training process produces upper layer representations that hardly contain any information other than what is necessary for the task encoded by the optimization criterion.
In fact, once the representation contains enough information to fulfill the training task, the optimization process has no reason to create and accumulate features that no longer help improve the training objective but might yet become useful when the data distribution changes.
F EATURE ACCUMULATION
How then can we construct a deep learning procedure that accumulates as many potentially useful features as possible into the internal representations of the network?
The answer depends of course on how we determine that a feature is potentially useful before knowing how the data distribution might change.
The Rich Feature Construction (RFC) algorithm of Zhang et al.
(2022, § 4.1) seeks features that are useful for at least some subset of the training examples.
The algorithm first performs multiple training episodes using adversarially re-weighted training data in a manner that ensures that these training episodes yield substantially different representations.
Then a multiple head distillation process (see
Figure 1, Left) constructs a single feature extractor that combines all these distinct representations into a single representation vector of equivalent size.
This cumbersome process has been shown to vastly improve the out-of-distribution performance of several invariant training algorithms.
[Figure]
C AT
[Figure] [Figure]
The present contribution exploits a far simpler intuition. Although the optimization process has no reason to accumulate features that no longer help the training objective, these same features might have been found useful when the network representation was still wanting.
Therefore we posit that we can obtain substantially different representations by simply performing multiple training episodes with different seeds, that is, different initial weights and different data orderings.
In order to validate this intuition
[Figure]
we train five RESNET 18 networks on the Caltech-UCSD Birds ( CUB ) dataset
(Wah et al., 2011) using different seeds, yielding five feature extractors Φ i and final linear layers v i .
We then compute the normalized least square errors (computed via QR decomposition) obtained when predicting the output of network j using the features of each network i ̸
= j ,
[Equation]
left) shows how these errors are substantially higher than those obtained when predicting the network outputs using the combined feature extractor obtained by distillation (Figure 1, left).
This indicates that merely training with the same data but different seeds produces different representations which can be combined into a richer representation that predicts the outputs of all the original networks better than any individual network representation.
Figure 2 (right) shows the same effect using instead five regularly spaced snapshots taken during a single training episode with a high step size.
Although this is computationally more attractive, the experimental results reported in the rest of this paper always use the more easily interpretable and independently reproducible multiple training approach.
Although distillation provides the means to combine the representations of multiple networks into a single feature extractor of equivalent dimension (Figure 1, left), the distillation process depends on the training data distribution and therefore might still exclude features that could become useful when the distribution changes.
The simplest distribution-independent way to combine representations consists of concatenating them into a vector whose dimension is the sum of the dimensions of the [Table] [Figure] source representations (Figure 1, right).
However, we must then compare the out-of-distribution performance of the concatenated representation with that of single networks of equivalent complexity.
The following sections show that substantial improvements can still be achieved under this condition.
S UPERVISED TRANSFER LEARNING EXPERIMENTS
This section focuses on supervised transfer learning scenarios in which the representations learned using an auxiliary supervised task, such as the I MAGE N ET (1k) object recognition task (Deng et al., 2009), are then used for the target tasks, such as, for our purposes, the C IFAR 10, C IFAR 100, and
I NATURALIST 18 (I NAT 18), object recognition tasks (Krizhevsky, 2009; Van Horn et al., 2018).
We distinguish the linear probing scenario where the penultimate layer features of the pre-trained network are used as inputs for linear classifiers trained on the target tasks, and the fine tuning scenario which uses back-propagation to further update the transferred features using the target task training data.
The first three rows of Table 2, labeled ERM, provide baselines for the linear probing scenario, using respectively a RESNET 50 network (He et al., 2016a), as well as larger variants RESNET 50 W n with n times wider internal representations and roughly n 2 times more parameters.
The following
$ !"% !" $ !"% !"
#$%"&’()#&*+)(#%,(&-.’/’&"0)%& 3’(#)%-(#)#&*+)(#%,(%&"4#%)%& ! " !
# ! $ " 5,(*&%#(&%#)6’(#$%-(#7)"#8"#1#(%&%’,(1)&(7)%"&’( */&11’6’#") !" [Figure]
two rows of Table 2 provides additional baseline results using networks n × RESNET 50 composed of respectively n separate RESNET 50 networks joined by concatenating their penultimate layers.
Although these networks perform relatively poorly on the pre-training task I MAGE N ET , their linear probing performance is substantially better than that of the ordinary RESNET s.
The following four lines of Table 2, labeled C AT n , are obtained by training n separate RESNETnetworks on I MAGE N ET using different random seeds and using their concatenated representations as inputs for a linear classifier trained on the target tasks.
This approach yields linear probing performances that substantially exceed the performances of comparably sized baseline networks.
It is interesting to note how C AT n , with separately trained components, outperforms the architecturally similar n × RESNET 50 trained as a single network.
The final line of Table 2, labeled D ISTILL 5, is obtained by combining the representation of five separately trained RESNET 50 by distillation and training a linear classifier on the target task.
The D ISTILL 5 linear probing performance exceeds that of the comparable RESNET 50 network but does not match C AT n .
These results are succinctly represented in the top row of Figure 3.
For each target task I NAT 18,
C IFAR 100, and C IFAR 10, two curves respectively show the linear probing performance of the baseline RESNET 50 W n (red, labeled “wide(lp)” ) and of the C AT n networks (blue, “cat(lp)” ) as a function of the number of parameters of their inference architecture.
An additional point (pink star, “distill(lp)” ) describes the performance of D ISTILL 5.
The left plot (double height) of Figure 3 provides the same information when the target task is the same as the pre-training task.
In order to save space, all further results in the main text of this contribution are presented with such plots, with result tables provided in the appendix B and C.
The top row of Figure 3 also plots the performance of deep ensemble averaging (logits averaging, green, labeled “ensemble” ).
Such ensembles rely on the same concatenated representation as the C AT n but differ because the final classifier is an average of n separately trained classifiers of dimension d instead of a full linear classifier of dimension nd .
We view this as a difference in training capacity that explains why ensembles match the C AT n performance on the C IFAR tasks but lag behind on the more challenging I NAT 18 task.
The bottom row of Figure 3 presents results for the fine-tuning scenario.
For each target task, the red curve (labeled “wide(ft)” ) plots the fine-tuning transfer learning performance of the baseline
RESNET 50 W n networks.
The brown curves ( “cat(ft)” ) represent the poor transfer performance obtained by concatenating n separately trained RESNET 50 feature extractors, adding a linear layer, and fine-tuning everything by back-propagation on the target task.
The poor performance of plain fine-tuning has already been pointed out (Kumar et al., 2022; Kirichenko et al., 2022).
In contrast, the blue curves ( “cat(2ft)” ) represent the superior performance of two-stage fine tuning (Figure 4), which consists of separately training n RESNET 50 on I MAGE N ET , separately fine-tuning these RESNETon the target task, and finally training a linear classifier on top of the concatenation of the n separately fine-tuned representations. [Figure]
)” ) matches equivalently sized baseline models, but with much easier training.
Due to the space limitation, we put other fine-tuning curves in appendix C.1.1.
Figure 5 shows that these observations also hold for large self-attention networks. We carried out supervised transfer experiments using the original vision transformers (ViT) (Dosovitskiy et al., 2020) and using a more advanced version (ViT(augreg)) with tuned data augmentations and regularization (Steiner et al., 2021). We use two transformers of two different sizes, ViT-B/16 and ViT-L/16, pretrained on I MAGE N ET 21 K . 1 Supervised transfer baselines are obtained by linear-probing and by fine-tuning on I MAGE N ET 1 K .
These baselines are outperformed by respectively linear-probing and two-stage fine tuning on top of the concatenation of their final representations (C AT 2).
Note that an even larger transformer architecture, ViT-H/14, yields about the same I MAGE N ET 1 K fine-tuning performance as ViT-L/16, despite having twice as many parameters Dosovitskiy et al.
S ELF - SUPERVISED TRANSFER LEARNING EXPERIMENTS
In self-supervised transfer learning (SSL), transferable representations are no longer constructed using a supervised auxiliary task, but using a training criterion that does not involve tedious manual labeling.
We focus on schemes that rely on the knowledge of a set of acceptable pattern transformations.
The training architecture then resembles a siamese network whose branches process different transformations of the same pattern.
The SSL training objective must then balance two terms: on the one hand, the representations computed by each branch must be close or, at least, related; on the other hand, they should be prevented from collapsing partially (Jing et al., 2021) or catastrophically (Chen and He, 2020).
Although this second term tends to fill the representation with useful features, what is necessary to balance the SSL training objective might still exclude potentially useful features for the target tasks.
This section presents results obtained using SWAV pre-training using 1.2 million I MAGE N ET images (Caron et al., 2020) and using SEER pre-training using 1 billion I NSTAGRAM 1B images (Goyal et al., 2022).
These experiments leverage the pre-trained models made available by the authors: four
RESNET 50, one RESNET 50 W 2, one RESNET 50 W 4 and one RESNET 50 W 5 for the SWAV experiments; 2 and one REGNET 32 GF , one REGNET 64 GF , one REGNET 128 GF , and one REGNET 256 GF (1.3B parameters) for the SEER experiments.
The first four columns of Figure 6 present linear probing results for four target object recognition tasks: supervised I MAGE N ET , I NATURALIST 18, C IFAR 100, and C IFAR 10.
The baseline curves (red, labeled “wide(lp)” or “wide&deep(lp)” ) plots the performance of linear classifiers trained on top of the pre-trained SSL representations.
The C AT n curves (blue, labeled “cat(lp)” ) were obtained by training a linear classifier on top of the concatenated representations of the n smallest SSL pre-trained Checkpoints SEER [Table]
[Figure] representations. The deep ensemble curve (green, labeled ) was obtained by averaging the outputs (before softmax) of linear classifiers separately trained on top of each of the n smallest SSL pre-trained representations.
The C AT n approach offers the best overall performance.
The last column of Figure 6 presents results with fine-tuning for the supervised I MAGE N ET task.
The C AT n approach with two-stage fine-tuning (Figure 4) matches the performance of equivalently sized baseline networks.
In particular, the largest C AT 4 model, with 2.3B parameters, achieves 85.5% correct classification rate, approaching the 85 .
8% rate of the largest network of Goyal et al.
REGNET 10B with 10B parameters.
Of course, separately training and fine-tuning the components of the C AT 4 network is far easier than training a single REGNET 10B network.
Additional results using S IM S IAM (Chen et al., 2020) and with D ISTILL are provided in appendix
C.3.
Other experiment details are provided in appendix C.
M ETA - LEARNING AND FEW - SHOTS LEARNING EXPERIMENTS
Each target task in the few-shots learning scenario comes with only a few training examples.
One must then consider a large collection of target tasks to obtain statistically meaningful results.
We follow the setup of Chen et al.
(2019) in which the base task is an image classification task with a substantial number of classes and examples per class,
and the target tasks are five-way classification problems involving novel classes that are distinct from the base classes and come with only a few examples.
Such a problem is often cast as a meta learning problem in which the base data is used to learn how to solve a classification problem with only a few examples.
Chen et al.
(2019) find that excellent performance can be achieved using simple baseline algorithms such as supervised transfer learning with linear probing, (B ASELINE , as in Section 5), or with a cosine-based final classifier (B ASELINE ++).
These baselines match and sometimes exceed the performance of common few shots algorithm such as MAML (Finn et al., 2017), R ELATION N ET (Sung et al., 2018), M ATCHING N ET (Vinyals et al., 2016), and P ROTO N ET (Snell et al., 2017).
Figure 7 reports results obtained with a RESNET 18 architecture on both the M INI I MAGE N ET and
C UB ) five ways classification tasks with either one or five examples per class as set up by Chen et al.
The MAML , R ELATION N ET , M ATCHING N ET , and P ROTO N ET results (red bars) are copied verbatim from (Chen et al., 2019, table A5).
The B ASELINE and B ASELINE ++ results were further improved by a systematic L2 weight decay search procedure (see appendix D.2).
All these results show substantial variations across runs, about 4% for C UB and 2% for M INI I MAGE N ET .
The D ISTILL n and C AT n results were then obtained by first training n RESNET 18 on the base data with different seeds, constructing a combined feature extractor by either distillation or concatenation, then, for each task, training a cosine distance classifier using these features as inputs.
Despite the high replication variance of the competing results, both D ISTILL and C AT show very strong performance.
The pink bar (D ISTILL 5-s) in Figure 7 shows that similar results can be obtained when one combines the feature extractors of five snapshots taken at regular intervals during a single training episode with a relatively high step size (0.8).
More results and details are shown in appendix D. [Table]
O UT - OF - DISTRIBUTION GENERALIZATION EXPERIMENTS
In the out-of-distribution generalization scenario [Table] we seek a model that performs well on a family of data distributions, also called environment, on the basis of a finite number of training sets distributed according to some of these distributions.
Arjovsky et al.
(2020) propose an invariance principle to solve such problems and propose the IRM V 1 algorithm which searches a good predictor whose final linear layer is simultaneously optimal for all training distributions.
Since then, a number of algorithms exploiting similar ideas have been proposed, such as V RE X (Krueger et al., 2020), F ISHR (Rame et al., 2022), or CLO V E (Wald et al., 2021).
Theoretical connections have been made with multi-calibration (Hebert-Johnson et al., 2018; Wah et al., 2011).
Alas the performance of these algorithms remains wanting (Gulrajani and Lopez-Paz, 2021).
Zhang et al.
(2022) attribute this poor performance to the numerical difficulty of optimizing the complicated objective associated with these algorithms.
They propose to work around these optimization problems by providing initial weights that already extract a rich palette of potentially interesting features constructed using the
RFC algorithm.
Following Zhang et al.
(2022), we use the C AMELYON 17 tumor classification dataset (Bandi et al., 2018) which contains medical images collected from five hospitals with potentially different devices and procedures.
As suggested in Koh et al.
(2021), we use the first three hospitals as training environments and the fifth hospital for testing.
OOD-tuned results are obtained by using the fourth hospital to tune the various hyper-parameters.
IID-tuned results only use the training distributions (see details in appendix E).
The purpose of our experiments is to investigate whether initializing with the D ISTILL or C AT algorithm provides a computationally attractive alternative to RFC.
Table 3 compares the test performance achieved by two algorithms, V RE X and ERM, after initializing with ERM, C AT n , D ISTILL n , and RFC2, in both the IID-tune and OOD-tune scenarios.
The C AT and D ISTILL initialization perform better than ERM but not as well as RFC.
This result clearly shows the need to research better ways to train networks in a manner that yields diverse representations.
Although this contribution shows that simply changing the seed (as in C AT and D ISTILL ) can achieve good results, the experience of deep ensembles (Gontijo-Lopes et al., 2022) suggests that more refined diversification methods might yield substantially better representations.
C ONCLUSION
We have presented an ensemble of experimental results for scenarios involving changing task and distributions such as transfer learning, few shots learning, and cross-domain robust learning. These results are quite good in their own right. But more importantly, they show that such scenarios are better served by representations that are richer than those obtained with a single optimization episode.
This observation provides a lot of room for new representation learning algorithms that move away from the currently dominant scheme, that is, relying solely on a single optimization episode.
A CKNOWLEGMENTS
Anonymized.
TASKS
AND DISTRIBUTIONS
Supplementary Material A C IFAR SUPERVISED TRANSFER LEARNING
C IFAR 10 supervised transfer learning experiments train a RESNET 18 network on the C IFAR 10 dataset with/without L2 weight decay (4e-5) for 200 epochs.
During training, we use a SGD optimizer (Bottou et al., 2018) with momentum=0.9, initial learning rate=0.1, cosine learning rate decay, and batch size=128.
As to data augmentation, we use R ANDOM R ESIZED C ROP (crop scale in [0 . 8 , 1 . 0] ), aspect ratio in [3 / 4 , 4 / 3] ) and R ANDOM H ORIZONTAL F LIP .
During testing, the input images are resized to 36 × 36 by bicubic interpolation and C ENTER C ROPED to 32 × 32 .
All input images are normalized by mean = (0 . 4914 , 0 . 4822 , 0 . 4465) , std = (0 . 2023 , 0 . 1994 , 0 . 2010) at the end.
Then transfer the learned representation to C IFAR 100 dataset by training a last-layer linear classifier (linear probing).
The linear layer weights are initialized by Gaussian distribution N (0 , 0 . 01) .
The linear probing process shares the same training hyper-parameters as the supervised training part except for a zero L2 weight decay in all cases.
The C IFAR 100 supervised transfer learning experiments swap the order of C IFAR 100 and C IFAR 10.
B I MAGE N ET SUPERVISED TRANSFER LEARNING B. E XPERIMENT SETTINGS
Image Preprocessing: Following He et al.
(2016b), we use R ANDOM H ORIZONTAL F LIP and
R ANDOM R ESIZED C ROP augmentations for all training tasks.
For I MAGE N ET and I NAT 18, the input images are normalized by mean = (0 . 485 , 0 . 456 , 0 . 406) , std = (0 . 229 , 0 . 224 , 0 . 225) .
For C IFAR , we use the same setting as Appendix A.
I MAGENET
Pretraining: The RESNET s are pre-trained on I MAGE N ET with the popular protocol of
Goyal et al.
(2017): a SGD optimizer with momentum=0.9, initial learning rate=0.1, batch size=256,
L2 weight decay=1e-4, and 90 training epochs.
The learning rate is multiplied by 0.1 every 30 epochs.
By default, the optimizer in all experiments is SGD with momentum=0.9.
D ISTILL : To distill the C AT n representations [ ϕ 1 , . . .
ϕ n ] ( n × RESNET 50) into a smaller representation Φ ( RESNET 50), we use the multi-head architecture as Figure 2.
Inspired by Hinton et al.
(2015), we use the Kullback–Leibler divergence loss to learn Φ as:
[Equation]
[Equation]
where s τ ( v ) i =
e vi/τ (cid:80) k
e vk/τ is a softmax function with temperature τ , v i is the learned last-layer classifier of i th sub-network of C AT n .
In the D ISTILL experiments, we distill five separately trained RESNET 50 into one RESNETaccording to Eq 2 with τ = 10 .
We use a SGD optimizer with momentum=0.9, batch size=2048, and weight decay=0.
The initial learning rate is 0.1 and warms up to 0.8 within the first 5 epochs.
Then learning rate decays to 0.16 and 0.032 at 210 th and 240 th epochs, respectively.
The total training epochs is 270.
Linear probing:
• I MAGE N ET : The I MAGE N ET linear probing experiments train a linear classifier with the same hyper-parameters as I MAGE N ET pretraining.
By default, the
• I NAT 18, C IFAR 100, C IFAR 10:
Following the settings of Goyal et al.
, the linear probing experiments (on I NAT 18, C IFAR 100, C IFAR 10) adds a B ATCH N ORM layer before the linear classifier to reduce the hyper-parameter tuning difficulty.
The learning rate is initialized to 0.01 and multiplied by 0.1 every 8 epochs.
Then train these linear probing tasks for 28 epochs by SGD Nesterov optimizer with momentum=0.9, batch size 256.
Note that B ATCH N ORM + a linear classifier is still a linear classifier during inference.
We tune
L2 weight decay from {1e-4, 5e-4, 1e-3, 5e-3, 1e-2, 5e-2} for C IFAR 100 and C IFAR 10, {1e-6, 1e-5, 1e-4} for I NAT 18.
Fine-tuning: As to the fine-tuning experiments (on C IFAR 100, C IFAR 10, and I NAT 18), we tune the initial learning rate from {0.005, 0.01, 0.05}, training epochs from {50, 100}.
We further tune
L2 weight decay from {0, 1e-5, 1e-4, 5e-4} for C IFAR 100 and C IFAR 10, {1e-6, 1e-5, 1e-4} for
I NAT 18.
A cosine learning rate scheduler is used in fine-tuning experiments.
A 0.01 learning rate and 100 training epochs usually provide the best performance for these three datasets.
So we fix these two hyperparameters in the following supervised learning two-stage fine-tuning experiments and self-supervised learning experiments.
Two-stage fine-tuning:
For the two-stage fine-tuning experiments, we separately fine-tune each sub-network (i.e. RESNET 50) of the C AT n architecture by the same protocol as the normal fine-tuning above.
Then train a last-layer linear classifier on top of the concatenated fine-tuned representation.
The last-layer linear classifier training can be very efficient with a proper weights initialization strategy.
In this work, we initialize the last-layer classifier w (including the bias term) by concatenating the last-layer classifier of each fine-tuned sub-network w i , w ← [ w ⊤ 0 , . . .
, w ⊤ n ] ⊤ /n , Then we only need to train the last-layer classifier w for 1 epoch with a learning rate = 0.001.
B. E XPERIMENTS ON A DEEPER ARCHITECTURE : RESNET
Similar to table 2 in section 5, table 4 provides similar experiments on a deeper architecture
RESNET 152.
C AT n exceeds ERM on I MAGE N ET , C IFAR 10, C IFAR 100 , and I NAT 18 linear probing tasks. [Table] B.
[Table]
F INE TUNING EXPERIMENTS
For reference, table 5 provides numerical results for the fine-tuning experiments of Figure 3.
B. V ISION TRANSFORMER E XPERIMENT SETTINGS
For all vision transformer experiments, we keep the input image resolution at 384 × 384. For linear-probing and (2-stage) fine-tuning, we follows a similar protocol as the one in {C IFAR 10, C IFAR 100, I NAT 18}.
Specifically, we use a weight decay=5e-4 and a batch size=256 for linear probing, a weight decay=0 and a batch size=512 (following the same settings as Dosovitskiy et al. for (2-stage) fine-tuning. Following Dosovitskiy et al.
(2020), all input images are normalized Various representations are pre-trained on I MAGE N ET then fine-tuned or two-stage fine
[Table]
C S ELF - SUPERVISED TRANSFER LEARNING C. SWAV ON I MAGE N ET
SWAV is a contrastive self-supervised learning algorithm proposed by Caron et al.
We train RESNET 50 on I MAGE N ET 4 by the SWAV algorithm four times, which gives us four pretrained
RESNET 50 models.
As to the rest four SWAV pre-trained models in this work, we use the public available RESNET 50 5 , RESNET 50 W 2 6 , RESNET 50 W 4 7 , and RESNET 50 W 5 8 checkpoints.
Linear probing:
Following the settings in Goyal et al.
, the linear probing experiments (on I MAGE N ET , I NAT 18, C IFAR 100, C IFAR 10) adds a B ATCH N ORM layer before the last-layer linear classifier to reduce the hyper-parameter tuning difficulty.
The learning rate is initialized to 0.01 and multiplied by 0.1 every 8 epochs.
Then train these linear probing tasks for 28 epochs by SGD Nesterov optimizer with momentum=0.9.
We search L2 weight decay from { 5 e − 4 } , { 5 e − 4 , 1 e − 3 , 5 e − 3 , 1 e − 2 } , and { 1 e − 6 , 1 e − 5 , 1 e − 4 } for I MAGE N ET , C IFAR , and I NATtasks, respectively.
Fine-tuning:
• I MAGE N ET : Inspired by the semi-supervised I MAGE N ET fine-tuning settings in Caron et al.
(2020), we attach a randomly initialized last-layer classifier on top of the SSL learned representation.
Then fine-tune all parameters, using a SGD optimizer with momentum=0.
and L2 weight decay=0.
Low-layers representation and last-layer classifier use different initial learning rates of 0.01 and 0.2, respectively.
The learning rate is multiplied by 0.
at 12 th and 16 th epochs.
We train 20 epochs for networks: RESNET 50, RESNET 50 W 2, RESNET 50 W 4.
We further search training epochs from { 10 , 20 } for the wide network (due to overfitting), RESNET 50 W 5 and then select the best one with 10 training epochs.
• C IFAR 10, C IFAR 100, I NAT 18 :
Same as the fine-tuning settings in supervised transfer learning in Appendix B.1.
Two-stage fine-tuning:
• I MAGE N ET : Similar to the two-stage fine-tuning settings in supervised transfer learning, we initialize the last-layer classifier w by concatenation and then train 1 epoch with learning rate=0.001, L2 weight decay=0.
• C IFAR 10, C IFAR 100, I NAT 18 : For C IFAR 10, C IFAR 100, we use same two-stage finetuning settings as in supervised transfer learning in Appendix B.1.
For I NAT 18, we attach a B ATCH N ORM layer before the last-layer linear classifier to reduce the training difficulty.
Note that B ATCH N ORM + a linear classifier is still a linear classifier during inference.
Following the linear probing protocol, we train the B ATCH N ORM and linear layers by a SGD optimizer with momentum=0.9, initial learning rate=0.01, and a 0.2 learning rate decay at 12 th and 16 th epochs.
As to L2 weight decay, we use the same searching space as in the fine-tuning.
C.1. A DDITIONAL RESULTS
Beside the SWAV I MAGE N ET fine-tuning experiments of figure 6, Figure 8 provides further finetune / two-stage fine-tune various SWAV pretrained RESNET s on NATURALIST 18, C IFAR 100, and
C IFAR 10 tasks.
We give the “cat(ft)” curve on the I MAGE N ET task, but omit the curves on other tasks ( NATURALIST 18, C IFAR 100, and C IFAR 10) because they are computational costly.
[Figure]
C. SEER ON I NSTAGRAM 1B
SEER (Goyal et al., 2022) trains large REGNET {32 GF , 64 GF , 128 GF , 256 GF , 10B} architectures on the I NSTAGRAM 1B dataset with 1 billion Instagram images, using the SWAV contrastive selfsupervised learning algorithm.
Linear Probing:
Same as the linear probing settings in SWAV.
Fine-tuning: We use SEER checkpoints 9 fine-tuned on
I MAGE N ET with 384 × 384 resolutions.
It is fine-tuned on I MAGE N ET for 15 epochs using SGD momentum 0.9, weight decay 1e-4, learning rate 0.04 and batch size 256.
The learning rate is multiplied by 0.1 at 8 th and 12 th epochs.
Two-stage Fine-tuning: We keep L2 weight decay 1e-4 the same as fine-tuning.
Then follow the other settings as in SWAV. C.
A DDITIONAL EXPERIMENT : S IM S IAM ON C IFAR
S IM S IAM Chen and He (2020) is a non-contrastive self-supervised learning algorithm.
In this section, we pre-train the networks using S IM S IAM on C IFAR 10, when transfer the learned representation by linear probing to C IFAR 10, C IFAR 100, C IFAR 10 with 1% training examples, and C IFAR 100 with 10% training examples. SEER
SimSiam Pretraining Following Chen and He (2020) we pretrain RESNET 18, RESNET 18 W 2,
RESNET 18 W 4, 2 RESNET 18, and 4 RESNET 18 on C IFAR 10 ( 32 × 32 resolution) by Simsiam forepochs, using a SGD optimizer with momentum = 0 .
9 , initial learning rate = 0 .
06 , batch size = 512 ,
L2 weight decay = 5 e − 4 , and cosine learning rate scheduler.
The data augmentations include R ANDOM R ESIZED C ROP (crop scale in [0 . 2 , 1] ), R ANDOM H ORIZONTAL F LIP , R ANDOM G RAY S CALE ( p = 0 . 2 ), and a random applied C OLOR J ITTER ( 0 . 4 , 0 . 4 , 0 . 4 , 0 . 1 ) with probability 0 . 8 .
All images are normalized by mean = (0 . 4914 , 0 . 4822 , 0 . 4465) , std = (0 . 2023 , 0 . 1994 , 0 . 2010) before training.
D ISTILL
Since self-supervised learning tasks don’t contain target labels as supervised learning, we apply knowledge distillation on representation directly.
Specifically, we set v 1 , . . .
v n in Figure(left) as Identity matrices.
Then we distill [ ϕ 1 , . . .
, ϕ n ] into Φ by use a cosine loss:
[Equation]
Linear Probing:
Following again the settings of Goyal et al.
, the linear probing experiments (on C IFAR 100, C IFAR 10, C IFAR 100(1%) with 10% training data, and C IFAR 10(1%) with 1% training data) adds a BATCHNORM layer before the last-layer linear classifier to reduce the hyperparameter tuning difficulty.
We use batch size = 256 for C IFAR 100 and C IFAR 10, use batch size =for corresponding sampled (10%/1%)version.
Then we search initial learning rate from { 0 . 1 , 0 . 01 } ,
L2 weight decay from {1e-4, 5e-4, 1e-3, 5e-3}.
The learning rate is multiplied by 0.1 every 8 epochs during the total 28 training epochs.
As to the optimizer, all experiments use a SGD Nesterov optimizer with momentum=0.9.
Results Table 6 shows the linear probing accuracy of Simsiam learned representation on various datasets and architectures.
When linear probing on the same C IFAR 10 dataset as training, the C AT n method performs slightly better than width architectures (e.g. RESNET 18 W 2 and RESNET 18 W 4).
When comparing them on the C IFAR 100 dataset (OOD), however, C AT n exceeds width architectures.
[Table]
C. N UMERICAL RESULTS
For reference, Tables 7 and 8 provide the numerical results for the linear probing, fine-tuning, and two-stage fine-tuning plots of Figure 6.
[Table] [Table]
- SEER
REGNET 128 GF 637M 75.
- SEER
REGNET 256 GF 1270M 77.
[Table]
C AT418M 76.
[Table] - 84.5 C AT1060M 77. [Table] [Table]
[Table]
- 85.1 C AT2330M
[Table]
[Table]
[Table]
D META - LEARNING / FEW - SHOTS LEARNING D. D ATASETS
CUB (Wah et al., 2011) dataset contains 11 , 788 images of 200 birds classes, 100 classes ( 5 ,images) for training and 100 classes ( 5 , 794 images) for testing.
M INI I MAGE N ET (Vinyals et al., 2016) dataset contains 60 , 000 images of 100 classes withimages per class, 64 classes for training, 36 classes for testing.
D. B ASELINE AND B ASELINE ++ EXPERIMENT S ETTINGS
For B ASELINE and B ASELINE ++ experiments, following Chen et al.
(2019), we use R ANDOMS IZED C ROP , I MAGE J ITTER (0.4, 0.4, 0.4), and H ORIZONTAL F LIP augmentations, as well as a image normalization mean = (0 . 485 , 0 .
456 , 0 . 406) , std = (0 . 229 , 0 .
Then use an A DAM optimizer with learning rate = 0.001, batch size = 16, input image size = 224 × 224 .
Finally, train
RESNET 18 on CUB and M INI I MAGE N ET datasets for 200 and 400 epochs, respectively.
We further tune L2 weight decay from {0, 1e-5, 1e-4, 1e-3, 1e-2} and choose 1e-4 for CUB , 1e-5 for M INII MAGE N ET experiments.
Compared with the B ASELINE and B ASELINE ++ performance reported by Chen et al.
(2019) (table A5), this L2 weight decay tuning process provides ∼ 5% and ∼ 1% improvement on MINI I MAGENET 5way-1shot and 5way-5shot, respectively.
In this work, we use this stronger setting in baseline methods.
As to the few-shots learning evaluation, following Chen et al.
(2019), we scale images by a factor of 1.15, C ENTER C ROP , and normalization.
Then randomly sample 1 or 5 images from 5 random classes from the test set (5way-1shot and 5way-5shot).
Finally, train a linear classifier on top of the learned representation with a SGD optimizer, momentum = 0.9, dampening = 0.9, learning rate = 0.1, L weight decay =
1e-3, batch size = 4, and epochs = 100.
We take the average of 600 such evaluation processes as the test score.
The B ASELINE and B ASELINE ++ results in Figure 7 report the mean of five runs with different training and evaluating seeds.
Implementation details of the cosine classifier
Here we summarize the technical details of the cosine classifier implementation used in this work which follows Chen et al.
Denote the representation vector as z .
The cosine classifier calculates the i th element of logits by:
[Equation]
(4) where u i is a vector with the same dimension of z , g i is a scalar,
h i is i th element of logits h .
Then minimize the cross entropy loss between the target label y and softmax output s ( h ) by updating w and g : min w,g L ce ( y, s ( h )) .
D. C AT AND D ISTILL EXPERIMENT SETTINGS
For C AT , we concatenate n representation separately trained by either B ASELINE or B ASELINE ++ as the settings above.
For D ISTILL , we use the same multi-head architecture as figure 2 together with a cross-entropy loss function:
[Equation]
, where L ce indicates a cross-entropy loss, α is a trade-off parameter between cross-entropy loss and kl-divergence loss.
We set L2 weight decay = 0, τ = 10 , search α ∈ { 0 . 8 , 0 . 9 , 1 } , and keep the other hyper-parameters as Appendix D.2.
We find the impact of α is limited in both CUB ( ≤ 1% ) and
M INI I MAGE N ET ( ≤ 0 .
5% ) tasks.
D. S NAPSHOTS EXPERIMENT SETTINGS
The previous section shows the C AT n and D ISTILL n methods applied on n separately trained
RESNET 18 representations.
In this section, we apply D ISTILL n on n snapshots sampled from one training episode.
We train CUB and M INI I MAGE N ET respectively for 1000 and 1200 epochs by naive
SGD optimizer with a relevant large learning rate 0.8.
Then we sample 5 snapshots, { 200 th , 400 th , 600 th , 800 th , 1000 th } and { 400 th , 600 th , 800 th , 1000 th , 1200 th } , for CUB and
M INI I MAGE N ET , respectively.
The other hyper-parameters are the same as Appendix D.2.
D. M ORE EXPERIMENTAL RESULTS
Table 10 provides the exact number in Figure 7, as well as additional C AT n and D ISTILL n few-shots learning results with a linear classifier (The orange and gray bars in figure 7 report the few-shots learning performance with a cosine classifier). Table 9 provides more D ISTILL 5-s results with either a linear classifier or a cosine-based classifier.
E O UT - OF - DISTRIBUTION LEARNING
Following Zhang et al.
(2022), we use the C AMELYON 17 (Koh et al., 2021) task to showcase the
C AT and D ISTILL constricted representation in out-of-distribution learning scenario.
The first row of table 3 is copied from Zhang et al.
The rest results use a frozen pre-trained representation, either by concatenating n ERM pre-trained representations (C AT n ), distilling of n ERM pre-trained representations (D ISTILL n ), or RFC constructed representations (RFC2).
Then train a linear classifier on top of the representation by vREx or ERM algorithms.
For the vREx algorithm, we search the penalty weights from {0.5, 1, 5, 10, 50, 100}.
For D ISTILL n representations in the C AMELYON 17 task, we follow the Algorithm 2 in Zhang et al.
, but [Table] [Table] use a slightly different dataset balance trick in the loss function (Zhang et al.
Algorithmline 13-14).
We instead balance two kinds of examples: one share the same predictions on all ERM pre-trained models, one doesn’t.
We keep other settings to be the same as Zhang et al.
F R EPLICABILITY
Code will of course be provided on github to replicate these experiments.
As explained in the paper, many of our experiments are based the setups defined by other authors and rely on the code and the data files they provide.
We are therefore very conscious of the value of making well organized replication resources available to the community.