How well do contrastively trained models transfer?
Abstract
There are two prevailing methods for pre-training on large datasets to learn transferable representations: 1) supervised pre-training on large but weakly-labeled datasets; 2) contrastive training on image only and on image-text pairs.
While supervised pre-training learns good representations that can be transferred to a wide range of tasks, contrastively trained models such as CLIP have demonstrated unprecedented zero-shot transfer.
In this work we compare the transferability of the two aforementioned methods to multiple downstream tasks.
The pre-training distributions we consider include YFCC, Conceptual Captions, and ImageNet-21K while pre-training objectives range from supervised to SimCLR, CLIP, and SLIP.
We observe that different pre-training methods with the same training source transfer similarly given their ImageNet accuracy.
1. Introduction
The last few years of computer vision have witnessed scaling in both dataset and model size.
Supervised pre-training on large but weakly-labeled datasets such as Instagram images (Mahajan et al., 2018) and JFT (Sun et al., 2017; Zhai et al., 2021) learns good representations which can be transferred to a wide range of tasks.
Abnar et al.
(2021) study the effect of pre-training on JFT-300 (Sun et al., 2017) and
ImageNet-21K (Deng et al., 2009) on transferring to multiple downstream tasks, across different architectures, upstream dataset and model sizes, concluding that learned representations could be transferred well to downstream tasks, however, such transfer performance is saturated.
In contrast to supervised pre-training, another promising direction to scale the required data for large models is selfsupervised learning with contrastive loss.
SimCLR (Chen et al., 2020) and MoCo (He et al., 2020) are two examples which learn the representation in image modality, while CLIP (Radford et al., 2021), SLIP (Mu et al., 2021),
ALIGN (Jia et al., 2021), and BASIC (Pham et al., 2021) leverage data from both image and text modalities, demonstrating unprecedented robustness to the challenging distribution shifts.
However, such robustness improvements are at best in the zero-shot setting.
Ericsson et al.
(2021) show that semi-supervised contrastive pre-training on ImageNet transfers well to downstream tasks related to object recognition in natural images.
Cole et al.
(2021) extend the ImageNet transfer study and show that the learned representations by supervised methods can transfer better than semi-supervised methods on non-ImageNet domains, e.g. , fine-grained classification.
Radford et al.
and Jia et al.
(2021) try finetuning contrastive image and text models and show that the accuracy of their few-shot fine-tuned models on downstream tasks is lower than that of the original zero-shot model, and such performance drop is compensated with higher shots.
Wortsman et al.
(2021) also propose a finetuning strategy (WiSE-FT) for pre-trained
CLIP models on downstream tasks to reduce this performance drop.
Their proposed method linearly interpolates the weights of the pre-trained model and that of the finetuned model.
In this work, we investigate the transferability of learned representations by different pre-training objectives to ImageNet,
CIFAR100, DTD, and CALTECH-101.
We compare supervised pre-training on ImageNet-21K (Deng et al., 2009) with 14M images to contrastive image and contrastive image+text pre-training on YFCC-15m
(Radford et al., 2021) and Conceptual Captions 12M (Changpinyo et al., 2021).
We explore different impacting factors on transferability and make the following observations:
• In few-shot transfer, supervised pre-training on ImageNet-21K shows higher accuracy than contrastive pre-training on YFCC-15m. • Among contrastive models pretrained on YFCC-15m, SLIP shows the best performance when transferring learned representations from YFCC-15m to the downstream tasks, followed by SimCLR and CLIP.
• Given the same ImageNet accuracy in few-shot transfer, supervised pre-training on ImageNet-21K shows lower
[Figure]
accuracy, while SimCLR shows higher accuracy than other methods.
• Different pre-training methods with the same training source transfer similarly given their ImageNet accuracy.
2. Background
In this section we first review pre-training methods used in this work and then discuss the details of the transfer learning framework which we used.
Details of different datasets can be viewed in Appendix Section A.
2.1. Methods
SimCLR (Chen et al., 2020) applies random transformations to each image to get a pair of two augmented images.
These images are then passed through an encoder.
The resulting representations then get transformed and projected via non-linear layers to obtain the final representations for images and their augmented views.
The similarity between two augmented versions of an image is calculated using cosine similarity.
The idea is to pull together similar and push away dissimilar images.
SUP-IN21K pre-trains ViT-B/16 on ImageNet-21K with cross-entropy loss.
SUP CLS is the supervised classification using crossentropy loss and from scratch on YFCC-15m-cls dataset.
SimCLR+CLS is a SimCLR model pre-trained on YFCC15m, and fine-tuned using cross-entropy on YFCC-15m-cls.
CLIP (Radford et al., 2021) is directly trained on images and their corresponding unstructured text from the web ( million image and text pairs).
Given a batch of M (image, text) pairs, CLIP learns a multi-modal embedding space, by jointly training an image-encoder and a text-encoder, such that the cosine similarity of the valid M (image, text) pairs is maximized.
The resulting models achieve decent robustness even on a series of challenging distribution shifts such as
ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-Adversarial (Hendrycks et al., 2021), and ObjectNet (Barbu et al., 2019).
SLIP (Mu et al., 2021) combines self-supervision of SimCLR with CLIP for better visual representations.
The CLIP and self supervised objectives are computed on the relevant embedding and then accumulated into a single scalar loss.
2.2. Transfer Learning
In this work we investigate the transferability of the learned representations in a few-shot learning setup.
This is motivated by the previous findings that as the number of downstream samples increases, the effect of transfer learning shrinks (Kornblith et al., 2019; Mensink et al., 2021;
Zoph et al., 2020; Abnar et al., 2021).
Therefore we mostly focus on comparison between different settings where transferability differs most i.e. , few-shot learning.
All pre-trained models are ViT-B/16 (Dosovitskiy et al., 2020).
For SupervisedImageNet-21K we used the pre-trained model from Wightman (2019).
SLIP, CLIP, and SimCLR checkpoints are from
Mu et al.
SimCLR+CLS and SUP CLS checkpoints are from Fang et al.
We use the training procedure from BeiT (Bao et al., 2021) for end-to-end finetuning on downstream tasks.
This procedure takes advantage of significant regularization and data augmentation, as well as layer-wise learning rate decay.
We finetune the pre-trained models for 100 epochs.
[Figure]
3. Experiments and Results
Figure 1 shows the accuracy of multiple methods after few-shot fine-tuning on three downstream tasks 12 .
When evaluated in few-shot setting, supervised pre-training on
ImageNet-21K shows higher accuracy than contrastive pretraining on YFCC-15m. The difference is large for CIFAR100 and smaller for DTD and CALTECH.
As stated in Section 2.2, when we use all samples from the downstream task for finetuning, the gap between different models shrinks.
Supervised pre
-training on a subset of YFCC (YFCC-15m-cls) performs worst.
Among contrastive pretraining methods, SLIP shows the best performance when transferring learned representations from YFCC-15m to the downstream tasks, followed by SimCLR and CLIP.
Below we run multiple dedicated studies to investigate the effect of various impacting factor on transferability.
3.1. Does supervised finetuning help transferability?
To answer this question, we compare SimCLR pre-trained on YFCC-15m, with the same method but further finetuned on YFCC-15m-cls using cross-entropy loss, referred to as
SimCLR and SimCLR+CLS in the figures, respectively.
Both methods follow almost the same transfer accuracy across different shots and datasets implying that supervised finetuning on the upstream dataset has no effect on transferability.
3.2. Does language supervision help transferability?
We compare SimCLR pre-trained on YFCC-15m images with CLIP pre-trained on YFCC-15m with both images and text data.
Figure 1 shows that when evaluated few-shot,
SimCLR pre-training shows higher accuracy than CLIP.
This finding is aligned with (Fang et al., 2022), in which they show that language supervision does not contribute to a model’s robustness, but simplifies training on a diverse distribution of images by removing the need for consistent annotation with class labels.
3.3. What is the effect of pre-training distribution?
In order to answer this question, we compare CLIP pretrained on YFCC-15m dataset with the same model but pretrained on Conceptual Captions-12M. Figure 3(a) shows that different pre-training sources, keeping the pre-training objective fixed (CLIP), result in different few-shot transfer performance.
Few-shot transfer of CC-12m learned representations shows higher accuracies.
The fine-tuning performance gap for these two sources is at smallest for DTD and larger for CIFAR100 and CALTECH-101.
Higher performance on Conceptual Captions could be attributed to the noisy captions in this dataset (Hendricks et al., 2021;
Hendricks & Nematzadeh, 2021).
We leave the the language similarity (measured by perplexity) between pre-training distribution and downstream tasks for future works.
3.4. What is the role of pre-training dataset size?
We compare CLIP models pre-trained on two different sizes of Conceptual Captions dataset: 3M and 12M. Figure 3(b) shows that the larger pre-training dataset results in higher few-shot transfer accuracy.
For Conceptual Captions experiments, we used checkpoints from SLIP GitHub repository (Mu et al., 2021).
3.5. What is the role of pre-training objective in transfer learning?
Comparison between SLIP and CLIP pre-training objectives shows the effect of adding self-supervised contrastive learning (SimCLR) objective.
Figure 1 shows that SLIP performs better than CLIP in all shots and datasets.
However, the gap
[Figure]
between these two methods shrinks as the number of shots increases; in the full-finetuning setting, SLIP performs only slightly better than CLIP.
3.6. Distance to pre-training dataset
Table 1 compares CLIP and SLIP in linear and fullfinetuning settings, across different datasets.
The difference between Full and Linear finetuning can be used as proxy of the distance to the upstream dataset, here YFCC-15m. Intuitively, a small gap between full and linear finetuning would suggest that the upstream features can be transferred more easily.
It is interesting to see that the ranking of differences between full and linear finetuning is the same for CLIP and
SLIP.
i.e. , CALTECH < DTD <
ImageNet < CIFAR100.
It is worth noting that the image resolution of downstream tasks differs which might be a confounding factor in such a ranking.
[Table]
3.7. Does better transfer to ImageNet correspond to better transfer to other downstream tasks?
Figure 2 shows the correlation between ImageNet few-shot evaluation and different downstream tasks.
We can observe that models with the same pre-training source which achieve higher ImageNet accuracy, also perform better on the downstream task.
This shows that ImageNet is a good proxy for transferability across different downstream tasks.
In contrast to Figure 1, given the same ImageNet accuracy when evaluating few-shot, supervised pre-training on ImageNet21K shows lower accuracy, while SimCLR shows higher accuracy than other methods.
Different pre-training methods with the same training source transfer similarly given their ImageNet accuracy.
3.8. Can we predict the full-finetune accuracy with lower computation
[Table]
1-shot accuracy as a proxy .
Table 2 shows the correlation between full-finetuning and 1 shot accuracy as a low computation proxy.
As the Kendall’s τ coefficients and their p-values indicate, 1 shot accuracy can be used as a proxy to choose the best pre-trained model to finetune on a downstream task without actually having to do so.
For ImageNet, we used the 3 full finetuning accuracies reported in Mu et al.
(2021) and another from Dosovitskiy et al.
For other datasets, we employ all six methods described in Section 2.1.
Exact values for full-finetuning accuracies can be seen in Table 4.
Appendix Section C.1 also shows the correlation between 1-NN and full-finetuning accuracy.
4. Discussion
In this work we compare supervised and contrastive selfsupervised pre-training on transferability of learned representations to multiple downstream tasks.
We consider
ImageNet-21K for supervised pre-training, YFCC-15m
and
Conceptual Captions-12m for contrastive image only and image+text pre-training.
Pre-training objectives include supervised, SimCLR, CLIP, and SLIP.
Our observations show that different pre-training methods with the same training source transfer similarly given their ImageNet accuracy.
We leave further investigations on more pre-training sources and downstream tasks for future works.
Appendix A. Datasets
YFCC .
We use the YFCC-15m
(Radford et al., 2021) dataset, a subset of YFCC-100M (Thomee et al., 2016) filtered to only images with English titles or descriptions.
The dataset contains 14,829,396 images with natural language captions associated with each image.
We also followed Fang et al.
(2022) to convert YFCC to a classification dataset ( YFCC-15m-cls ) with class labels.
They assign ImageNet labels to each image using a simple strategy: if the title or description contains the name of an ImageNet synset or synonym Miller (1995), they assign the corresponding synset label to the image.
If an image contains no or multiple ImageNet synsets, they discard that image.
This results in 1,694,125 images (11.4% of the full dataset) covering 953 ILSVRC classes.
ImageNet-21K .
To have same number of images as YFCC-15m, we chose ImageNet-21K (Deng et al., 2009) for supervised pretraining, which consists of 14,197,122 images, each tagged in a single-label fashion by one of 21,841 possible classes.
Conceptual-Captions .
To investigate the effect of pre-training dataset distribution and dataset size we also use pretrained checkpoints on Conceptual Captions 3M (CC-3M) and Conceptual Captions 12M (CC-12M) (Sharma et al., 2018;
Changpinyo et al., 2021).
CC-3M represtns a wide variety of images and caption styles and is built via extraction and filtering of images and their associated texts from billions of web pages.
To build CC-12M dataset, the authors take a step further and relax the data collection pipeline used to build the previous CC-3M version and obtain 12 million image and text pairs.
Downstream tasks .
We measure the transferablity of YFCC learned representations using different methods on CIFAR100 (Krizhevsky et al., 2009), DTD (Cimpoi et al., 2014), and Caltech-101 (Fei
[Table]
B. Training Hyperparameters B.1. Pretraining on YFCC
We used CLIP, SLIP and SimCLR checkpoints from SLIP GitHub repository (Mu et al., 2021).
For SUP-CLS and
SimCLR+CLS we use the checkpoints trained in Fang et al.
B.2. Transfer to downstream tasks
We follow the procedure from BeiT (Bao et al., 2021) by using fine-tuning scripts from SLIP GitHub repository (Mu et al., 2021) for our full fine-tuning and few-shot training experiments.
Mu et al.
(2021) use DeiT training procedure for smaller datasets but we followed the BeiT training procedure for all of our experiments, including few-shot learning experiments.
We use a batch size of 800 for full tuning and 100 for few-shot training experiments (except for 1 shot experiments on DTD dataset where we use a batch size of 32 due to lower number of classes in this dataset).
We use the learning rate of 4e-3 when using a batch size of 800 and adjust it proportional to batch size whenever it is changed; according to Goyal et al.
Moreover, we employ layerwise learning rate decay to exponentially decrease the learning rate across layers in the model.
All the experiments in this study are trained for 100 epochs using AdamW optimizer with a weight decay of 0.05 and by employing different data augmentation methods and regularization techniques.
We also set drop path to 0.1 and layer decay to 0.65 for all the experiments.
[Table]
C. Full fine-tuning performance on downstream tasks C.1. KNN as a proxy for full-finetuning
While full-finetuning is computationally expensive, calculating KNN ( K = 1 ) is cheap and if we can show that the 1-NN and full-finetuning accuracies are correlated, 1-NN can be used as a low computation proxy to select the best pre-training method without actual finetuning.
Here we use K-NN with K = 1 as follows:
For each pre-training method, we first obtain the final embeddings of all training and test images.
Then, for each test sample, the predicted label is the label of the closest sample from the training set, i.e. , the sample with the highest cosine similarity to its embedding vector.
We used Kendall’s τ coefficient (Kendall, 1938) to calculate the ordinal association between ground truth full-finetuning results and the 1-NN results for each dataset.
Table 5 shows the correlation between full-finetuning and 1-NN accuracy.
The Kendall’s τ coefficient for correlation between ranking of different methods on CIFAR100, DTD, and CALTECH101 datasets are 0.60, 1.00, and 0.80, respectively.
This indicates a good correlation between 1-NN and fine tuning, showing that 1-NN can be used to select the best pre-training method without actual finetuning.
[Table]