Is Self-Supervised Contrastive Learning More Robust Than Supervised Learning?
Abstract
Self-supervised contrastive learning is a powerful tool to learn visual representation without labels.
Prior work has primarily focused on the recognition accuracy of contrastive pre-training algorithms, but has overlooked other behavioral aspects.
In addition to accuracy, distributional robustness plays a critical role in the reliability of machine learning models.
We design and conduct a series of robustness tests to quantify the behavioral differences between contrastive learning and supervised learning.
These tests leverage data corruptions at multiple levels, ranging from pixellevel gamma distortion to patch-level shuffling and to dataset-level distribution shift.
Our tests unveil intriguing robustness behaviors of contrastive and supervised learning.
On the one hand, under downstream corruptions, we generally observe that contrastive learning is surprisingly more robust than supervised learning.
On the other hand, under pre-training corruptions, we find contrastive learning vulnerable to patch shuffling and pixel intensity change, yet less sensitive to dataset-level distribution change.
We attempt to explain these results through the role of data augmentation and feature space properties. Our insight has implications in improving the downstream robustness of supervised learning.
1. Introduction
In recent years, self-supervised contrastive learning (CL) has demonstrated tremendous potential in learning generalizable representations from unlabeled datasets (Chen et al., 2020a;
First Workshop of Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022 , Baltimore, Maryland, USA, PMLR 162,
[Figure]
He et al., 2020; Grill et al., 2020; Chen & He, 2021; Caron et al., 2020; Zhong et al., 2021b). Current state-of-the-art CL algorithms can learn representations from ImageNet (Deng et al., 2009) that match or even exceed the accuracy of their supervised learning (SL) counterparts in terms of ImageNet linear evaluation and downstream task performance (Chen et al., 2020a;
He et al., 2020; Grill et al., 2020; Chen & He, 2021; Caron et al., 2020).
However, beyond accuracy, relatively less attention is paid on analyzing other behavioral differences between contrastive learning and supervised learning.
Robustness is an important aspect to evaluate machine learning algorithms
For example, robustness to long-tail or noisy training data allows the learning algorithm to work well in a wide variety of imperfect real-world scenarios (Wang et al., 2017).
Robustness of the model output across training iterations enables anytime early-stop (Hu et al., 2019) and smoother continual learning (Shen et al., 2020).
Robustness to input corruptions at test-time plays an important role in reliable deployment of trained models in safety critical applications, as signified by the existence of adversarial examples (Goodfellow et al., 2015; Salman et al., 2020) and the negative impact of domain shift (Zhao et al., 2019).
In this paper, we investigate whether CL and SL behave robustly to data distribution changes.
In particular, how does the change in data affect the behavior of algorithms? Do SL and CL behave similarly? To this end, we design a wide-spectrum of corruptions as shown in Figure 1 to alter data distribution and conduct comprehensive experiments, with different backbones, CL algorithms and datasets.
The corruptions are carefully selected to be multi-level, corrupt different structural information (not necessarily humanrecognizable), and are rooted in prior literature.
Our main results consist of two sets of experiments: The first investigates the downstream robustness of pre-trained models towards corruptions of downstream data.
The second studies the robustness under pre-training data corruptions. We deliver a set of intriguing new discoveries .
We generally observe that CL is consistently more robust than SL to downstream corruptions.
On the other hand, contrastive learning on corrupted pre-training data leads to diverging observations depending on the corruption type: CL is more robust to dataset-level corruption than SL, but less so to pixel- and patch-level corruptions.
To explain why CL models are more robust to downstream corruptions, our analysis of the learning dynamics through feature space metrics reveals that CL yields larger overall and steadily-increasing per-class feature uniformity (Wang & Isola, 2020) and higher stability than SL.
An immediate consequence is an improvement to supervised pre-training’s downstream robustness by adding a uniformity regularization term to explicitly promote intra-class variance.
As for CL’s vulnerability to pre-training data corruption types such as patch shuffling, we hypothesize that the interference with random crop augmentation is the main culprit, as we find switching the order of data corruption and standard augmentations recovers a substantial amount of robustness.
We summarize our contributions as follows.
(1) We design extensive robustness tests to study the behavioral differences of CL and SL systematically.
(2) We discover diverging robustness behaviors between CL and SL, and even among different CL algorithms.
(3) We offer initial analyses and explanations for such observations, and show a simple way to improve the downstream robustness of supervised learning by encouraging uniformity.
We claim our paper as an empirical study.
We hope our findings can serve as an initial step to fully understand CL’s behaviors beyond accuracy and inspire more future studies to explore such aspects through larger-scale experiments and theoretical analysis.
2. Method 2.1. Robustness Tests
Robustness Test I: Downstream data corruption.
In this test, the pre-training algorithm is run on the clean version of the pre-training dataset.
For a given downstream dataset, we evaluate the pre-trained model’s accuracy on its original version and various corrupted versions.
This assesses the robustness of the algorithm by looking at the pre-trained model’s robustness behaviors.
Robustness Test II: Pre-training data corruption.
This assesses the algorithm’s robustness to pre-training data corruptions.
We run the pre-training algorithm on the corrupted version of the dataset, and then evaluate the final model’s accuracy on either the corrupted test set or the original test set.
The test set can be in-domain or out-domain to the train set.
Since data corruption destroys certain information by design, the model pre-trained on corrupted data is expected to yield degraded performance than the model pre-trained on the uncorrupted dataset.
Robustness Metric.
In both cases, the robustness is measured by the degradation in accuracy caused by certain data corruption.
An algorithm is more robust if the degradation is smaller.
Denote D original as the original dataset and D corrupted as the corrupted dataset.
We use the same notation for both pre-training and downstream corruptions.
For an algorithm Alg ∈ { CL , SL } , we define
[Equation]
We use two methods to obtain the test accuracy in the above equation.
The first is linear evaluation , where we train a linear classifier on top of the representation learned by the pre-training algorithm on the train split, and then evaluate the classifier on the test split.
The second is KNN evaluation .
We use the weighted KNN classifier from Wu et al. (2018), where the prediction is the exponential-distance weighted average of the K nearest neighbors in the train split of any test data point, measured by the normalized feature vectors.
The KNN evaluation effectively leverages an non-parametric classifier, therefore no classifier training is required.
Depending on the context, we use either linear or
KNN evaluation in our experiments.
The essential question we would like to ask is whether ∆( CL ) is consistently larger or smaller than ∆( SL ) across different data corruptions.
2.2. Types of Data Corruption
There is a natural hierarchy of data corruptions ranging conceptually from micro-level to macro-level.
We will describe the several variations illustrated in Figure 1.
Note that our data corruption is different from data augmentation.
In data augmentation, the corruption is applied randomly on a perimage basis.
In our case, a fixed random transformation ( e.g. , the gamma in gamma distortion or the permutation order in shuffling) is decided first then applied consistently across all the images.
We effectively transform the entire train or test dataset with the corruption method.
Pixel-Level Corruption.
The pixel intensity distribution is altered, but neither the spatial layout of each image nor the overall data distribution is changed.
The popular color jittering augmentation randomly modifies the brightness, contrast, saturation, and hue of training images.
Here, we deliberately pick gamma distortion, as it is not already part of the conventional data augmentation pipeline
• Gamma distortion: Gamma distortion remaps each R,G,B pixel intensity ( ∈ [0 , 255] ) according to x → ⌊ 255 × ( x/ 255) γ ⌋ , where γ > 0 is a tunable parameter.
γ = 1 recovers the original intensity.
Larger or smaller γ shifts the intensities darker or brighter, respectively.
Due to quantization error, there will be part of the intensity information lost during the process.
Patch-Level Corruption.
Pixel-level corruption does not change the spatial layout of pixels. The next type of corruption in the hierarchy is at patch level.
Inspired by prior work (Zhang et al., 2017), we employ global and local shuffling.
Note that pixel shuffling is not commonly used in the standard augmentation pipeline of visual recognition training (He et al., 2016).
We are curious what kind of behaviors CL and SL will exhibit under the patch-level shuffling data corruption.
• Global shuffling: Global shuffling breaks down the image into patches and shuffles the patches according to a fixed random order.
Specifically, suppose the image size is s × s and the size of each patch is p × p , and then the image is divided into s/p × s/p patches.
Global shuffling destroys the global spatial structure of an image but preserves the local structure.
The image becomes less structured with a smaller patch size.
• Local shuffling: Inversely to global shuffling, local shuffling randomly permutes the pixels inside each local p × p patch by a fixed random order, but keeps the global ordering of patches.
It damages the local image structure while preserving the overall global structure.
The image becomes less structured with a larger patch size.
Dataset-Level Corruption
Finally, we consider corruptions happening at the whole dataset distribution level.
The previous two types of corruptions change the images, but not the distribution of the images.
• Synthesized data: Synthesized data is popularizing ( e.g. , DALL · E 2 (Ramesh et al., 2022)) and studied to replace real data (Jahanian et al., 2021). We utilize generative methods such as a conditional GAN (Karras et al., 2020) to generate a synthesized dataset D GAN and replace D original .
We can then measure and compare ∆( Alg ) between these two datasets of different distributions.
Oftentimes, the generated distribution is not perfectly aligned with the real distribution, therefore training with the generative data source may lead to degradation in accuracy of clean data or downstream performance.
• Class imbalance: Real-world data often follows a longtail distribution of semantic classes, where head classes have many examples while tail classes have few examples (Kang et al., 2020; Samuel & Chechik, 2021).
However, benchmark datasets such as CIFAR and ImageNet are curated and class-balanced.
We consider the widelyused variant of ImageNet, ImageNet-LT (long-tail) (Liu et al., 2019), with maximally 1280 images and minimally 5 images per class.
For comparison, we construct ImageNet-UF (uniform) which is a class-balanced subset of ImageNet that contains the same number of images as ImageNet-LT (115K).
Then we test whether moving from pre-training on ImageNet-UF to ImageNet-LT leads to different behaviors between CL and SL.
2.3. Experiment Setup
Datasets.
We pre-train on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) variants to evaluate the robustness differences between CL and SL methods subject to pre-training data corruptions.
We use CIFAR10/100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and two fine-grained classification datasets, Cars (Krause et al., 2013) and Aircrafts (Maji et al., 2013), to analyze the performance of the pre-trained models on the corrupted downstream tasks.
For comparable results, we fix the data augmentation for all settings that involve training
Models and Algorithms.
We benchmark a variety of self-supervised contrastive algorithms.
These methods are carefully-sampled to be representative.
They include contrastive learning with negatives (SimCLR-v2 (Chen et al., 2020a;b), MoCo-v2 (He et al., 2020; Chen et al., 2020d)), without negatives (SimSiam (Chen & He, 2021), and the momentum counterpart, BYOL (Grill et al., 2020)), with redundancy reduction (BarlowTwins (Zbontar et al., 2021)), and contrasting clustering assignments (DeepCluster-v2 (Caron et al., 2020), SwAV (Caron et al., 2020)).
We test both
CNN (standard ResNet-18/50 (He et al., 2016)) and Vision
Transformer (ViT) (Dosovitskiy et al., 2021) backbones.
For transformer, we leverage pre-trained models on ImageNet (Deng et al., 2009) from ViT (Dosovitskiy et al., 2021),
DeiT (Touvron et al., 2021), DINO (Caron et al., 2021),
MoCo-v3 (Chen et al., 2021b), and MAE (He et al., 2022).
3. Results 3.1. CL is more robust to downstream data corruption
We conduct downstream robustness tests on various datasets with frozen pre-trained ResNet-
(He et al., 2016) in Table 1 and ViT (Dosovitskiy et al., 2021).
The fully fine-tuned
ResNet-50 results and the ViT results are deferred to the appendix as the observations are similar.
The model checkpoints are obtained from VISSL (Goyal et al., 2021b) and [Table] [Table] [Table]
the official code bases.
They are pre-trained on the clean version of ImageNet.
We employ pixel-level and patch-level corruptions in these tests.
The CL methods have demonstrated higher robustness (lower average ∆ ) than the SL method.
Interestingly, not all CL methods are equally robust; even within the same method, models trained with different hyper-parameters (such as epochs) exhibit different levels of robustness ( e.g. , comparing SwAV-a and SwAV-b).
With ResNet-50, we notice that SimSiam, SwAV and BarlowTwins behave slightly more robust than others.
CLs and SL expose different degrees of robustness to different types of pre-training data corruption
Pixel-Level and Patch-Level Corruption.
Tabledemonstrates the impacts of gamma distortion and patch global/local shuffling during pre-training.
The drop of accuracy of SL due to gamma distortion is 2 .
4% , which is smaller than all the tested CL methods.
In terms of robustness to pre-training patch shuffling corruption, all CL methods behave similarly and less robust than SL, except for the L8x8 case.
The same observation carries over to the larger-scale
ImageNet-100 experiment result in Table 3, where we compare MoCo-v2 (since it appears to be the most robust on
CIFAR-10) and Sup in a total of 8 corruption settings.
MoCo-v2 on average yields a 28 . 0% degradation, while
Sup only degraded 6 .
Dataset-Level Corruption.
In Table 4, MoCo has shown more robustness on average to dataset-level corruption [Table] Linear evaluation.
ImageNet-LT is a longtail subset of ImageNet (Liu et al., 2019), ImageNet-UF is a uniform subset.
MoCo shows less sensitivity to class imbalance than SL. [Table] caused by an imperfect GAN. We adopt a class-conditional
StyleGAN2-ADA (Karras et al., 2020) generator trained on
CIFAR-10 to synthesize another copy of CIFAR-10.
When training with synthesized data and testing on the original
CIFAR-10, MoCo-v2 only has 2 .
58% accuracy drop, outperforming SL, which drops for 8 . 44% .
Evaluating on a GANsynthesized test set yields similar observation – MoCo-v shows almost no drop while SL drops 6% .
We also conduct downstream experiments for this corruption: testing the classification accuracy of the pre-trained representation on CIFAR-100, and make a similar observation.
Table 5 shows the impact of the other type of dataset-level corruption, the class imbalance.
Although there is a gap between the baseline top-1 accuracy of MoCo-v2 and SL, we observe that, the decline of MoCo resulting from pretraining on the long-tail rather than the uniform version is much smaller than SL.
In fact, the MoCo performance appears to be insensitive to class balance or imbalance (the top-1 ∆ is only 0 .
This is in contrary to SL, which shows a larger drop.
The difference is more salient by looking at the low-shot ( < 20 images per class), medium-shot and many-shot ( > 100 images per class) accuracy separately.
SL on ImageNet-LT sacrifices the low-shot accuracy for higher many-shot accuracy, whereas MoCo shows insignificant difference among the shots.
Our observation is consistent with a contemporary work (Liu et al., 2022).
4. Analysis
CL and SL behave differently during training.
The robustness discrepancy between CL ( e.g. , MoCo) and SL is
200 Epoch
P e r - C l a ss A cc u r a c y =2.
MoCo-v   40 Epoch    =2.
Supervised not only reflected in the final trained models, but is in fact also attributed in the training process.
To analyze how the feature space evolves during training, we measure the following three metrics.
Feature Semantic Fluctuation.
We can monitor the classification ability of the feature extractor by the accuracy of a KNN probe.
We define feature semantic fluctuation of class i as the total variation of per-class accuracy of class
i (as a function of epoch t ) averaged over all epochs: T V i =
[Equation] (cid:80) T [Equation] t =
| Acc ( i ) t + [Equation] − Acc ( i ) t | .
We further define the mean feature semantic fluctuation as the mean of T V i over all classes.
Larger semantic fluctuation indicates less stable feature space.
Feature Uniformity.
We can measure the uniformity of all the features or class-wise features as the log-mean of Gaussian potentials of the normalized features: U ( f t , D ) =
− log E x 0 ,x 1 ∼D (cid:104) e −
∥ f t ( x 0 ) − f t ( x 1 ) ∥ 22 (cid:105) .
Here f t is the network at epoch t , D is the dataset, and x 0 and x 1 are images sampled from the dataset.
The use of this measure to study contrastive learning is exemplified in Wang & Isola (2020).
Intuitively, a greater U means more uniformly distributed features on the unit sphere, while a smaller value means more concentrated features.
Feature Distance.
We also measure the average feature squared ℓ 2 distance between two classes.
A larger distance could mean more linear separability.
Denoting D i and D j as feature matrices of two classes, the feature distance is calculated as: d ( f t , D i , D j ) =
E x 0 ∼D i ,x 1 ∼D j (cid:2)
∥
f t ( x 0 ) − f t ( x 1 ) ∥ 22 (cid:3) .
When D i =
D j , it actually measures the intra-class variance of class i .
We train ResNet-18 (He et al., 2016) on the original {+CIFAR
 (Krizhevsky+} et al., 2009) train split and measure the above metrics on the test split.
Figure 3 shows the dynamics of feature uniformity and distances of MoCo-v2 (He et al., 2020; Chen et al., 2020d), supervised contrastive (SupCon) (Khosla et al., 2020), and supervised learning.
We are interested in SupCon, because it bridges CL and SL by leveraging a similar contrastive loss.
As illustrated, the overall feature uniformity of MoCo-v [Figure] (Chen et al., 2020d) is greater than 2.5 and approaching 3, while the overall feature uniformity of SupCon and supervised methods range from 1.25 to 2.2.
This means that features from contrastive learning methods are more uniformly distributed on the unit sphere.
By looking at the class-wise feature uniformity and distance, we notice that the supervised model tends to compress (and maybe overcompress) the features of each class.
Figure 2 shows that the accuracy of a KNN probe during supervised learning also fluctuates more dramatically.
We can interpret it as that the classes are competing with each other and SL cannot improve the performance on all classes at the same time like CL methods.
Further analysis on augmentation dependency and application of uniformity regularization to improve SL’s performance are included in the appendix. 5. Conclusion This paper systematically studies the robustness of contrastive learning and supervised learning through a diverse set of multi-level data corruption robustness tests. We discover diverging robustness behaviors of contrastive learning to different corruption settings.
We hope our findings can draw more attention towards understanding the behavior differences of pre-training algorithms beyond accuracy. Acknowledgement: This work was supported in part by NSF Grant 2106825, the Jump ARCHES endowment through the Health Care Engineering Systems Center, the New Frontiers Initiative, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign through the NCSA Fellows program, and the IBM-Illinois Discovery Accelerator Institute.
The different degree of dependency on data augmentation of CL and SL may explain why CL algorithms are less robust to pixel-level and patch-level pre-training corruptions. Contrastive learning relies heavily on well-defined data augmentation, while supervised learning can train without data augmentation as Row 1 of Table A.1. The central assumption in using data augmentation is that the augmented image falls within or close to the natural image statistics, e
[Table]
g. , a random cropped image is still plausible. However, our data corruption such as patch shuffling destroys the structure of an image and thus renders the random resize-crop augmentation inappropriate, as the cropped image no longer has the same global structure. To support our claim
[Table]
we switch the order of the patch shuffling corruption and the standard augmentation in the MoCo experiments in Table A.1 (KNN evaluation)
[Table]
since shuffling after crop gives consistent image structure. We find that this switching reverts much of the accuracy drop, making MoCo comparably robust to Sup.
[Table]
A.2. Uniformity regularization improves downstream robustness of supervised pre-training
The analysis above shows that MoCo appears to yield a more uniformly distributed feature space and tends not to compress the intra-class variance of semantic classes.
Could the larger uniformity be one reason behind the higher downstream robustness of MoCo?
We introduce a small uniformity-promoting regularization term in addition to the cross-entropy loss in SL.
This regularization is computed from the mini-batch itself without using the Siamese architecture or memory bank in
MoCo.
In Table A. [Table] there is no surprise that adding (or subtracting) the uniformity regularization produces a more (or less) uniform test feature space.
We also notice a correlation between the KNN evaluation accuracy and the test set uniformity under 3 corruption conditions.
This experiment suggests that we could improve SL by leveraging loss functions from CL and potentially get the best of both worlds.
[Table]
B. Related Work
Supervised Learning (SL).
Supervised deep learning, large labeled dataset, and computation have been a success recipe for cracking many visual recognition problems (Russakovsky et al., 2015; He et al., 2016).
Typically, one first collects a large-scale dataset for the target vision problem and crowdsources labels for the specific task from human annotators.
A machine learning model, e.g. , a neural net, is then trained by minimizing a loss function defined on the prediction-label pairs.
Self-Supervised Learning (SSL) and Contrastive Learning (CL).
Remarkable progress has been made in self-supervised representation learning from unlabeled datasets (Chen et al., 2020a;
He et al., 2020; Grill et al., 2020; Chen & He, 2021;
Caron et al., 2020).
This paper focuses on a particular kind of SSL algorithm, contrastive learning, that learns augmentation invariance with a Siamese network.
To prevent trivial solution, contrastive learning pushes negative examples apart (MoCo (He et al., 2020;
Chen et al., 2020d; 2021b), SimCLR (Chen et al., 2020a;b)), makes use of stop-gradient operation or asymmetric predictor without using negatives (SimSiam (Chen & He, 2021), BYOL (Grill et al., 2020), DINO (Caron et al., 2021)), or leverages redundancy reduction (BarlowTwins (Zbontar et al., 2021)) and clustering (DeepCluster-v2 and SwAV (Caron et al., 2020)).
In addition to augmentation invariance, generative pre-training (Ramesh et al., 2021; Bao et al., 2022; He et al., 2022) and visual-language pre-training (Radford et al., 2021) are promising ways to learn transferable representations.
There is a growing body of literature on understanding self-supervised learning.
Wang & Liu (2021) decomposes the contrastive objective into alignment (between augmentations) and uniformity (across entire feature space) terms.
Uniformity can be thought of as an estimate of the feature entropy, which we leverage as a metric to study the feature space dynamics during training.
Wang & Isola (2020) makes connection between uniformity and the temperature parameter in a contrastive loss, and finds that a good temperature can balance uniformity and tolerance of semantically similar examples.
Zhao et al. (2021) discovers that SSL transferring better than SL can be due to better low- and mid-level features, and the intra-class invariance objective in SL weakens transferability by causing more pre-training and downstream task misalignment.
Ericsson et al. (2021) studies the downstream task accuracy of a variety of pre-trained models and finds that SSL outperforms SL on many tasks.
Cole et al. (2022) investigates the impact of pre-training data size, domain quality, and task granularity on downstream performance.
Chen et al. (2021a) identifies three intriguing properties of contrastive learning: a generalized version of the loss, learning with the presence of multiple objects, and feature suppression induced by competing augmentations.
Our work falls into the same line of research that attempts to understand SSL better.
However, we investigate from the angle of robustness behavior comparison between SSL/CL and SL.
Robustness and Data Corruption.
The success of learning algorithms is often measured by some form of task accuracy, such as the top-1 accuracy for image classification (Deng et al., 2009; Krizhevsky et al., 2009; Coates et al., 2011; Wah et al., 2011), or the mean average precision for object detection (He et al., 2020; Zhong et al., 2021a; 2020).
Beyond accuracy, robustness is another important measure (Hendrycks & Dietterich, 2019), and there are benchmarks and metrics proposed for SL (Nado et al., 2021).
Robustness is more and more studied in SSL settings (Chuang et al., 2022; Goyal et al., 2022).
Chuang et al. (2022) tries to improve CL’s robustness to noisy positive views.
Recent large-scale study reveals that vision models are more robust and fair when pre-trained on uncurated images without supervision (Goyal et al., 2022).
We use “robustness” to refer to the ability of learning algorithms to cope with systematic train or test data corruptions.
Under the supervised setting, deep models are shown to train successfully (albeit not to generalize) under pixel shuffling corruption and random labels, even though they are not human-recognizable anymore (Zhang et al., 2017).
Adversarial robustness (Szegedy et al., 2014; Goodfellow et al., 2015; Madry et al., 2018; Shafahi et al., 2019; Chen et al., 2020c) is a related but different concept, which refers to the model’s ability to defend against adversarial attacks.
An adversarial attack (Szegedy et al., 2014; Goodfellow et al., 2015) is a perceptually indistinguishable perturbation to a single image that fools the model.
Adversarial training (Madry et al., 2018; Shafahi et al., 2019) is a technique to achieve adversarial robustness.
Self-supervised perturbation is explored in adversarial attack and training (Naseer et al., 2020;
Kim et al., 2020).
Hendrycks et al. (2019) shows that SSL models possess better adversarial robustness.
Fan et al. (2021) improves the adversarial robustness transferability of CL.
Our definition of robustness differs from adversarial robustness – we use robustness to analyze the tolerance of learning methods to systematic data corruptions (rather than per-image imperceptible perturbation).
There are many types of data corruptions in prior work.
The most common data corruptions, such as random resizing and cropping, flipping, and color jittering, appear as data augmentation in SL and SSL (He et al., 2016; 2020; Chen et al., 2020a).
The learned representation is encouraged to be invariant to such corruptions.
Hendrycks & Dietterich (2019) proposes a set of corruptions complementary to ours.
Block shuffling (our image global shuffling) has been used to study what is transferred in transfer learning (Neyshabur et al., 2020) and as negative views with diminished semantics in contrastive learning (Ge et al., 2021).
Cole et al. (2022) tampers data quality in SimCLR and SL training by salt-and-pepper noise, JPEG, resizing, and downsampling, and tests on clean data.
We use a different set of data corruptions and test on the corrupted data as well.
A recent work (Jahanian et al., 2021) also studies generative models as an alternative data source for contrastive learning.
They focus on comparison with real data, while we emphasize the behavior difference of SSL and SL in response to the generative data source.
Feature backward-compatibility (Shen et al., 2020) is related to our stability analysis of feature dynamics.
Recently, Goyal et al. (2021a) studies the effectiveness of SSL on uncurated class-imbalanced data.
Liu et al. (2022) also notices that SSL tends to be more robust to class imbalance than SL. We bring extra insights over them.
We consider both pre-training and downstream robustness and compare CL and SL behaviors , while Goyal et al.
(2021a) only focuses on downstream and compares dataset scale.
Our investigation suggests that pre-train behavior can be opposite to downstream. Liu et al. (2022) only studies class imbalance, but we consider broader corruptions. C. Additional Implementation Details
Table C.3 below lists the experiment configurations for each pre-training robustness table of the main paper.
We train our own ResNets (He et al., 2016) on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet variants (Deng et al., 2009).
ImageNet- LT/UF are the long-tail and Uniform-subsampled versions.
ImageNet-100 is a 100 class subset of full ImageNet-1K. We mainly list Sup and MoCo-v2 (Chen et al., 2020d) hyper-parameters here.
The other CL methods follow their recommended hyper-parameter values in the Solo-Learn package (da Costa et al., 2022).
[Table]
[Table]
D. Additional Results D.1. Pre-training robustness test with Transformer backbone
In the main paper, we compare pre-training robustness with a CNN backbone
[Table]
and show Vision Transformer (ViT) downstream robustness test results.
Here, we supplement ViT pre-training robustness test results.
Specifically, we leverage
MoCo-v3 (Chen et al., 2021b), the ViT version of MoCo, and Supervised ViT.
The results are in Table D.1.
We find that the MoCo-v3 degradation is larger with patch shuffling, but smaller with gamma distortion.
Interestingly, the impact of patch shuffling is much smaller than a CNN (despite the Orig performance gap between ViT and CNN).
We suspect this is due to the unique patching and attention network structure of ViT.
Essentially, if we do not take into consideration the data augmentation, with the right patch size, the shuffling within a small patch does not affect the learning of ViT much, and the global ordering of patches also does not matter much, because of learned positional embeddings and global attention. [Table]
D.2. Pre-training robustness test with longer epochs
In Table 2 of the main paper [Table] we mostly report results of short pre-training schedules: Sup 30 epochs and CL 200 epochs, in order to make the baseline results comparable.
We report CIFAR-10 longer training epochs in Table D.2.
Training longer does not change our observation that MoCo appears less robust to patch and pixel-level corruptions than SL during pre-training on this dataset.
D.3. Downstream robustness test with ResNet-50: detailed KNN accuracy numbers
Table D.3 show the detailed accuracy numbers for computing the summary statistics in Table 1 of the main paper. [Table]
D.4. Downstream robustness test with ViT backbone
[Table]
Table 1 in the main paper and Table D.3 above are generated with the KNN evaluation protocol.
We also experiment with full fine-tuning on the downstream datasets.
The results are in Table D.5.
Since different pre-trained checkpoints are optimized with different optimizers (SGD for Sup, SimSiam (Chen & He, 2021), MoCo-v2 (Chen et al., 2020d), and
SimCLR-v2 (Chen et al., 2020b); LARS (You et al., 2017) for BYOL (Grill et al., 2020), BarlowTwins (Zbontar et al., 2021), DeepCluster2, and SwAV (Caron et al., 2020)), we use SGD (lr 0.002 cosine) for Sup, SimSiam, MoCo, and SimCLR, and AdamW (lr 0.001 cosine)
(Loshchilov & Hutter, 2019) for others during fine-tuning.
All models are fine-tuned forepochs.
We find this strategy of using different optimizers is able to make the baseline results on original images comparable across methods.
We note that fine-tuning drastically improves the accuracy on downstream datasets, while the general observation that CL methods are more robust to downstream corruption than SL still holds, except for BarlowTwins which is slightly worse than SL.
Another interesting observation here is that different CL methods actually yield different robustness behaviors , although they are all doing some form of contrastive learning and have similar baseline accuracies.
D.6. Variance of pre-training results
We repeat MoCo-v2 on the original CIFAR-10 200ep three times: The KNN evaluation mean and std is 82 .
44 ± 0 .
Repeating MoCo-v2 on the global 8x8 shuffling corrupted CIFAR-10 gives KNN evaluation mean and std 59 .
24 ± 0 .
The linear evaluation variance is similar.
The randomness has a smaller order than the gap between MoCo and Supervised results.
[Table] [Table] D.7. Pre-train on corrupted CIFAR-10, but test on uncorrupted images
In the main paper [Table] we show the results when both the pre-training and evaluation datasets are corrupted in the same consistent way.
In the following Table D. [Table] we report the accuracy numbers obtained from KNN evaluation on the original uncorrupted images.
Since these models are pre-trained on the pixel or patch-level corrupted dataset, the results reflect the transfer capability of the pre-trained representation from corrupted data to original data.
We find that the trend is similar to evaluating on corrupted data that Sup appears more robust.
[Table]
E. Additional Visualization E.1. Visualizing corrupted images
Please check Figure E.4 for more visual examples of the pixel-level gamma distortion and patch-level shuffling corruptions we used.
E.2. Visualizing Grad-CAM attention maps
Figure E.5 visualizes the Grad-CAM (Selvaraju et al., 2017) attentions maps of ResNet-18 models pre-trained and linearly fine-tuned on either uncorrupted or 4x4 global patch shuffled images.
We discover some difference in terms of the equivariant property: Sup models are largely equivariant to 4x4 global patch shuffling – the attention is focused on the object parts even after patch shuffling, whereas the MoCo model pre-trained on 4x4 global shuffled images are not – it is rather focused on distracting parts.
The quality of attention map correlates with the top-1 validation accuracy, where Sup on 4x4 achieves 65% and MoCo achieves 35%.
Intuitively, a model can be more robust to such global patch shuffling if it possesses such equivariant property.
This shows the robustness of SL from another aspect, because it can robustly learn the same feature even under the shuffling disturbance.
Orig =0.
=2.
G2x
G4x
G8x
G32x32 G96x96/L1x
L2x
L4x
L8x
L32x
[Figure]