On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning
Abstract
Self-Supervised Learning (SSL) has become a very active area of Deep Learning research where it is heavily used as a pre-training method for classification and other tasks.
However, the rapid pace of advancements in this area comes at a price: training pipelines vary significantly across papers, which presents a potentially crucial confounding factor.
Here, we show that, indeed, the choice of hyperparameters and data augmentation strategies can have a dramatic impact on performance.
To shed light on these neglected factors and help maximize the power of SSL, we hyperparameterize these components and optimize them with
Bayesian optimization, showing improvements across multiple datasets for the SimSiam SSL approach.
Realizing the importance of data augmentations for SSL, we also introduce a new automated data augmentation algorithm, GroupAugment , which considers groups of augmentations and optimizes the sampling across groups.
In contrast to algorithms designed for supervised learning, GroupAugment achieved consistently high linear evaluation accuracy across all datasets we considered.
Overall, our results indicate the importance and likely underestimated role of data augmentation for SSL.
1. Introduction
Self-supervised learning (SSL) has seen an explosion of research interest in recent years, with significant progress using SSL as a pre-training method for classification (Grill et al., 2020; Chen & He, 2021; Chen et al., 2020a;
He et al., 2020; Chen et al., 2020b).
A large variety of SSL s) methods have been developed, for example using different optimization paradigms (van den Oord et al., 2018; Pathak et al., 2016), different objective functions (Giradis et al., 2018; Doersch et al., 2015; Zhang et al., 2016), or different data modalities (Radford et al., 2021).
An aspect of SSL performance that is less researched is the effect of other choices, such as the training hyperparameters or the augmentation strategy.
To shed light on these neglected factors, we use Bayesian optimization (Mockus et al., 1978; Shahriari et al., 2016) to search for configurations of the SimSiam SSL algorithm (Chen & He, 2021) on
CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and the medical dataset DermaMNIST (Yang et al., 2021a;b).
We consider, on the one hand, a search over training hyperparameters and, on the other hand, a search over data augmentation strategies.
Among other findings, our results suggest the importance of data augmentation for SSL.
Motivated by the apparent importance of data augmentation for SSL, we then develop a new automated data augmentation algorithm, GroupAugment , that covers a more diverse space of augmentation strategies than existing methods and can, e.g., design augmentation strategies that resemble manually-designed SSL augmentation strategies.
In summary, our main contributions are:
• We study the effect of training hyperparameters and augmentation strategies for the SimSiam SSL approach (Section 3).
Our results indicate the importance of data augmentation for SSL.
• We introduce the automated data augmentation algorithm GroupAugment and demonstrate its high performance for SSL (Section 4).
2. Background and Related Work
Self-Supervised Learning The most cited works in SelfSupervised Learning, such as SimCLR, BYOL, SimSiam,
MoCo, or DINO (Grill et al., 2020; Chen & He, 2021; Chen et al., 2020a; He et al., 2020; Chen et al., 2020b) all apply a similar or identical data augmentation protocol (random horizontal flip, color distortion, and nowadays also Gaus- sian blur and solarization).
Most relevant to our work are the findings of Grill et al.
and
Chen et al.
(2020a), who identify and address the sensitivity to choosing color distortions in their methods.
Moreover, Chen et al.
(2020a) identified that a sophisticated supervised data augmentation strategy does not perform better than simple cropping with strong color distortion in the self-supervised setting.
These observations motivate the contributions regarding data augmentation for SSL, which we will revisit next.
Data Augmentation Algorithms
In supervised learning, data augmentation algorithms usually outperform manually selected augmentation strategies: Three algorithms that rely on randomly sampling augmentations from a fixed set of augmentations are TrivialAugment (M¨uller & Hutter, 2021) and SmartSamplingAugment (Negassi et al., 2022).
Other algorithms, such as AutoAugment (Cubuk et al., 2019),
RandAugment (Cubuk et al., 2020), and SmartAugment (Negassi et al., 2022) perform a gradient-free search in the space of augmentation policies.
It is also possible to apply gradient-based optimization to meta-learn pre-training hyperparameters (Raghu et al., 2021) for ECG data.
In SelfAugment (Reed et al., 2021), the authors bootstrap from the correlation between supervised and self-supervised evaluation performance to incorporate a rotation task for generating augmentation policies efficiently.
SelfAugment is qualitatively different from the previous approaches in that it does not optimize for general downstream performance but rotation task performance. Additionally, it will likely not include rotations into the selected augmentation policy.
In contrast to these, our approach GroupAugment (Section 4) covers a more diverse space of augmentation strategies than existing methods and can, e.g., design augmentation strategies that resemble manually-designed SSL augmentation strategies.
It generalizes existing approaches by optimizing group-specific sampling probabilities, the number of group-specific augmentations, and the total number of augmentations applied while imposing no limitations on the set of augmentations such as SelfAugment.
3. Study on the Importance of Hyperparameters and Data Augmentation
We study the following research questions:
• What role does data augmentation play in SSL, and can better data augmentation strategies lead to better performance?
• Which hyperparameters may be notorious for resulting in model collapse when set incorrectly?
• Which hyperparameters are important to optimize in SSL to achieve good performance and outperform baselines?
3.1. Study Design
To answer the presented questions, we conducted the study described below.
Models, Datasets and Hyperparameter Search Spaces
We perform all our experiments on the CIFAR-10, CIFAR100 and DermaMNIST datasets (Krizhevsky, 2009; Yang et al., 2021a;b) and use the SimSiam (Chen & He, 2021) approach with the ResNet-18 (He et al., 2016) architecture.
We optimize a wide range of training pipeline hyperparameters (which we refer to as Training Hypers ) such as the learning rate, warmup and weight decay, and optimizer, as well as data augmentation hyperparameters (which we refer to as Augmentations ) involving magnitudes and probabilities of image distortions.
Please see Appendix A for more details on our search spaces.
We point out that all hyperparameter search spaces of the baselines are chosen identically across all datasets except for the pre-training epochs, where we used 800 for CIFAR-10/100 and 100 for DermaMNIST.
Lastly, we report how the train, validation, and test splits were chosen in Appendix B.
Search Algorithm To optimize over the search spaces listed above, we use Bayesian optimization (BO) (Mockus et al., 1978) with expert priors (Hvarfner et al., 2021) as implemented by Stoll et al.
For details on the chosen priors see Appendix B.
Performance Evaluation All reported performance values are based on the standard linear evaluation protocol (Dalal & Triggs, 2005; Grill et al., 2020) from the SSL literature that trains a linear classifier on top of the frozen ResNet backbone weights.
3.2. Results
Hyperparameters vs Data Augmentation Strategy Table 1 shows that (a) optimizing six training hyperparameters (detailed in Table 5 in the appendix) of SimSiam only lead to marginal improvements or even deterioration of performance (due to differences in validation and test split); (b) optimizing the data augmentation strategy lead to consistent significant performance improvements (at least 1%, and up to 2.3% for CIFAR-100).
This shows that SimSiam’s training hyperparameters were already very well-tuned.
How to Avoid Collapsing Chen & He (2021) already analyzed which factors, e.g., stop-gradient, can cause collapsing solutions for SimSiam.
A collapsing solution is an undesired solution where all the outputs collapse to a constant vector.
We continue this study and give insights into which hyperparameters may cause collapsing if chosen suboptimally.
While we observed collapsing solutions in some of our experiments on the CIFAR-10 and CIFAR-
[Table]
[Table]
[Table]
datasets, for DermaMNIST, we surprisingly observed no collapsing solutions.
We show violin plots in Appendix D giving some insights on which hyperparameters may be responsible for collapsing solutions.
For example, Figure 1, an excerpt from Appendix D, indicates that a probability of applying grayscale to CIFAR-100 around 0 .
5 might cause collapsing solutions.
[Figure]
Importance of Hyperparameters
In the following, we study the importance of individual hyperparameters for achieving good performance.
First, we compare the manually designed hyperparameter settings from Chen & He (2021) to the optimized settings found in our study.
As
Chen & He (2021) do not report results on CIFAR-100 and
DermaMNIST, we focus on CIFAR-10 here.
For training hyperparameters, we do not observe significant differences in performance (cf.
Table 1), and the settings found by (Chen & He, 2021) seem to be quite optimal.
For the parameterized data augmentation config space, we observe that the grayscale probability hyperparameter should be set half as high as in the baseline and that the saturation strength from the color jitter needs to be higher than in the baseline.
Further, adding solarize seems important, which is also observed by Grill et al.
The hyperparameter importance study in Table 2 also supports the importance of the above parameters.
Additionally, Figure 2 suggests that for the above parameters, good and bad configurations differ, supporting our findings.
Moreover, our results also give insights into how individual hyperparameters should be optimally set.
As an example, Figure 1 shows how the probability of applying grayscale should be optimally set for the CIFAR-100 dataset.
In the best-performing configurations, this probability hyperparameter has been sampled below 0 .and
in bad-performing configurations above 0 .
We show violin plots of all the other individual hyperparameters of the different config spaces and datasets in Appendix D.
[Table]
4. GroupAugment
In this section, we introduce GroupAugment , an automated data augmentation algorithm that operates on groups of augmentations (such as color or quality transformations) and designs sampling strategies over these groups.
Further, we present an empirical study where we find that, in contrast to the data augmentation algorithms designed for supervised learning, GroupAugment robustly outperforms the baseline in all the settings we analyzed.
4.1. Algorithm
Augmentation Sampling Given some groups of data augmentations { g i } , GroupAugment uses one global hyperparameter and two sets of group-specific hyperparameters to create a list of augmentation sequences that will be consecutively applied to an image.
The global hyperparameter T determines the number of augmentation sequences in that list.
The group-specific hyperparameter P g i determines the probability
that augmentation group
g
i is chosen to create the following augmentation sequence.
N g
i determines the number of augmentations to sample uniformly without replacement to form an augmentation sequence for augmentation group g i .
We provide pseudocode for a GroupAugment policy in Algorithm 1.
Algorithm
A GroupAugment policy applied to an image.
Input: Image I , augmentation groups { g i } , group-specific sampling probabilities { P g i } , group-specific #augmentations { N g i } , total #group-samples T
Initialize empty list of augmentation sequences A for augmentation sequence 1 , . .
, T do
Sample group g according to { P g i }
Sample N g augmentations from group g
Append augmentations to A end for
Apply sampled augmentation sequences A to I
Novelty of Group Sampling While Negassi et al.
(2022) explored optimizing the parameters of color and geometric augmentation groups, GroupAugment generalizes this notion to any set of augmentation groups and searches over more general spaces of sampling strategies.
In our study, we instantiate GroupAugment with five groups: color, geometric, non-rigid, quality, and exotic.
See Table 9 in Appendix C for the specific augmentations we used and Tablein Appendix A for a detailed description of GroupAugment’s search space.
Search Algorithm To optimize over the resulting search space, we use the same Bayesian optimization (BO) approach with expert priors as above.
Further, we normalize the sampled probabilities, as BO samples each group probability individually, and therefore, they do not necessarily add up to 1.
4.2. Comparisons and Results
We compare GroupAugment to several automated data augmentation algorithms.
On the one hand, we observe that applying standard automated data augmentation algorithms, e [Table] g., RandAugment, to the self-supervised learning setting worsens the results for most of our analyzed datasets, al-
[Table]
though those algorithms perform well in the supervisedlearning setting (see Table 3).
This observation is in line with the finding of Chen et al.
(2020a) that a sophisticated supervised data augmentation strategy did not perform better than simple cropping with strong color distortion in the SSL setting.
On the other hand, contrary to standard supervisedlearning data augmentation algorithms, GroupAugment robustly outperforms the baseline in all the settings we analyzed.
We give more details on the experimental settings in Appendix B.
Further, we observe a performance improvement over the baseline when tuning its data augmentation strategy’s magnitudes and application probabilities with the same computational budget we used for the GroupAugment search space.
However, as GroupAugment outperforms the tuned
SimSiam data augmentation for most of our results, allowing stronger parameterization, we recommend optimizing the data augmentation with GroupAugment, especially for datasets having no intuition about which data augmentation might be helpful.
5. Conclusion and Limitations
While SimCLR and BYOL have analyzed the role of data augmentation, our results show that it is beneficial to analyze it in much greater detail.
In summary, we provide evidence for the underestimated role of data augmentation for SSL and present a novel automated data augmentation algorithm, GroupAugment, which outperforms vanilla-SimSiam across all datasets we study.
Limitations We conducted our study on CIFAR-10,
CIFAR-100 (Krizhevsky, 2009), and DermaMNIST (Yang et al., 2021a;b).
More datasets should be considered to gather more insights and strengthen our experimental con- clusions.
In particular, results on ImageNet (Deng et al., 2009) and more medical datasets are of interest.
Further, as automated data augmentation (e.g., our GroupAugment) optimizes the augmentation strategy on a validation set, outof-distribution test sets can pose a challenge if the validation set is in-distribution.
Acknowledgements
We acknowledge funding by Robert Bosch GmbH, by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828, by European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765), and by BrainLinksBrainTools which was funded by the German Research
Foundation (DFG, grant no. EXC 1086) and is currently funded by the Federal Ministry of Economics, Science and Arts of Baden W ¨ urttemberg within the sustainability program for projects of the excellence initiative. Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC.
Neither the
European Union nor the ERC can be held responsible for them.
A. Search Spaces
[Table]
[Table]
[Table]
[Table]
[Table]
B. Experimental Details B.1. Expert Priors
We use Bayesian optimization (BO) (Mockus et al., 1978) with expert priors (Hvarfner et al., 2021) as implemented in the
NePS python package (Stoll et al., 2022).
Therefore, we set expert priors to guide the search.
The priors in NePS are, in the continuous case, Gaussian distributions centered at a default value with the standard deviation determined via a confidential setting.
We always use a “medium” confidence and set the default value as described below.
Training Hyperparameters and Data Augmentation Strategy We set the defaults to the baseline for the training hyperparameters and data augmentation strategy.
As no solarization is used in the SimSiam baseline and Chen & He (2021) shows that adding solarize might improve the performance, we add solarize to the search space and set the user prior default to the values reported by Chen & He (2021).
RandAugment For the RandAugment search space, we set the defaults according to the optimal data augmentation policy for CIFAR-10 following Cubuk et al.
SmartAugment As we are the first applying SmartAugment to classification, we set the user prior defaults based on the optimal data augmentation policy for CIFAR-10 following Cubuk et al.
GroupAugment For the GroupAugment config space, we set the default user priors to one augmentation per group.
As only color and geometric augmentations occur in the baseline, we set the user prior defaults for these group probabilities to 0.5 and the other group probabilities user prior defaults to 0.
B.2. Resources and Compute Budget
For our Bayesian optimization runs, we allowed a budget of 50 evaluations.
For CIFAR-10 and CIFAR-100, one configuration evaluation took ≈ 8h with one GeForce RTX 2080 Ti GPU, for DermaMNIST ≈ 10min with 1 GPU.
We used 10 GPUs in parallel for CIFAR-10, 20 GPUs in parallel for CIFAR-100, and 5 GPUs in parallel for DermaMNIST.
In order to take noise on the validation set during our HPO into account, we evaluate the best-performing configurations multiple times on the validation set.
B.3. GroupAugment Comparative Study: RandAugment Search Space
While SmartAugment and GroupAugment were designed with BO in mind, for RandAugment, which originally used Grid
Search, we follow Negassi et al.
and consider an extended search space for the number of operations between 1 and 15.
Negassi et al.
(2022) showed that a larger number of operations than 3 can be beneficial for the performance.
See also
Table 7. B.4. Dataset Splits
Since CIFAR-10 and CIFAR-100 (Krizhevsky, 2009) do not provide a validation set, we split the training set and randomly sample a fixed validation set containing 10% of the training data for our hyperparameter optimization.
We use the entire training set for training for our final test evaluations of the best-performing validation configurations.
For DermaMNIST (Yang et al., 2021a;b), we have adopted the provided training, validation, and test split.
C. GroupAugment Details
[Table]
D. Individual Hyperparameter Analysis
Here, we individually analyze each hyperparameter and augmentation strategy parameter considered in Section 3.
In particular, we plot density estimates for the values sampled in our search.
We show density estimates for the best and worst-performing configurations (top and bad), all configurations (all), and configurations leading to collapsing.
For categorical values, we show the sample distribution.
D.1. Tuned SimSiam Data Augmentation Strategy
[Table]
top all bad collapsing
[Table]
p_ c
o l o r ji tt e r
[Table]
top all bad collapsing
[Table] [Table]
a r i z e
[Table]
collapsing
[Table]
top
[Table]
collapsing
[Figure]
top
[Figure]
collapsing
[Figure]
b r i gh t ne ss _ s t r eng t h top all bad collapsing
[Figure] [Figure]
c on t r a s t _ s t r eng t
[Figure]
top all bad collapsing
[Figure]
[Figure]
[Figure] [Figure] [Figure]
[Figure]
hue_ s t r eng t
[Figure]
l
[Figure] j i tt [Figure] top all bad collapsing
[Figure]
ysc
[Figure]
top all bad collapsing
[Figure] [Figure] p_ho r i z on t
[Figure]
l _ f
[Figure]
p
[Figure]
bad collapsing
[Figure]
p_ s o l a r i z e [Figure] collapsing
[Figure]
top all bad 0.
b r i gh t ne ss _ s t r eng t
h top all bad 0.
c on t r a s t _ s t r eng t h top all bad 0.
hue_ s t r eng t
h top all bad 0.
p_ c o l o r ji tt e r top all bad 0.
p_g r a ysc a l e top all bad 0.
p_ho
r i z on t a l _ f
li p top all bad 0.
p_ s o l a r i z e top all bad 0.
[Figure]
eng
[Figure]
h top all bad
[Figure]
D.2. Tuned SimSiam Training Hyperparameters
top all bad collapsing 0.
p t _ l ea r n i ng _ r a t e top all bad collapsing       # c on f i g s w i t h op t i m i z e r sgd adamw lars top all bad collapsing 0. p t _ w e i gh t _ de c a y _ end top all bad collapsing 0. p t _ w e i gh t _ de c a y _ s t a r t top all bad collapsing       w a r m up_ epo
[Figure]
top all bad collapsing 1.
w a r m up_ m u l t
i p li e r
[Figure]
p t
_ l ea r n i ng _ r a t e
[Figure]
[Figure]
f
[Figure]
g
[Figure]
i
[Figure]
op
[Figure]
z
[Figure]
sgd
[Figure]
lars
[Figure]
p t _ w e i gh t _ de c a y _ end
[Figure]
i gh
[Figure]
_ de
[Figure]
y _
[Figure]
a
[Figure]
[Figure]
up
[Figure]
l t i
[Figure]
[Figure] t _ l ea r n i ng [Figure] a t
[Figure]
p t _ w e i gh t _ de c a y _ s t a r t
[Figure]
[Figure]
[Figure]
w a r m up_ m u l t
i p li e r