R ETHINKING THE E FFECT OF D ATA A UGMENTATION IN A DVERSARIAL C ONTRASTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent works have shown that self-supervised learning can achieve remarkable robustness when integrated with adversarial training (AT).
However, the robustness gap between supervised AT (sup-AT) and self-supervised AT (self-AT) remains significant.
Motivated by this observation, we revisit existing self-AT methods and discover an inherent dilemma that affects self-AT robustness: either strong or weak data augmentations are harmful to self-AT, and a medium strength is insufficient to bridge the gap.
To resolve this dilemma, we propose a simple remedy named
DynACL (Dynamic Adversarial Contrastive Learning).
In particular, we propose an augmentation schedule that gradually anneals from a strong augmentation to a weak one to benefit from both extreme cases.
Besides, we adopt a fast post-processing stage for adapting it to downstream tasks.
Through extensive experiments, we show that DynACL can improve the state-of-the-art self-AT robustness by 8.84% under
Auto-Attack on the CIFAR-10 dataset, and can even outperform vanilla supervised adversarial training.
We demonstrate that self-supervised AT can attain even better robustness than supervised AT for the first time.
I NTRODUCTION
Learning low-dimensional representations of inputs without supervision is one of the ultimate goals of machine learning.
As a promising approach, self-supervised learning is rapidly closing the performance gap with respect to supervised learning (He et al., 2016; Chen et al., 2020b) in downstream tasks.
However, for whatever supervised and self-supervised learning models, adversarial vulnerability remains a widely-concerned security issue, i.e., natural inputs injected by small and human imperceptible adversarial perturbations can fool the deep neural networks (DNNs) into making wrong predictions.
For supervised learning, one popular approach to enhance adversarial robustness is adversarial training ( sup-AT ), where DNNs are trained on adversarial examples (Goodfellow et al., 2014; Madry et al., 2017).
However, sup-AT requires true labels to craft adversarial examples.
For self-supervised learning, recent works including RoCL (Kim et al., 2020)
, ACL (Jiang et al., 2020), and AdvCL (Fan et al., 2021) have proposed the corresponding adversarial training counterpart ( self-AT ).
Despite obtaining a certain degree of robustness, these methods failed to eliminate the large gap between sup-AT and self-AT.
As a reference, for standard training (ST), the performance gap between sup-ST and self-ST is lower than 1% on CIFAR-10 (da Costa et al., 2022).
However, for adversarial training (AT), the robustness gap remains > 8% (sup-AT 46.2% v.s. self-AT 37.6%), as shown in Figure 1(a).
This phenomenon leads us to the following question:
What is the key factor that prevents self-AT from obtaining comparable robustness to sup-AT?
To answer this question, we examine the difference between sup-AT and self-AT.
As they share the same perturbation strategy, the main difference mainly lies in their learning paradigms.
Different from sup-AT which directly utilizes labels, existing self-AT methods (RoCL, ACL, and AdvCL) all adopt the contrastive learning paradigm, which instead relies crucially on strong data augmentations to obtain generalizable representations.
As noted by recent theoretical understandings of contrastive learning (HaoChen et al., 2021; Wang et al., 2022), strong augmentations create support overlap between intra-class samples such that the alignment between augmented pairs could cluster these intra-class samples together.
Under weaker augmentations, the accuracy of standard contrastive learning (SimCLR) will dramatically decrease, as the red line shows in Figure 1(b).
Nevertheless, the blue line in Figure 1(b) demonstrates that strong data augmentations are very harmful to the
[Figure]
[Figure]
obu
[Figure]
t a
[Figure]
u r a cy (
A u t o - A tt a ck )
[Figure]
[Figure]
adversarial robustness of ACL. This reveals a critical dilemma of self-AT, i e., the default strong augmentations, although useful for standard accuracy, will hurt adversarial robustness.
Further, we dive into the strong augmentations adopted by self-AT in Figure 1(c) and find that significant semantic shifts to the original images are observed.
Therefore, there will be a much larger distribution gap between training and test data, and the local ℓ p robustness on training examples becomes less transferable to test data.
More severely, some augmented samples from different classes become even inseparable (the bird and the cat samples in red squares), and thus largely distort the decision boundary.
The necessity and harmfulness of data augmentations form a fundamental dilemma in self-AT.
Therefore, as data augmentations play an essential role in self-AT, we need to strike a balance between utilizing strong augmentations for representation learning and avoiding large image distortions for good robustness.
Although it seems impossible under the current static augmentation strategy, we notice that we can alleviate this dilemma by adopting a dynamic augmentation schedule.
In particular, we could learn good representations with aggressive augmentations at the beginning stage, and gradually transfer the training robustness to the test data by adopting milder and milder augmentations at later stages.
We name this method Dynamic Adversarial Contrastive Learning (DynACL).
In this way, DynACL could benefit from both sides, gaining both representation power and robustness aligned with test distribution.
Built upon DynACL, we further design a fast post-processing stage for bridging the difference between the pretraining and downstream tasks, dubbed DynACL++.
As a preview of the results, Figure 1(a) shows that the proposed DynACL and DynACL++ bring a significant improvement over state-of-the-art self-AT methods, and achieve comparable, or even superior robustness, to vanilla sup-AT.
Our main contributions are:
• We reveal the reason behind the robustness gap between self-AT and sup-AT, i.e., the widely adopted aggressive data augmentations in self-supervised learning may bring the issues of training-test distribution shift and class inseparability.
• We propose a dynamic augmentation strategy along the training process to balance the need for strong augmentations for representation and mild augmentations for robustness, called Dynamic Adversarial Contrastive Learning (DynACL).
• Experiments show that our proposed methods improve both clean accuracy and robustness over existing self-AT methods by a large margin.
Notably, DynACL++ improves the robustness of ACL (Jiang et al., 2020) from 37.62% to 46.46% on CIFAR-10, which is even slightly better than vanilla supervised AT (Madry et al., 2017).
Meanwhile, it is also more computationally efficient as it requires less training time than ACL.
B ACKGROUND AND R ELATED W ORK
Sup-AT.
Given a labeled training dataset D l = { (¯ x, y ) | ¯ x ∈ R n , y ∈ [ K ] } , to be resistant to ℓ pbounded adversarial attack, supervised adversarial training (sup-AT) generally adopts the following
[Equation]
Here the classifier f : R n → R K is learned on the adversarially perturbed data (¯ x + δ ¯ x , y ) , and ∆ =
{ δ : ∥ δ ∥ p ≤ ε } denotes the feasible set for adversrial perturbation.
Contrastive Learning.
For each ¯ x ∈ X , we draw two positive samples x, x + from the augmentation distribution A ( ·| ¯ x ) , and draw M independent negative samples { x − m } Mm =1 from the marginal distribution A ( · ) =
E ¯ x A ( ·| ¯ x ) .
Then, we train an encoder g : X → Z by the widely adopted InfoNCE loss using the augmented data pair ( x, x + , { x − m } )
(Oord et al., 2018):
[Equation]
Here sim( · , · ) is the cosine similarity between two vectors, and τ is a temperature hyperparameter.
Surged from InfoNCE (Oord et al., 2018), contrastive learning undergoes a rapid growth (Tian et al., 2020; Misra & Maaten, 2020;
Chen et al., 2020a; He et al., 2020) and has demonstrated state-of-theart performance on self-supervised tasks (Chen et al., 2020b; Kong et al., 2020).
Nevertheless, several works also point out contrastive learning is still vulnerable to adversarial attack when we transfer the learned features to the downstream classification (Ho & Nvasconcelos, 2020; Kim et al., 2020).
Self-AT.
To enhance the robustness of contrastive learning, adversarial training was similarly adapted to self-supervised settings (self-AT).
Since there are no available labels, adversarial examples are generated by maximizing the contrastive loss (Eq. 2) w.r.t.
all input samples,
[Equation]
Several previous works, including ACL (Jiang et al., 2020), RoCL (Kim et al., 2020), and CLAE (Ho & Nvasconcelos, 2020), adopt Eq.
3 to perform self-AT.
In addition, ACL (Jiang et al., 2020) further incorporates the dual-BN technique (Xie et al., 2020) for better performance.
Specifically, given a backbone model like ResNet, they duplicate its BN modules and regard the encoder g as two different branches (all parameters are shared except for BN): the clean branch g c and the adversarial branch g a are trained by clean examples and adversarial examples respectively:
[Equation]
After training, they use the adversarial branch g a as the robust encoder.
They empirically show that ACL obtains better robustness than the single branch version.
Built upon ACL, AdvCL (Fan et al., 2021) further leverages an additional ImageNet-pretrained model to generate pseudo-labels for sup-AT via clustering.
Different from them, we explore an orthogonal direction to previous work by investigating the influence of augmentation strength on the robustness of self-AT.
R ETHINKING THE E FFECT OF D ATA A UGMENTATIONS IN S ELF -AT
Different from the case of supervised learning, data augmentation is an indispensable ingredient in contrastive self-supervised learning.
Existing self-AT methods inherit the aggressive augmentation strategy adopted in standard contrastive learning ( e.g., SimCLR (Chen et al., 2020a)), i.e., a composition of augmentations like random resized crop, color jitter, and grayscale.
However, this strategy is not necessarily optimal for adversarial training.
In this section, we provide an in-depth investigation of the effect of this default data augmentation strategy on self-supervised adversarial training (self-AT).
For intuitive exposure, we mainly resort to empirical evidence on real-world data.
Rigorously, we also provide theoretical justifications for these phenomena in Appendix E.
3. P ROBLEMS OF A GGRESSIVE D ATA A UGMENTATIONS IN S ELF -AT
As shown in Figure 1(c), the default augmentations in self-AT are much more aggressive than that of sup-AT, which leads to two obvious drawbacks: large training-test distribution gap and class inseparability, as we quantitatively characterize below.
For comparison, we evaluate 4 kinds of clean adversarial sup-AT self-AT    t
r
a i n i ng - t e s t MM
D (r e l a t i v e ) (a) training-test distribution gap clean adversarial sup-AT self-AT 0.
c l a ss w i s e
d
i s t an c e on t r a i n i ng s e t 16 /adversarial inseparable
[Figure]
data augmentation methods on CIFAR-10: 1) clean (no augmentation), 2) pre-generated adversarial augmentation by a pretrained classifier, 3) the default augmentations (padding-crop and horizontal flip) adopted by sup-AT (Madry et al., 2017), and 4) the default augmentations (random resized crop, color jitter, grayscale, and horizontal flip) adopted by self-AT (Jiang et al., 2020;
Kim et al., 2020).
Large Training-test Distribution Gap.
Although training and test samples are supposed to follow the same distribution, random augmentations are often adopted in the training stage, which creates a training-test distribution gap.
To study it quantitatively in practice, we measure this distribution gap with Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) calculated between the augmented training set and the raw test set of CIFAR-10 (details in Appendix B.1).
1 As shown in Figure 2(a), “adversarial” and “sup-AT” augmentations only produce a small distribution gap, while the aggressive “self-AT” augmentations yield a much larger gap (6 times) than “sup-AT”.
Therefore, the self-AT training data could be very distinct from test data, and the robustness obtained w.r.t.
augmented training data could be severely degraded when transferred to the test data.
Class Inseparability.
Yang et al.
(2020) found that under supervised augmentations and adversarial perturbations with ε = 8 / 255 , the minimal ℓ ∞ distance between training samples from different classes, namely the classwise distance , is larger than 2 · ε = 16 / 255 .
In other words, CIFAR-10 data are adversarially separable in sup-AT.
However, we notice that this no longer holds in self-AT.
As shown in Figure 2(b), self-AT augmentations have a much smaller classwise distance that is close to zero.
One possible cause is those CIFAR-10 samples with complete black or white regions, which could result in identical augmentations after aggressive random cropping (Figure 2(c)).
As a result of class inseparability, self-AT will mix adversarial examples from different classes together, and these noisy examples will degrade the robustness of the learned decision boundary.
3. T HE D ILEMMA OF S ELF -AT UNDER S TATIC A UGMENTATION S TRENGTH
The discussion above shows that aggressive augmentations are harmful to self-AT, which motivates us to quantitatively study the effect of different augmentation strengths on self-AT.
Quantifying Augmentation Strength.
In order to give a quantitative measure of the augmentation strength, we design a set of data augmentations T ( s ) that vary according to a hyperparameter s ∈ [0 , 1] : 1) when s = 0 , it is mild and contains only horizontal flip; 2) when s = 1 , it is the aggressive augmentations of self-AT; and 3) when s ∈ (0 , 1) , we linearly interpolate the augmentation hyperparameters to create a middle degree of augmentation.
For example, in random resized cropping, a larger s will crop to a smaller region in an image, which amounts to a larger training-test distribution gap, as shown in Figure 3.
See a detailed configuration of T ( s ) in Appendix A.1.
Static Augmentation Cannot Fully Resolve the Dilemma of Self-AT.
The discussion in Section 3.1 reveals that strong augmentations are harmful to adversarial training while being critical for representation learning (Figure 1(b), Appendix D.3 for more details).
The augmentation strength thus becomes a critical dilemma of self-AT.
Here, we study whether we could resolve this issue by tuning the augmentation strength s for an optimal tradeoff.
As shown in Figures 4(a) and 4(b), decreasing augmentation strength s (left to right) indeed brings smaller training-test distribution gap
[Figure]
t r a i n i n g - t e s t
[Figure]
D ( r e l a t i v
[Figure]
and eliminates class inseparability, which benefits adversarial robustness (blue line in Figure 4(c)) and hurts standard accuracy at the same time (the red line in Figure 4(c)).
Roughly, s = 0 .
5 seems a good trade-off between accuracy and robustness, while its 41.79% robustness is still much lower than the 46.23% robustness of sup-AT (orange line in Figure 4(c)).
This suggests that the dilemma of augmentation strength cannot be fully resolved by tuning a static augmentation strength.
The training process in Figure 4(d) presents that strong or weak augmentations have their own defects while medium augmentations only bring a limited gain (green line).
In the next section, we will introduce a new dynamic augmentation strategy to resolve this dilemma (a preview of performance is shown as the red line in Figure 4(d)).
P ROPOSED D YN AMIC A DVERSARIAL C ONTRASTIVE L EARNING (D YN ACL)
In this section, we propose a simple but effective method DynACL to resolve the dilemma of self-AT augmentations.
In Section 4.1, we introduce our dynamic pretraining method for self-AT, named
DynACL.
In Section 4.2, we further propose an improved version of DynACL with a fast postprocessing stage to mitigate the gap between pretraining and downstream tasks, named DynACL++.
Figure 6 in Appendix A.3 demonstrates the framework of the proposed DynACL.
4. D YN ACL: D YNAMIC A DVERSARIAL C ONTRASTIVE L EARNING
In view of the dilemma of static augmentation strength (Section 3.2), we propose a new dynamic training recipe for self-AT, named Dynamic Adversarial Contrastive Learning ( DynACL ), which consists of a dynamic schedule for data augmentation and a dynamic objective for model training.
Dynamic Augmentation.
Different from prior works that all adopt a static augmentation strength, we propose to treat the augmentation strength s as a dynamic parameter s ( t ) that varies along different epochs t .
In this way, at earlier training stages, we can enforce self-AT to learn meaningful representations by adopting aggressive augmentations with a large augmentation strength s ( t ) .
Afterwards, we could gradually mitigate the training-test distribution gap and class inseparability by annealing down the augmentation strength s ( t ) (see Figures 4(a) and 4(b)).
With this dynamic augmentation schedule, we could combine the best of both worlds: on the one hand, we can benefit from the representation ability of contrastive learning at the beginning stage; on the other hand, we can achieve better adversarial robustness by closing the distribution gap at the later stage.
Following the analysis above, we design the following piecewise decay augmentation schedule:
[Equation]
where T is the total number of training epochs, K is the decay period, and ⌊·⌋ is the floor operation.
As illustrated in Figure 3, this schedule s ( t ) starts from strong augmentations with s = 1 at the start epoch, and gradually decays to a weak augmentation with s = 0 by reducing K/T every K epochs.
Dynamic ACL Loss.
Following the same idea above, we can also design a dynamic version of the
ACL loss in Eq.
Instead of adopting an equal average of the clean and adversarial contrastive losses, we can learn meaningful representations in the beginning stage with the clean contrastive loss ℓ NCE (Eq. 2).
Later, we can gradually turn our focus to adversarial robustness by down-weighting the clean contrastive loss and up-weighting the adversarial loss ℓ AdvNCE (Eq. 3).
Following this principle, we propose a dynamic ACL loss with a dynamic reweighting function w ( t ) for the t -th epoch:
[Equation]
For simplicity, we assign the dynamic reweighting function w ( t ) as a function of the dynamic augmentation schedule s ( t ) :
[Equation]
where λ ∈ [0 , 1] denotes the reweighting rate.
In this way, when the training starts ( t = 0 ), we adopt an equal combination of the clean and adversarial losses as in ACL.
As training continues, the augmentation strength s ( t ) anneals down, and the reweighting coefficient w ( t ) becomes larger.
Accordingly, the clean loss ℓ NCE will have a smaller weight while the adversarial loss ℓ AdvNCE will have a larger weight.
At last ( t = T ) , the weight of clean loss is decreased to 1 − λ , and the weight of adversarial loss is increased to 1 + λ .
A larger reweighting rate λ indicates more aggressive reweighting.
Our dynamic ACL loss degenerates to vanilla ACL loss with λ = 0 as a special case.
4. D YN ACL++: D YN ACL WITH F AST P OST - PROCESSING
The main dynamic pretraining phase proposed above can help mitigate the training-test data distribution gap.
Nevertheless, there remains a training-test task discrepancy lying between the contrastive pretraining and downstream classification tasks: the former is an instance-level discriminative task, while the latter is a class-level discriminative task.
Therefore, the pretrained instance-wise robustness might not transfer perfectly to the classwise robustness to be evaluated at test time.
To achieve this, we propose a simple post-processing phase with Pseudo Adversarial Training (PAT) to mitigate this “task gap”.
Given a pretrained encoder with a clean branch g c and an adversarial branch g a , we simply generate pseudo labels { ˆ y i } by k-means clustering using the clean branch g c , which natural accuracy is relatively high because it is fed with natural samples during training.
Inspired by the Linear Probing then full FineTuning (LP-FT) strategy (Kumar et al., 2022), we propose to learn a classifier with Linear Probing then Adversarially full FineTuning (LP-AFT) for better feature robustness: we first use standard training to train a linear head h :
R m → R k ( k is the number of pseudo-classes) with fixed adversarial branch g a , and then adversarially finetune the entire classifier h ◦ g a on generated pseudo pairs ( x, ˆ y ) using the TRADES loss (Zhang et al., 2019).
In practice, we only perform PAT for a few epochs ( e.g., 25 ), which is much shorter than the pretraining stage with typically 1000 epochs.
Thus, we regard this as a fast post-processing stage of the learned features to match the downstream classification task and name the improved version DynACL++.
Dicsusions.
Our DynACL is built upon the dual-stream framework of ACL (Jiang et al., 2020).
Compared to ACL with static augmentation strength and learning objective, DynACL adopts a dynamic augmentation schedule together with a dynamic learning objective.
Also built upon ACL,
AdvCL (Fan et al., 2021) leverages a third view for high-frequency components and pseudo labels into the entire pretraining phase, which brings significant computation overhead.
In comparison, we devise the LP-AFT process to align linear encoders with pretrained features and exploit self-generated pseudo labels to mitigate the task gap.
Compared to AdvCL, our method brings more significant improvements whil being much more computionally efficient.
E XPERIMENTS
In this section, we evaluate DynACL and DynACL++ under the benchmark datasets: CIFAR-10,
CIFAR-100 (Krizhevsky, 2009), and STL-10 (Coates et al., 2011), compared with baseline methods: [Table]
RoCL (Kim et al., 2020), ACL ( [Table] et al., 2020), and AdvCL (Fan et al., 2021).
Additional experimental details can be found in Appendix C.
Pretraining.
We adopt ResNet-18 (He et al., 2016) as the encoder following existing self-AT methods (Kim et al., 2020; Jiang et al., 2020; Fan et al., 2021).
We set the decay period K = 50 , and reweighting rate λ
= 2 / 3 .
As for other parts, we mainly follow the training recipe of ACL (Jiang et al., 2020), while we additionally borrow two training techniques from modern contrastive learning variants: the stop gradient operation and the momentum encoder.
Both are proposed by BYOL (Grill et al., 2020) and later incorporated in MoCo-v3 (Chen et al., 2021) and many other methods.
An ablation study on these two components can be found in Appendix D.2, where we show that the improvement of DynACL mainly comes from the proposed dynamic training strategy.
Evaluation Protocols.
We evaluate the learned representations with three protocols: standard linear finetuning (SLF), adversarial linear finetuning (ALF), and adversarial full finetuning (AFF).
The former two protocols freeze the learned encoder and tune the linear classifier using cross-entropy loss with natural (SLF) or adversarial (ALF) samples, respectively.
As for AFF, we adopt the pretrained encoder as weight initialization and train the whole classifier following ACL (Jiang et al., 2020).
5. B ENCHMARKING THE P ERFORMANCE OF D YN ACL
Robustness on Various Datasets.
We evaluate the robustness of sup-AT and self-AT methods on CIFAR-10, CIFAR-100, and STL-10 in Table 1.
As we can see, DynACL outperforms all previous self-AT methods in terms of robustness without using additional data 2 .
Specifically, DynACL brings a giant leap of robustness by improving previous state-of-the-art self-AT methods by 8.84% (from 37.62% to 46.46%) on CIFAR-10, 4.37% (from 15.68% to 20.05%) on CIFAR-100, and 1.95% on STL-10 (from 45.26% to 47.21%) under the auto-attack benchmark.
Meanwhile, we notice that
DynACL alone leads to a slight decrease in standard accuracy in CIFAR-10 and STL-10, suggesting that annealed augmentations at a later stage might slightly hurt the discriminative power to trade off robustness.
Nevertheless, post-processing in DynACL++ can largely mitigate this issue and improve standard accuracy and robustness.
As a result, DynACL++ achieves even higher AA accuracy than the supervised vanilla AT (with heavily tuned default hyperparameters (Pang et al., 2020)) on CIFAR-10.
It is the first time a self-supervised AT method outperforms its supervised counterpart.
Robustness under Different Evaluation Protocols.
Table 2 shows that DynACL and DynACL++ obtain state-of-the-art robustness among self-AT methods across different evaluation protocols.
In particular, under ALF settings, DynACL++ outperforms all existing self-AT methods and even the sup-AT method.
Furthermore, under AFF settings, DynACL could improve the heavily tuned sup-AT baseline (Pang et al., 2020) by 1.58% AA accuracy (48.96% → 50.54%), and is superior to other pretraining methods.
In conclusion, DynACL and DynACL++ have achieved state-of-theart performance across various evaluation protocols. [Table] [Table]
Performance under Semi-supervised Settings.
Following Jiang et al.
(2020), we evaluate our proposed DynACL++ under semi-supervised settings.
Baseline methods are state-of-the-art semisupervised AT method UAT++ 3 (Uesato et al., 2019), and self-supervised method ACL Jiang et al.
The results are shown in Table 3.
We observe that our DynACL++ can demonstrate fairly good performance even when only 1% of labels are available.
5. E MPIRICAL U NDERSTANDINGS
In this part, we conduct comprehensive experiments on CIFAR-10 to have a deep understanding of the proposed DynACL and DynACL++.
More analysis results are deferred to Appendix D.
Ablation Studies.
We conduct an ablation study of our two major contributions (augmentation annealing in Section 4.1 and post-processing in Section 4.2) in Table 4.
We can see that both annealing and post-processing can significantly boost the model’s robustness, which demonstrates the superiority of our method.
Robust Overfitting.
A well-known pitfall of sup-AT is Robust Overfitting (Rice et al., 2020): when training for longer epochs, the test robustness will dramatically degrade.
Surprisingly, we find that this phenomenon does not exist in DynACL.
In Figure 5(a), we compare DynACL and sup-AT for very long (pre)training epochs.
We can see that sup-AT overfits quickly after the 100th epoch and suffers a 12% test robustness decrease eventually.
Nevertheless, the test robustness of DynACL keeps increasing along the training process even under 2000 epochs.
Loss Landscape.
Previous studies (Prabhu et al., 2019; Yu et al., 2018; Wu et al., 2020) have shown that a flatter weight landscape often leads to better robust generalization.
Motivated by that discovery, we visualize the loss landscape of ACL and DynACL++ following (Li et al., 2018).
Figure 5(b) and 5(c) show that DynACL++ has a much flatter landscape than ACL, explaining the superiority of
DynACL++ in robust generalization.
Training Speed.
We evaluate the total training time of self-AT methods with a single RTX 3090 GPU.
Specifically, the total training time is 32 . 7 , 105 . 0 , 29 . 4 , and 30 .
3 hours for ACL, AdvCL, DynACL, and DynACL++, respectively.
Built upon ACL, DynACL’s dynamic strategy (augmentation and loss) introduces little computation overhead, and it spends even less time due to the stop gradient on the target branch.
Moreover, since the post-processing only lasts a few epochs, DynACL++ is also faster [Table] [Figure]
than ACL. In comparison, AdvCL is much slower and even triples the training time of
These our both efficient and effective.
Effect of Annealing Schedule.
We evaluate DynACL++ with different decay period K in Table 5(a).
We observe that our dynamic schedule significantly outperforms the constant one.
Also, we notice that the step-wise annealing strategy performs slightly better.
This may be because over-frequent data reloading (i.e. over-frequent distribution shift) may make the model fail to generalize well under each distribution.
Effect of Reweighting Rate.
We also study the effect of annealing loss (Eq. 7) with different reweighting rates λ .
Table 5(b) shows that compared to no annealing ( λ = 0 ), the annealing loss ( λ > 0 ) consistently improves both accuracy and robustness for around 1% , and the best robustness is obtained with a moderate reweighting rate λ
= 2 / 3 .
[Table]
C ONCLUSIONS
In this paper, we observe a dilemma about the augmentation strength that either weak or strong augmentations degrade the model robustness through a quantitative investigation of data augmentations in self-supervised adversarial training (self-AT).
Going beyond simple trade-off by selecting a medium augmentation strength, we propose to adopt a dynamic schedule that gradually anneals the strength from strong to weak, named DynACL.
Afterward, we further devise a fast clustering-based post-processing technique to adapt the model for downstream finetuning, named DynACL++.
Experiments show that the proposed DynACL and DynACL++ successfully boost self-AT’s robustness and even outperform its vanilla supervised counterpart on CIFAR-10 under Auto-Attack.
A A LGORITHM D ETAILS A. D ETAILED C ONFIGURATION OF D ATA A UGMENTATIONS T ( s )
We formulate the augmentation configuration T ( s ) with respect to augmentation strength s in the following PyTorch-style code: from torchvision import transforms def get_transforms(strength):rnd_color_jitter=transforms.
RandomApply([transforms.
ColorJitter( 0.4*strength,0.4*strength,0.4*strength,0.1*strength)],p=0.8*strength) rnd_gray=transforms.
RandomGrayscale(p=0.2*strength)transform=transforms.
Compose([ transforms.
RandomResizedCrop(32,scale=(1.0-0.9*strength,1.0)), #Noneedtodecayhorizontalfliptransforms.
RandomHorizontalFlip(p=0.5), rnd_color_jitter,rnd_gray, transforms.
ToTensor(),]) returntransform
A. P SEUDO C ODE OF D YN ACL AND D YN ACL++ A LGORITHM
Algorithm 1: DynACL algorithm input : Encoder g and linear classification head h .
Function of the augmentation set T ( · ) , number of pretraining epoch T , and post-processing epoch T ′ , T ′′ .
output : Pretrained and post-processed encoder g .
* Phase 1: Momentum contrastive pretraining with augmentation annealing (DynACL)
*/ 1 forall t ∈ { 1 , 2 , · · · , T } do for sampled mini-batch X do augment samples from X by augmentation set T ( s ( t ))
L ← L DynACL ( g ; t ) update parameters in g to minimize L /* Phase 2:
Pseudo label based post-processing (DynACL++) */ 6 Extract the normal route of the momentum encoder, and denote it by g c 7 { ˆ y
i
Algorithm 1 demonstrates the pseudo-code of our proposed DynACL and DynACL++.
Note that
DynACL++ adds a fast and simple post-processing phase to DynACL and enjoys higher robustness.
A. P IPELINE OF D YN ACL AND D YN ACL++
Figure 6 demonstrates the pipeline of the proposed DynACL and DynACL++.
For DynACL (upper part of the figure), we adopt our proposed augmentation annealing strategy and the momentum technique (He et al., 2020) in the pretraining phase.
Projection heads are added on the top of both branches of the ResNet backbone.
For simplicity, this architecture is not explicitly shown in the figure.
For DynACL++, we add a simple and fast post-processing phase to DynACL (bottom part of the figure).
Specifically, we replace the projection head with a linear classification head and adopt our PAT algorithm.
Note that both two phases require NO true labels.
B D ETAILS OF THE M EASUREMENT OF A UGMENTATION E FFECTS B. C ALCULATION P ROTOCOLS
Calculation of minimal classwise distance.
We randomly augment each sample from the training set 50 times and calculate the distance in terms of ℓ ∞ norm in the input space.
We believe that the input space class separability is a vital indicator of adversarial robustness.
Typical adversarial perturbation budget ε is defined in the input space, e.g., 8/255 under ℓ ∞ -norm.
Thus, given an augmented dataset
D , whether the minimal input space classwise distance is smaller than 2 · 8 / 255 = 16 / 255 decides whether this dataset is adversarially separable, i.e., whether there exists a classifier that achieves 100% training robust accuracy.
Calculation of Maximum Mean Discrepancy (MMD).
Denote the whole training set by X tr and the whole test set by X ts .
We first transform each set into a batch of tensors and then flatten them into two dimensions.
Finally, we adopt Radical Basis Function as kernel function with bandwidth 10 , 15 , 20 , 50 to calculate the MMD between X tr and X ts .
B. L ATENT S PACE D ISTRIBUTION G AP AND M INIMAL C LASSWISE D ISTANCE
In Section 3.1, we have demonstrated the input space distribution gap between the training set and test set and found that stronger data augmentation will result in a wider distribution gap.
To further justify our findings, we calculate the statistics using the distance of sample features in the latent space.
Specifically, we utilize the Perceptual Similarity proposed in (Zhang et al., 2018) to calculate latent space distance.
Latent Space Distribution Gap.
Specifically, for MMD, given perceptual similarity function f ( x, y ) which takes two images x, y as inputs, the kernel function in MMD is defined as 1 − f ( x, y ) .
Note that the perceptual similarity is lower for similar images and 0 for identical images.
Figure 7(a) demonstrates the latent space MMD between the augmented training set and clean test set.
Similar
[Figure]
t r a i n i n g - t e s t
MM D ( r e l a t i v e )
[Figure]
i
[Figure]
to input space MMD, stronger data augmentation will bring a bigger distribution gap, thus harming robustness generalization.
Latent Space Minimal Classwise Distance.
Similarly, we can calculate the classwise distance using latent space distance.
Specifically, we augment each sample from the training set 20 times and use a pretrained AlexNet to extract its latent space features.
After normalization, We calculate the ℓdistance between augmented samples from different classes and record the minimum.
The results are shown in Figure 7(b).
We can see that the latent classwise distance will also become significantly smaller under stronger augmentations (larger s ), which generally agrees with the trend in the input space.
C E XPERIMENTAL D ETAILS C. P RETRAINING S ETTINGS
Except that we adopt the momentum backbone and stop-gradient operation (Grill et al., 2020), we follow ACL on other pretraining configurations.
Specifically, we train the backbone forepochs with LARS optimizer and cosine learning rate scheduler.
The projection MLP has 3 layers.
For DynACL pretraining, we set the decay period K = 50 , and reweighting rate λ
= 2 / 3 .
For
DynACL++, we generate pseudo labels for training data by K-means clustering.
First, we tune the classification head only by standard training for 10 epochs, then we tune both the encoder and the classification head by TRADES (Zhang et al., 2019) for 25 epochs.
C. E VALUATION P ROTOCOLS
As for SLF and ALF we train the linear classifier for 25 epochs with an initial learning rate of 0 .
01 on
CIFAR-10 and 0 .
1 on CIFAR-100 and STL-10 (decays at the 10th and 20th epoch by 0.1) and batch size 512.
For AFF, we employ the TRADES (Zhang et al., 2019) loss with default parameters, except that in DynACL and DynACL++, we feed the clean samples to the normal encoder when generating adversarial perturbations.
Finetuning lasts 25 epochs with an initial learning rate of 0 .
1 (decays by 0.1 at the 15th and 20th epoch) and batch size 128.
For all baselines and DynACL, we report the last result in the evaluation phase.
In the original implementation of AdvCL (Fan et al., 2021), they incorporate extra data from ImageNet to train a feature extractor based on SimCLR.
For a fair comparison, we replace AdvCL’s ImageNetpretrained model with a CIFAR-pretrained model.
This model is trained following the default hyperparameters and official implementations and reaches a clean accuracy of 88 .
15% on CIFARand 65 .
78% on CIFAR-100.
D A DDITIONAL E XPERIMENTS D. V ISUALIZATION OF T RAINING D YNAMICS
Figure 8 demonstrates the training dynamics of ACL, DynACL, and supervised-AT (all with cosine learning rate schedule).
We observe that the robust accuracy (PGD-20) of both DynACL and ACL increases continuously throughout the training process, while that of sup-AT suffers a significant decrease in the later period of training.
[Figure]
Figure 8 also depicts the effectiveness of annealing the augmentation strength s to 0.
To further examine this point, we consider an alternative schedule, where we threshold the augmentation strength when it reaches s = 0 .
3 at the 700-th epoch, and adopt a constant augmentation strength s =
.afterwards.
Table 6 shows that the thresholding strategy performs a little bit worse than the original strategy.
[Table]
D. A DDITIONAL A BLATION S TUDY
Here we conduct an ablation study on the momentum encoder and stop-gradient operation that is additional adopted by DynACL.
We can see without both techniques, DynACL still achieves very high AA robustness (44.9%) that is significantly better than ACL (Jiang et al., 2020).
As for the two techniques, we notice that the stop gradient alone has limited improvement, also it helps improve training efficiency.
The momentum encoder, instead, help improves the standard accuracy by nearly 2%, while attains similar AA robustness to the vanilla
DynACL (ACL + dynamic strategy) [Table] [Table] D.
[Figure]
[Figure] s t a n d a r d a cc u r a c y [Figure]
r o b u s t a cc u r a c y
( A u t o - A tt a c k )
Effect of Augmentation Strength in Sup-AT.
Popular sup-AT methods (Zhang et al., 2019; Pang et al., 2020) typically adopt very weak data augmentations, including only horizontal flip and padding-crop.
As shown in Figure 9, aggressive data augmentations are detrimental to the model’s performance.
[Table]
Comparison among Augmentation Strategies. Data Augmentation is an effective approach to enhance data diversity and prevent overfitting. Canonical augmentations are input-agnostic and static across the training process. Recently, a few works have explored class-wise or instance-wise adaptive augmentation strategies, such as AdaAug (Cheung & Yeung, 2021b)
MetaAugment (Zhou et al., 2021).
We instead explore an augmentation schedule that dynamically changes during the training process .
Compared to meta-learning strategies that bring additional training costs, our augmentation schedule is simple and does not introduce training or parameter overhead.
In practice, we also compare our method against AdaAug (Cheung & Yeung, 2021a).
We observe that it can bring small improvements, but is much less effective to DynACL++ (Jiang et al., 2020)
(Table 8).
This shows that our dynamic strategy is very suitable for the self-AT task compared to general-purpose adaptive augmentations.
[Table]
Effect of Augmentation Strength in Self-supervised Standard Training.
In Table 9, we provide the effect of data augmentation strength in sup-ST.
It shows that weaker data augmentations will bring worse performance, and our proposed dynamic schedule may also yield performance degradation.
This phenomenon demonstrates the difference between self-ST and self-AT, which is consistent with our discussion in Section 3.2.
D. G ENERALIZATION A BILITY TO O THER M ETHODS
Applicability on Other Self-AT Methods.
In Table 4, we evaluate the effectiveness of our strategy when built upon ACL.
Here, we further apply it to two other self-AT backbones, RoCL (Kim et al.,
[Table]
and AdvCL (Fan et al., 2021).
Results are shown in Table 10.
We can see that our DynACL++ also attains significant improvements in robustness when applied to RoCL and AdvCL.
[Table]
Applicability with Other SSL Frameworks.
We also show that our method can be applied to different contrastive learning frameworks for self-AT, such as the bootstrapping method SimSiam (Chen & He, 2021).
As shown in Table 11(a), DynACL still obtains a significant improvement (4%
AA) over the vanilla ACL method, demonstrating that our dynamic strategy is also effective for other
SSL frameworks.
Applicability with Different Backbone Networks.
We also compare ACL and our DynACL++ on a different network, AlexNet.
From Table 11(b), we can see that our method brings 12% absolute improvements in AA robustness, which is over 100% relative improvement over ACL.
E T HEORETICAL A NALYSIS ON THE E FFECT OF A UGMENTATION S TRENGTH
In Section 3, we have analyzed the effect of augmentation strength on self-AT from an empirical perspective.
In this section, we further provide theoretical justifications on how strong augmentations will bring training-test distribution gap (Section E.1) and smaller classwise distance (Section E.2).
E. E FFECT ON T RAINING - TEST D ISTRIBUTION G AP
In this section, following the theoretical analysis of adversarial training (Sinha et al., 2017), we provide a solid theoretical justification for our analysis.
Specifically, we show that a large training-test distribution gap caused by aggressive data augmentations will induce more significant test errors.
Setup.
Consider an input data space X and a bounded loss function l ( x ; θ ) satisfying | l ( x ; θ ) | ≤
M .
For the ease of theoretical exposure, we consider the Wasserstein distance for measuring the distribution gap.
In particular, the Wasserstein distance between P and Q is
[Equation]
where Π( P, Q ) denotes the couplings of P and Q , and c : X → X defines a non-negative, lower semicontinuous cost function satisfying c ( x, x ) = 0 , ∀ x ∈ X .
Accordingly, we adopt a robust surrogate objective ϕ γ ( θ ; x 0 ) :
= sup x ∈X { ℓ ( θ ; x ) − γc ( x, x 0 ) } , which allows adversarial perturbations of the data x in the Wasserstein space, modulated by the penalty γ .
Considering the equivalence of norms in the finite-dimensional space, our discussion here in the Wasserstein space also sheds light on ℓ p -norm bounded adversarial training.
Suppose the original data distribution (without augmentation) is P test , and the augmented data distribution is P aug .
Denote the empirical augmented training data distribution as ˆ P aug with n samples.
Below, we are ready to establish theoretical guarantees on the out-of-distribution (OOD) generalization of robust training on the augmented data, i.e., ˆ P aug to the original test data distribution ˆ P test .
Theorem 1 For any and a fixed t > 0 , the following inequality holds with probability at least 1 − e − t , simultaneously for all θ ∈ Θ , ρ ≥ 0 , γ ≥ 0 :
[Equation]
Here, ϵ n [Equation]
) := [Equation]
Mn [Equation] tn , where F is the hypothesis class, N ( F , ε, ∥ · ∥ ) is the corresponding cover number, and b 1 and b 2 are numerical constants.
Analysis.
From Theorem 1, we can see that ρ controls the distribution gap between the original data distribution and the augmented data distribution, and a larger ρ indeed introduces a larger error term in the upper bound of the test performance.
Specifically, when ρ is very large, e.g., with aggressive augmentations, the test performance guarantees to be poor.
This shows that theoretically, a too large training-test distribution gap introduced by aggressive augmentation will indeed contribute to worse generalization, which echoes with our empirical investigation in Section 3.
Consequently, if we gradually anneal the augmentation strength s from 1 to 0 , as we have done in the proposed DynACL, the distribution discrepancy ρ will gradually shrink (as it is nearly 0 when s =at last), leading to a smaller upper bound.
Thus, our augmentation annealing strategy will indeed help bridge the distribution gap and improve generalization to test data.
E.1. P ROOF OF T HEOREM
We first state the following lemma from Sinha et al.
(Sinha et al., 2017) that gives a duality result between the objectives on any two distributions P and Q .
Lemma 2 Let ℓ : Θ × X
→ R and c : X × X
→ R + be continuous.
Let ϕ γ ( θ ; z 0 ) =
sup z ∈X { ℓ ( θ ; z ) − γc ( z, z 0 ) } be the robust surrogate (2 b) .
For any distribution Q and any ρ > 0 ,
[Equation]
and for any γ ≥ 0 , we have
[Equation]
Applying Lemma 2 to our problem, we have the following deterministic result
[Equation]
for all ρ > 0 , distributions P aug , and γ ≥ 0 .
Next, we show that E (cid:98) P aug [ ϕ γ ( θ ; X )] concentrates around its population counterpart at the usual rate.
Also remind that we have that ϕ γ ( θ ; z ) ∈ [ − M, M ] , because − M ≤ ℓ ( θ ; x ) ≤ ϕ γ ( θ ; x ) ≤ sup x ℓ ( θ ; x ) ≤ M .
Thus, the functional θ (cid:55)→ F n ( θ ) satisfies bounded differences, and applying standard results on Rademacher complexity and entropy integrals gives the result. E. E FFECT ON C LASSWISE D ISTANCE
Setup.
{+Without loss of generality, we consider a binary classification task ( y = { 0 , 1 } ) with separable data ¯ D 
+}
=
{+{ ¯ x 0 i } from class y = 
+}
and ¯ D 1 = { ¯ x 1 j } from class y = 1 .
Denote the classwise distance as the the minimal ℓ ∞ distance between all cross-class pairs,
[Equation]
Because the data are classiwise spearable, we have d > 0 .
Then, we consider a simple data augmentation scheme, that adds a small noise
δ uniformly drawn from a ℓ ∞ -ball B ε = { δ |∥ δ ∥ ≤ ε } with norm ε to each raw sample ¯ x , resulting in an augmented sample x = ¯ x + δ .
Here, the norm ε controls the augmentation strength: a large ε adds stronger perturbations to the original sample.
{+Theorem 
+}
Denote the classwise distance of the augmentated data with strength ε as d ε , we have
[Equation]
Therefore, as augmentations become stronger (larger ε ), the classwise distance will shrink.
When augmentations become too strong ( ε > d/ 2 ), the classwise distance becomes zero, i.e., d = 0 .
Theorem 3 shows that a strong augmentation strength will indeed cause smaller classwise distance, or even make the data inseparable, which echos our empirical observations in Section 3.1.
E.2. P ROOF OF T HEOREM
Denote a sample pair that attains the smallest classwise distance as (¯ x 0 ∗ , ¯ x
[Equation]
) , i.e
[Equation]
[Equation]
∥ ∞ .
Then, after augmentation, the classwise distance d ε satisfies
[Equation]
[Equation]
[Equation]
[Equation]
where D ε 0 = { ¯ x 0 i + δ | ¯ x 0 i ∈ ¯ D 0 , δ ∈ B ε } and D
ε 1 = { ¯ x 1 j + δ | ¯ x 0 j ∈ ¯ D 1 , δ ∈ B ε } denote the classwise augmented data with augmentation strength ε .