Paper: On Adversarial Training without Perturbing all Examples (ICLR-2024)

Reviewer comment (Evaluation):
This paper mainly contains various experiments and their results, but lack the important analysis. Specifically, there are no analysis of the transferability observed from CSAT and ESAT. I cannot get any insightful information after reading this paper, although the results are informative.

Author response:
1. **no analysis:**
We agree with the reviewer, that an additional analysis regarding the 'why' would be of interest. At this point, we cannot offer one but refer the reviewer to their own statement that the "*results are informative*". We also believe that our results are informative and are likely to spur follow up work. We believe this is a strong reason to publish this work. 
We restate the contribution of our paper: in contrast to related work on robustness and its transfer capabilities (as discussed in related work and our method section), we conduct a systematic analysis where we perturb only a fixed subset of training examples. This reveals a surprising phenomenon: adv. training a single class, transfers robustness to all other classes above non-trivial levels. Moreover, the hardest class provides the best transfer. As a consequence, we observe that just perturbing $50$% of training examples is sufficient to reach baseline AT performance and only $30$% of examples is sufficient in the task transfer setting.

Reviewer comment (Theory):
There is no theoretical analysis to rethink the observation as a special property of adversarial training. Additionally, from Figure 4, we can find that the clean accuracy has a similar tendency as the robust accuracy, therefore, it is possible that CSAT and ESAT are just because of the generalizability of the deep learning models.

Author response:
a) **no theory:** While additional theory is useful, we think that a lack of it does not reduce the importance of our work. Our work describes a phenomenon of adversarial training --few attacked examples/classes transfer robustness surprisingly well-- comprehensively.
    This is confirmed by the reviewer itself and also R3h75, RgZFF and RQzqJ.
  
    b) **effect is just side-effect of generalizability:** We thank the reviewer for a highly relevant question that is addressed by providing additional figures in the new paper revision reporting the defense rate -- robustness independent of clean accuracy. While the trend between clean and robust accuracy seems highly correlative, in section A.6 and figure 17 in the appendix, we show that the increase in robustness is not due to clean accuracy alone. By providing robustness on accurately classified examples only, we remove the influence of clean accuracy entirely. We can see that robustness transfer is best when SAT is performed using the hardest examples.

Reviewer comment (Experiments):
Only PGD-AT is considered. More advanced methods, like TRADES and AWP, should be evaluated under the same settings.

Author response:
3. **no TRADES and AWP evaluation:** We agree with the reviewer that integration of TRADES is especially interesting in our setting, since it provides a means to trade-off clean with robust accuracy. However, we highlight that selecting the best trade-off between clean and robust accuracy with TRADES does not alter the main findings of our paper: adv. training on few examples/classes transfers robustness surprisingly well. Moreover, adversarial weight perturbation would be interesting to investigate, yet we believe that it goes beyond the scope of this work. Regarding TRADES, we have added results for CIFAR-10 for CSAT $L_2 \epsilon=0.5$ $|A|=1$ and the general case. For $|A|=1$ we conducted a coarse hyperparameter search for TRADES $\beta$ parameter and found $\beta=12.0$ to provide good robustness transfer (see table below). This configuration achieves $42.9$% $(+10.7)$ robust accuracy on $B$ while also increasing rob. acc. on $A$ to $78.2$% $(+28.6)$, with the expected decrease in overall clean accuracy ($81.0$% $(-10.2)$).

|$A=\{cat\}$|$\beta$|clean acc on CIFAR-10|rob. acc on $A$|rob. acc on $B$|
|--|--|--|--|--|
|w/o TRADES|N/A|91.0|49.6|37.8|
|w/ TRADES|1.0|92.4|41.5|32.0|
||6.0|85.9|70.2|42.1|
||12.0|**81.0**|**78.2**|**42.9**|
||24.0|81.6|80.1|41.8|

In a second experiment, we use linearly decreasing $\beta$, from $12$ down to $6$, for our ESAT $L_2 \epsilon=0.5$ experiment on CIFAR-10, corresponding to figure 6.
That is, when $|A|$ is small, we trade off some of the gained clean accuracy for increased robustness with large $\beta=12$, but decrease to $\beta=6$ with increasing size of $|A|$. We choose $\beta=6.0$ for $|A|=9$ based on the same parameter choice for the full CIFAR-10 set in (Zhang et al. 2019). We compare *ESAT-hardest first* with and without TRADES in the table below.
We observe quicker robust accuracy convergence to the baseline ($66$% robust accuracy is achieved at $20k$ samples, instead of at $25k$) with an expected drop in clean accuracy below full AT baseline clean accuracy ($85.6$% clean accuracy at $20k$ vs $89.7$% at $25k$ vs $89.2$% for full AT).


|TRADES||5k|10k|15k|20k|25k|30k|35k|40k|45k|full AT|
|--|--|--|--|--|--|--|--|--|--|--|--|
|yes|Rob. acc|57.2|62.1|65.7|66.1|67.7|67.6|68.2|66.4|69.0|69.0|
||Clean acc|87.0|87.4|84.2|85.6|84.3|87.0|84.0|88.0|88.4|88.6|
|no|Rob. acc|52.1|59.1|63.3|64.9|66.3|66.8|67.7|68.7|68.5|69.0|
||Clean acc|91.9|91.2|90.5|89.7|89.4|89.4|89.2|89.4|89.5|89.2|

Reviewer comment (Novelty):
For downstream task transferability, it is similar, but not exactly the same, as contrastive adversarial training. The authors should discuss the similarity and difference between these two methods in this case.

Author response:
4. **what is difference to contrastive AT?:** We base our discussion around (Kim et al.), in which the authors provide a self-supervised adversarial training method, which does not require explicit labels. We highlight, that our proposed analysis is complementary to this work. (Kim et al.) perturb *all* training examples, similar to the vanilla AT approach. Ours, instead, investigates the impact of robustness transfer when we perturb only a *fixed subset*. Our SAT analysis is therefore applicable to the proposed contrastive learning approach.

    Kim, Minseon, Jihoon Tack, and Sung Ju Hwang. "Adversarial self-supervised contrastive learning." Advances in Neural Information Processing Systems 33 (2020).

Reviewer comment (Presentation):
Some figures' labels are blocked.

Author response:
5. **some figures' labels are blocked:** We thank the reviewer for raising this issue. We have fixed the blocked labels in all figures.

Reviewer comment (Experiments):
I notice that the authors use stronger data augmentation than usual. For example, when training on CIFAR-10, we usually only use random crop and flip. I hope the authors can provide ablation studies on these data augmentation methods.

Author response:
6. **data augmentation different to related work:** We adopted the adversarial training methodology from the GitHub repository *github.com/MadryLab/robustness*. Consequently, we use the same data augmentation as they did. That is, on CIFAR-10, the data augmentation involves cropping, flipping, color jittering and rotating (for details see *github.com/MadryLab/robustness/blob/master/robustness/data_augmentation.py*).
Irrespectively, we repeated our CSAT $|A|=1$ experiments with the reviewers suggested data augmentation pipeline: only performing cropping and flipping. Results, reported in the table below, show similar results to our reported in our submission.
We conclude that the data augmentation is not the reason for our observed robustness transfer.

|data augmentation|class in $A$ | clean acc on CIFAR-10|rob. acc. on $A$|rob. acc. on $B$|
|--|--|--|--|--|
|crop,flip|plane|88.6|88.8|26.7|
||car|89.2|95.4|15.4|
||bird|84.1|80.6|34.4|
||cat|91.0|50.4|39.9|
||deer|84.6|85.5|34.5|
||dog|91.5|61.2|35.0|
||frog|86.8|90.2|28.3|
||horse|88.2|90.2|27.6|
||ship|89.0|91.2|24.0|
||truck|91.9|87.0|20.3|
||
|crop,flip,jitter,rotate|plane|87.2|89.4|25.7|
||car|89.8|93.5|17.1|
||bird|84.4|80.4|34.3|
||cat|91.0|49.6|37.8|
||deer|85.4|84.0|34.2|
||dog|91.5|60.9|34.9|
||frog|85.7|92.7|28.4|
||horse|88.1|88.6|27.3|
||ship|89.0|91.8|24.5|
||truck|87.6|92.3|21.9|

Reviewer comment (Theory):
While this paper presents intriguing empirical results on the SAT approach, it falls short of providing a clear explanation for the observed transferability of adversarial robustness from subset A to subset B.

Author response:
**no (theoretical) explanation for the observed transferability:**
While we agree with the reviewer, that additional theoretical or empirical based explanations would be beneficial, we believe that our paper has a valuable contribution to the scientific community. We highlight a surprising phenomenon and counter-intuitive results -- e.g. (i) CIFAR-10 SAT on *cat* provides better robust transfer to *car* than *truck* on *car* or (ii) $30$% of training examples achieve baseline robust accuracies in the task transfer setting -- that could lead to new research directions.

Reviewer comment (Theory):
Can you provide more theoretical justification or intuitive explanations for the observed efficacy of constructing AEs on only a subset of the training data? Specifically, what underpins the phenomenon where harder examples offer better robustness transfer?

Author response:
**theoretical justification for why harder examples transfer robustness better:** At this point, we cannot provide an underlying theory. We speculate however, that classifying hard examples requires the detection of more diverse features, which are trained adv. robust. Given the compositional nature of deep networks, it is plausible to assume that these robust features are reused by other classes.

Reviewer comment (Evaluation):
You mentioned that as dataset complexity increases, the trend of harder examples providing better robustness transfer diminishes. Can you explain the reasons behind this observation? Are there specific characteristics or properties of complex datasets that might be influencing this behavior?

Author response:
**reasons behind diminishing returns when dataset complexity increases:** We conjecture this to be similar to our point made for the previous question. Datasets with a larger number of classes and larger input images likely require the detection of more diverse features.

Reviewer comment (Evaluation):
Could you explain more on why the robustness transfer notably increases for smaller $\epsilon$ and decreases for larger $\epsilon$?

Author response:
**why does robustness transfer increase for smaller $\epsilon$?:** We conjecture, that with increasing $\epsilon$ each feature becomes more class-specific and is thus less reusable. In contrast, with decreasing $\epsilon$ the generalizability is improved.

Reviewer comment (Novelty):
I have a single but potentially critical concern regarding the significance of this work. In particular, I am unsure what are the benefits of considering a subset of the adversarial examples during adversarial training to improve the efficiency of the process.

Author response:
1. **what are the benefits of considering a subset:?** 
    
    The benefits of considering a subset are twofold: (a) practical, and (b) analytical. 
    
    **Practical:** We observe significant savings in terms of training cost with our protocol. For example, training our ResNet-18 models on CIFAR-10 takes approx. 3h without any adversarial perturbations. Full adversarial training takes about 7h, and only perturbing 50\% of the data results in a training time of 5h. Thus  reducing the cost of training considerably. These savings are expected to be even larger at scale, that is when training large models on large datasets. E.g. consider the power consumption of training foundational models, which consume hundreds of Mega-Watt-hours without adversarial training (e.g. nearly 500MWh for LLaMa-65B [LLama]). Adding adversarial training roughly increases the training time -- and thus the power consumption -- by a factor of PGD-steps. Our insights suggest that training time/consumption could be at least halved.


    **Analytical:**
    While it is true that "the proposed method does not lead to better robust accuracy than full adversarial training", we show that you can trade-off a minor loss in robust accuracy against significant savings in training time -- by not perturbing all classes/examples. However, we also believe that this loss in robust accuracy should motivate further research into more effective defenses against adversarial attacks: Why **do** we need to perturb all classes/examples to achieve the full potential of adversarial training? Can we train models to be robust in a class-agnostic manner? These are important questions raised by our work, which we believe to be of interest to the scientific community.
    
    [LLama] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

Reviewer comment (Experiments):
As I understand, the key objective of this paper is to limit the size of the set of examples used for adversarial training, to achieve a similar robust accuracy than with full adversarial training. The reason is that full adversarial training is computationally expensive. Considering the settings described in the experimental protocol, I fail to understand why the proposed process is more efficient than full adversarial training. In both cases, the full datasets need to be labeled and the same amount of computational resources is needed, as only the set of available examples for adversarial training differs, not the total number of examples used during adversarial training (since the subset of adversarial examples is completed with the clean examples not used for generating adversarial examples). One could argue that we save 70% of the time to generate the adversarial examples, but the largest cost still comes from model training.

Author response:
1. **what are the benefits of considering a subset:?** 
    
    The benefits of considering a subset are twofold: (a) practical, and (b) analytical. 
    
    **Practical:** We observe significant savings in terms of training cost with our protocol. For example, training our ResNet-18 models on CIFAR-10 takes approx. 3h without any adversarial perturbations. Full adversarial training takes about 7h, and only perturbing 50\% of the data results in a training time of 5h. Thus  reducing the cost of training considerably. These savings are expected to be even larger at scale, that is when training large models on large datasets. E.g. consider the power consumption of training foundational models, which consume hundreds of Mega-Watt-hours without adversarial training (e.g. nearly 500MWh for LLaMa-65B [LLama]). Adding adversarial training roughly increases the training time -- and thus the power consumption -- by a factor of PGD-steps. Our insights suggest that training time/consumption could be at least halved.


    **Analytical:**
    While it is true that "the proposed method does not lead to better robust accuracy than full adversarial training", we show that you can trade-off a minor loss in robust accuracy against significant savings in training time -- by not perturbing all classes/examples. However, we also believe that this loss in robust accuracy should motivate further research into more effective defenses against adversarial attacks: Why **do** we need to perturb all classes/examples to achieve the full potential of adversarial training? Can we train models to be robust in a class-agnostic manner? These are important questions raised by our work, which we believe to be of interest to the scientific community.
    
    [LLama] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

Reviewer comment (Evaluation):
Considering the empirical results that demonstrate that the proposed method does not lead to better robust accuracy than full adversarial training, I do not see the added value for model robustness efficiency/effectiveness. It would be good if the authors could clarify the benefits of using only a subset of examples in the adversarial training process.

Author response:
Dear reviewer, thank you for the response.

We would like to emphasize that our paper should not **exclusively** be read as providing “guidelines“ for more efficient adversarial training. As described earlier in this thread, there is also an analytical component to our paper centered on the question: Do we need to adversarially perturb images from every class to attain good robustness, and what are the implications? 

There is a “glass half full” answer that we back up with our experiments, namely that one can get away with a minor loss of robustness by **not** perturbing examples from every class. This has positive practical implications that we have discussed at length.

There is however also an interesting “glass half empty” answer, namely that to achieve the full potential of adversarial training in terms of robustness, it would seem that one **indeed** has to perturb representative images from every class, but also that the choice of classes is inferior when random. This has all kinds of interesting implications and suggests further questions and directions for research, e.g. to name a few: While we provide some measures that correlate with robustness transfer, what really are the underlying mechanisms? What theoretical tools are needed to distinguish between individual object and scene classes in terms of their visual characteristics? What does this say about adversarial training as a general framework if robustness ultimately has to be learned for every class? Can we be satisfied with visual recognition methods that must gain robustness in this manner? Is this scalable?

Note: Regarding the fast adversarial methods (esp Wong et al 2020) our initial experiments show that these may well be complementary but we acknowledge that further investigation is warranted.

Reviewer comment (Reproducibility):
What is the exact process of adversarial training used in the experiments? Are all examples adversarially perturbed? Are some examples perturbed and others clean? In what proportion? In total is the number of perturbation executions the same for adversarial training as subset adversarial training?

Author response:
2. **adversarial training protocol:** For vanilla adversarial training, we perturb all training examples. There is no mix between clean and perturbed examples in a batch as described in earlier work on adv. training. For SAT on the other hand, we randomly sample a training batch and perturb only those examples that are contained in $A$. We have updated the appendix, section A.1 in our manuscript to make this clear.

Reviewer comment (Experiments):
What is the objective of transferring the robustness across classes and tasks? We need to retrain models to adapt to these new classes and tasks; isn’t it simpler to adversarial train them at retraining, therefore empirically obtaining a more robust model?

Author response:
3. **during task transfer, why not retrain with full adv. training?** Our argument is similar to question 1. We see an increase in foundational models, which provide strong off-the-shelf performance which can be improved via fine-tuning. Given the computational demand that is required to train such large models, it is often sufficient to only train a shallow classifier on top of intermediate outputs.
In this case, adversarial robustness can only be provided when the foundational model is robust.
We have shown that it is sufficient to perturb only $30\%$ of examples during the initial (foundational) training to achieve high adversarial robustness after fine-tuning.

Reviewer comment (Presentation):
On page 6, lines 6 and 11, of the paragraph left to fig 3, describe the results for the class “car” while the numerical value corresponds to the line “plane” on fig 2.

Author response:
4. **mismatched reporting on page 6:** Thank you for highlighting this issue. We have fixed it in the new revision. Note that, now, the numerical values are actually better than what was reported in our initial text.

Reviewer comment (Presentation):
Figures are not readable when printed.

Author response:
5. **unreadable figures:** Thank you for highlighting a readability issue. We will improve the readability for the final revision.