Paper: Going Further: Flatness at the Rescue of Early Stopping for Adversarial Example Transferability (ICLR-2024)

Reviewer comment (Reproducibility):
There are no descriptions of SAM variants (GSAM [Zhuang et al. 2022], ASAM [Kwon et al. 2022], and LookSAM [Liu et al. 2022]) used in their experiments, making their experiments difficult to interpret.

Author response:
**(W1)** Thanks for your suggestions. We will improve the presentation by including formal mathematical formulations of non-robust features, SAM and its variants. Due to lack of space, we moved the description of the five SAM variants to the Appendix. We will do our best to improve their presentation.

Reviewer comment (Theory):
There is no perceived contradiction between the existing hypothesis and the experimental results in Section 3 regarding the relation between the non-robust features hypothesis and early stopping. As far as understood, [Benz et al. 2021] claim robust features transfer better than non-robust features. The authors' experiments show that "early-stopped non-robust features" are more transferable than "fully-trained non-robust features," which is aligned with [Benz et al. 2021] since "early-stopped non-robust features" should be more robust than "fully-trained non-robust features." This discussion requires mathematical formulation to discuss precisely.

Author response:
**(W2)** We respectfully and kindly disagree: our empirical observations in Section 3 are in contradiction with the conclusion of \[Benz et al. 2021\] that "DNNs first mainly learn Robust Features (RFs) and then Non-Robust Features (NRFs)", and that "to get a substitute model with more Robust Features, a straightforward idea inspired by this finding is to train the substitute model with an early stop" (page 8 of \[Benz et al. 2021\]). We agree with you that early learned NRFs are more transferable than fully learned NRFs (our Figure 2, green line). This conclusion conflicts with the explanation of \[Benz et al. 2021\] that early stopped improves transferability due to an inherent trade-off between RFs and NRFs. As a complement, Figure 3 shows that early stopped representations are best to target both NRFs and RFs targets. Considering transferability as a proxy to measure representation similarity, we do not observe such a trade-off between RFs and NRFs, along epochs. We will clarify our position.

Reviewer comment (Novelty):
The fact that the loss-surface sharpness of a surrogate model affects the transferability has already been shown by [Gubri et al. 2022], hence reducing the scientific contribution of this paper.

Author response:
**(W3)** Thanks. We agree that we have the same motivation as \[Gubri et al. 2022\]. Our work complements \[Gubri et al. 2022\] for two reasons.  

1. Contrary to \[Gubri et al. 2022\], we explicitly optimize the sharpness of the surrogate model. The high learning rate used by LGV \[Gubri et al. 2022\] implicitly affects sharpness.
2. \[Gubri et al. 2022\] trains an ensemble of surrogates, whereas we train a single DNN. \[Gubri et al. 2022\] observes that a single LGV model transfers poorly, leaving as an open question whether we can improve the transferability of a single representation using flatness. We address this question.  

We will improve our current discussion of LGV \[Gubri et al. 2022\] in the related work.

Reviewer comment (Novelty):
[Gubri et al. 2022] proposed a method called LGV that aims to enhance the loss-surface flatness of the surrogate model, which is the same motivation as this paper. In fact, in Table 2, the difference between LGV and LGV+l-SAM is minor, making the contribution weak.

Author response:
**(W3)** Thanks. We agree that we have the same motivation as \[Gubri et al. 2022\]. Our work complements \[Gubri et al. 2022\] for two reasons.  

1. Contrary to \[Gubri et al. 2022\], we explicitly optimize the sharpness of the surrogate model. The high learning rate used by LGV \[Gubri et al. 2022\] implicitly affects sharpness.
2. \[Gubri et al. 2022\] trains an ensemble of surrogates, whereas we train a single DNN. \[Gubri et al. 2022\] observes that a single LGV model transfers poorly, leaving as an open question whether we can improve the transferability of a single representation using flatness. We address this question.  

We will improve our current discussion of LGV \[Gubri et al. 2022\] in the related work.

Reviewer comment (Novelty):
The authors should make clear what's the difference between LGV and their approach and their contribution.

Author response:
**(W3)** Thanks. We agree that we have the same motivation as \[Gubri et al. 2022\]. Our work complements \[Gubri et al. 2022\] for two reasons.  

1. Contrary to \[Gubri et al. 2022\], we explicitly optimize the sharpness of the surrogate model. The high learning rate used by LGV \[Gubri et al. 2022\] implicitly affects sharpness.
2. \[Gubri et al. 2022\] trains an ensemble of surrogates, whereas we train a single DNN. \[Gubri et al. 2022\] observes that a single LGV model transfers poorly, leaving as an open question whether we can improve the transferability of a single representation using flatness. We address this question.  

We will improve our current discussion of LGV \[Gubri et al. 2022\] in the related work.

Reviewer comment (Experiments):
It lacks discussion of attacking adversarially trained models, as discussed by [Wu et al. 2020], [Springer et al. 2021]. Is l-SAM useful for attacking adversarially trained models as well?

Author response:
**Q2.** Thanks for your suggestion. We will add the evaluation on adversarially trained target model in the next revision.

Reviewer comment (Evaluation):
What is the loss-surface sharpness value for other methods, such as SAT [Springer et al. 2021]? It is expected that SAT would be sharper than l-SAM based on the paper's claim. If it is, it can strengthen the contribution. If it's not, an explanation is needed for why the SAT is less transferable. In other words, does the sharpness metric correlate well with the transferability?

Author response:
**Q3**. Thanks for the suggestion. We will add the sharpness metrics for SAT and other surrogate training techniques in Table 1 in the next revision. We will discuss the results accordingly.

Reviewer comment (Experiments):
The evaluation experiments are only conducted on standardly trained models. I wonder how l-SAM works when attacking adversarially trained robust models.

Author response:
Thanks for your suggestion. We will add the evaluation on adversarially trained target model in the next revision.

Reviewer comment (Theory):
As acknowledged in Section 2, the underlying mechanism of SAM is still a popular research topic and remains controversial [1, 2]. In particular, it is still uncertain whether the effectiveness of SAM is caused by sharpness. Therefore, I suggest toning down some claims on the proposed method, such as:  
> the effect of early stopping on transferability is closely related to the dynamics of the exploration of the loss surface

Author response:
Thanks for the suggestion. We agree with your view: we also mentioned in the related work that the topic of sharpness is controversial. We will better reflect both sides of the controversy in the rest of the paper, as you suggested.

Reviewer comment (Writing):
In Section 2, I suggest adding more details (e.g. training objective and algorithm) to improve readability, particularly for readers who are not familiar with SAM.

Author response:
Thank you for recommending improving the presentation of SAM. For the sake of space, we moved the illustration of SAM to Appendix E. We will improve the presentation of SAM in the main paper.

Reviewer comment (Novelty):
In the context of adversarial robustness, leveraging SAM is not a completely new idea. While I'm not critiquing the novelty of this paper, I think the following papers [3,4] should be mentioned in the related work:  
a. [3] proposes leveraging SAM to craft adversarial examples to find the common weaknesses of different models, which can boost the transferability of adversarial attacks.  
b. [4] shows that using SAM with a larger $\rho$ (exactly l-SAM in this paper) in standard training can improve adversarial robustness. Additionally, as shown in [5], a little robustness can improve adversarial transferability, so the effectiveness of using SAM in surrogate training is somewhat expected. Please discuss this viewpoint in your revision.

Author response:
Thanks for the relevant references. We already evaluate the technique of \[5\], called SAT (slight adversarial training) in Section 6, and discuss its relation with sharpness in the related work. We will add and discuss \[3\] and \[4\] in the related work, as you suggested.

Reviewer comment (Presentation):
First of all, the teaser figure (Fig. 1) is very confusing. I think the illustration is not clear enough. I cannot extract any information of the "transferrability" other than from the texts. I would recommend you to make the training curve in a color system that brighter colors represent high transferability, right now it seems SGD has the best transferability and the early stopping one has the worst. Also, why is this statistic-based or just sketch map. If it is the latter case, I would recommend the authors to use the former to make this more convincing.

Author response:
Thanks for your suggestion. We will use the suggested color maps to clarify the relation with transferability. Figure 1 is a sketch illustration. This figure represents a schematic basin of attraction, inspired by Kaddour et al. (2022). We will clarify in the legend that this is *not* a loss plot.

Reviewer comment (Theory):
I think both the notion used as well as the literature review of the sharpness for neural network in this work is very limited. First of all, the concept of sharpness has different meanings or evaluation method. Below are a lot of literature that this work missed:

> [1] Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape, AISTATS 2022  
> [2] A modern look at the relationship between sharpness and generalization, ICML 2023  
> [3] On large-batch training for deep learning: Generalization gap and sharp minima, ICLR 2017  
> [4] Fantastic generalization measures and where to find them, ICLR 2020  
> [5] Rethinking parameter counting in deep models: Effective dimensionality revisited (The Hessian-based metrics).

In fact, sharpness even has its own meaning in the context of adversarial training:

> [6] Evaluating and understanding the robustness of adversarial logit pairing.

To be frank, I do not believe the SAM-based sharpness is a very good metric for sharpness, and the authors need to prove it is meaningful in the context of adversarial training.

Author response:
Thanks for the references. We mentioned in our related work that the relationship between sharpness and natural generalization is subject to scientific controversy. We tried to reflect on both sides of the topic. Nevertheless, will improve our coverage of this extensive topic. Thanks for your suggestion.   
   We would like to respectfully correct one point: our surrogate models are **not** trained with adversarial training. We train our surrogate model using SGD or SAM, then we craft adversarial examples against them that we feed to other target models to evaluate transferability. Our work does not consider sharpness in the context of adversarial training.

Reviewer comment (Presentation):
If early stopping improves adversarial example's transferability, I think Figure 2 can be removed or moved to Appendix. Also, please improve the presentation by referring to the figure when making arguments. For example, on page 4 paragraph "Early stopping indeed increases transferability.", there is no figure referred and it is not friendly to readers.

Author response:
We think that there might be a misunderstanding here. The Figure used in the paragraph "Early stopping indeed increases transferability." (page 4) is already in Appendix B, as you suggested. The point of Figure 2 is not to show that early stopping improves transferability overall. Figure 2 provides evidence against the hypothesis currently accepted by the literature that early stopping learns more robust features than non-robust features. This Figure is key to supporting our strength 1 that you recognized as the most valuable of the paper. We will improve the clarity of the captions of Figures 2 and 3.

Reviewer comment (Evaluation):
I think the most significant problem of the evaluation is that, it lacks a serious evaluation on the flatness. For example, the literature [1] provides a list of methods to evaluate the flatness following different metrics. The authors claim better flatness improves tranferability, but forget to measure the flatness. This is not convincing.

Author response:
Thanks for your suggestion. Figure 5 uses two Hessian-based metrics to measure sharpness: the top-1 eigenvalue and the trace. Most importantly, we observe the effect of sharpness on transferability by evaluating optimizers that are known to decrease sharpness: SAM and five of its variants, that minimizes sharpness explicitly and SWA that does it implicitly. We will add other metrics of sharpness in a new revision of the paper.

Reviewer comment (Theory):
The main issue of the paper is the conceptual mismatch: empirical evidence shows that early stopping helps transferability of the attacks, and also that transferability spikes when learning rate decays. Nevertheless, the statement "when LR decays, the sharpness of the parameter space drops" (page 5) is not valid overall. Learning rate does not take part in computation of the sharpness and does not affect it. Smaller learning rate is conjectured to drive models to "sharper" places, but up till now this is only a conjecture. Therefore, the connection from LR experiments to sharpness experiments is not justified.

Author response:
Thanks for pointing out a potential ambiguity. Our central claim is that flat regions of the loss landscape contain better surrogate models. In Section 4, we study the impact of the learning rate on both sharpness and transferability, to provide a more fine-grained analysis of the transferability of early stopping: the transferability along epochs does not evolve smoothly. We uncover the peaks of transferability that happen when the learning decays. To these peaks of transferability correspond peaks of sharpness, suggesting a relationship. We confirm this relationship in Sections 5 and 6. We show that optimizers known to decrease sharpness (SWA, SAM and 5 SAM variants) train better surrogate models. In particular, the large flat neighborhoods found by SAM and its variants, improve transferability further. 

Section 3 challenges the accepted hypothesis that early stopping improves transferability due to a trade-off between robust and non-robust features along epochs. If such a trade-off had applied, we would have observed an “X-shaped” transferability
curves in Figure 2: increasing transferability from non-robust features and strictly decreasing transferability from robust features. Instead, the transferabilities of both robust and non-robust features evolve similarly along epochs. Considering transferability as a proxy to measure representation similarity, we do not observe such a trade-off between robust and non-robust features, since early stopping is best to transfer from and to both robust and non-robust features.

Reviewer comment (Theory):
In the works on adversarial attacks it is very important to describe precisely the attack mode assumed. As understood from the paper, the knowledge of the attacker is assumed to be almost absolute—so they can train a surrogate model with exactly the same architecture, dataset, and even training setup. This significantly weakens the challenge of attacking.

Author response:
Thanks for your question. The threat model used here is standard in the literature on the transferability of adversarial examples. As in most of the related work, we assume that the attacker can train a model on the same training set, but not using the same architecture, nor the same random seed. This is reflected in our experimental settings, since we report the transferability from one architecture to another (e.g., from ResNet-50 to ViT-B-16), trained with different random initialization and random batches. We will write explicitly the threat model considered, and in particular the assumption about the attacker's knowledge.

Reviewer comment (Theory):
The discussion about robust features (RF) and non-robust features (NRF) is very convoluted. According to the definition provided, it should be very hard to create adversarial attacks on the RF, but if they are learned first then it is unclear how early stopping can improve transferability. Alternatively, it could make it harder to create attacks, but they are more universal.

Author response:
Thanks for pointing out a potential ambiguity. Our central claim is that flat regions of the loss landscape contain better surrogate models. In Section 4, we study the impact of the learning rate on both sharpness and transferability, to provide a more fine-grained analysis of the transferability of early stopping: the transferability along epochs does not evolve smoothly. We uncover the peaks of transferability that happen when the learning decays. To these peaks of transferability correspond peaks of sharpness, suggesting a relationship. We confirm this relationship in Sections 5 and 6. We show that optimizers known to decrease sharpness (SWA, SAM and 5 SAM variants) train better surrogate models. In particular, the large flat neighborhoods found by SAM and its variants, improve transferability further. 

Section 3 challenges the accepted hypothesis that early stopping improves transferability due to a trade-off between robust and non-robust features along epochs. If such a trade-off had applied, we would have observed an “X-shaped” transferability
curves in Figure 2: increasing transferability from non-robust features and strictly decreasing transferability from robust features. Instead, the transferabilities of both robust and non-robust features evolve similarly along epochs. Considering transferability as a proxy to measure representation similarity, we do not observe such a trade-off between robust and non-robust features, since early stopping is best to transfer from and to both robust and non-robust features.

Reviewer comment (Theory):
The conclusion on page 4, from the robust and non-robust surrogate models training, seems to be inverted: early learned NRFs would have low transferability (since it grows with training), but early learned RFs might have low transferability this way.

Author response:
Thanks for pointing out a potential ambiguity. Our central claim is that flat regions of the loss landscape contain better surrogate models. In Section 4, we study the impact of the learning rate on both sharpness and transferability, to provide a more fine-grained analysis of the transferability of early stopping: the transferability along epochs does not evolve smoothly. We uncover the peaks of transferability that happen when the learning decays. To these peaks of transferability correspond peaks of sharpness, suggesting a relationship. We confirm this relationship in Sections 5 and 6. We show that optimizers known to decrease sharpness (SWA, SAM and 5 SAM variants) train better surrogate models. In particular, the large flat neighborhoods found by SAM and its variants, improve transferability further. 

Section 3 challenges the accepted hypothesis that early stopping improves transferability due to a trade-off between robust and non-robust features along epochs. If such a trade-off had applied, we would have observed an “X-shaped” transferability
curves in Figure 2: increasing transferability from non-robust features and strictly decreasing transferability from robust features. Instead, the transferabilities of both robust and non-robust features evolve similarly along epochs. Considering transferability as a proxy to measure representation similarity, we do not observe such a trade-off between robust and non-robust features, since early stopping is best to transfer from and to both robust and non-robust features.

Reviewer comment (Evaluation):
The experiment with robust and non-robust surrogate models does not seem to prove that the hypothesis about the sequence of the learning process (model first learns RFs and then NRFs) is wrong. There is still a significant difference in the success rate between robust and non-robust features. The conclusion that can be made is rather that early stopping's effect on transferability is not connected with robustness of features, but not that the models do not learn some features earlier than others and this does not affect transferability.

Author response:
Thanks for pointing out a potential ambiguity. Our central claim is that flat regions of the loss landscape contain better surrogate models. In Section 4, we study the impact of the learning rate on both sharpness and transferability, to provide a more fine-grained analysis of the transferability of early stopping: the transferability along epochs does not evolve smoothly. We uncover the peaks of transferability that happen when the learning decays. To these peaks of transferability correspond peaks of sharpness, suggesting a relationship. We confirm this relationship in Sections 5 and 6. We show that optimizers known to decrease sharpness (SWA, SAM and 5 SAM variants) train better surrogate models. In particular, the large flat neighborhoods found by SAM and its variants, improve transferability further. 

Section 3 challenges the accepted hypothesis that early stopping improves transferability due to a trade-off between robust and non-robust features along epochs. If such a trade-off had applied, we would have observed an “X-shaped” transferability
curves in Figure 2: increasing transferability from non-robust features and strictly decreasing transferability from robust features. Instead, the transferabilities of both robust and non-robust features evolve similarly along epochs. Considering transferability as a proxy to measure representation similarity, we do not observe such a trade-off between robust and non-robust features, since early stopping is best to transfer from and to both robust and non-robust features.

Reviewer comment (Theory):
What is the precise statement that is made by the paper? If it is that transferability of attacks requires very flat-in-the-parameters models, then how do learning rate and exploration of the loss surface connect to this statement? What exactly shows that better transferability in the beginning is not affected by the difference in features learned by the network?

Author response:
Thanks for pointing out a potential ambiguity. Our central claim is that flat regions of the loss landscape contain better surrogate models. In Section 4, we study the impact of the learning rate on both sharpness and transferability, to provide a more fine-grained analysis of the transferability of early stopping: the transferability along epochs does not evolve smoothly. We uncover the peaks of transferability that happen when the learning decays. To these peaks of transferability correspond peaks of sharpness, suggesting a relationship. We confirm this relationship in Sections 5 and 6. We show that optimizers known to decrease sharpness (SWA, SAM and 5 SAM variants) train better surrogate models. In particular, the large flat neighborhoods found by SAM and its variants, improve transferability further. 

Section 3 challenges the accepted hypothesis that early stopping improves transferability due to a trade-off between robust and non-robust features along epochs. If such a trade-off had applied, we would have observed an “X-shaped” transferability
curves in Figure 2: increasing transferability from non-robust features and strictly decreasing transferability from robust features. Instead, the transferabilities of both robust and non-robust features evolve similarly along epochs. Considering transferability as a proxy to measure representation similarity, we do not observe such a trade-off between robust and non-robust features, since early stopping is best to transfer from and to both robust and non-robust features.

Reviewer comment (Theory):
What is the attack model that is assumed in the paper?

Author response:
Thanks for your question. The threat model used here is standard in the literature on the transferability of adversarial examples. As in most of the related work, we assume that the attacker can train a model on the same training set, but not using the same architecture, nor the same random seed. This is reflected in our experimental settings, since we report the transferability from one architecture to another (e.g., from ResNet-50 to ViT-B-16), trained with different random initialization and random batches. We will write explicitly the threat model considered, and in particular the assumption about the attacker's knowledge.