Paper: PubDef: Defending Against Transfer Attacks From Public Models (ICLR-2024)

Reviewer comment (Evaluation):
No analysis on how expensive the method is, compared, e.g., to adversarial training.

Author response:
**PubDef is about 2$\times$ less expensive than white-box adversarial training.** The training time of PubDef is approximately $16T + 0.2NT$ where $T$ is the time for one epoch of white-box adversarial training and $N$ is the number of epochs of training. In comparison, adversarial training takes $NT$ time. For $N=50$ epochs, PubDef's training time is $26T$ vs $50T$ for adversarial training. 

The first term in the formula ($16T$) is a one-time cost for generating all the transfer attacks used in training: 4 source models * 4 instances per training sample * 10 PGD steps = 16T. Regardless of how many PubDef models are trained or how many hyperparameters are swept, this step has to be done once. The second term is the cost of the training loop which depends on the exact loss function used (Section 5.1). For the “Random” loss function, the cost is 0.2T per epoch.

Reviewer comment (Experiments):
The defense should be tested fully, also against white-box evasion attacks. This is motivated by the fact that the authors are trying to make the model robust, hence they should demonstrate the robustness against worst-case adversaries. Otherwise, there is a high risk that the defense might be broken by stronger attacks. As the approach is similar to AT, the authors should demonstrate that the decision boundary of the model is in fact becoming more robust to unseen adversarial attacks and perturbations, thus they should also test gradient-based attacks against the model itself.

Author response:
There appears to be a misunderstanding. The paper makes no claims about the robustness of PubDef against white-box attacks nor about the decision boundary. Against white-box attacks, PubDef is no better than that of an undefended model (0% accuracy against white-box attacks).  PubDef is not designed to provide security against white-box attacks and is not appropriate in scenarios where white-box attacks are possible, so such attacks are beyond the scope of this paper. We do not claim "robustness" in general; rather, we claim robustness against TAPM attacks. Achieving robustness against all white-box attacks, without significant loss of clean accuracy, is an open problem that has yet to be achieved despite concerted effort from the community.

We would like to re-emphasize that a core argument of our paper is that **it is not necessary in practice to make every model robust against white-box attacks.** Instead, the defender should identify the most likely threats to their system and focus on those. If the model has to be completely public, then robustness against white-box attacks is necessary. Otherwise, using white-box adversarial training is overkill, like “using a sledgehammer to crack a nut.” 

We believe that a more common situation is that the model does not need to be public. In that case, we can use systems-level defenses to make white-box attacks, query-based attacks, and transfer attacks from non-public models unattractive to an attacker, so that the top priority is to provide robustness against TAPM attacks; and we show that we are able to achieve a surprising level of robustness against such attacks, with low impact on clean accuracy. This is, as far as we are aware, a new insight, and it has significant implications for how to defend models in practice.

Reviewer comment (Experiments):
The authors should test against an attack that is generated ensembling the gradients of multiple models, which should improve transferability and also test effectively the robustness claim that the authors are making.

Author response:
We already evaluated PubDef against transfer attacks generated with gradients from an ensemble of multiple models. We gather three different ensembles and use all the 11 attack algorithms. Please see “Attacks from ensembles of the public source models” in Section 7.1 for more details. To summarize, the best attack from the ensembles turns out to be weaker than the best attack from a single model (92% vs 89% adversarial accuracy on PubDef on CIFAR-10).

Reviewer comment (Experiments):
The model should be tested with variations of the attacks, e.g., with PGD with logit loss, or against APGD with DLR loss. The authors should demonstrate that the defense generalizes against other unseen attacks and variations to really claim robustness. The defense should be robust regardless of the method used to find the adversarial attacks.

Author response:
There is already extensive research literature on variations of transfer attacks, and we experimented with 11 state-of-the-art schemes in the literature (which also includes APGD with DLR loss). We ensure that they are diverse by choosing ones that employ different mechanisms for improving transferability. While it is always possible to try even more variations, it is not our intention to innovate in transfer attacks, nor do we believe it to be necessary in our case. We also found that our defense generalizes well to “unseen” transfer attack algorithms that it wasn't trained against. See Sections 6.2 and A.5.2 (supplementary material).

Reviewer comment (Reproducibility):
All the parameters of the attacks should be specified to ensure they are conducted properly and they are not suffering from optimization issues.

Author response:
Please see Section A.1.2 (supplementary material). For evaluation, we set the common attack parameters as follows: step size of 0.001, 100 steps, and 3 random restarts. To generate adversarial examples for training, we use 10-step PGD with step size $\epsilon/4$ and 1 restart. We note that, unlike white-box adversarial training, the transfer attack has to be generated only once outside of the training loop (see the cost of training section below). For the attack-specific parameters, we follow the existing implementation (either an official one or one from [this survey paper](https://github.com/ZhengyuZhao/TransferAttackEval)) to ensure that the parameter choices are reasonable and already tested. We will include this additional detail in the appendix.

Reviewer comment (Evaluation):
The authors should discuss how expensive it is to train a model with this technique, compared, e.g., with AT. Additionally, they should specify how costly it is to add further heuristics.

Author response:
**PubDef is about 2$\times$ less expensive than white-box adversarial training.** The training time of PubDef is approximately $16T + 0.2NT$ where $T$ is the time for one epoch of white-box adversarial training and $N$ is the number of epochs of training. In comparison, adversarial training takes $NT$ time. For $N=50$ epochs, PubDef's training time is $26T$ vs $50T$ for adversarial training. 

The first term in the formula ($16T$) is a one-time cost for generating all the transfer attacks used in training: 4 source models * 4 instances per training sample * 10 PGD steps = 16T. Regardless of how many PubDef models are trained or how many hyperparameters are swept, this step has to be done once. The second term is the cost of the training loop which depends on the exact loss function used (Section 5.1). For the “Random” loss function, the cost is 0.2T per epoch.

Reviewer comment (Experiments):
The authors use a heuristic to choose the models to use for creating the transfer attacks. However, this heuristic seems expensive, as it requires computing the adversarial accuracy (in transfer) against all available models. The authors should clarify whether this requires launching a full evaluation against all models.

Author response:
Computing the adversarial accuracy is cheap, compared to the cost of PubDef training, because it suffices to evaluate the accuracy on only a small random subset of the validation set. In the paper, we report results on the entire validation set, but in practice, it would suffice to use a subset of the validation set.

Reviewer comment (Writing):
The authors state that they propose "using all available system-level defenses", however the presented method is clearly a ML-level defense. The authors should clarify this aspect, as it is confusing to read and there is no discussion in the rest of the paper on which system-level defenses are mounted on the model, or if they are mounted at all.

Author response:
We will revise the paper accordingly. We propose that PubDef's training method (an ML-level defense) be combined with all relevant systems-level defenses (as was stated in Section 1, "We propose using...") to protect against attackers under all threat models without a loss in the model utility. The specific systems-level defenses are listed in the last sentence of Section 2. See also assumption 3 in Section 3 and point 3 in Section 7.3.

Reviewer comment (Evaluation):
The authors state that the drawback of AT is that it degrades clean accuracy, however this does not seem to be the case according to https://robustbench.github.io. The authors should clarify this aspect by detailing this statement.

Author response:
Yes, adversarial training does degrade clean accuracy [1]. For instance, for ImageNet (with no additional training data), a ResNet-50 can achieve 79-80% clean accuracy. In comparison, the best adversarial training recipe for ResNet-50 achieves 63-64% clean accuracy [2]. Similarly, for CIFAR-10, undefended WRN-34-10 models achieve about 96% clean accuracy but adversarial training achieves about 85% clean accuracy [3] (when reviewing RobustBench, make sure to compare only to schemes with no extra training data or synthetic images for a fair comparison).

Reviewer comment (Theory):
The method assumes that the defender is aware of the same set of public models as the attacker. However, given the proliferation of public models (also acknowledged in the introduction of the paper), it might not be realistic to assume that the defender is aware of all possible models available. Furthermore, assuming that the attacker has $s \dot a$ attack strategies might lead to an excessive number of combinations. The authors should clarify these statements and propose solutions for these limitations.

Author response:
**The paper shows that not much robustness is lost if a defender is unaware of some public models that the attacker is aware of.** This surprising effectiveness or generalization is a result of our technique to choose one source model from each group. Please see the discussion of unseen source models in Sections 6.2, 7.2, Table 2, Figure 4. To summarize, the accuracy under attack drops by less than 2 percentage points on CIFAR-10/100 and by 8 points on ImageNet under unseen attacks (unseen attack algorithm and unseen source models). For any other table or figure, we always report the accuracy against the strongest attack including both seen and unseen. In practice, we believe it is likely that defenders can identify most or all publicly available models.

That said, we have conducted an ablation study where **the defender only sees one random public model from each group.** See Table 9 in Appendix A.4.1. Here, PubDef is still significantly better than all the baseline models; on CIFAR-10, it has 10 and 9 percentage points higher clean and adversarial accuracy (7 and 11 pp. for CIFAR-100) than the best white-box adversarially trained models. We note that choosing randomly is completely unrealistic and is the worst possible strategy for the defender. There is no reason to believe that the defender would just randomly choose the source models to train against, so we expect that in practice the defense will perform significantly better than this, even if some models are not known to the defender.