Paper: Towards Reliable Backdoor Attacks on Vision Transformers (ICLR-2024)

Reviewer comment (Experiments):
When testing existing backdoor attacks against ViT, the authors only use CNN-based backdoor attacks without ViT-specific backdoor attack methods. Thus, the possibility exists that existing ViT-specific backdoor attacks can also evade well-adapted backdoor defenses.

Author response:
**Q1:**  Thus, the possibility exists that existing ViT-specific backdoor attacks can also evade well-adapted backdoor defenses.

**A1:** TrojViT [1] and DBIA [2] are the existing two kinds of ViT-specific attacks. **In Table 5 and Table 10 in the paper, we have shown that both of them fail to evade well-adapted backdoor defenses**: they only achieve a very low ASR (<1%) across all model-agnostic defense approaches while the benign accuracy maintains a relatively high level (>75%). Here we further illuminate that our proposed adapted approaches remain essential for better defending TrojViT and DBIA. The experiments are performed on ImageNet. The ASR and ACC before and after adaptations are shown as follows:

**ASR** (%)

TrojViT:

|                    | Before | FT   | FP   | NAD  | ANP  | AWM  |
| ------------------ | ------ | ---- | ---- | ---- | ---- | ---- |
| Before adaptations | 91.58  | 0.16 | 0.19 | 0.19 | 0.29 | 0.00 |
| After adaptations  | 91.58  | 0.14 | 0.14 | 0.16 | 0.46 | 0.18 |

DBIA:

|                    | Before | FT   | FP   | NAD  | ANP  | AWM  |
| ------------------ | ------ | ---- | ---- | ---- | ---- | ---- |
| Before adaptations | 99.58  | 0.07 | 0.10 | 0.24 | 0.13 | 0.06 |
| After adaptations  | 99.58  | 0.09 | 0.07 | 0.10 | 0.10 | 0.05 |

**ACC** (%)

TrojViT:

|                    | Before | FT        | FP        | NAD       | ANP       | AWM       |
| ------------------ | ------ | --------- | --------- | --------- | --------- | --------- |
| Before adaptations | 80.59  | 68.92     | 69.00     | 69.88     | 69.72     | 0.10      |
| After adaptations  | 80.59  | **76.82** | **76.93** | **77.55** | **76.31** | **77.18** |

DBIA:

|                    | Before | FT        | FP        | NAD       | ANP       | AWM       |
| ------------------ | ------ | --------- | --------- | --------- | --------- | --------- |
| Before adaptations | 79.52  | 69.36     | 64.05     | 67.51     | 71.04     | 57.00     |
| After adaptations  | 79.52  | **78.30** | **75.20** | **77.18** | **76.49** | **78.94** |

The results demonstrate that although vanilla settings can successfully decrease the ASR, they also impair the benign function of the backdoored models a lot. In contrast, our proposed adaptations better preserve the benign ACC. For example,  AWM achieves the ACC of 77.18% after adaptations against TrojViT while it totally collapses without any adaptations.

Reviewer comment (Experiments):
Lack of enough baselines to prove the effectiveness of the proposed attack method. After proposing a new backdoor attack, the authors should compare it with SOTA backdoor attacks, especially advanced ViT-specific backdoor attacks, to show its superiority.

Author response:
**Q3:** After proposing a new backdoor attack, the authors should compare it with SOTA backdoor attacks, especially advanced ViT-specific backdoor attacks, to show its superiority.

**A3:**  **In fact, we have compared CAT with two ViT-specific attacks: TrojViT [1] and DBIA [2] in Table 5 and 10 in the paper.**  We summarize the average ASR of all attack methods after performing five model-agnostic defenses in the following table.

|                | DBIA | TrojViT | badnets+CAT | blend+CAT |
| -------------- | ---- | ------- | ----------- | --------- |
| Average ASR(%) | 0.08 | 0.21    | 44.12       | 32.84     |

The results demonstrate that current ViT-specific attacks can not be immune to the adapted defenses and  CAT can outperform them with a higher ASR.

Reviewer comment (Evaluation):
There is insufficient evaluation to explore whether the proposed attack can evade the SOTA backdoor defenses designed for ViT.

Author response:
**Q4:** There is insufficient evaluation to explore whether the proposed attack can evade the SOTA backdoor defenses designed for ViT.

**A4:** **In Table 5 and Table 10 in the paper, we have shown that CAT can evade the AB (Attention block) [5] which is one of the ViT-specific backdoor defenses.** Here we further perform experiments on patch-processing [6] which is another ViT-specific defense.  Following their original paper, we select TPR (the proportion of benign samples that are successfully identified) and TNR (the proportion of backdoor samples that are successfully detected) as our metrics. The experiments are performed on ImageNet with ViT-B architecture. 

|           | TPR   | TNR   |
| --------- | ----- | ----- |
| Blend     | 80.12 | 13.03 |
| Blend+CAT | 81.09 | **5.15**  |

Compared to the model-agnostic defense in Table 5, patch-processing only obtains limited performance on both Blend and Blend+CAT. We speculate this is because ImageNet is a large and complex dataset. It makes the classification results more susceptible to change if certain patches of the image are masked, thereby increasing the difficulty of distinguishing between clean and backdoor images. In addition, combining with CAT helps Blend obtain lower TNR, representing the backdoor samples are harder to detect. We also perform experiments on CIFAR-10 to further investigate whether CAT can provide better robustness on small datasets. The experiments are also performed on ViT-B:

|           | TPR   | TNR   |
| --------- | ----- | ----- |
| Blend     | 83.08 | 97.11 |
| Blend+CAT | 84.11 | **48.69** |

Similar to the experiments on Imagenet, combining with CAT can obtain comparable TPR and lower TNR (<50%). This demonstrates that CAT can evade the SOTA backdoor defenses designed for ViT.

Reviewer comment (Experiments):
There is a lack of enough complex datasets, such as Imagenet, to evaluate the effectiveness of the proposed attacks.

Author response:
**Q5:** There is a lack of enough complex datasets, such as ImageNet, to evaluate the effectiveness of the proposed attacks. 

**A5:** **We actually perform experiments on ImageNet in Section 5.2 in the paper.** The results show that our proposed attacks are still effective on large datasets.

Reviewer comment (Experiments):
Study of backdoor attack with only CIFAR10 and single vision transformer architecture is not convincing, and any conclusion based on these limited settings won’t be accurate. Note that study of backdoor attack on the Vision Transformer has been conducted before.

Author response:
Firstly, we would like to reclaim our contributions compared to previous works [1-4] studying the backdoor robustness for ViTs:

- **Backdoor Attack:** Zheng et al. in [1] and Lv et al. in [2] propose two ViT-specific attacks respectively: TrojViT and DBIA. However, our works show that both of them fail to evade the backdoor defenses after adaptations. **Only our proposed CAT can maintain a high ASR.**
- **Backdoor Defense:** In [3], they first apply model-agnostic backdoor defense on ViTs. However, due to the lack of any adaptation, they either significantly reduce ACC or are unable to reduce ASR. Therefore, they conclude that: the effective defense that has been verified on the CNN architecture may not be suitable for the ViT architecture. In contrast to their conclusions, **we find that with proper adaptation, current defense methods are still demonstrated to be effective on ViTs.** Subramanya et al. in [4] propose a ViT-specific defense: AB (Attention Blocking). We show that **CAT can improve the ASR even for ViT-specific defense like AB.**

Furthermore, to illustrate that our conclusion is general and convincing, we conduct experiments on the CIFAR-100 dataset with the ViT-B and Swin-B architectures as follows:

**Comparison of Optimizers**

We perform FT using SGD and AdamW optimizer, respectively in the following tables:

ViT-B

|        | Before | SGD   | AdamW     |
| ------ | ------ | ----- | --------- |
| ACC(%) | 89.54  | 42.76 | **69.12** |
| ASR(%) | 100.00 | 1.37  | 2.31      |

Swin-B

|        | Before | SGD  | AdamW     |
| ------ | ------ | ---- | --------- |
| ACC(%) | 92.24  | 2.81 | **78.56** |
| ASR(%) | 100    | 0    | 1.39      |

While the SGD optimizer can reduce the ASR to almost zero, it will significantly sacrifice the benign ACC, which makes the purified models unusable. By contrast, the ACC for fine-tuning using AdamW (69.12%) is much higher than the ACC for fine-tuning using SGD (42.76%). This observation is consistent with our findings on the CIFAR-10 dataset.

**Comparison of Fine-tuning Epochs**

On CIFAR-100, we also fine-tuned ViT-B and Swin-B with various epochs to investigate the effect of longer training on ViTs’ performance. The benign accuracy is shown in the following table,

|        | Before | 20 epoch  | 100 epoch |
| ------ | ------ | --------- | --------- |
| ViT-B  | 89.54  | **69.12** | 58.93     |
| Swin-B | 92.24  | **78.56** | 72.66     |

where we find the severe overfitting issue caused by longer training. For example, the ACC drop of 100 epochs with ViT-B is more than 10% (69.12 → 58.93). This is consistent with the phenomenon in our paper.

**Comparison of Puning Granularity**

Taking AWM as an example, we investigate the impact of pruning granularity on the performance of defense approaches. Here, we compare the default granularity and our adapted granularity on CIFAR-100 with both ViT-B and Swin-B in the following tables.

ViT-B

|        | Before | Vanilla | Adapted   |
| ------ | ------ | ------- | --------- |
| ACC(%) | 89.54  | 61.04   | **82.07** |
| ASR(%) | 100.00 | 1.17    | 1.18      |

Swin-B

|        | Before | Vanilla | Adapted   |
| ------ | ------ | ------- | --------- |
| ACC(%) | 92.24  | 39.97   | **81.81** |
| ASR(%) | 100.00 | 0.41    | 2.60      |

Similarly, our proposed coarse granularity (pruning the output channel of the linear projection layer) surpasses the default granularity (pruning ViTs using an element-wise mask) in preserving ACC. For example, our proposed adaptation improves ACC of ViT-B by a notable margin (61.04% → 82.07%) while keeping the ASR almost unchanged. The results demonstrate that pruning the output channel is a better choice across datasets or architectures.

Reviewer comment (Experiments):
What is your thread model in your proposed attack? Do you assume that adversary have access to the model during training? Current setting is confusing to me since there are two thread models: 1. Both source and target being same model 2. Source and target are different.

Author response:
Sorry for the ambiguity in Section 5. We have revised the paper to make it clearer. Our primary focus is on the most common threat model, a **black-box scenario** (the source model is ViT-B, while the target model is another) where the attacker only knows the training data but is unaware of the training process and model. However, due to the difficulty of the black-box scenario, we initially considered a simpler **white-box scenario** (the source and target model are both ViT-B), where the attacker is aware of both the training data and the pre-trained model used for backdoor training. It's important to note that we have no access to the victim model to generate poisoned data since this victim model is fine-tuned from the pre-trained model based on already generated poisoned data.
Therefore, this paper considers two threat models in experiments: 1. **white-box scenario**: both source and target have the same architecture and backdoor training from the same pre-trained model; 2. **black-box scenario**: source and target are different. Among these, the second threat model is our primary focus.

Reviewer comment (Experiments):
In ImageNet experiments, both source and target are ViT-B. Does this means that adversary has access to model architecture and its parameters. This is not a practical scenario in my opinion and limits the impact of the paper.

Author response:
As mentioned in A2, the white-box scenario serves as a simpler and transitional phase in our research, while the more complex black-box scenario is our actual objective. We admit that performance on the white-box scenario is insufficient to demonstrate the efficacy of our approach with ViT. To address this, we supplement the results in the black-box scenario, where the source model is ViT-B, while the target model is Swin-B. The ACC and ASR  on ImageNet with or without CAT are summarized as follows:

ASR (%):

|             | Before | FT        | FP        | NAD       | ANP       | AWM       |
| ----------- | ------ | --------- | --------- | --------- | --------- | --------- |
| Badnets     | 100    | 31.91     | 22.23     | 42.36     | 42.29     | 35.92     |
| Badnets+CAT | 100    | **79.17** | **35.96** | **61.18** | **73.62** | **59.58** |
| Blend       | 100    | 6.10      | 3.01      | 18.00     | 10.85     | 31.96     |
| Blend+CAT   | 100    | **21.62** | **22.32** | **35.11** | **43.84** | **45.92** |

ACC (%):

|             | Before | FT    | FP    | NAD   | ANP   | AWM   |
| ----------- | ------ | ----- | ----- | ----- | ----- | ----- |
| Badnets     | 83.02  | 77.28 | 76.06 | 77.45 | 68.93 | 75.78 |
| Badnets+CAT | 83.11  | 77.27 | 76.10 | 77.86 | 69.54 | 76.84 |
| Blend       | 82.93  | 76.70 | 76.37 | 77.81 | 68.23 | 76.22 |
| Blend+CAT   | 83.09  | 76.87 | 76.51 | 77.26 | 70.16 | 75.96 |

Similar to the results in Table 5, CAT brings negligible impact on ACC. Meanwhile, CAT improves ASR by a notable margin. For example, after combining CAT, the ASR for FP increases from 22.23% to 35.96% (+13.73%) against Badnets attack. This demonstrates that our proposed method still works on the black-box scenario.

Reviewer comment (Novelty):
The contributions of this paper seem incremental, especially in the defense part. The experiments indicate that optimizing the choice of optimizer, adjusting epoch numbers, and selecting appropriate granularity for pruning can improve defense performance on ViT. To apply fine-tuning-based methods to ViT, the authors adjust optimizers and epochs. However, these improvements are based on experimental trials, and there is no methodology to guide us on how to pick good hyperparameters.

Author response:
**A1:** We believe that when solving problems, the greatest contribution to the community is to explore the fundamental issues and solve them with straightforward strategies, rather than a seemingly novel and complex approach. Here, we attribute the failure of the current defense to the following two points:

- **The mismatch between training and defense**: In fact, the first paper proposing ViTs [1] uses AdamW for pretraining and finetuning with SGD. Therefore, multiple works such as [2] adopt this strategy as default and they only obtain 42.00% ACC. We first identify this problem and propose that the AdamW optimizer is essential for fine-tuning-based defense.
- **The overfitting problem during the defense:** There is a typical problem when only a small amount of data is available for defense. In this situation, fine-tuning a large-capacity model is risky as it can easily lead to overfitting, especially for ViTs which are known to lack inductive bias [1]. Therefore, applying coarser granularity to reduce the number of learnable parameters and decreasing the number of iterations to lower the actual exploration parameter space can both reduce overfitting issues.

Therefore, we believe that our approaches provide the community with sufficient insight and contributions. From our investigations, there are three methodologies that can be concluded to guide us in picking good hyperparameters.

- If the defense methods are based on fine-tuning, the AdamW optimizer is recommended.
- If the defense methods are based on pruning, pruning the channel of the linear layer might be a good choice.
- For all defense methods, additional measures are needed to be taken to prevent overfitting.

Reviewer comment (Experiments):
In Section 3.2, the impact of the epoch on fine-tuning defense is explored. The curve for the first 20 epochs differs significantly from the first 20 epochs when setting the experiment to 100 epochs, particularly in the left plot of (a) left. The variability in experimental results raises concerns about the reliability of the findings, considering the potential instability.

Author response:
**A2**: This is because we employ a cosine learning rate (lr) schedule, which starts with a very large learning rate (0.0003) and gradually decreases until it reaches 0 in the final epoch. lr decreases at a different pace between training for 20 epochs and training for 100 epochs. Specifically, in the former case, lr drops to 0 at the 20th epoch (20 epochs in total), whereas in the latter case, it still keeps a large value (0.000274) at the 20th epoch (100 epochs in total). As a result, the curve seems quite different for the first 20 epochs.

In addition, we repeated our experiments 3 times with various numbers of epochs in the following table, in which we can find our method is stable with small variability and the findings are reliable.

**Badnets Attack**

| fine-tuning epoch | 20          | 40          | 60          | 80          | 100         |
| ----------------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| ACC               | 93.71 $\pm$ 0.14 | 92.42 $\pm$ 0.10 | 92.23 $\pm$ 0.14 | 91.25 $\pm$ 0.10 | 90.48 $\pm$ 0.17 |
| ASR               | 2.08 $\pm$ 0.60  | 1.92 $\pm$ 0.49  | 1.70 $\pm$ 0.28  | 1.62 $\pm$ 0.52  | 1.54 $\pm$ 0.10   |

Reviewer comment (Evaluation):
Table 4 illustrates that the CAT attack method improves ASR, but the enhancement is limited, as most unsuccessful attacks do not become successful.

Author response:
**A3**: Note that even a single successful attack can lead to substantial losses. When ASR increases from 1% to 10%, the incident frequency becomes 10 times larger, turning a negligible risk into a significant threat. In contrast, models with ACC of either 1% or 10% are both considered unusable in practice.

In the following table, we have listed the average improvement of CAT against various defense algorithms under a given attack method, along with the changes in the incident frequency. It is evident that our method significantly increases the incident frequency. Notably, it even increases the incident frequency of BadNets to ~34 times larger compared to its original value.

| ASR (Average over five defense)   | BadNets | Blend | CLB   | SIG   |
| --------------------------------- | ------- | ----- | ----- | ----- |
| Vanilla                           | 1.41    | 8.00  | 6.13  | 2.23  |
| CAT                               | 47.80   | 46.28 | 14.18 | 27.18 |
| Improvement of indecent frequency | 33.9x   | 5.8x  | 2.3x  | 12.2x |