Paper: Adversarial Defense using Targeted Manifold Manipulation (ICLR-2024)

Reviewer comment (Experiments):
The "Trapclass" filter mentioned in the paper primarily detects untargeted attacks, while the "Entropy" filter mainly identifies attacks that strive to minimize perturbation. Such a design might not be stable under certain specific attack strategies.

Author response:
Ans: To stress-test our defense we designed an adaptive attack that assumes the knowledge of the defense algorithm and the associated parameters to defeat the detection system. This should represent the worst of all the unknown attack strategies (sec 4.4). However, we showed that our detection methods still provided respectable accuracies against such extreme attack scenario.

Reviewer comment (Evaluation):
It is crucial to evaluate the efficiency and speed of the algorithm when considering applying the method to large-scale datasets or real-time application scenarios. The paper doesn't seem to delve deeply into this aspect.

Author response:
Ans: We have already covered that under the 'Computational time' in the experiment section (the last paragraph).

Reviewer comment (Reproducibility):
The L function in the article combines multiple loss components, but it does not clearly state how these components interact or how their weights are balanced.

Author response:
Ans: Loss components are indirectly given weight by sampling the data instances corresponding to individual losses at different ratios (sec 4.1). Thus we don't need to provide individual weights in the combined loss function.

Reviewer comment (Experiments):
The paper mentions different thresholds (such as ξ and ρ) used for detecting and identifying adversarial samples. However, the selection of these thresholds appears to be based on experience rather than systematic optimization. For these hyperparameters, the article does not provide a sensitivity analysis of their impact on the results, nor does it give clear guidance on how to choose these parameters for different datasets or tasks.

Author response:
Ans: We did specify a method to set the thresholds in sec 4.2, para 'Offline attack detection performance'. They are set to cause at most 0.5% false positive rate in the training dataset.

Reviewer comment (Experiments):
During the model training process, the setting of triggers is randomly selected. This raises the concern of whether some genuine data might be mistakenly identified as data containing triggers, leading to false alarms. Additionally, this random selection introduces a level of uncertainty. After numerous attempts, an adversary might be able to identify and bypass these triggers. Researchers should consider using a more stable and reliable method for setting triggers and balance the pros and cons of randomness and determinacy when designing.

Author response:
Ans: The randomness of the trigger (both the norm and the position) as discussed in sec 3.1 is by choice. Fig 3 shows the effect of our trigger placement. The trapring design as seen in there is only possible by randomly placing the trigger and the thick skin of the trapring is possible by using the random trigger norm. Thus they are the necessity for our method and not the negative aspects.

Reviewer comment (Presentation):
A lot of the tables and figures feel insufficiently described. Non-exhaustive list: the units for the FP column in the tables is unclear; there are no units or explanations of Table 5; it’s quite hard to see what is being shown in Figure 2 (and it looks like the markings were made with pen); Figure 2 claims a 2x2 checkerboard pattern when the image shows a 4x4; in a final version of the paper it would be great to add more detail to each table.

Author response:
- _**Q. Style Issue**_

   - Figure 2 caption is wrongly written. The trigger patch is 4X4.
   - Yes, the citations sometimes came wrongly. We will fix it.
   - In the Table 5, the Brightness column refers to the percentage of the original brightness. And the Gaussian blur $\sigma$ represents the standard deviation.

Reviewer comment (Writing):
Citations are often formatted weirdly or incorrectly. For example: “Zhain et. al. Zhai et al. (2019)…” on Page 3.

Author response:
- _**Q. Style Issue**_

   - Figure 2 caption is wrongly written. The trigger patch is 4X4.
   - Yes, the citations sometimes came wrongly. We will fix it.
   - In the Table 5, the Brightness column refers to the percentage of the original brightness. And the Gaussian blur $\sigma$ represents the standard deviation.

Reviewer comment (Theory):
The threat model is unclear. In particular, it is unclear what the defender knows about the attack: do they have an epsilon in mind? Do they know an attacker’s parameters? Do they know an attacker’s algorithm?

Author response:
- _**On the threat model and defender's knowledge about attack parameters**_


    Ans:We are extremely sorry for the confusion created due to overloading the same symbol ($\epsilon$) for both the trigger's lowest norm ($\epsilon_{training}$, used during training) and attack's perturbation budget ($\epsilon_{attack}$, used during attack). Trigger's lowest norm is set as small as possible so that clean accuracy is not hampered much. In our training, we used 0.01 (2.55/255) as the minimum $\epsilon$ for training. This means that any attack using a larger perturbation budget than 0.01 will end up in the Trapring. Attacks with smaller perturbation budget either fails because of extreme closeness requirement to the genuine data or still gets detected because even though the Traprings are enforced starting at 0.01, it starts to induce Trapring formation even before that. In short, the defender does not need to know the attacker's perturbation budget ($\epsilon_{attack}$) and it needs to use just the minimum perturbation ($\epsilon_{training}$) that it can use to avoid drastic reduction in the clean data performance, which can usually be very small. Thus, our method can provide defense across a wide spectrum of attack parameters.

Reviewer comment (Novelty):
The work does not appear to be well-differentiated from adversarial training regimes. Essentially, they train on three datasets (via a parameterized loss), one of which is clean and two of which are altered in traditional ways. This distinction could be better addressed.

Author response:
- _**TMM on the point of view of Adversarial Training**_

    Ans:In the adversarial training (AT), we use adversarial samples to make the model robust. The adversarial training has two significant drawbacks: 1) the robustness obtained by AT depends on the attack methods used in the adversarial sample generation process. It has been shown that adversarial robustness against one attack method does not transfer to other attack methods. 2) AT can have large impact on the clean data accuracy as it can blur the class boundaries between genuine classes. In contrast, our method do not need to know the attack method as our design ensures universal defense and it has a least effect as instead of blurring the class boundaries it creates Traprings around the genuine data which has a very low impact on the clean data accuracy. Thus our method offers a superior choice.

Reviewer comment (Experiments):
The experimental section feels somewhat limited. One of the major claims is that their defense works on many attacks, but the analysis of attacker parameters ($\epsilon$, learning rate, etc.) is limited to a single case in the main text. It is unclear whether their method is robust to a higher learning rate attacker that ‘jumps’ over their trap-ring regions in the training data.

Author response:
- _**Use of higher learning rates during attack**_


    **Table 1**:Detection accuracies against attacks with different learning rates. 

      -------------------------------------------------------------
      |   Lr   | Method| FGSM | PGD |  CW |   DF  |  PA   |   AA  |
      |------------------------------------------------------------
      | 1/255  | TMM-O |  100 |99.99|90.67| 99.98 | 98.50 |  100  |
      | 1/255  | TMM-L |  100 | 100 | 100 | 99.99 | 85.24 |  100  |
      | 2/255  | TMM-O |  100 | 100 |88.71| 99.98 | 97.31 |  100  |
      | 2/255  | TMM-L |  100 | 100 | 100 | 98.99 | 83.09 |  100  |
      | 4/255  | TMM-O |  100 | 100 |96.44| 98.34 | 98.49 |  100  |
      | 4/255  | TMM-L |  100 | 100 | 100 | 99.73 | 85.17 |  100  |
      | 8/255  | TMM-O |  100 | 100 | 100 | 99.97 | 98.57 |  100  |
      | 8/255  | TMM-L |  100 | 100 | 100 | 99.91 | 87.01 |  100  |
      | 16/255 | TMM-O |  100 | 100 |98.78| 99.13 | 97.09 |  100  |
      | 16/255 | TMM-L |  100 | 100 | 100 |  100  | 92.14 |  100  |
      -------------------------------------------------------------

    Ans: As we discussed in sec 3.1, the norm of our trigger is sampled between [$\epsilon$, 0.99], which means that the Trapring can be quite wide, sort of creating an effect of filling up the vacant spaces between the data instances with Traps. Thus, it is actually easier for the attacks to fall into the Trapring with higher learning rate (step-size) as shown in the below table.

Reviewer comment (Experiments):
The method is based on an attacker ‘landing’ an adversarially perturbed point in the trap-ring of a training datapoint, but this inevitability could easily be countered by an attacker using a larger step-size to avoid the trapdoor region altogether. This potential limitation is unaddressed.

Author response:
- _**Use of higher learning rates during attack**_


    **Table 1**:Detection accuracies against attacks with different learning rates. 

      -------------------------------------------------------------
      |   Lr   | Method| FGSM | PGD |  CW |   DF  |  PA   |   AA  |
      |------------------------------------------------------------
      | 1/255  | TMM-O |  100 |99.99|90.67| 99.98 | 98.50 |  100  |
      | 1/255  | TMM-L |  100 | 100 | 100 | 99.99 | 85.24 |  100  |
      | 2/255  | TMM-O |  100 | 100 |88.71| 99.98 | 97.31 |  100  |
      | 2/255  | TMM-L |  100 | 100 | 100 | 98.99 | 83.09 |  100  |
      | 4/255  | TMM-O |  100 | 100 |96.44| 98.34 | 98.49 |  100  |
      | 4/255  | TMM-L |  100 | 100 | 100 | 99.73 | 85.17 |  100  |
      | 8/255  | TMM-O |  100 | 100 | 100 | 99.97 | 98.57 |  100  |
      | 8/255  | TMM-L |  100 | 100 | 100 | 99.91 | 87.01 |  100  |
      | 16/255 | TMM-O |  100 | 100 |98.78| 99.13 | 97.09 |  100  |
      | 16/255 | TMM-L |  100 | 100 | 100 |  100  | 92.14 |  100  |
      -------------------------------------------------------------

    Ans: As we discussed in sec 3.1, the norm of our trigger is sampled between [$\epsilon$, 0.99], which means that the Trapring can be quite wide, sort of creating an effect of filling up the vacant spaces between the data instances with Traps. Thus, it is actually easier for the attacks to fall into the Trapring with higher learning rate (step-size) as shown in the below table.

Reviewer comment (Experiments):
It would be useful to see how the method responds to CIFAR-10.1. It should maintain high performance there, but it is unclear how the distribution shift would affect the proposed method.

Author response:
- _**Q. CIFAR10.1 dataset results**_

    Ans:As suggested, we have conducted the experiment on CIFAR10.1 dataset. Given table shows the performance. We can see only 1.81% increment in the False Positive (FP) rate without any drop in the detection accuracy.

      ----------------------------------------------------------------------
      |   Dataset  | Method| FP | FGSM | PGD |  CW |   DF  |  PA   |   AA  |
      |---------------------------------------------------------------------
      | CIFAR10    | TMM-O |5.33|  100 |98.87|94.21| 99.98 | 98.24 |  100  |
      | CIFAR10    | TMM-L | 0.0|  100 | 100 |99.98| 99.80 | 81.54 |  100  |
      | CIFAR10.1  | TMM-O |7.14|  100 |98.69|96.24| 98.73 | 98.20 |  100  |
      | CIFAR10.1  | TMM-L | 0.0|  100 | 100 | 100 | 98.99 | 83.24 |  100  |
      ----------------------------------------------------------------------

Reviewer comment (Writing):
One of the main defenses proposed (TMM-LA) has very little description associated with it (Page 8 bottom).

Author response:
- _**Explanation of Live detection method (TMM-L)**_

  Ans:TMM-L is meant for the cloud based applications where the models are being watched continuously. Our hypothesis is that when attack algorithm seeks to optimize the perturbation through gradient, it goes through the Trapclass, and we have noticed that within the very first 2-3 number of iterations, the perturbed samples fall in the trap region and gets classified as $y_{Trap}$, and thus gets detected. Since the detection method in this case is just comparison of the class label with $y_{Trap}$, and computation of few statistics based on the model output, the cost of detection is very low. Due to these significant benefits, we proposed TMM-L as the special version for the cloud-based deployment scenario.

Reviewer comment (Experiments):
Doesn’t Tiny ImageNet have 100K examples in the training set?

Author response:
- _**On Tiny imagenet sample count**_

   Ans: Sorry, it was a typo and your numbers are correct. Thank you.

Reviewer comment (Theory):
Regarding the statement “Almost all attacks need to go through the trap-ring”: what is preventing an attacker from taking a learning rate high enough to jump over the trap-rings? Does the analysis of the trap rings fail when an adversarial point jumps over the trap-ring?

Author response:
- _**Use of higher learning rates during attack**_


    **Table 1**:Detection accuracies against attacks with different learning rates. 

      -------------------------------------------------------------
      |   Lr   | Method| FGSM | PGD |  CW |   DF  |  PA   |   AA  |
      |------------------------------------------------------------
      | 1/255  | TMM-O |  100 |99.99|90.67| 99.98 | 98.50 |  100  |
      | 1/255  | TMM-L |  100 | 100 | 100 | 99.99 | 85.24 |  100  |
      | 2/255  | TMM-O |  100 | 100 |88.71| 99.98 | 97.31 |  100  |
      | 2/255  | TMM-L |  100 | 100 | 100 | 98.99 | 83.09 |  100  |
      | 4/255  | TMM-O |  100 | 100 |96.44| 98.34 | 98.49 |  100  |
      | 4/255  | TMM-L |  100 | 100 | 100 | 99.73 | 85.17 |  100  |
      | 8/255  | TMM-O |  100 | 100 | 100 | 99.97 | 98.57 |  100  |
      | 8/255  | TMM-L |  100 | 100 | 100 | 99.91 | 87.01 |  100  |
      | 16/255 | TMM-O |  100 | 100 |98.78| 99.13 | 97.09 |  100  |
      | 16/255 | TMM-L |  100 | 100 | 100 |  100  | 92.14 |  100  |
      -------------------------------------------------------------

    Ans: As we discussed in sec 3.1, the norm of our trigger is sampled between [$\epsilon$, 0.99], which means that the Trapring can be quite wide, sort of creating an effect of filling up the vacant spaces between the data instances with Traps. Thus, it is actually easier for the attacks to fall into the Trapring with higher learning rate (step-size) as shown in the below table.

Reviewer comment (Novelty):
Please elaborate on the distinction between this work and existing adversarial training literature.

Author response:
- _**TMM on the point of view of Adversarial Training**_

    Ans:In the adversarial training (AT), we use adversarial samples to make the model robust. The adversarial training has two significant drawbacks: 1) the robustness obtained by AT depends on the attack methods used in the adversarial sample generation process. It has been shown that adversarial robustness against one attack method does not transfer to other attack methods. 2) AT can have large impact on the clean data accuracy as it can blur the class boundaries between genuine classes. In contrast, our method do not need to know the attack method as our design ensures universal defense and it has a least effect as instead of blurring the class boundaries it creates Traprings around the genuine data which has a very low impact on the clean data accuracy. Thus our method offers a superior choice.

Reviewer comment (Evaluation):
The trigger creation process might induce higher sample complexity compared to the baseline adversarial training techniques, since the defender must generate additional trapdoor data, in addition to the robust region data. The additional data burden isn't measured.

Author response:
- _**Size of the extra dataset**_

    Ans: The extra datasets corresponding to robust and the backdoor loss functions can be created on the fly during the batch creation and thus we don't need to create separate datasets (more in section 4.1, and Table 4 in the supplementary). However, due to the effect of so many loss functions, we needed to train our models for longer time. For example, to achieve similar level of clean data accuracy on WideResNet (width-8, depth-20) we had to train our model on CIFAR-10 for 3,000 epochs, which is roughly 3-4 times that of learning a model only on the clean data. This is one of the significant requirement of our method.

Reviewer comment (Experiments):
The evaluation against semantic preserving attack was weak considering previous work already formulated strong perceptual attacks beyond simple brightness and blurring modification [1, 2]. The significance would be improved if the authors compare against these techniques.

Author response:
- _**Performance against semantic preserving attacks**_

    Ans:We implemented “strong perceptual attack” , as suggested, on CIFAR10 dataset. Result his shown below. As we can see our methods (TMM-L and TMM-O) both provided very high detection accuracies, and much higher than the baselines.Here, we also like to take the opportunity to state that in our paper we used semantic-preserving transformations to show that our detection methods do not detect them as adversarial/ood samples unlike other baselines. This is to show the benefit of using single model for both classification and detection and the alignment it offers, compared to methods that use separate model for detection (mentioned also in the Introduction, second para).

      **Table 5**: Attacks defended by proposed method and the baselines in % on CIFAR10
      -------------------------------------------------------------------
      |  Attack |   TMM-O  |  TMM-L |   AT  |  Mahalanobis |  Trapdoor |
      |------------------------------------------------------------------
      |    LPA  |  98.84   |   100  |  0.25 |     65.40    |   82.95   |
      -------------------------------------------------------------------

Reviewer comment (Experiments):
There is no comparison to standard robustification techniques such as random smoothing [3] or vanilla adversarial training [4]. Without these it is difficult to measure the impact of the work to the broader community.

Author response:
- _**Q. Comparison with Vanilla adversarial training**_

    Ans:FGSM adversarial training (AT) is done with the perturbation bound of $\epsilon$ =4/255. Number of iteration is 1000. Clean accuracy obtained = 89.96 %, adversarial robustness obtained = 89.50% against FGSM with perturbation norm of 4/255. Given below table describes the performance of the vanilla adversarial trained model against multiple attacks at multiple perturbation norms. We can notice that vanilla AT model is not at all useful against unseen attacks. and against the same attack but with different perturbation budgets On the other hand, TMM fared similarly across different attacks and variable perturbation budgets.

      **Table 3**: Performance of an adversarially trained model in blocking adversarial attacks
      -----------------------------------------------------------------------------
      |  Dataset | Clean Acc.|  eps  |  FGSM  |  PGD  |  CW  |   DF |   PA  |   AA |
      |-----------------------------------------------------------------------------
      | CIFAR10  |   89.93   | 4/255 |  89.50 | 88.94 | 0.11 | 0.28 | 17.59 | 93.24 |
      | CIFAR10  |   89.93   | 8/255 |  63.25 |  62.00| 0.09 | 0.37 | 16.14 | 33.75 |
      -------------------------------------------------------------------------------

Reviewer comment (Writing):
The description of the trapdoor mask creation (Loss function for trapdoor) was difficult to follow and could be simplified. Some of the notation seemed superfluous, e.g., trying to describe every span of coordinates and every span of values of the location-parameterized mask. IMO it would be better to just define a tensor $T \in \mathbb{R}^{ch\times m\times n}$ which is alpha-blended and applied as in Equation 1 centered at a coordinate sampled from a range. Let values in $T$ sample from a predefined range.

Author response:
- _**Presentation of Trigger formation equations**_

    Ans:Yes, we will simplify the presentation using Tensor representation. Thank you for the suggestion.