Paper: Embodied Active Defense: Leveraging Recurrent Feedback to Counter Adversarial Patches (ICLR-2024)

Reviewer comment (Theory):
In Section 4.2, the authors mention the effectiveness against adaptive attack; however, I find it hard to understand the reason why a surrogate **uniform** superset policy distribution would necessitate an optimized patch to handle various action policies, as this uniform surrogate may not contain any useful information about the policy model in EAD for the adversary to attack.

Author response:
1. **Effectiveness of proposed adaptive attack**.

    Your inquiry into the efficacy of a surrogate uniform superset policy distribution is highly relevant. In our initial submission, attacking EAD across a 4-step trajectory was impractical due to excessive GPU memory requirements (approximately 90 GB for 4 steps). Besides our original technique, we have explored gradient checkpointing [1], a method that allows for recalculating parts of the computation graph established by the $\tau$-step EAD inference process, rather than storing them directly. This reduces memory usage while increasing computational demands. Employing this method, we successfully attacked the entire pipeline along a 4-step trajectory using an NVIDIA RTX 3090 Ti.

    To empirically assess the Uniform Superset Policy (USP), we compared it with our new adaptive attack leveraging gradient checkpointing (denoted as GC) across the entire trajectory.

    

    *Table a: Evaluation of Adaptive Attacks on the EAD Model. Columns marked **USP** display results from optimizing the patch using an expected gradient over USP. **GC** columns represent results from attacking EAD following the gradients along the 4-step trajectory with gradient checkpointing.*

    |              | **Dodging** |        | **Impersonation** |        |
    | :----------: | :---------: | :----: | :---------------: | :----: |
    |              |   **USP**   | **GC** |      **USP**      | **GC** |
    | **ASR (\%)** |  **22.11**  | 15.79  |       **8.33**        |  7.29  |

    

    As Table a demonstrates, our original adaptive attack using USP was more effective than tracing the authentic policy of EAD. This may be attributed to vanishing or exploding gradients impeding optimization [2]. This problem is potentially mitigated by our approach that computes expectations over a uniform policy distribution. Additionally, our method maintains its effectiveness against various adaptive attacks. This underscores that EAD's defense stems from the synergistic integration of its policy and perception models, encouraging strategic data collection rather than merely countering adversarial patches with specific viewpoints. Further details and evaluation results on these adaptive attacks are elaborated in the revised Appendix D.3 and D.7 respectively.

Reviewer comment (Writing):
The paper could benefit from improved clarity, especially for readers with a foundational understanding of adversarial robustness but limited exposure to RL/Robotics. As it stands, the document is dense with jargon, making it challenging to navigate and comprehend upon the initial read.

Author response:
2. **Clarity improvement**.

    Thanks for your pointing it out. Regarding your concerns about clarity for readers with foundational knowledge of adversarial robustness but limited exposure to RL/Robotics, we have added a section in simplified language to introduce the relevant background, enhancing readability. The comprehensive background information is detailed in the Appendix B. We hope that these revisions make our work more accessible to a broader audience.

Reviewer comment (Experiments):
For those adversarially-trained passive defense models, would it be beneficial to enhance them with 3D-augmented data, e.g., feeding multiple views of the same patch-attacked human face during training, if the lack of 3D environment awareness is the problem here?

Author response:
3. **Enhancing passive defense models with 3D-augmented data.**

    This is a potential misunderstanding, and we humbly clarify that we have already incorporated 3D augmented images to train the passive baseline, aiming to optimize baseline performance for a fair comparison. Additionally, we conducted training exclusively with 2D data. We have summarized the performance of baselines trained with 2D data (SAC-2D, PZ-2D), 3D augmented data (SAC, PZ), and our EAD model respectively.

    

    *Table b: Performance comparison of passive baseline with different types of data.* 

    | Method         | Acc. (\%) | Dodging ASR (\%) |         |         |          |          | Impersonation ASR (\%) |         |         |         |         |
    | -------------- | --------: | ---------------: | ------: | ------: | -------: | -------: | ---------------------: | ------: | ------: | ------: | :-----: |
    |                |           |              MIM |     EoT |   GenAP |    3DAdv |     Adpt |                    MIM |     EoT |   GenAP |   3DAdv |  Adpt   |
    | *SAC-2D*       |   *86.22* |           *74.5* |  *82.9* |  *81.9* |   *75.5* |   *48.9* |                 *45.0* |  *49.0* |  *68.0* |  *61.0* |  *N/A*  |
    | *PZ-2D*        |   *88.78* |           *61.1* |  *66.7* |  *85.6* |   *76.7* |     74.0 |                 *48.5* |  *51.5* |  *84.8* |  *75.8* | *69.7*  |
    | SAC            |     80.55 |             78.8 |    78.6 |    79.6 |     85.8 |     85.0 |                    3.2 |     3.2 |    18.9 |    22.1 |  51.7   |
    | PZ             |     85.85 |              6.1 |     6.2 |    14.3 |     20.4 |     69.4 |                **3.1** |     3.2 |    19.1 |    27.4 |  61.0   |
    | **EAD (ours)** | **90.45** |          **0.0** | **0.0** | **2.1** | **13.7** | **22.1** |                    4.1 | **3.1** | **5.1** | **7.2** | **8.3** |

    

    As illustrated in Table b, our EAD model outperforms the others, and the 3D-augmented passive baselines are more robust than those trained solely with 2D data. This suggests that enhanced 3D environmental awareness improves adversarial robustness. Furthermore, the active policy of EAD effectively utilizes 3D environmental information to defend against adversarial patches, resulting in superior performance.

Reviewer comment (Reproducibility):
There is one part I feel confused about in the algorithm box: $O_t′ \gets A(O_t, P; S_t)$ and $O_{t+1}'\gets A(O_{t+1},P;S_{t+1})$, where do $O_t'$ and $O_{t+1}'$ go? They are not explicitly used or referenced anywhere else post-assignment. Are clean observations $O_t$ and $O_{t+1}$ overwritten by $O_t'$ and $O_{t+1}'$ after applied with the adversarial patch by default?

Author response:
4. **Typo in the algorithm box.**

    We are so sorry for the typographical errors in the algorithm box, which may have led to confusion. Your interpretation regarding the notation is correct; the perturbed observations $O'$ were intended for training, but we inadvertently omitted the $'$. We have amended these errors in the revised paper.

Reviewer comment (Writing):
Overall, the paper writing style and organization needs improvement to allow the reader to easily understand the paper.

Author response:
We understand your concern regarding the paper's writing style and organization. We have worked diligently to address these issues to make readers easily grasp the content. Specifically, we have made the following improvements:

    1.1 A detailed introduction to the tasks of impersonation and dodging attacks has been added in the updated Appendix D.2 for improved clarity.

    1.2 The environmental setup for object detection, similar to that used in face recognition, primarily varies in the feasible region of viewpoints. We've included a comprehensive description of this setting, incorporating the CARLA simulator and ShapeNet EG3D in the tail of Appendix C.

    1.3 The explanation for the used attacks including MIM, EoT, GenAP and Face3DAdv, has been elaborated in  Appendix D.2 for enhanced clarity.

    1.4 Annotation definitions and terms including model prediction are now more clearly defined in the revised methodology section (Sec. 3.1).

    1.5 The codebase outline and a detailed description have been provided to ensure reproducibility.

Reviewer comment (Experiments):
The paper makes a claim that passive adversarial defense approaches are not sufficient for dynamic environments. This claim should be supported by a passive defense used as a baseline. Additionally, it would be great to see the extra amount of training resources used to train the active embodied model vs a passive defense approach (for example Madry's PGD adversarial training). It would also help the readers if some other active defenses were used as a baseline in order to establish the efficacy of the proposed approach.

Author response:
2. **Evidence supporting the inadequacy of passive defenses in dynamic environments and baseline concerns**.

    2.1 **Passive defenses are inadequate in dynamic environments**.

    We want to humbly point out that we have incorporated various passive defenses in our submission, including Preprocessed methods (e.g., JPEG Compression, Local Gradients Smoothing (LGS), Segment and Complete (SAC) and PatchZero (PZ)) alongside an adversarial training based method for patch attacks, namely Defense against Occlusion Attacks (DOA), which is similar to Madry's PGD adversarial training. These methods represent common and state-of-the-art passive defenses against patch attacks. The comparative results are detailed in Table 1 and Figure 2 of our original submission.    

    2.2 **Extra training resources**.

    Regarding training resources, our method demonstrates an obviously reduced training overhead compared to traditional adversarial training approaches, such as DOA. Adversarial training typically incurs substantially higher computational costs than standard training methods [1], particularly when designed to counter a broad range of attacks. Despite our model necessitating an additional policy model and a temporal fusion model, its training resource requirements remain considerably lower compared to those of adversarial training-based methods (e.g., DOA). For the training of EAD only requires patches from Uniform Superset Approximation thereby bypassing the expensive computational cost of inner maximization of adversarial training. Furthermore, our approach is orthogonal to adversarial training, making it feasible to integrate the latter to enhance EAD's resilience against specific attacks. We also provide detailed discussions on computational resources below in "Computational overhead for EAD defense".

    2.3 **Active defense baselines.** 

    To the best of our knowledge, our work represents the first attempt to promote the adversarial robustness of perception in an active paradigm.

Reviewer comment (Experiments):
The experimental (both training and evaluation) setting does not seem sufficient and scalable enough to make conclusions that it would generalize. For example: “To train EAD for face recognition, we randomly sample images from 2,500 distinct identities from the training set of CelebA-3D.” and “we report the white-box attack success rate (ASR) on 100 identity pairs in both impersonation and dodging attacks with various attack methods”.

Author response:
3. **Sufficiency and scalability of experimental setting.**

    As for the data sufficiency in our experimental settings,  we would like to clarify that our training dataset encompasses over 3M unique images, rendered from various 3D facial models and viewpoints. Each identity has approximately 20 distinct 3D facial models, with nearly 60 images rendered from each model. This quantity of face data is exactly the same as previous adversarial defense work like SAC [2]. Besides, this amount has reached the average volume of current top-tier FR dataset (e.g., 1.2M for Digi-Face 1M [3], 3.3M for VGG Face2 [4], 0.5M for CASIA-WebFace [5]). As for testing, we adhere to the standard testing protocol in [6]. The testing dataset is equally abundant and diverse with ones in the training phase, which involves various 3D facial models and viewpoints. Furthermore, we are also generating additional data to further validate the scalability and generalizability of our findings, with plans to update our findings accordingly.

Reviewer comment (Writing):
Minor typo/suggestions:  
- “EAD comprises two primary submodules:” → “EAD comprises of two primary submodules:”  
- “Notely, it maintains or even improves standard accuracy due to” → “Notably, it maintains or even improves standard accuracy due to”  
- “Formally, It derives a strategic action” → “Formally, it derives”  
- “they have now developed to perceive various perception models”  
- “It is noteworthy this presents a more versatile 3D formulation for adversarial patches”  
- “from given scene x with its annotation y from another modal like CLIP” → “from given scene x with its annotation y from another model like CLIP”  
- “In D.5 MORE EVALUATION RESULTS ON EG3D, We present the iterative defense process in Figure ??”

Author response:
4. **Minor typos/suggestions.**

    We deeply apologize for any ambiguity or errors in our initial submission and have meticulously corrected these in the revised version. We have thoroughly refined the entire paper, paying special attention to clarity and precision in our descriptions. Additionally, following your advice, we have reevaluated the ethical implications of our work. In light of this, we have carefully modified relevant statements to more accurately reflect the potential impacts of our research, ensuring a more responsible presentation.

Reviewer comment (Reproducibility):
Question — How much training time/wall clock time and memory resources does it take for the EAD defense (both at training and inference phase)?

Author response:
5. **Computational overhead for EAD defense.**

    Thanks for pointing out the computational cost of our method compared to the baselines. In response, we have conducted a detailed comparison of computational overhead, including parameter amount and computation speed, in both training and inference phases, summarized in the following table:

    

    *Table a: Computational overhead comparison of different defense methods in face recognition. We report the training and inference time of defense on a NVIDIA GeForce RTX 3090 Ti and an AMD EPYC 7302 16-Core Processor with the training batch size as 64.* 

    | **Method** | **\# Params (M)** | **Description**                                              | **Training Epochs** | **Training Time per batch (s)** | Overall Training time (GPU hours) | **Inference Time per step (ms)** |
    | ---------- | ----------------- | ------------------------------------------------------------ | ------------------- | ------------------------------- | --------------------------------- | -------------------------------- |
    | JPEG       | -                 | non-parametric                                               | -                   | -                               | -                                 | 9.65                             |
    | LGS        | -                 | non-parametric                                               | -                   | -                               | -                                 | 26.22                            |
    | SAC        | 44.71             | Train a  segmenter to detect a patch area, which consists of two phases: training with pre-generated adversarial images and self-adversarial training | 50 + 10             | 0.152 + 4.018                   | 104                               | 26.43                            |
    | PZ         | 44.71             | Share the same training procedure and segmenter as SAC       | 50 + 10             | 0.152 + 4.018                   | 104                               | 11.88                            |
    | DOA        | 43.63             | An adversarial training method that retrains the feature extractor. | 100                 | 1.732                           | 376                               | 8.10                             |
    | EAD (ours) | 57.30             | Train the policy model and perception model in two stages    | 50 + 50             | 0.595 + 1.021                   | 175                               | 11.51                            |

    

    As indicated in Table a, in the training phase, the total training time of our EAD model is effectively balanced between the pure adversarial training method DOA and the partially adversarial methods like SAC and PZ. This efficiency stems mainly from our unique USAP approach, which bypasses the need for generating adversarial examples, thereby boosting training efficiency.
        In terms of model inference, our EAD, along with PZ and DOA, demonstrates superior speed compared to LGS and SAC. This is attributed to the latter methods requiring CPU-intensive, rule-based image preprocessing, which diminishes their inference efficiency. Further details, including loss curves and more precise time consumption metrics, are elaborated in Appendix D.6.

Reviewer comment (Experiments):
Question — The general assumption about the adversary is that it is not bound by computation resources. For example, in the statement in the paper for adaptive attacks: “While the deterministic and differential approximation could enable backpropagation through the entire inference trajectory of EAD, the computational cost is prohibitive due to rapid GPU memory consumption as trajectory length τ increases”, here for face recognition can we do a white box attack by following the gradients for all the 4-step trajectory?

Author response:
6. **Computational overhead for adaptive attack.**

    In our initial submission,  attacking EAD by following the gradients for all the 4-step trajectory is prohibitive due to the substantial GPU memory consumption  (4 steps require over 90 GB video memory). Besides the technique presented in our submission, we further investigate a new technique, namely gradient checkpointing [7]. By checkpointing nodes in the computation graph defined by the $\tau$-step EAD inference procedure, this technique recomputes the parts of the graph instead of directly storing them, thereby reducing memory cost with extra computation. With this technique, we succeed in attacking the whole pipeline along 4-step trajectory (overall) on a NVIDIA RTX 3090 Ti. 

    We conduct the adaptive attack by following the gradients along the whole trajectory with gradient checkpointing technique (denoted by GC) and show the results along with the adaptive attack with surrogate Uniform Superset Policy (denoted by USP) which is proposed in our previous submission.  

   *Table b: Evaluation of adaptive attacks on the EAD Model. Columns with **USP** represent results obtained by optimizing the patch with an expected gradient over the Uniform Superset Policy (USP). And **GC** denotes attacking EAD by following the gradients for along the 4-step trajectory with gradient checkpointing.* 

    |              | **Dodging** |        | **Impersonation** |        |
    | :----------: | :---------: | :----: | :---------------: | :----: |
    |              |   **USP**   | **GC** |      **USP**      | **GC** |
    | **ASR (\%)** |  **22.11**  | 15.79  |       8.33        |  7.29  |

    As depicted from the above results,  EAD stills maintains its robustness when faced adaptive attack against its whole pipeline. It further shows that EAD’s defensive capabilities arise from the synergistic integration of its policy and perception models, facilitating strategic observation collection rather than learning a short-cut strategy to neutralize adversarial patches from specific viewpoints. Moreover, the detailed methodologies of these adaptive attacks and relevant results are now illustrated in Appendix D.3 and D.7 respectively.

Reviewer comment (Writing):
Question — In the paper the description for Figure 3 mentions that “subsequent active interactions with the environment progressively reduce the similarity between the positive pair”. Is this a typo, should this be “reduce the dissimilarity” or “increase the similarity”?

Author response:
7. **Question about the description of Figure 3.**

    Thank you for your insightful query regarding the description of Figure 3 in our paper. Upon reviewing your question, we have identified that there is a typographical error in our manuscript. The correct phrasing should be “reduce the dissimilarity” or, equivalently, “increase the similarity” between the positive pair. We have corrected this typographical error, aligning it with the standards of academic excellence.

Reviewer comment (Writing):
Question — In Algorithm 1, the perturbed observation O’t is not being used, is there a typo there?

Author response:
8. **Question about the notation in Algorithm 1.**

    Thank you for your attentive reading of our manuscript and for highlighting the concern regarding Algorithm 1. Upon revisiting the algorithm, this is indeed a typographical error in our paper.

    To clarify, $O_t'$ should be integrated into the algorithm at 7th and 9th line, where it plays a crucial role in building patch-countering capability of the EAD. We appreciate your keen eye in identifying this oversight, and we will ensure that this correction is made to accurately represent the workings of the algorithm in the revised manuscript.

Reviewer comment (Evaluation):
There is no discussion regarding the computational cost of the proposed method and how it compares to the undefended baseline as well as prior defenses.

Author response:
**Discussion over computational overhead** 

    Thanks for pointing out the computational cost of our method compared to the baselines. In response, we have conducted a detailed comparison of computational overhead, encompassing aspects such as parameter amount and computation speed during both training and inference phases. The outcomes of this analysis are presented in Table a.

    

    *Table a: Computational overhead comparison of different defense methods in face recognition. We report the training and inference time of defense on a NVIDIA GeForce RTX 3090 Ti and an AMD EPYC 7302 16-Core Processor with the training batch size as 64.*     

    | **Method** | **\# Params (M)** | **Description**                                              | **Training Epochs** | **Training Time per batch (s)** | Overall Training time (GPU hours) | **Inference Time per step (ms)** |
    | ---------- | ----------------- | ------------------------------------------------------------ | ------------------- | ------------------------------- | --------------------------------- | -------------------------------- |
    | JPEG       | -                 | non-parametric                                               | -                   | -                               | -                                 | 9.65                             |
    | LGS        | -                 | non-parametric                                               | -                   | -                               | -                                 | 26.22                            |
    | SAC        | 44.71             | Train a segmenter to detect a patch area, which consists of two phases: training with pre-generated adversarial images and self-adversarial training | 50 + 10             | 0.152 + 4.018                   | 104                               | 26.43                            |
    | PZ         | 44.71             | Share the same training procedure and segmenter as SAC       | 50 + 10             | 0.152 + 4.018                   | 104                               | 11.88                            |
    | DOA        | 43.63             | An adversarial training method that retrains the feature extractor | 100                 | 1.732                           | 376                               | 8.10                             |
    | EAD (ours) | 57.30             | Train the policy model and perception model in two stages    | 50 + 50             | 0.595 + 1.021                   | 175                               | 11.51                            |

    

    As indicated in Table a, in the training phase, the total training time of our EAD model is effectively balanced between the pure adversarial training method DOA and the partially adversarial methods like SAC and PZ. This efficiency stems mainly from our unique USAP approach, which bypasses the need for generating adversarial examples, thereby boosting training efficiency.
    In terms of model inference, our EAD, along with PZ and DOA, demonstrates superior speed compared to LGS and SAC. This is attributed to the latter methods requiring CPU-intensive, rule-based image preprocessing, which diminishes their inference efficiency. Further details, including loss curves and more precise time consumption metrics, are elaborated in Appendix D.6.

Reviewer comment (Theory):
There are no theoretical results establishing how the approximations used in Sec 3.3 relate to the actual quantities. If possible to obtain, this would be nice to have.

Author response:
**Theoretical results relating to approximations in Sec 3.3.**

    Thanks for your suggestion. We agree that while our method is fundamentally grounded in information theory, there are several approximations in our implementation that merit deeper theoretical exploration. We would like to further achieve this as a potential area for future work, aiming to provide a more comprehensive theoretical analysis that aligns closely with our practical implementations.

Reviewer comment (Evaluation):
How does the computational cost of the proposed method compares to the baselines?

Author response:
**Discussion over computational overhead** 

    Thanks for pointing out the computational cost of our method compared to the baselines. In response, we have conducted a detailed comparison of computational overhead, encompassing aspects such as parameter amount and computation speed during both training and inference phases. The outcomes of this analysis are presented in Table a.

    

    *Table a: Computational overhead comparison of different defense methods in face recognition. We report the training and inference time of defense on a NVIDIA GeForce RTX 3090 Ti and an AMD EPYC 7302 16-Core Processor with the training batch size as 64.*     

    | **Method** | **\# Params (M)** | **Description**                                              | **Training Epochs** | **Training Time per batch (s)** | Overall Training time (GPU hours) | **Inference Time per step (ms)** |
    | ---------- | ----------------- | ------------------------------------------------------------ | ------------------- | ------------------------------- | --------------------------------- | -------------------------------- |
    | JPEG       | -                 | non-parametric                                               | -                   | -                               | -                                 | 9.65                             |
    | LGS        | -                 | non-parametric                                               | -                   | -                               | -                                 | 26.22                            |
    | SAC        | 44.71             | Train a segmenter to detect a patch area, which consists of two phases: training with pre-generated adversarial images and self-adversarial training | 50 + 10             | 0.152 + 4.018                   | 104                               | 26.43                            |
    | PZ         | 44.71             | Share the same training procedure and segmenter as SAC       | 50 + 10             | 0.152 + 4.018                   | 104                               | 11.88                            |
    | DOA        | 43.63             | An adversarial training method that retrains the feature extractor | 100                 | 1.732                           | 376                               | 8.10                             |
    | EAD (ours) | 57.30             | Train the policy model and perception model in two stages    | 50 + 50             | 0.595 + 1.021                   | 175                               | 11.51                            |

    

    As indicated in Table a, in the training phase, the total training time of our EAD model is effectively balanced between the pure adversarial training method DOA and the partially adversarial methods like SAC and PZ. This efficiency stems mainly from our unique USAP approach, which bypasses the need for generating adversarial examples, thereby boosting training efficiency.
    In terms of model inference, our EAD, along with PZ and DOA, demonstrates superior speed compared to LGS and SAC. This is attributed to the latter methods requiring CPU-intensive, rule-based image preprocessing, which diminishes their inference efficiency. Further details, including loss curves and more precise time consumption metrics, are elaborated in Appendix D.6.

Reviewer comment (Evaluation):
Why is the enhanced version of SAC less robust against dodging attacks than regular SAC (in table 1)?

Author response:
**Enhanced SAC is less robust than the regular one against dodging attacks.**

    SAC is a preprocessing-based method that adopts a segmentation model to detect patch areas, followed by a "shape completion" technique to extend the predicted area into a larger square, and remove the suspicious area. We observed that the enhanced SAC, while exhibiting superior segmentation performance, inadvertently increases the likelihood of masking critical facial features such as eyes and noses in scenarios like face recognition. As a result, the Face Recognition (FR) model is more likely to misidentify the masked faces as different identities, thereby reducing the efficacy of dodging attacks. Besides, we also provide an illustrative example (Figure 10) and more details in the Appendix D.8 to elucidate this phenomenon.