Paper: BATTLE: Towards Behavior-oriented Adversarial Attacks against Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
The writing needs improvement, particularly in clarifying several terms and diagrams. For example, some terms like "find a precise weighting function to balance the state distribution" need better explanation. Clarification is also needed for the diagram in Figure 2.

Author response:
We apologize for the confusion. Regarding the term 'find a precise weighting function to balance the state distribution,' our intent is to convey the process of training a weighting function that can appropriately assign weights to various states. This function plays a critical role in balancing the influence of different states during the learning process. We will provide a more detailed description of this and update Figure 2 to enhance its clarity in the next version.

Reviewer comment (Presentation):
The presentation of experimental results is somewhat confusing. The differences of scenarios in Figure 4 and 5 are not clear, and additional explanations are required for the target coordinates mentioned for Figure 4. Captions of Figures 7 (a) and (b) might need to be swapped, and sections (c) and (d) require clearer explanations.

Author response:
Thanks for your question. Regarding Figures 4 and 5, the scenarios presented in these figures differ in the objectives set for the adversary. In the manipulation scenario, as depicted in Figure 4, the adversary's objective is to manipulate the victim into grasping fixed objects located far from the original target location. On the other hand, in the opposite behavior scenario shown in Figure 5, the attacker's goal is to induce the victim to behave in a manner that is contrary to its original behavior. More detailed can be found in Appendix F.4. As for Figure 7, we will revise it in the revison.

Reviewer comment (Presentation):
The paper lacks a discussion of limitations, which should be addressed.

Author response:
Thanks for your question. In the revised version of our paper, we will include a section dedicated to discussing the limitations.

Reviewer comment (Experiments):
It's disappointing that the paper doesn't include RADIAL-RL[1] or WocaR-RL[2] as baselines when discussing robust training. Even if only ATLA is selected as a robust baseline, it would be valuable to mention other adversarial robust RL papers in the related work.

Author response:
We appreciate the reviewer's insightful feedback. While our paper primarily focuses on introducing a novel adversarial attack method, we are pleasant to include more powerful baselines when discussing robust training to make our discussion more comprehensive.

Reviewer comment (Novelty):
In the introduction, the paper illustrates the practical implications of targeted attacks on robotics, but the concern is raised that BATTLE is a white-box attack applying perturbations to states. In the context of robotics, its practical applicability is very limited. The paper could benefit from a more thorough clarification or discussion of this concern and its potential implications for practical applications. It fails to persuasively underscore the significance and relevance of this work within the field.

Author response:
Thanks for your question. It is crucial to note that BATTLE is designed as a black-box attack method, not a white-box attack as might have been perceived. This means that BATTLE does not require access to the gradients of the victim model. Instead, it operates by observing the external outputs of the system, making it much more applicable in real-world scenarios where internal access to a system, such as a robotic platform, is typically not feasible.

Reviewer comment (Experiments):
The method's reliance on extensive human labeling hampers its real-world applicability, raising concerns about scalability.

Author response:
**A3**: In the manipulation scenario, we utilized approximately 9000 labels across all tasks. For the opposite behaviors scenario, the number of labels varied based on the specific task: 1000 for Window Close, 3000 for Drawer Close, 5000 each for Faucet Open, Faucet Close, and Window Open, and 7000 each for Drawer Open, Door Lock, and Door Unlock. Detailed information about the implementation and labeling process is provided in Section 5.1 and Appendix F of our paper.  
>
>It is important to emphasize that BATTLE represents an advancement over previous work in two key aspects:  
> 1. BATTLE does not require knowledge of the victim's reward function. Labeling preference is more feasible and less challenging, as acquiring the victim's reward function is often impractical.  
>2. Unlike some previous methods, BATTLE does not require access to the victim's gradients. It operates by observing the external outputs of the system, which makes it significantly more practical.

Reviewer comment (Reproducibility):
The paper lacks some critical methodological details. For example, it doesn't specify how the victim policy approximator is trained or the volume of data required, leaving gaps in the understanding of the implementation.

Author response:
**A4**: Thanks for your question. We train the victim models on the Meta-world tasks using the SAC algorithm [1]. Specificially, we employ fully connected neural networks for the policy and Q function. The detailed hyperparameters used in our experiments are provided in Table 5. As for the amount of data, we train each agent for 1 million time steps, which can achieve around 100% success rate on original tasks.  
>
**Reference**\  
[1] Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.

Reviewer comment (Evaluation):
Expanding the experimental results to include additional comparison metrics would be valuable. Specifically, how does PALM fare against targeted attacks and various other defense methods?

Author response:
**A2**: Thanks for your question. As detailed in Table 1 of our paper, we subjected robustly trained agents to three different types of attacks to evaluate their robustness. Among these, our BATTLE-ATLA method demonstrated the most robust performance under multiple attack scenarios. This suggests that BATTLE-ATLA is highly effective in challenging even well-defended agents. Furthermore, we applied various defense methods to gauge the robustness of these agents. Our findings indicate that BATTLE consistently outperformed other strategies, showcasing its strong capability in executing universal attacks against robust models. These results collectively highlight BATTLE's efficacy as both an offensive and defensive tool in adversarial settings.

Reviewer comment (Reproducibility):
Could you provide details on the training process for the victim policy approximator, including the amount of data needed for effective training?

Author response:
**A4**: Thanks for your question. We train the victim models on the Meta-world tasks using the SAC algorithm [1]. Specificially, we employ fully connected neural networks for the policy and Q function. The detailed hyperparameters used in our experiments are provided in Table 5. As for the amount of data, we train each agent for 1 million time steps, which can achieve around 100% success rate on original tasks.  
>
**Reference**\  
[1] Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.

Reviewer comment (Writing):
I have found some of the writing to be confusing and verbose. For example, “intention policy” is not defined until multiple mentions in. The description of what it is in the abstract is very ambiguous. “Inner” and “outer” level loss are not described. I don’t see a definition for “success rate” in the paper. I don’t think the paper does as good of a job as it could with laying out things in a way that is clear and quick to understand.

Author response:
**A1**: Thank you for your feedback on our writing. We'd like to clarify the following points.
> (a) The primary goal is to ensure that the victim's behavior under adversarial attacks is aligned with intention policy, which is trained following the framework of PbRL. (b) The inner and outer losses correspond to Equations (6) and (7) in our paper, respectively. \(c\) The success rate is determined by evaluating whether the victim performs behaviors as intended by humans. For example, if the human intention is for the victim to open a window, the success rate is calculated based on whether the window is actually opened by the victim under adversarial attacks. Specifically, we have adopted the success metric directly from Meta-world [1], which is typically based on the distance between the task-relevant object and its final goal pose. For the complete list of success metrics and thresholds for each task, see Appendix 12 of Meta-world. (d) In Figure 2, the arrow connecting the reward learning component to the replay buffer is intended to represent reward estimation, we will revise it in the revision.

Reviewer comment (Theory):
I do not understand the rationale in the first paragraph of section 4.1. I do not think this makes sense as an explanation of why an intention policy is needed instead of a direct attack.

Author response:
**A2**: Thank you for your valuable feedback. As shown in the results presented in Table 2 of our paper, specifically the column 'BATTLE w/o $\pi_\theta$', which indicates that directly training the adversarial policy using PbRL was not as effective. The crucial insight from our research is that the intention policy offers essential step-by-step guidance that significantly enhances the efficiency and effectiveness of the adversarial policy development process.

Reviewer comment (Presentation):
I do not understand figure 1. Why is there an arrow between reward learning and the replay buffer?

Author response:
**A1**: Thank you for your feedback on our writing. We'd like to clarify the following points.
> (a) The primary goal is to ensure that the victim's behavior under adversarial attacks is aligned with intention policy, which is trained following the framework of PbRL. (b) The inner and outer losses correspond to Equations (6) and (7) in our paper, respectively. \(c\) The success rate is determined by evaluating whether the victim performs behaviors as intended by humans. For example, if the human intention is for the victim to open a window, the success rate is calculated based on whether the window is actually opened by the victim under adversarial attacks. Specifically, we have adopted the success metric directly from Meta-world [1], which is typically based on the distance between the task-relevant object and its final goal pose. For the complete list of success metrics and thresholds for each task, see Appendix 12 of Meta-world. (d) In Figure 2, the arrow connecting the reward learning component to the replay buffer is intended to represent reward estimation, we will revise it in the revision.

Reviewer comment (Experiments):
This may speak to either issues with my reading of the paper, its writing, or the quality of the experiments. But I am unsure why BATTLE trains an intention policy with PbRL instead of just training the adversarial policy directly with PbRL. This would seem to be substantially simpler.

Author response:
**A2**: Thank you for your valuable feedback. As shown in the results presented in Table 2 of our paper, specifically the column 'BATTLE w/o $\pi_\theta$', which indicates that directly training the adversarial policy using PbRL was not as effective. The crucial insight from our research is that the intention policy offers essential step-by-step guidance that significantly enhances the efficiency and effectiveness of the adversarial policy development process.

To address your concerns effectively, I'd like to first clarify the threat model of the proposed universal attack. Threat Model:
1. **Adversary's Limited Access**: The adversary cannot access the victim's gradients or reward function.
2. **Dynamic Learning of Attack Goal**: Unlike previous approaches with pre-defined goals, our attack goal requires dynamic learning, which is one key aspect of our method's novelty.
3. **Adversary's Strategy and Constraints**: The adversary perturbs the state $s$ into $\tilde{s}$ restricted by $\mathcal{B}(s)$ (i.e., $\tilde{s} \in \mathcal{B}(s)$). $\mathcal{B}(s)$ is defined as a small set {$\{\tilde{s} \in \mathcal{S}: \parallel s-\tilde{s} \parallel_p \le \epsilon\}$} which limits the attack power of the adversary, and $\epsilon$ is the attack budget. This follows the settings in previous methods.

Therefore, the success of attacking the victim hinges on the algorithm's effectiveness, the amount of feedback provided, and the attack budget.

We see that SA-RL (PbRL) and PA-AD (PbRL), which essentially involve training the adversary or director directly with PbRL, do not work as well when there are strict limits, like less feedback and smaller attack budgets. Our additional experiments specifically examined the impact of varying amounts of feedback and attack budgets. When provided with more feedback and larger attack budgets, the performance of SA-RL (PbRL) improved noticeably. Attack budgets, in particular, have a significant influence. SA-RL (oracle) utilizes ground-truth rewards but still exhibits suboptimal performance in some scenarios. These findings indicate that training the adversarial policy directly with PbRL can work. However, this requires a higher volume of feedback and a larger attack budget, making it costly and impractical. In contrast, BATTLE can significantly enhance training efficiency and performance by introducing an intention policy and utilizing the proposed learning framework.

Reviewer comment (Evaluation):
I do not understand how SA-RL can perform so poorly relative to BATTLE unless it is just due to reward shaping. What is the reward function used for SA-RL? If an adversarial policy is directly trained to minimize the agent’s reward, how can this do worse than BATTLE at making the target agents’ reward be minimized? If the key difference truly is just reward shaping, then this paper would just seem to be one about how PbRL makes reward shaping automatic and implicit. And if so, then this paper would seem to have no novelty.

Author response:
**A3**: Thank you for your question. In our paper, the focus is on a universal adversary that can manipulate the victim to perform behaviors as desired by humans, rather than simply minimizing the victim's cumulative reward. In our experiments, the reward model for all methods is derived from human preferences. The key difference lies in how the adversarial policy is trained. BATTLE's superior performance is largely due to its bi-level optimization framework, which integrates the intention policy and a weighting function.

To address your concerns effectively, I'd like to first clarify the threat model of the proposed universal attack. Threat Model:
1. **Adversary's Limited Access**: The adversary cannot access the victim's gradients or reward function.
2. **Dynamic Learning of Attack Goal**: Unlike previous approaches with pre-defined goals, our attack goal requires dynamic learning, which is one key aspect of our method's novelty.
3. **Adversary's Strategy and Constraints**: The adversary perturbs the state $s$ into $\tilde{s}$ restricted by $\mathcal{B}(s)$ (i.e., $\tilde{s} \in \mathcal{B}(s)$). $\mathcal{B}(s)$ is defined as a small set {$\{\tilde{s} \in \mathcal{S}: \parallel s-\tilde{s} \parallel_p \le \epsilon\}$} which limits the attack power of the adversary, and $\epsilon$ is the attack budget. This follows the settings in previous methods.

Therefore, the success of attacking the victim hinges on the algorithm's effectiveness, the amount of feedback provided, and the attack budget.

We see that SA-RL (PbRL) and PA-AD (PbRL), which essentially involve training the adversary or director directly with PbRL, do not work as well when there are strict limits, like less feedback and smaller attack budgets. Our additional experiments specifically examined the impact of varying amounts of feedback and attack budgets. When provided with more feedback and larger attack budgets, the performance of SA-RL (PbRL) improved noticeably. Attack budgets, in particular, have a significant influence. SA-RL (oracle) utilizes ground-truth rewards but still exhibits suboptimal performance in some scenarios. These findings indicate that training the adversarial policy directly with PbRL can work. However, this requires a higher volume of feedback and a larger attack budget, making it costly and impractical. In contrast, BATTLE can significantly enhance training efficiency and performance by introducing an intention policy and utilizing the proposed learning framework.

Reviewer comment (Novelty):
Relatedly, I do not understand why this paper is about adversarial attacks. BATTLE could be used to make RL policies do anything — not just to targetedly adversarially attack them. But in reality, to make RL policies do things, we just finetune them directly. This relates to why I do not understand why the intention policy was used. Why not just finetune the target agent or adversary directly with PbRL?

Author response:
**A4**: Thank you for your question. Our work aims to propose a universal adversarial attack rather than policy tuning, where techniques like RLHF [1] or DPO [2] involve directly training the policy. Instead, our focus is on how to train a universal adversary capable of manipulating a victim to act according to human-desired behaviors.

To address your concerns effectively, I'd like to first clarify the threat model of the proposed universal attack. Threat Model:
1. **Adversary's Limited Access**: The adversary cannot access the victim's gradients or reward function.
2. **Dynamic Learning of Attack Goal**: Unlike previous approaches with pre-defined goals, our attack goal requires dynamic learning, which is one key aspect of our method's novelty.
3. **Adversary's Strategy and Constraints**: The adversary perturbs the state $s$ into $\tilde{s}$ restricted by $\mathcal{B}(s)$ (i.e., $\tilde{s} \in \mathcal{B}(s)$). $\mathcal{B}(s)$ is defined as a small set {$\{\tilde{s} \in \mathcal{S}: \parallel s-\tilde{s} \parallel_p \le \epsilon\}$} which limits the attack power of the adversary, and $\epsilon$ is the attack budget. This follows the settings in previous methods.

Therefore, the success of attacking the victim hinges on the algorithm's effectiveness, the amount of feedback provided, and the attack budget.

We see that SA-RL (PbRL) and PA-AD (PbRL), which essentially involve training the adversary or director directly with PbRL, do not work as well when there are strict limits, like less feedback and smaller attack budgets. Our additional experiments specifically examined the impact of varying amounts of feedback and attack budgets. When provided with more feedback and larger attack budgets, the performance of SA-RL (PbRL) improved noticeably. Attack budgets, in particular, have a significant influence. SA-RL (oracle) utilizes ground-truth rewards but still exhibits suboptimal performance in some scenarios. These findings indicate that training the adversarial policy directly with PbRL can work. However, this requires a higher volume of feedback and a larger attack budget, making it costly and impractical. In contrast, BATTLE can significantly enhance training efficiency and performance by introducing an intention policy and utilizing the proposed learning framework.

Reviewer comment (Experiments):
The human preference-based intention policy learning brings extra requirement and uncertainty to the process - the collection of human preference data can be expensive. More importantly, to obtain human preference labels, one need to first collect diverse behavior data so that human can pick the intended policy. Would the collection of the behavior data already involve a pre-defined target policy? (If that's the case, why not directly use the target policy for attacks?)

Author response:
**A1**: Thanks for your question regarding the requirement and uncertainty involved in collecting human preference data in PbRL. In the PbRL paradigm, humans are required to provide preferences over pairs of trajectories generated from the victim policy. Therefore, the process does not involve a pre-defined target policy. In fact, the intention policy acts as an adaptive target policy, dynamically aligning with human preferences. There are several reasons why we do not use a pre-defined target policy: (1). It is often challenging to find a pre-defined target policy that precisely meets the specific requirements of a given task or scenario. (2). Relying on a pre-defined target policy often leads to suboptimal outcomes, which fails to adapt to the nuances and variations in different environments.

Reviewer comment (Experiments):
For baselines, the authors used the codebases of SA-RL and PA-AD and modified their attack's reward as the learned reward. However, this straightforward modification of the PA-AD baseline contradicts with the original method (PA-AD's formulation is for a reward-minimizing adversary, so directly replacing the attacker's reward may not work). Since the original PA-AD method is to use an RL director to find the target policy and to use an actor to conduct targeted attack, a more natural modification of PA-AD in the behavior attack scenario can be to directly use the learned intention policy as the target of actor ($\hat{a}$ in Equation (G) in the original paper).

Author response:
**A3**: Thank you for your insightful comments. Let me address each point separately:\
**Reward Model**: From a general perspective, the reward model represents the direction of optimization for the adversarial policy. In the original SA-RL [2] and PA-AD [3], the adversary's reward model is the negative of the victim's reward function $r_v$, i.e. $-r_v$. In our work, however, the reward model is learned from human preferences, which allows expression of various intentions and thus enables universal adversarial attacks. In our experiments, the reward learning is the same across all methods, with the primary differences being in the adversary's learning. BATTLE's outstanding performance is attributed to its bi-level optimization framework, where the intention policy and the weighting function effectively collaborate to enhance the adversary's training.\
**Modification of PA-AD**: We also provided oracle versions of SA-RL and PA-AD, utilizing ground-truth rewards as the reward model. As shown in Figure 4, SA-RL demonstrated significant improvement in most tasks, but the same was not observed for PA-AD. A possible explanation for this could be related to the inherent design of PA-AD, which performs optimally by accessing the victim's gradients in scenarios using a fixed reward model. However, when it comes to dynamically learning the reward model, this approach may not be as successful.

Reviewer comment (Experiments):
How are the behavior sequences generated for human preference labeling?

Author response:
**A4**: Thanks for your question. To provide a clear explanation, let's consider an example from our preference labeling process. We collect pairs of victim trajectories under adversary attacks, denoted as $(\sigma_v^0, \sigma_v^1)$, which correspond to two adversary-generated trajectories $(\sigma_a^0, \sigma_a^1)$. When a human prefers $\sigma_v^0$ over $\sigma_v^1$ (i.e., $\sigma_v^0 \succ \sigma_v^1$), it implies that the attack in $\sigma_a^0$ is more effective or aligns better with human intent, leading to $\sigma_a^0 \succ \sigma_a^1$. Using the preference data on the adversary's trajectories, we can learn an adversary that aligns with human intentions.