Paper: SHINE: Shielding Backdoors in Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
No formal definition of attack and defense is provided. It is not even clear when does attacker attack, does it use any attack policy? The notations used in section 3.3 is not sufficiently clear.

Author response:
**1. About the definition of the attacks and defenses.**

Thank the reviewer for the comment. As stated in the introduction, the existing attacks against DRL policy train a victim policy $\pi$ for an agent in a given environment. The agent performs normally in the clean environment, i.e., will finish its corresponding task with a high success rate and achieve a high reward. The victim policy will take suboptimal actions when the trigger shows in the environment state. This will lower the victim agent's reward. 

Specifically, perturbation-based attacks add a small patch as the trigger to the environment state and the suboptimal actions of the victim policy can be a random action (Non-targeted attacks) or a specific action (targeted attacks). As stated in the evaluation section, we follow the original attack papers and add the trigger patch to each state with a certain probability. 

Adversarial-agent attacks target two-player competitive games with an adversarial agent and a victim agent. The trigger is a sequence of continuous actions of the adversarial agent. Once the adversarial agent takes the trigger actions, the victim agent will perform suboptimally. As stated in the evaluation, we follow the original attack paper and let the adversarial agents take the trigger action at the start of a game. 
We are sorry for the confusion. We will summarize these attack descriptions in one paragraph in the next version.

Regarding our defense, we are given a shielding agent’s policy $\pi$ and identify a trigger $\tau$ at the trigger restoration step (Section 3.2). Then, we use the restored trigger to retrain the policy and obtain a shielded policy $\hat{\pi}$.

Reviewer comment (Theory):
In adversarial agent attack, how are actions embedded in state as you claim in the beginning of page 5? Actions usually come from a different space of discrete state than states.

Author response:
**2. About Adversarial agent attacks.**

Thank the reviewer for the comment. We follow the setups and arguments in the backdoorl paper. For MuJoCo environments, the action space of both agents is continuous rather than discrete. Second, in the state representation of one agent, there are certain dimensions corresponding to the other agent’s action. As such, the actions of one agent are embedded in the state representation of the other agent. We consider MuJoCo environments because this is the only environment in which backdoorl apply their attacks.

Reviewer comment (Experiments):
Learning a feature explanation mask for each pixel in a state does not look very scalable especially in environments like Atari games where state space could get really large. How do you address this problem?

Author response:
**3. Questions about the explanation methods.**

Thank the reviewer for the questions. First, our proposed feature-level explanation learns a mask matrix for all pixels in the state representation, where all the individual masks are optimized together. As such, our method can handle environments with high-dimensional state representations, such as Atari games (as shown in our evaluation). 

Regarding the local linear constraint in the step-level explanation (EDGE), it is inherent in the AlvarezMelis & Jaakkola (2018) paper. It helps make the explanation method perform locally linear around the given samples, which improves the explainability of the approximation model. It is equivalent to conducting a piece-wise linear approximation of a complicated non-linear function, where the piece-wise linear function is explainable. We will clarify this in the next version.

Reviewer comment (Writing):
What is local linear constraint in line above Feature-level explaination paragraph? How does it help?

Author response:
**3. Questions about the explanation methods.**

Thank the reviewer for the questions. First, our proposed feature-level explanation learns a mask matrix for all pixels in the state representation, where all the individual masks are optimized together. As such, our method can handle environments with high-dimensional state representations, such as Atari games (as shown in our evaluation). 

Regarding the local linear constraint in the step-level explanation (EDGE), it is inherent in the AlvarezMelis & Jaakkola (2018) paper. It helps make the explanation method perform locally linear around the given samples, which improves the explainability of the approximation model. It is equivalent to conducting a piece-wise linear approximation of a complicated non-linear function, where the piece-wise linear function is explainable. We will clarify this in the next version.

Reviewer comment (Theory):
Without any assumption, it may happen that the adversary may not attack when the algorithm is run. Does the algorithm still work?

Author response:
Thanks the reviewer F9HD for the additional comment. We have responded in points 1&4, a clear setup of our attack and defense, together with our assumption. In point 5, we responded to the corner cases mentioned by the reviewer. It would be super helpful if the reviewer could point out which part of the setup and assumption is still not clear. Currently, we have made it clear that our defense is effective against both perturbation-based attacks and adversarial agent attacks under single- and multi-agent environments. We follow the attack setups of all three main-stream backdoor attacks against DRL. That is, the attacker adds a trigger (either a patch or a sequence of actions) to the environment will significantly reduce the total reward of the victim agent collected in one episode. The patch can be all zero (as pointed out in response point 5). The additional assumption our defense requires is the trigger needs to be presented. We really appreciate it if the reviewer could point out more specifically which part is still clear. Thank you very much!

Reviewer comment (Experiments):
It would be helpful to test the proposed methods against a broader set of backdoor attacks.

Author response:
Thank the reviewer for the comment. In our evaluation, we have evaluated our method against all three existing mainstreaming backdoor attacks in RL. These attacks cover both single-agent and multi-agent RL. They also consider different types of triggers: state perturbations and adversarial agents. We also consider different attack variations and adaptive attacks. With this large set of attacks, we humbly believe that our evaluation has already covered most attacks against RL and is able to demonstrate the effectiveness of our defense. If the reviewer believes we need to evaluate SHINE against other specific attacks, we are happy to add those additional evaluations.

Reviewer comment (Experiments):
Could you also compare your method with other types of defenses, e.g. non-trigger-inversion?

Author response:
Thank the reviewer for the comment. We follow the reviewer’s comment and add one more baseline defense: STRIP [1]. This defense assumes the inputs containing the trigger, and it tries to pinpoint the trigger. It does not conduct trigger inversion (like FeatureRE or NE). We use this method to detect the trigger for perturbation-based attacks (given that it can only be applied to games with discrete action spaces). After detecting the trigger, we apply our retraining method to remove the backdoor.  We selected the pong environment for attacks against single-agent RL and the QMIX setup for attacks against multi-agent RL. The results in the following table show that SHINE is better than this method. In addition, SHINE can apply to adversarial agent attacks, while STRIP cannot. 

|           | Method | Pong  | QMIX  |
|-----------|--------|-------|-------|
| Fidelity  | SHINE  | 0.998 | 0.936 |
| Fidelity  | STRIP  | 0.853 | 0.811 |
| Operating | SHINE  | 0.728 | 99.1  |
| Operating | STRIP  | 0.624 | 85.3  |
| Clean     | SHINE  | 0.734 | 99.2  |
| Clean     | STRIP  | 0.612 | 87.5  |

[1] STRIP: A Defence Against Trojan Attacks on Deep Neural Networks

Reviewer comment (Novelty):
Related work: The related work section could use some more efforts in elaborating how adversarial based attack works, as well as more prior work on the DRL explanation.

Author response:
**2. Questions about the related work.**  

Thank the reviewer for the comment. The review first asked for more description of “ how adversarial based attack works.” We would like to kindly confirm with the reviewer if the reviewer referred to the “adversarial-agent attack” or other attacks. The adversarial agent attack is designed for two-agent competitive games, where one agent is the adversarial agent that takes the trigger action, and the other is the victim agent that contains the backdoor. At a high level, this attack designs the trigger as a sequence of continuous actions of the adversarial agent. Then, it constructs two policies for the victim agent. One policy is a normal policy, and the other is a policy that reacts to the trigger actions and fails the task. Finally, it fuses these two policies into one policy network via behavior cloning and uses it as the victim agent’s policy. We will add this to the next version (Please let us know if the reviewer referred to some other attacks by saying “adversarial based attack”). The reviewer also suggests providing more literature about the DRL explanation. We had some related descriptions in the Section 3. We will follow the reviewer’s suggestion and provide more literature review in Section 2.

Reviewer comment (Experiments):
Need more elaboration on how step and feature-level explanation would be applied if attack is adversarial based, and how multi-agent RL can benefit from this approach.

Author response:
**3. Questions about the generalizability.**  

Thank the reviewer for the comment. Again, we would kindly ask for the reviewer to explain a little bit more concretely what the “attack is adversarial based” refers to. If it refers to adversarial agent-based attacks, we evaluated such attacks in the evaluation. Since we target trojan attacks, which are launched by the attacker, they are naturally adversarial attacks. In our evaluation, we also evaluate our defense against the state-of-the-art backdoor attacks in multi-agent RL and demonstrate that SHINE is still effective against this attack. In Appendix A.4, we explained how we extend SHINE to this attack. At the high level, we shield each backdoored agent in turn. We just need to change the input to explanation methods (local or global observation) based on the agents’ training methods.

Reviewer comment (Theory):
Need more explanation on the two constraints in Eq (4), which is the key of the re-training process. In general the writing of the Sec 3.3 is too abstract to derive to the final objective.

Author response:
**4. Questions about the constraints in Eqn (4).**  

Thank the reviewer for the comment. The first constraint ($\hat{K}$) constrains the agent’s performance change in the poisoned (operating) environment. This is the constraint used on the PPO algorithm when updating the policy. Using it together with the optimization objective function in Eqn. (4) enforces the agent to achieve a better performance in the poisoned environment (removing the backdoor). The second constraint ($K$) comes from Theorem 2. It constrains the agent’s performance change in the clean environment. This constraint helps maintain the agent’s performance in the clean environment (maintaining the agent’s utility, i.e., clean performance). $M_{\pi}(\hat{\pi}) \leq M_{\pi}(\pi)$ because we maximize $M_{\pi}(\hat{\pi})$, so the $M_{\pi}$ of the updated policy $\hat{\pi}$ is at least the same as the $M_{\pi}$ of the previous policy $M_{\pi}(\pi)$. $M_{\pi}(\pi) = \hat{\eta}(\pi)$ because the KL term equals zero, and the advantage function is also zero when $\hat{\pi} = \pi$. We will follow the reviewer’s comment and make this part more clear in the next version.

Reviewer comment (Experiments):
Does the number of pre-trained trajectories matter to the shielding capability?

Author response:
**1. Sensitivity analysis regarding the impact of the number of trajectories used for explanation.**

Thank the reviewer for the comment. Following the reviewer’s comment, we conducted an additional experiment to evaluate the impact of the number of trajectories on our method. We used the Pong game for this experiment. We vary the number of trajectories as 200/500/1000 and report the trigger fidelity and final retraining performance. The results in Table 1 show that our method is not that sensitive to this factor. It also shows that SHINE does not require extensive failed cases to apply shielding. In addition, as stated in Section 5-Postmortem defense, we discuss that our goal is to quickly react to the backdoor attack once it is triggered. 
| # of trajectories             | Trigger fidelity | Operating | Clean |
|-------------------------------|------------------|-----------|-------|
| Failure: 100 and Succeed: 900 | 0.998            | 0.728     | 0.734 |
| Failure: 50 and Succeed: 450  | 0.991            | 0.721     | 0.729 |
| Failure: 20 and Succeed: 180  | 0.978            | 0.698     | 0.709 |

Reviewer comment (Experiments):
How this method applies to Multi-agent DRL scenarios?

Author response:
**3. Questions about the generalizability.**  

Thank the reviewer for the comment. Again, we would kindly ask for the reviewer to explain a little bit more concretely what the “attack is adversarial based” refers to. If it refers to adversarial agent-based attacks, we evaluated such attacks in the evaluation. Since we target trojan attacks, which are launched by the attacker, they are naturally adversarial attacks. In our evaluation, we also evaluate our defense against the state-of-the-art backdoor attacks in multi-agent RL and demonstrate that SHINE is still effective against this attack. In Appendix A.4, we explained how we extend SHINE to this attack. At the high level, we shield each backdoored agent in turn. We just need to change the input to explanation methods (local or global observation) based on the agents’ training methods.

Reviewer comment (Theory):
It is not clear to me why $M_\pi(\hat{\pi}) \geq M_\pi(\pi)=\hat{\eta}(\pi)$.

Author response:
**4. Questions about the constraints in Eqn (4).**  

Thank the reviewer for the comment. The first constraint ($\hat{K}$) constrains the agent’s performance change in the poisoned (operating) environment. This is the constraint used on the PPO algorithm when updating the policy. Using it together with the optimization objective function in Eqn. (4) enforces the agent to achieve a better performance in the poisoned environment (removing the backdoor). The second constraint ($K$) comes from Theorem 2. It constrains the agent’s performance change in the clean environment. This constraint helps maintain the agent’s performance in the clean environment (maintaining the agent’s utility, i.e., clean performance). $M_{\pi}(\hat{\pi}) \leq M_{\pi}(\pi)$ because we maximize $M_{\pi}(\hat{\pi})$, so the $M_{\pi}$ of the updated policy $\hat{\pi}$ is at least the same as the $M_{\pi}$ of the previous policy $M_{\pi}(\pi)$. $M_{\pi}(\pi) = \hat{\eta}(\pi)$ because the KL term equals zero, and the advantage function is also zero when $\hat{\pi} = \pi$. We will follow the reviewer’s comment and make this part more clear in the next version.