Paper: Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR-2024)

Reviewer comment (Theory):
Although it makes sense to consider a constrained attack model to obtain better stealthiness, the proposed formulation is a bit confusing. In particular, if the agent is only partially controlled by the attacker, there is a three-player game between the victim, the agent partially controlled by the attacker, and the attacker. In this case, why should the agent adopt a fixed policy without adapting to the victim's policy?

Author response:
In line with our answer to Question 2:

- **Fixed policy assumption at test time**: It is a plausible assumption that a fixed policy is employed during the test phase. This is consistent with the premise adopted in related works [1, 2, 3] where the victim agent is also presumed to utilize a fixed policy at test time. Consequently, it becomes unnecessary for agent $\alpha$ to implement an adaptive strategy in response to the victim's actions, given the static nature of the interaction during testing.
- **Pre-attack training and framework applicability**: Prior to any attack, the policy pair $(\hat{\pi}\_\alpha, \hat{\pi}\_\nu)$ undergoes joint training within game scenarios, enabling them to adapt to each other. Importantly, our framework is versatile and applicable to any policy $\hat{\pi}\_\alpha$, including those that have been co-trained with $\hat{\pi}\_\nu$. This adaptability enhances the robustness and relevance of our approach across various scenarios.

Reviewer comment (Experiments):
In Figure 1, it says that an unconstrained adversarial policy and a constrained policy with the same winning rate are chosen. How is this achieved? Also, is the same victim policy used in both cases? Intuitively, a smaller epsilon should produce a lower state distribution shift and a stealthier attack, but at the same time, the victim's (best possible) performance should improve, given that the attacker becomes weaker.

Author response:
- Firstly, the same victim policy used in both cases.
- **Secondly, it is true that varying attack budgets will not yield the same rewards or win rates.** In **Appendix E4**, we shows the trade-off: reduced budgets boost stealth but might limit the potential of achieving higher rewards.
- **Thirdly, the selected policies for the unconstrained attack do NOT necessarily reflect the algorithms' final state**. It's important to clarify that the policies selected for comparison in the unconstrained attack scenario do not necessarily represent the ultimate outcome of the algorithms. For instance, if an adversarial policy developed through our method attains a win rate of 0.7, we juxtapose it with an unconstrained adversarial policy also having a win rate of 0.7. However, such an unconstrained policy may not reflect the final convergence point of the baseline algorithm. The key takeaway is that when an attacker prioritizes stealthiness while targeting a specific win rate, our constrained attack methodology demonstrates superior stealth performance.

Reviewer comment (Theory):
Stealthiness is defined as the difference in state distributions. This is a step in the right direction and seems like a good proxy for real-world problems. However, two identical state distributions can come from very different trajectories (e.g. just inverting the order of states within all trajectories does not change the state distribution). This should be considered and discussed in more detail.

Author response:
> Q1. Stealthiness is defined as the difference in state distributions. This is a step in the right direction and seems like a good proxy for real-world problems. However, two identical state distributions can come from very different trajectories (e.g. just inverting the order of states within all trajectories does not change the state distribution). This should be considered and discussed in more detail.

- **Firstly, according to our standard definition of state occupancy, inverting the order of states within all trajectories could still change the state distribution.** The reason for this lies in Definition 2.1, where the probability is expressed as a sum of discounted probabilities. This formulation implies that states appearing later in a trajectory contribute less to the overall state distribution. Consequently, reversing the order of states within all trajectories can result in a subtle alteration of the state distribution. This distinction, rooted in the mathematical structure of the probability definition, highlights the impact of temporal positioning on state contributions.
- **We agree with the reviewer that the discrepancy on trajectory distributions can be a more fine-grained metric**. The concept of a trajectory-level distribution is indeed capable of effectively characterizing the stealthiness of an adversarial policy. However, a significant reason for not adopting this metric in our analysis is the considerably larger size of the trajectory space compared to the state space. This poses a challenge in accurately computing discrepancies within trajectory distributions and in implementing constraints on them. The complexity and computational demands associated with managing this larger space are key factors in our decision to focus on state space metrics.

Reviewer comment (Experiments):
The empirical results wrt stealthiness are very limited. Only three videos are provided for the qualitative comparison. A larger study and/or quantitative results would much improve the paper.

Author response:
> Q2. The empirical results wrt stealthiness are very limited. Only three videos are provided for the qualitative comparison. A larger study and/or quantitative results would much improve the paper.

We thank the reviewer for the kind suggestions from the reviewer and shall add more videos in the link we provided in our paper.

We thank the reviewer for appreciating our responses. We apologize for not publishing our updated website correctly. The updated animations can be found in https://sites.google.com/view/stealthy-attack , where we have added two environments for further displaying the stealthiness of our approaches.

If there are additional concerns from the reviewer, please do not hesitate to let us know. We believe your suggestion/comments can improve our paper greatly

Reviewer comment (Novelty):
In my understanding, the novelty of the paper is mainly the "partial control" and "attack budget", compared with the previous work: Gleave et al. (2019); Wu et al. (2021b); Guo et al. (2021). However, the partial control is previously introduced in PR-MDP, Tessler et al. (2019).

Author response:
> Q1. In my understanding, the novelty of the paper is mainly the "partial control" and "attack budget", compared with the previous work: Gleave et al. (2019); Wu et al. (2021b); Guo et al. (2021). However, the partial control is previously introduced in PR-MDP, Tessler et al. (2019).
- **Firstly, our contribution to the generalized attack formulation is not proposing "partial control", but understanding the impacts of a constrained formulation on stealthiness and defenses in a multi-agent setting.** The fact that PR-MDP also considered the partial control does not weaken our novelty but well justifies our problem formulation since our discussions on how a constrained formulation could bring benefits to stealthiness and defenses is unique to our settings of adversarial policies, which are not discussed in PR-MDP, Tessler et al. (2019).
- **Our provable defense algorithm is one of our key novelties, which does not appear in Gleave et al. (2019); Wu et al. (2021b); Guo et al. (2021); Tessler et al. (2019).** Note that Gleave et al. (2019); Wu et al. (2021b); Guo et al. (2021) only considered heuristic defense mechanism while the defense of Tessler et al. (2019) highly depends on the single-agent setup such that the policy iteration algorithm can be efficient. Therefore, our paper improves the defense in Gleave et al. (2019); Wu et al. (2021b); Guo et al. (2021) and the algorithm is significantly different from that of Tessler et al. (2019), thus highlighting the novelty of our paper.

Reviewer comment (Theory):
The motivation of the attack budget and partial control is not enough convincing. From the attack side, it is true that the setting of attack budget can improve the stealthiness, but it is hard to say that the proposed attack is optimal subject to the limitation of the discrepancy between the state distributions (or marginalized transition dynamics).

Author response:
> Q2.1. The motivation of the attack budget and partial control is not enough convincing. From the attack side, it is true that the setting of attack budget can improve the stealthiness, but it is hard to say that the proposed attack is optimal subject to the limitation of the discrepancy between the state distributions (or marginalized transition dynamics).

- **We believe the discrepancy between the state distributions (or marginalized transition dynamics) itself is a good metric to evaluate stealthiness but not a natural metric to model real-world constraints that attackers could encounter.**  As in our response to Q1, our paper focuses on the understanding of a generalized attack formulation for real-world applications that often exhibit certain natural constraints on the attacker, instead of proposing a specific type of attacking strategy to ensure optimal stealthiness.
- **The optimal adversarial policy subject to the stealthiness conditions suggested by the reviewer could potentially violate the common and well-accepted constraint on the attacker (Equation 3.2 and also Tessler et al. (2019)).** Therefore, we believe developing such optimal attacks under the stealthiness constraints is a promising and interesting direction but beyond the scope of our paper of understanding the generalized formulation and provable defense mechanisms of adversarial policies.

Reviewer comment (Presentation):
The result in Figure 1 is confusing. 1. Why the bottom-right subfigure is empty? 2. Is original state distribution means that the distribution without any attack? If it is true, should not the original state distribution be same in the bottom and top subfigures? The original state distributions in top-mid and bottom-mid subfigures are different. Could you explain the reason? 3. Where do the results in Figure 1 come from? The paper only mentions that two attack polices are assessed under the same winning rate. However, according to the Figure 6, the unconstrained policy has higher wining rate.

Author response:
> Q3.1. The result in Figure 1 is confusing. 1. Why the bottom-right subfigure is empty? 

From the authors' side, we are not aware of any empty figures and we suggest the reviewer try to refresh/reopen our pdf. In case the bottom-right subfigure is still empty, we also provide Figure 1. in the anonymous link (https://drive.google.com/file/d/1LCo824MlmxMT4bOe6QaxNrMAEn3buY9L/view?usp=sharing).

> Q3.2 Is original state distribution means that the distribution without any attack? If it is true, should not the original state distribution be same in the bottom and top subfigures? The original state distributions in top-mid and bottom-mid subfigures are different. Could you explain the reason? 

Firstly, what the reviewer understands is correct that the original state distribution means there is no attack.

Secondly, the original state distribution is **marked by the blue points**. Therefore, the original state distributions in top-mid and bottom-mid subfigures are exactly the same and we are not aware of any differences from Figure 1. We guess the reason why the reviewer feels they are different may be because some blue points in the top-mid subfigure overlap with the orange points.

> Q3.3. Where do the results in Figure 1 come from? The paper only mentions that two attack polices are assessed under the same winning rate. However, according to the Figure 6, the unconstrained policy has higher wining rate.
- **Firstly, we don't claim that varying attack budgets will yield the same rewards or win rates.** What the reviewer understands is correct. Our Appendix E4 shows the trade-off: reduced budgets boost stealth but might limit the potential of achieving higher rewards.
- **Secondly, the selected policies for the unconstrained attack do not necessarily reflect the algorithms' final state**. It's important to clarify that the policies selected for comparison in the unconstrained attack scenario do not necessarily represent the ultimate outcome of the algorithms. For instance, if an adversarial policy developed through our method attains a win rate of 0.7, we juxtapose it with an unconstrained adversarial policy also having a win rate of 0.7. However, such an unconstrained policy may not reflect the final convergence point of the baseline algorithm. The key takeaway is that when an attacker prioritizes stealthiness while targeting a specific win rate, our constrained attack methodology demonstrates superior stealth performance.

Reviewer comment (Presentation):
Despite numerous references to previous work, there is no clean depicted picture of how this work on multi agent robustness fits in the context of previous work. I am not complaining for lack of references here, I just think the presentation should be more clear.

Author response:
We thank for the kind suggestions of the reviewers. The most closely related works to us are [1, 2, 3] and we discuss how our paper fits in the context of them as follows:

- **Our research offers a more comprehensive attack formulation compared to references [1, 2, 3], incorporating additional aspects such as stealth**. Our paper advances the understanding of adversarial policies in practical scenarios, where attackers are often limited by physical world constraints and motivated to maintain stealth.
- **We introduce the first provably effective defense algorithm against adversarial policy threats**, a significant step beyond the primarily heuristic approaches of prior work, which mainly focus on retraining against specific victims.

Reviewer comment (Writing):
Some examples of typos: Def 5.1 : "has" -> 'have'.

Author response:
We thank the reviewer for pointing out these typos and we revised our paper accordingly.

Reviewer comment (Writing):
In paragraph "(c) Connection to action adversarial RL." in page 4, there is "\tilde{\pi}\alpha" in which alpha needs to be the index.

Author response:
We thank the reviewer for pointing out these typos and we revised our paper accordingly.

Reviewer comment (Writing):
In Definition 5.2., please give an intuition of what this is before giving a full line math formula.

Author response:
We apologize the brevity in our explanations of these concepts. Definition 5.2 essentially provides an intuitive measure of the difficulty an agent faces while exploring the environment. This is achieved by comparing the stationary state occupancy frequencies under certain policies against the initial state distribution. In simpler terms, a smaller value of this quantity indicates that the environment is more easily explorable. We have also included more detailed explanations in our revised paper.

Reviewer comment (Theory):
You say "performance. This contrasts with supervised learning attacks, where small perturbations can result in large performance shifts." but this seems to be only applicable to test time attacks, as poisoning has a limited effect if it is limited to eps-fraction of data (e.g., ERM will be effected by eps increase in its probability of error as well).

Author response:
We agree with the reviewer that our argument mainly applies to test-time attacks. We have made our argument more accurate in the revision. Since our setting is focusing on test-time attacks for RL, we are comparing it with test-time attacks of supervised learning. For training-time attacks, the effect of the poisons can have very different effects on the victim policy, which we believe is an interesting future work.

Reviewer comment (Writing):
When you say "However, it has been demonstrated that while re-training against a specific adversarial policy does augment robustness against it, the performance against other policies may be compromised." what is the reference?

Author response:
This has been validated experimentally by [1] in Section 6. Intuitively, if the victim is retrained against a specific attacker, its policy might be overfitted to the trained attacker. Hence, this highlights the necessity of more advanced adversarial training algorithms instead of naive retraining.