Paper: Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations (ICLR-2024)

Reviewer comment (Theory):
This paper does not have a clear mathematical representation of the problem it intends to address.

Author response:
We appreciate the reviewer's feedback regarding the clarity of our problem formulation. In response to this concern, we have enhanced the paper by providing a more explicit and detailed formulation of our problem in Section 2 based on the zero-sum game. We thank the reviewer for highlighting the mathematical problem formulation.

Reviewer comment (Novelty):
The article claims its primary contribution lies in using zero-sum games to formulate the robust RL problem. However, employing zero-sum games to account for uncertainties, whether in single-agent or multi-agent RL, is well-established, as seen in works like Robust Adversarial Reinforcement Learning, Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, and Robust Multi-Agent Reinforcement Learning with State Uncertainty. Given this widespread application, the paper's stated novelty becomes questionable.

Author response:
While we acknowledge the precedent exploration of zero-sum games in robust RL within existing works, it is crucial to emphasize the distinctive contribution of our method, GRAD.

* **Converge to approximate equilibriums on an adversary population.**

During training, our approach achieves an approximate equilibrium on an adversary policy set, which is totally different from other robust RL papers which use the zero-sum game formulation but only learn with an adversary during alternating training. GRAD has the capability to continuously explore and learn new policies as best responses that are not present in the current policy set, which allows for a more thorough exploration of the policy space.

* **Adaptable to different adversaries.**

Significantly, prior works, such as [1, 2, 3], primarily focus on a specific adversary or only consider the worst-case scenario, often adopting a highly pessimistic outlook that limits their adaptability to adversaries with different constraints. However, GRAD does not specifically target certain adversaries during training. Instead, at each epoch, it learns the best response to the adversary policy sampled from the adversary population, **making it adaptable to different adversarial scenarios, including both temporally-coupled and non-temporally-coupled adversaries**.

Furthermore, previous approaches exclusively consider specific attack domains for adversaries, such as perturbations on environmental dynamics [1, 2] and action perturbations [3]. In contrast, GRAD represents a pioneering effort by introducing the first robust RL framework that accommodates various adversaries including action, state, and transition adversaries, **without any constraints on attack domains**. This broader scope positions GRAD as a more practical and scalable general solution for robust RL.
* **Solving adversarial RL rather than uncertainty-aware RL.**

Our method is proposed to defend against adversarial perturbations and improve adversarial robustness, while many papers such as [4] only focus on uncertainty-aware RL. Though GRAD exhibits strong performance even under model uncertainty, as shown in our experiments. Adversarial RL where the learning process is intentionally disrupted by an adaptive adversary (as a maxmin problem) is more challenging than uncertainty-aware RL (a max problem under uncertainty). 

To provide further clarity on this aspect, we have included a detailed illustration of robust RL formulated as a zero-sum game in **Appendix B (Additional Related Work)** and improved our writing in sections of introduction and methodology. We believe these additions effectively address your concerns.

[1] Robust Adversarial Reinforcement Learning, Pinto et al. 2017.

[2] Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, HUang et al. 2021.

[3] Action Robust Reinforcement Learning and Applications in Continuous Control, Tessler et al. 2019.

Reviewer comment (Novelty):
In terms of algorithmic design, the proposed method is largely an application of the Policy-Space Response Oracles (PSRO). The novelty seems limited, and it's unclear how PSRO uniquely addresses the issue of temporally-coupled perturbations.

Author response:
The question prompts us to articulate our novelty concerning the PSRO framework more explicitly. Although GRAD draws inspiration from the PSRO framework, it innovatively extends and adapts the original concept to effectively tackle robust RL challenges. The novelty of GRAD lies in its scalability and adaptability in robust RL against diverse types of adversaries. **Our primary contribution centers on demonstrating that game-theoretic RL can be a general and scalable solution for the robust RL objective**, transcending the confines of a particular game-theoretic RL algorithm like PSRO. In our revised paper, we have enriched the introduction and method sections with additional illustrations to provide a clearer depiction of our contributions.

* **How GRAD uniquely addresses temporally-coupled perturbations**

GRAD introduces attackers under temporally-coupled constraints during training, enhancing efficiency, and resulting in a robust model against temporally-coupled and non-temporally-coupledadversaries. GRAD's unique ability to address diverse adversaries from its achievement of the approximate equilibrium after training with a varied adversary policy set. Notably, while GRAD is not exclusively designed for temporally-coupled attacks, its effectiveness in this specific and meaningful case highlights the contribution of our framework.

Reviewer comment (Novelty):
Considering that the PSRO algorithm converges to an NE in two-player, zero-sum games and has seen recent extensions to other equilibria types [1, 2], the paper's proposed method, essentially a reiteration of PSRO, makes the convergence proof for GRAD appear somewhat lackluster in its contribution.

Author response:
We appreciate the reviewer's attention to the convergence proof aspect of our proposed method, GRAD. While it is true that PSRO has seen various extensions, our primary contribution lies in pioneering the application of PSRO to the robust RL setting. This adaptation is a novel and significant step in leveraging the power of PSRO to enhance the adversarial robustness. While theoretical convergence provides valuable insights, our emphasis is on the practical contribution of GRAD in achieving robustness against diverse adversarial attacks. The convergence proof serves as a foundational aspect supporting the effectiveness of our method.

Reviewer comment (Theory):
This paper has ample room for improvement in writing, problem formulation, and the method itself. For instance, by simply using the triangle inequality, we could give the range of $\bar{\epsilon}$ that break the temporally-coupled property, rather than merely stating, "By setting \epsilon ̄ to a large value, it converges to the non-coupled attack scenario." Additionally, the motivation behind temporally-coupled perturbations lacks clarity and persuasiveness, leaving me unconvinced of its pressing relevance.

Author response:
We express our sincere gratitude for the valuable feedback provided by the reviewer, and we have worked to address the writing improvements and revised our introduction, preliminary, and methodology sections. In response to the reviewer's concerns, we offer the following clarifications for the motivation behind temporally-coupled perturbations:

* **Practical realism**: We emphasize that the introduction of temporally-coupled constraints serves to create a more practical and realistic setting for adversarial perturbations, aligning with real-world scenarios where adversaries may exhibit temporal correlations. 
For example, the occurrence of abrupt forces on robots often comes from the same direction or disturbance source over a period of time. This necessitates our consideration of the temporal correlation of perturbations.

* **Efficient search space**: Temporally-coupled constraints significantly reduce the search space for potential attackers. This reduction enhances the efficiency of the exploration process, contributing to the scalability and applicability of our approach.

* **Novel Case to challenge existing methods**: Temporally-coupled perturbations represent a novel case to challenge existing methods that only consider l-p attacks. Our empirical experiments demonstrate that, even with the introduction of temporally-coupled constraints, these perturbations can have a notable impact on existing robust models, showcasing the need for considering such scenarios in robust RL. 

We sincerely hope that these clarifications and paper improvements align with the reviewer's expectations. We welcome further discussions and suggestions to enhance the quality of our paper.

Reviewer comment (Novelty):
The current framework considers robustness against state and action uncertainty, but more closely related work [1] and subsequent research are not included. Model uncertainty is justified in the framework by stating that the evolution of the environment depends on the perturbed actions. However, model uncertainty in robust RL is defined more generally [2-10]. It would be better to include a more detailed Related Works section incorporating [2-10] and other relevant works in the revision. While this work includes experiments with model uncertainty, the baselines are also only action-robust algorithms. It would be preferable to see more extensive writing and experiments for model uncertainty, or alternatively, for the current work to focus solely on state-action uncertainty as a significant contribution. The point is that the related work discussion is inadequate.

Author response:
Thanks Reviewer JmX3 for acknowledging and encouraging our paper's contributions. We provide a detailed response to your insightful feedback and make detailed revisions to our paper accordingly.

> 1. The current framework considers robustness against state and action uncertainty. It will be better to include more detailed Related Works including and more relevant works in the revision.

We appreciate your valuable suggestion to provide a more detailed discussion on related works concerning robustness against uncertainty. Following your advice, we have expanded the discussion in **Appendix B to include the mentioned papers**, aiming to make our literature survey more comprehensive. 

Regarding the experiments on state robustness, we present results under the most robust state adversary. Other state attack methods or state uncertainty sets are less effective in attacking both robust baselines and our approach. Furthermore, we need to clarify that our method is proposed to **defend against adversarial perturbations and improve adversarial robustness**, which differs from many papers that focus on uncertainty-aware RL. Adversarial RL where the learning process is intentionally disrupted by an adaptive adversary (as a max-min problem) is more challenging than uncertainty-aware RL (a max problem under uncertainty)many papers focus on uncertainty-aware RL. Remarkably, GRAD exhibits strong performance even under model uncertainty, as shown in our experiments. 

We have included **an additional set of experiments and updated results in Appendix C**. These experiments consider model mismatch or transition uncertainty following [1], across three perturbed MuJoCo environments with changes in different physical parameters. The observed performance disparity underscores the effectiveness of GRAD and its robustness against model uncertainty.

|  Perturbed Environments  |                    | RNAC-PPO(DS) | RNAC-PPO(IPM) | **GRAD** |
|----------------|----------------|------------|------------|-----------------|
| Hopper | natural reward | **3502 ± 256**       | 3254 ± 138       |  3482 ± 209   | 
|                                   | 'leg_joint_stiffness' value: 30   | 2359 ± 182       | 2289  ± 124       |  **2692  ± 236**     | 
| Walker | natural reward | 4322 ± 289        | 4248 ± 89       |  **4359 ± 141**   | 
|                                   | 'foot_joint_stiffness' value: 30    | 4078 ± 297       |  4129 ± 78       |   **4204 ± 132**     | 
| Halfcheetah | natural reward | 5524 ± 178        | 5569 ± 232      |  **6047 ± 241**   | 
|                                   | Bound on 'back_actuator_range': 0.5   | 768 ± 102       |  1143 ± 45       |   **1369 ± 117**     | 

[1] Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation, Zhou et al. 2023.

Reviewer comment (Novelty):
GRAD shares a similar idea to the RARL algorithm (Pinto et al., 2017), namely the zero-sum structure to obtain a robust policy against the nature adversary. More detail needs to be added to point out key differences (such as inbuilt state-action uncertainty) and appropriate references should be provided. For example, Pinto et al. (2017) model the competition between the agent and the attacker as a zero-sum two-player game and train the agent under a learned attacker to tolerate both environment shifts and adversarial disturbances.

Author response:
> 2. GRAD shares similar idea of RARL algorithm (Pinto et al., 2017), that is, zero-sum structure to get the robust policy against the nature adversary. 

This is indeed a valuable feedback. In Appendix B, we have extensively discussed other robust RL works that utilize zero-sum games and highlighted the distinctions between GRAD and these approaches. 

* **Converge to approximate equilibriums on an adversary population**

During training, our approach achieves an approximate equilibrium on an adversary policy set, which is totally different from other robust RL papers which use the zero-sum game formulation but only learn with an adversary during alternating training. GRAD has the capability to continuously explore and learn new policies as best responses that are not present in the current policy set, which allows for a more thorough exploration of the policy space.

* **Adaptable to different adversaries**

Significantly, prior works, such as [2, 3, 4], primarily focus on a specific adversary or only consider the worst-case scenario, often adopting a highly pessimistic outlook that limits their adaptability to adversaries with different constraints. However, GRAD does not specifically target certain adversaries during training. Instead, at each epoch, it learns the best response to the adversary policy sampled from the adversary population, **making it adaptable to different adversarial scenarios, including both temporally-coupled and non-temporally-coupled adversaries**.

Furthermore, previous approaches exclusively consider specific attack domains for adversaries, such as perturbations on environmental dynamics [2, 3] and action perturbations [4]. In contrast, GRAD represents a pioneering effort by introducing the first robust RL framework that accommodates various adversaries including action, state and transition adversaries, **without any constraints on attack domains**. This broader scope positions GRAD as a more practical and scalable general solution for robust RL.


In summary, we appreciate the reviewer's identification of weaknesses in our related works discussion and our novelty in problem formulation, while also acknowledging our contributions. We look forward to further discussions.


[2] Robust Adversarial Reinforcement Learning, Pinto et al. 2017. 

[3] Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, HUang et al. 2021.

[4] Action Robust Reinforcement Learning and Applications in Continuous Control, Tessler et al. 2019.

Reviewer comment (Theory):
Although the temporally-coupled adversarial perturbation seems new, it is quite limited. Definition 3.2 only considers the temporally-coupled perturbation from the last time step. Even if the authors don't consider the general partially observable MDP, they should consider a more general case, e.g., m-order MDP [Efroni et al. 2022, Provable Reinforcement Learning with a short-term memory].

Author response:
According to the reviewer's suggestion, we have expanded our experiments to include a more general scenario and introduce a memorized temporally-coupled attack. We calculate the mean of perturbations from the 10 past steps and apply the temporally-coupled constraint on this past perturbation mean. We add this experiment in **Appendix C.2** to showcase the effectiveness of GRAD under this extended adversarial setting. The added results indicate that **GRAD exhibits enhanced robustness compared to other robust baselines under the memorized temporally-coupled attack**. We believe this addition contributes valuable insights to the paper and addresses your concerns regarding the temporal scope of perturbations.

Moreover, the primary reason we focused on perturbations from the last time step is grounded in the common practice of state adversaries, which typically perturb the current state without explicitly attacking short-term memory, but we agree that considering the short-term memory is a reasonable and considerable setting for adversaries. We eagerly anticipate any additional discussions or suggestions the review may have. 

| Memorized temporally-coupled Attacks| Hopper | Walker2d | Halfcheetah | Ant | Humanoid |
| -------- | -------- | -------- |-------- | -------- |-------- |
| PA-ATLA-PPO     | 2334 ± 249     | 2137 ± 258    | 3669 ± 312    | 2689 ± 189     | 1573 ± 232
| WocaR-PPO     | 2256 ± 332     | 2619 ± 198    | 4228 ± 283     | 3229 ± 178     | 2017 ± 213
| **GRAD** |  **2869 ± 228**     | **3134 ± 251**    | **4439 ± 287**    | **3617 ± 188** | **2736 ± 269**

Reviewer comment (Novelty):
The zero-sum game-based approach is not new for robust training in RL, e.g., [Tessler et al., 2019].

Author response:
While we acknowledge the precedent exploration of zero-sum games in robust RL within the existing literature, it is crucial to emphasize the distinctive contribution of our method, GRAD.

* **Converge to approximate equilibriums on an adversary population.**

During training, our approach achieves an approximate equilibrium on an adversary policy set, which is totally different from other robust RL papers which use the zero-sum game formulation but only learn with an adversary during alternating training. GRAD has the capability to continuously explore and learn new policies as best responses that are not present in the current policy set, which allows for a more thorough exploration of the policy space.

* **Adaptable to different adversaries.**

Significantly, prior works, such as [1, 2, 3], primarily focus on a specific adversary or only consider the worst-case scenario, often adopting a highly pessimistic outlook that limits their adaptability to adversaries with different constraints. However, GRAD does not specifically target certain adversaries during training. Instead, at each epoch, it learns the best response to the adversary policy sampled from the adversary population, **making it adaptable to different adversarial scenarios, including both temporally-coupled and non-temporally-coupled adversaries**.

Furthermore, previous approaches exclusively consider specific attack domains for adversaries, such as perturbations on environmental dynamics [1, 2] and action perturbations [3]. In contrast, GRAD represents a pioneering effort by introducing the first robust RL framework that accommodates various adversaries including action, state, and transition adversaries, **without any constraints on attack domains**. This broader scope positions GRAD as a more practical and scalable general solution for robust RL.

To provide further clarity on this aspect, we have included a detailed illustration of robust RL formulated as a zero-sum game in **Appendix B (Additional Related Work)** and improved our writing in sections of introduction and methodology. We believe these additions effectively address your concerns.

[1] Robust Adversarial Reinforcement Learning, Pinto et al. 2017.

[2] Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, HUang et al. 2021.

[3] Action Robust Reinforcement Learning and Applications in Continuous Control, Tessler et al. 2019.

Reviewer comment (Experiments):
This paper misses one classic setting of robust MDP (i.e., transition adversaries) [e.g., Iyengar'05, Robust Dynamic Programming] as well as related baselines [e.g., Zhou et al. 2023, Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation].

Author response:
We deeply appreciate the insightful suggestion regarding the incorporation of experiments involving transition adversaries. To attempt to study this great point, we conducted a meticulous comparative study pitting GRAD against the baseline RNAC-PPO trained with DS and IPM uncertainty sets across three perturbed MuJoCo environments with changed physic parameters, following the experimental protocols outlined in [4]. 

The results unequivocally demonstrate that **GRAD outperforms RNAC-PPO across all tasks under transition uncertainty**. This performance disparity underscores the efficacy of GRAD and its adaptability across diverse attack domains. We believe these results findings make substantial empirical contributions to the paper, effectively addressing the concern raised by the reviewer. We also add this experiment in **Appendix C.2**.

|  Perturbed Environments  |                    | RNAC-PPO(DS) | RNAC-PPO(IPM) | **GRAD** |
|----------------|----------------|------------|------------|-----------------|
| Hopper | natural reward | **3502 ± 256**       | 3254 ± 138       |  3482 ± 209   | 
|                                   | 'leg_joint_stiffness' value: 30   | 2359 ± 182       | 2289  ± 124       |  **2692  ± 236**     | 
| Walker | natural reward | 4322 ± 289        | 4248 ± 89       |  **4359 ± 141**   | 
|                                   | 'foot_joint_stiffness' value: 30    | 4078 ± 297       |  4129 ± 78       |   **4204 ± 132**     | 
| Halfcheetah | natural reward | 5524 ± 178        | 5569 ± 232      |  **6047 ± 241**   | 
|                                   | Bound on 'back_actuator_range': 0.5   | 768 ± 102       |  1143 ± 45       |   **1369 ± 117**     |

Reviewer comment (Presentation):
Figure 1 doesn't make sense to me. The robust baseline considers the worst-case scenario, which should be more stable for different kinds of attacks compared with the less conservative model that is proposed in this work.

Author response:
We have revised the caption of Figure 1 to explain why the baseline is not robust to the temporally-coupled adversaries, 

While it is true that WocaR-PPO aims to enhance its robustness in the worst-case scenarios or improve its lower bound of performance, **it does not guarantee superiority under other case scenarios, such as temporally-coupled perturbations**. The baseline's focus on worst case estimations and improvement does not inherently ensure robustness across all conceivable situations. It is noteworthy that WocaR-PPO's relatively higher risk of stumbling under temporally-coupled attacks, as demonstrated in our experiments. This is precisely where GRAD excels. GRAD finds the equilibrium with an adversary policy set, showcasing a significant advantage in diverse adversarial settings. We believe these aspects underscore the substantial advantages of GRAD and welcome further discussion on this matter.

In summary, we have responded to the reviewer's concern by providing additional experiments in temporally-coupled settings and diverse attack domains, illustrating GRAD's robustness. We also discuss our noverlty different from related works to better elucidate GRAD's strengths. Regarding Figure 1, we explained the observations and modified Figure 1's illustration in our paper. We appreciate the valuable feedback, which enhances the quality of our paper, and we look forward to further discussion and additional suggestions.

[4] Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation, Zhou et al. 2023.