Paper: Efficient Action Robust Reinforcement Learning with Probabilistic Policy Execution Uncertainty (ICLR-2024)

Reviewer comment (Novelty):
Theorem 1 results in this work are already shown in Proposition 1 of (Tessler et al, 2019). Moreover, even though the latter work does not explicitly define "action robust Bellman equation" and the corresponding optimality equations, they can be inferred from (Eq.(11), Tessler et al, 2019). Although an independent analysis in this work is made for finite-horizon setting whereas (Tessler et al, 2019) is for infinite horizon, the proof ideas share similar characteristics due to the structure of the uncertainty set itself (helping commutability of the expectation and the minimization operations). Thus Theorem 1 proof largely follows the proofs in (Iyengar, 2005) for the finite-horizon, getting insights from Proposition 1 of (Tessler et al, 2019).

Author response:
**To weakness 1.**

We agree with the reviewer that the contributions in Section 4 are incremental. However, the results in Section 4 are not the core contributions of our paper. The main purpose of results in Section 4 is to provide motivation and insights for the design of the proposed two efficient algorithms in later sections. The approaches of Pinto et al. (2017) and Tessler et al. (2019) used an alternating update method that repeatedly solves MDP by fixing either the adversary policy or the agent policy and are sample inefficient. In our proposed algorithm, we use the iteration of action robust Bellman equation to solve the robust problem. Our proposed method can avoid the alternating update between the adversary policy and agent policy. Thus, we need to first introduce the results in Section 4. The proposed action robust Bellman equation in Section 4 tells the reader how to directly solve the robust problem for finite-horizon setting when we know the reward function and transition probabilities, and makes ARRLC more easily understandable.

To reflect the discussion above, we have changed the Theorem 1 to Proposition 1 in the revised version so as to de-emphasize the results in Section 4, changed the description of the main contributions in page 2 and added the discussion about the importance of introducing action robust Bellman optimality equation in the revised paper.

Reviewer comment (Novelty):
With the separable form of action robust Bellman equation ($V^*(.) = (1-\rho) \max_{a}Q^*(.,a) + \rho \min_{a}Q^*(.,a)$), ARRLC is an extension of ORLC (Dann et al, 2019) with less technical challenges. The analysis is made finer by replacing $Var(\overline{V})$ with $Var((\overline{V}+\underline{V})/2)$, which is straightforward from Eq.31 due to monotonicity and induction.

Author response:
**To weakness 2.**

ARRLC shares a similar form with ORLC, but is not an simple extension of ORLC with less technical challenges. Similar to ORLC, ARRLC considers the upper bound and the lower bound of $Q$-values and $V$-values. However, the trajectory sampling and the update of the value functions in ARRLC are different from those in ORLC, as we considered the action robust Bellman equation. More specifically, ARRLC uses two different actions $\overline{\pi}(s)$ and $\underline{\pi}(s)$ to update the value functions but only samples one of them at each step, while ORLC uses the sampled action $\pi(s)$ to update the value function (refer to Line 12 Algorithm 1 in Dann et al, (2019)). We need to do this because the update of the action robust Bellman equation requires both the maximum and minimum over the actions. Thus, some new technical challenges arises. For example, the empirical variance estimators $Var((\overline{V}+\underline{V})/2)$ sum not only over all episode $k$ and step $h$ but also over two different action $\overline{\pi}(s)$ and $\underline{\pi}(s)$, but only one action, $\overline{\pi}(s)$ or $\underline{\pi}(s)$, is sampled. Thus, the sum of variance is no longer simply the variance of the sum of rewards per episode, and we need new techniques to bound the sum of variance. We deal with this challenge in Appendix page 21-24. Some new techniques are then introduced, e.g., $\text{Var}((\overline{V}+\underline{V})/2)$ is further replaced with $C^{\pi^{k_*,\pi^k,\rho}}$, and different confidence bounds are used in our paper. We added these discussion in Section 5.1 at the top of page 6.

Reviewer comment (Novelty):
Given the results in [Tessler, 2019], the first contribution listed on page 2 is not very original (no big difference between episodic setting and infinite discounted horizon setting).

Author response:
We agree with the reviewer that the concept of the probabilistic action robust MDP and the existence of its optimal policy is not novel, which was introduced by Tessler et al. (2019). 
However, Tessler et al. (2019) and other related papers did not discuss the action robust bellman equation. In Section 2, we showed that the optimal action robust policy can be solved via the iteration of the action robust Bellman optimality equation. 
This motivated us to use the action robust Bellman optimality equation to directly solve the robust problem, and then we designed two efficient algorithms based on the action robust Bellman optimality equation.

Using the iteration of the proposed action robust Bellman equation to solve the robust problem can avoid the alternating update between the adversary policy and agent policy. The alternating update method is used in the approaches of Pinto et al. (2017) and Tessler et al. (2019), and is sample inefficient. 

Thus, we emphasize that the optimal action robust policy can be solved via the iteration of the action robust Bellman optimality equation in Section 2. and is the important idea of our proposed efficient algorithm.

To reflect the above, we changed the Theorem 1 to Proposition 1 in the revised version so as to de-emphasize the results in Section 4, changed the description of the main contributions in page 2 and added the discussion about the importance of introducing action robust Bellman optimality equation in the revised paper.

Reviewer comment (Experiments):
It appears that the policy execution uncertainty set may not be suitable for addressing large-scale problems, as it necessitates computing the minimum over the action space. While this paper concentrates on a tabular environment, it would be more prudent to opt for a scalable uncertainty set, e.g., [Zhou et al., 2023, Natural Actor Critic for Robust Reinforcement Learning with Function Approximation].

Author response:
This paper focuses on the tabular environment. It is true that the current theoretical guarantee on the sample complexity of the proposed algorithms is based on the tabular setting. However, the policy execution uncertainty set itself is also applicable for large-scale problems. For example, Tessler et al. (2019) introduced the probabilistic policy execution uncertainty set and perform extensive evaluation across several MuJoCo environments.

Designing provably sample efficient algorithm for large-scale problems is an important future direction for us to pursue. 
From our work, we found two insights that could be useful for large-scale problems: (1) The adversary policy and the agent policy can be simultaneously updated to efficiently sample trajectories; (2) The adversary policies at each episode does not necessarily need be the minimum over the actions, an approximation of the minimum also works. 
Based on these insights, a policy gradient method could potentially be designed handle the continuous action space in the robust problem. We can use policy gradient method, such like PPO, to find an approximation of the adversary policy (the minimum over actions). 

We also thank the reviewer for bring the paper [Zhou et al., 2023, Natural Actor Critic for Robust Reinforcement Learning with Function Approximation] to our attention. Consider a variant of the proposed probabilistic policy execution uncertainty set is also a good idea.
We added these discussions in the conclusion section of the revised version.

Reviewer comment (Novelty):
One important piece of literature on robust MDP is missing in the paper [Iyengar'05, Robust dynamic programming].

Author response:
We added the discussion related to the paper [Iyengar'05, Robust dynamic programming] in the revised version.

Reviewer comment (Novelty):
Pertaining to my previous concern, related work is described without assessing what each reference misses that this paper adds. This section should emphasize more on what previous work did and what is still missing that this study fulfills. This would not only give motivation to this work but also strengthen its significance.

Author response:
We updated the discussion on the related works in the revised version (marked in red color). 
The main update are: (1) the previous works on action robust RL fail to design efficient algorithm with theoretical guarantee on sample complexity or regret. We develop a minimax sample efficient algorithm and fill in this gap. (2) the works on sample complexity guarantees for episodic tabular RL can not directly use in action robust MDP with small technical changes.  We develop new adversarial trajectory sampling and action robust value iteration method in the proposed algorithm, and new techniques to bound the sum of variance so that our algorithm suits for action robust MDPs. (3) the efficient multi-agent RL algorithms, such as (Liu et al., 2021; Jin et al., 2021), can be used to solve the action robust optimal policy but are not minimax optimal. They are a factor of $A$ or $H^2$ above the minimax lower bound. Our algorithm ARRLC is minimax optimal.

Reviewer comment (Presentation):
The algorithm is too far away after the text referring to it.

Author response:
Regarding the algorithmic approach, Tessler et al. (2019) requires repeatedly solving the optimal policy given a fixed adversary policy, which is sample inefficient and computationally inefficient. Our algorithm solves the robust problem by the iteration of the proposed action robust Bellman equation. The proposed algorithm directly learns the robust policy and simultaneously updates the adversary policy and the agent policy without fixing any of them.

Reviewer comment (Writing):
The uncertainty set definition in Eq. (3) is confusing: it says $\tilde{\pi}$ is a mixture between $\pi$ and $\pi'$ without specifying $\pi'$, whereas the text after states that $\pi'$ should be an argmin. Once I understood what the authors meant, I think it would have been clearer to write: $\\Pi^{\\rho}(\\pi):= \\{ \\tilde{\\pi}: \\forall s, \\forall h, \\exists \\pi_{h}'(\\cdot|s) \\in\\Delta_{\\mathcal{A}} \\text{ such that } $ $\\tilde{\\pi_{h}}$ $(\\cdot|s)=$ $(1-\\rho)\\pi_{h}(\\cdot|s)+ \\rho \\pi_{h}'(\\cdot |s) \\},$ or even simpler, $\\Pi^{\\rho}(\\pi):=  (1-\\rho)\\pi + \\rho $ $(\\Delta_{\\mathcal{A}})^{|\\mathcal{S}|\\times H}$.

Author response:
We fixed typos in the revised version and changed the position of Algorithm 1. We agree with the reviewer that the original uncertainty set definition in Eq. (3) may by confusing. We have rewritten it in a form as the reviewer suggested.

Reviewer comment (Novelty):
It is not clear to me how this work improves upon Tessler et al.. It does give regret bounds and sample complexity, but how is the algorithmic approach essentially different?

Author response:
Regarding the algorithmic approach, Tessler et al. (2019) requires repeatedly solving the optimal policy given a fixed adversary policy, which is sample inefficient and computationally inefficient. Our algorithm solves the robust problem by the iteration of the proposed action robust Bellman equation. The proposed algorithm directly learns the robust policy and simultaneously updates the adversary policy and the agent policy without fixing any of them.

Reviewer comment (Experiments):
Why did the authors not compare ARRLC to the approaches of Pinto et al. (2017) or Tessler et al. (2019) in the experiments section, but chose a comparison to a non-robust approach ORLC instead?

Author response:
Our experimental results consists of three parts. 
(1) We compared our proposed method ARRLC with a non-robust algorithm ORLC and show the robustness of our algorithm.
(2) We compared ARRLC with robust TD and demonstrates the efficiency of our ARRLC algorithm. ARRLC converges much faster than robust TD algorithm.
(3) In the appendix, we compared our algorithm with the algorithms in Pinto et al. (2017) and Tessler et al. (2019) under different $\rho$ and $p$. We used a black-box MDP solver (Q-learning) to solve the optimal agent policy when the adversarial policy is given. We trained our method in 2000 episodes and others in 30000 episodes. Compared with our algorithm, the approaches of Pinto et al. (2017) and Tessler et al. (2019) are sample inefficient .

Due to the page limits, we put some experimental results of (3) into Appendix. We agree with the reviewer that providing the numerical comparison with the approaches of Pinto et al. (2017) and Tessler et al. (2019) in the main body of the paper can provide more insights. Hence, in the revised paper, we reorganized the experiment section. We moved the the numerical comparison with the approaches of Pinto et al. (2017) and Tessler et al. (2019) into the main body of the paper, and moved other experimental results into Appendix due to the page limit.

Reviewer comment (Theory):
I simply remark that $\\Pi^{\\rho}\\subset\\Pi^{D, \\rho}$, is that correct? Also, what does the relationship between those two types of uncertainty sets bring to the analysis?

Author response:
We agree with the reviewer. The relationship between those two types of uncertainty sets does not impact the analysis. We deleted this discussion in the revised version.

Reviewer comment (Theory):
As far as I remember, Tessler et al. (2019) proved the existence of an optimal policy for the action robust setting under probabilistic execution. Although their proof does not rely on Bellman recursion but a Markov game formulation, they show equivalence between this setting and the robust MDP setting, itself being solvable through Bellman recursion [1]. Could the authors please clarify?

Author response:
We agree with the reviewer that Tessler et al. (2019) proved the existence of an optimal policy for the action robust setting under probabilistic execution and showed the equivalence between this setting and one special robust MDP setting.

However the equivalent robust MDP setting in Tessler et al. (2019) is different with the setting in Iyengar et al. (2005). The equivalent robust MDP setting in Tessler et al. (2019) includes both the reward uncertainty and the transition probabilities uncertainty, while the Robust MDP setting in Iyengar et al. (2005) only considers the uncertainty in transition probabilities. Thus, the robust Bellman equations in two papers are different and the results in Iyengar et al. (2005) can not directly apply in action robust setting.