Paper: Relaxed State-Adversarial Offline Reinforcement Learning: A Leap Towards Robust Model-Free Policies from Historical Data (ICLR-2024)

Reviewer comment (Reproducibility):
The methods section appears somewhat unclear. It is recommended that the authors include a subsection on the implementation of RAORL, detailing the learning objectives for both value functions and policy.

Author response:
> Q: The methods section appears somewhat unclear. It is recommended that the authors include a subsection on the implementation of RAORL, detailing the learning objectives for both value functions and policy.

A: Our algorithm can be found in algorithm 1. Specifically, inspired by TD3+BC (Fujimoto & Gu, 2021) and ReBrac (Tarasov et al., 2023), Objective function for the policy: 

$$
\pi = argmax_{\pi} \mathbb{E}_{(s,a) \sim D} \left[ \lambda Q(s, \pi(s)) - (\pi(s) - a)^2 \right]
$$

The default values of the hyper-parameters($\lambda$) were used in the experiments (same as ReBrac (Tarasov et al., 2023)).

Objective function for the average scenario value function is followed the lemma2:

$$
L_{TD}(\theta) = E_{(s,a,s') \sim D} \left[ \left( r(s,a) + \gamma \big( \alpha Q_{\bar{\theta}}(s', \pi(s’)) + (1-\alpha) Q_{\bar{\theta}}(adv(s'), \pi(s’)) \big) - Q_{\theta}(s, a) \right)^2 \right],
$$

We set perturbation radius from the set {0.03, 0.05, 0.08, 0.1} multiplied by state differences (in Section 4.4). The setting of attack budgets were chosen to reflect the mean magnitude of the state affected by actions taken in each environment.

Reviewer comment (Novelty):
RORL [1] is already a model-free state-adversarial method that achieves state-of-the-art performance on the MuJoCo benchmark and robust experiments. A thorough discussion and comparison with this work are necessary. Moreover, the proposed method seems quite similar to the adversarial state training in S4RL [2]. A more comprehensive discussion with closely related works is required to emphasize the significance of the method; otherwise, the novelty may be questioned.

Author response:
> Q: A thorough discussion and comparison with this work are necessary. Moreover, the proposed method seems quite similar to the adversarial state training in S4RL [2]. A more comprehensive discussion with closely related works is required to emphasize the significance of the method; otherwise, the novelty may be questioned.

A: The main contribution of RAORL over S4RL is a theoretical guarantee for the lower bound of performance, further justifying the use of state-adversaries . Besides, we acknowledge the importance and “relaxed” technique beyond state-adversarial technique. In S4RL, the state-adversarial technique was identified as the most effective augmentation method based on their experiments. However, applying the state-adversarial technique in a straightforward manner can lead to overly cautious results (Lien et al., 2023). This is primarily because the state-adversarial technique inherently prepares for the worst-case scenario. Therefore, in practical applications, particularly in real-world problems, it becomes essential to adopt a "relaxed" version of the state-adversarial technique. The following table shows a comparative study of RAORL and S4RL in various experimental environments.

|                          | RAORL         | S4RL           |
|--------------------------|---------------|----------------|
| Halfcheetah-medium       | **66.1**          | 48.6           |
| Halfcheetah-medium-replay| **51.0**          | **51.7**         |
| Halfcheetah-medium-expert| **107.0**        | 78.1           |
| hopper-medium            | **102.3**         | 81.3           |
| hopper-medium-replay     | **100.4**        | 36.8           |
| hopper-medium-expert     | 108.9         | **117.9**         |
| walker2d-medium          | 86.8          | **93.1**           |
| walker2d-medium-replay   | **85.0**          | 35.0           |
| walker2d-medium-expert   | **112.2**        | 107.1          |
| Average                  | **90.9**          | 61.6           |


|                       | RAORL         | S4RL          |
|-----------------------|---------------|---------------|
| antmaze-umaze         | **97.5**      | 94.1          |
| antmaze-umaze-diverse | 87.7          | **88.0**      |
| antmaze-medium-play   | **91.5**      | 61.6          |
| antmaze-medium-diverse| **86.7**      | 82.3          |
| antmaze-large-play    | **71.2**      | 25.1          |
| antmaze-large-diverse | **68**        | 26.2          |
| Average               | **83.8**      | 62.8          |

Reviewer comment (Experiments):
It appears that the method can be applied to different base algorithms besides ReBrac. Incorporating the method with approaches like ATAC and IQL could enhance the comprehensiveness of the experiments and support the claims made in the paper.

Author response:
> Q: It appears that the method can be applied to different base algorithms besides ReBrac. Incorporating the method with approaches like ATAC and IQL could enhance the comprehensiveness of the experiments and support the claims made in the paper.

A: We are currently conducting experiments to test our method with different base algorithm IQL, to further validate its applicability and effectiveness. The following table are results from integrating the Relaxed Adversarial State technique with IQL are promising. These experiments, run on three different seeds, show notable improvements in performance.

|                          | IQL            | IQL+RA         |
|--------------------------|----------------|----------------|
| Halfcheetah-medium-expert| 86.7           | **93.3 ± 1.5**     |
| hopper-medium-expert     | 109.6          | **112 ± 1.9**      |
| walker2d-medium-expert   | 91.5           | **112.4 ± 0.6**    |

Reviewer comment (Experiments):
In the state perturbation experiments, a comparison with the prior state adversarial baseline RORL is needed.

Author response:
> Q: In the state perturbation experiments, a comparison with the prior state adversarial baseline RORL is needed.

A: Our methodology is primarily based on the ReBrac approach, making the performance comparison between our method and RORL particularly crucial. RORL utilizes an ensemble-based model, with "RORL-n" indicating the use of n ensemble models. In the following analysis on Hopper-medium-expert over 4 seeds, we demonstrate that our method achieves comparable results to RORL-10, which involves an ensemble size five times larger than ours. Furthermore, our approach notably outperforms RORL-2, which has the same numbers of critics of our model.

| Attack magnitude | RAORL | ReBrac | RORL-10 | RORL-2 |
|--------|-------|--------|---------|--------|
| 0.025  | 72.1  | 67.1   | 75.8    | 48.1   |
| 0.05   | 61.2  | 40.2   | 65.0    | 33.2   |
| 0.075  | 53.1  | 35.3   | 53.5    | 32.6   |
| 0.1    | 41.7  | 30.1   | 44.3    | 29.1   |

Reviewer comment (Writing):
The quality of writing in this paper could benefit from further refinement. Some explanations are unclear. For instance, in Lemma 1, formula 1 and formula 2 look identical, I had thought that formula 1 might have been inadvertently repeated. But it turns out there are slight differences. I think there should be a better presentation such as using the absolute difference is bounded. Furthermore, the content of Lemma 1 raises questions for me. The textual description seems to discuss the relationship of the policy's value learned from two different properties of P_B. However, the formula appears to depict the policy's value learned from U and P_B. I hope the authors recognize this distinction. One refers to the value function learned under a specific P_B, while the other pertains to the average value function learned on U.

Author response:
A: We revise Lemma 1 in absolute difference and clarify the details in Lemma 1 as follows.

1. Relevance of U and P_B: As shown in Figure 1,  U represents the set of all possible real-world transition kernels, encompassing a wide range of scenarios and environmental dynamics. In contrast, P_B is specific to the offline dataset and represents the transitions recorded in that particular dataset.  

2. Objective of Lemma 1: The lemma intends to quantify the performance gap between a policy learned exclusively from the offline dataset (P_B) and the policy evaluated against the universal set of transition kernels (U). This gap highlights the potential discrepancies and challenges in applying policies learned in a limited offline context to a broader range of real-world scenarios. 

3. Lemma 1 in absolute difference :  \begin{align}
\left| J_{\rho_0^B}(\pi, P_B) -E_{P_0 \sim U}(J_{\rho_0}(\pi, P_0)) \right| \leq & \frac{2R_{\text{max}}}{1-\gamma} E_{P_0 \sim U}[D_{\text{TV}}(\rho_0,\rho_0^B)]  + \frac{2 \gamma R_{\text{max}}}{(1-\gamma)^2} \beta + \frac{2R_{\text{max}}}{1-\gamma} E_{P_0 \sim U}{E}[\gamma^{\text{T}_{V}^\pi}].
\end{align}

Reviewer comment (Writing):
While this paper offers a plethora of proofs and new definitions, it lacks a coherent logical thread to effectively connect them. Additionally, the absence of adequate textual explanations for the theorems and lemmas makes the paper hard to follow and comprehend. There is no formal problem formulation for the problem the paper wants to solve. A more structured approach and clear expositions would greatly enhance its readability.

Author response:
A: In the second paragraph of Section 4, we provided a logical thread of sections of 4.1, 4.2 and 4.3. In Section 4.1, we systematically measure the performance discrepancies between the policy shaped solely from the offline dataset ($P_B$) and its application in the universal set of transition kernels (U). Sections 4.2 and 4.3 further build upon this by demonstrating methods to enhance policy performance.

Reviewer comment (Novelty):
This paper introduces the concept of the Relaxed State-Adversarial Transition Kernel. However, the name doesn't seem to genuinely capture the essence of this transition kernel. I'm uncertain where the "State-Adversarial" aspect is present. The approach of crafting a new transition kernel using a linear combination appears to be already employed by many robust RL under model uncertainty methods, such as the R-contamination uncertainty set. You may check this paper "Policy Gradient Method For Robust Reinforcement Learning".

Author response:
A: The term "State-Adversarial" in our Relaxed State-Adversarial Transition Kernel is defined in Definition 2 of our paper. It refers to the process where state transitions are adversarially perturbed towards those with lower value estimates, a concept critical for robustness in continuous state-action spaces. Our kernel thus represents a bridge between standard and adversarial MDPs, a novel contribution particularly tailored for continuous environments.
While the "Relaxed" aspect of our kernel does resemble approaches like the R-contamination uncertainty set found in robust RL literature, the "State-Adversarial" facet specifically addresses continuous domains. This differentiates our work from that in "Policy Gradient Method For Robust Reinforcement Learning," which primarily addresses discrete spaces (as their algorithm requires the summation over all possible states and actions) and does not extend to the continuous settings that our paper focuses on.

Reviewer comment (Presentation):
The paper's structure is not well-organized, making it challenging to grasp the primary contributions and follow the narrative. Furthermore, the writing is convoluted, containing multiple unclear sentences that hinder comprehension.

Author response:
In the second paragraph of Section 4, we provided a logical thread of sections of 4.1, 4.2 and 4.3. In Section 4.1, we systematically measure the performance discrepancies between the policy shaped solely from the offline dataset (P_B) and its application in the universal set of transition kernels (U). Sections 4.2 and 4.3 further build upon this by demonstrating methods to enhance policy performance.

Reviewer comment (Experiments):
The experimental results lack persuasiveness. Firstly, the authors do not compare their work with state-of-the-art algorithms such as SAC-N, EDAC [1], and LB-SAC [2]. Even when implementing the algorithms based on https://github.com/tinkoff-ai/CORL, this paper's results are inferior to previous methods in most Mujoco and Adroit environments. Hence, I believe there are significant limitations in the paper's empirical contributions.

Author response:
SAC-N, EDAC, and LB-SAC are all ensemble-based algorithms where the ensemble sizes are 500, 50, and 50, respectively. These ensemble-based algorithms often require extensive computational resources due to their reliance on large ensembles for uncertainty estimation (Nikulin et al., 2022). We did not include these algorithms in the comparative study because our method is not only model-free but also ensemble-free.

Reviewer comment (Novelty):
While the paper focuses on model-free offline RL, I find it puzzling that model-based offline RL is included in the related work. Robust RL works like [3, 4] (and others) dealing with state adversaries seem more relevant, yet the authors do not discuss them in the related work section.

Author response:
The inclusion of model-based offline RL in the related work is to present a comprehensive background, as it also addresses offline scenarios. We acknowledge the relevance of robust RL literature, particularly works that handle state adversaries. We will incorporate references [3, 4] into our discussion on robust reinforcement learning in Section 2.3 to provide a more complete overview of the field and its developments.

Reviewer comment (Experiments):
How to determine the attack budget ($\epsilon$) in RAORL?

Author response:
We set perturbation radius from the set {0.03, 0.05, 0.08, 0.1} multiplied by state differences (in Section 4.4). The setting of attack budgets were chosen to reflect the mean magnitude of the state affected by actions taken in each environment.

Reviewer comment (Evaluation):
I am seeking clarity regarding the primary advantage of RAORL. Is it intended to be more effective than other offline model-free algorithms, aimed at improving adversarial robustness, or targeting enhanced offline performance? While the paper mentions these advantages, the results do not strongly demonstrate superior performance or robustness with enough evidence and explanation.

Author response:
The primary objective of RAORL is to offer an effective and efficient solution in the realm of offline model-free algorithms.  It significantly reduces the computational and memory overhead typically associated with ensemble methods. In terms of performance and robustness, RAORL is designed to be competitive with other state-of-the-art offline model-free algorithms.

Reviewer comment (Novelty):
Similarities between this paper’s theoretical results and those of the MoREL paper (Kidambi et al. 2020) are not highlighted. For example, Lemma 1 here looks nearly identical to Theorem 1 in that paper, with the addition of expectation over the uncertainty set $\mathcal{U}$. It would be helpful for the reader if you include some comments on the similarities/differences, perhaps in an appendix.

Author response:
Our Lemma 1 and MoREL's Theorem 1 differ primarily in their treatment of transition probabilities. MoREL assumes known transition probabilities, using them to create a pessimistic MDP. In Theorem 1, MoREL work with the constraint that the Total Variation Distance (DTV) between the model transition probability $\hat{P}(\cdot|s,a)$ and the true probability $P(\cdot|s,a)$ is less than or equal to $\alpha$. However, in practice, $P(\cdot|s,a)$ is unknown since we only have access to the transition kernel derived from the offline dataset. In contrast, RAORL operates without this assumption, reflecting real-world scenarios where true dynamics are often unknown.

Reviewer comment (Novelty):
Am I missing other differences between RAORL and RAPPO?

Author response:
The implementation of RAORL is similar to RAPPO, yet the two approaches diverge fundamentally in their theoretical perspectives. Our research establishes a theoretical bound on the gap between offline datasets and real-world scenarios, substantiating the assertion that introducing state adversaries significantly bolsters the resilience of policies derived from static datasets. To the best of our understanding, RAORL stands out as a pioneering method in its integration of uncertainty sets within the framework of solving offline RL challenges. Relaxed State-Adversarial method (RAPPO) is only a step utilized to aggregate uncertain transitions when training policies. **Specifically, the goal of our study is to address two pivotal questions:**


**Q1: To what extent and through what mechanisms can we reduce the performance gap between offline datasets and real-world scenarios?**

A1:  

Our primary aim is to lessen the performance gap typically observed when a policy developed using offline datasets (denoted as $P_B$) is applied in real-world scenarios, characterized by a diverse set of transition kernels (represented by $U$). We approach this in two key ways:

1. **Risk-Aware Policy Training:** We demonstrate that training a risk-aware policy over a carefully constructed uncertainty set $U_r$ can reduce the gap between its performance in training and real-world applications. This reduction is quantified by: $\frac{4 \gamma R_{\text{max}}}{(1-\gamma)^2} (\beta - \frac{1}{2}p_r \beta_r)$ (details in Theorem 2).This formula represents a theoretical measure of the improvement we can expect when using this risk-aware approach.

2. **Constructing a Practical Uncertainty Set:** We create an uncertainty set, $U_r$, using a state-adversarial uncertainty set  $U_\epsilon^{\pi}$ as outlined in Definition 4. The theoretical benefit of this approach is captured by $\frac{4 \gamma R_{\text{max}}}{(1-\gamma)^2} (\beta - \frac{1}{2}p_{\text{RA}} (1-\alpha))$ (details in Theorem 3) , indicating our method can bridge both the theoretical and practical performance of policies.

**Q2: Is it possible to effectively incorporate uncertainty sets into offline datasets in a model-free context?**

A2: 

In offline reinforcement learning (RL), our sole data source is the offline dataset. To develop a robust model that accounts for uncertainty in transition probabilities, traditional methods often rely on model-based approaches, such as the one proposed in MoREL (Kidambi et al. 2020). These methods aim to achieve robustness by explicitly modeling the transition probabilities. However, our paper introduces a novel approach that targets the same goal of transition probability robustness but eliminates the need to train a separate transition model. We achieve this by constructing an uncertainty set within a state-adversarial Markov Decision Process (MDP). This method allows us to prepare for uncertainties in transition probabilities using only the available offline dataset, without the additional complexity of model-based methods.

Reviewer comment (Writing):
In Definition 1, is the expectation over $s,a$ taken over all transitions in the offline dataset? (I think the answer is yes, but wanted to confirm. The notation could be modified to clarify this.)

Author response:
In Definition 1, the expectation indeed goes beyond the transitions present in the offline dataset. It encompasses all possible transition kernels within a universal uncertainty set, representing potential dynamics in real-world environments. This approach extends our analysis beyond limited empirical data, preparing the model for a broader range of scenarios.
However, it's important to note that while the concept of a universal uncertainty set might seem overly extensive, in practice, we refine this to a more manageable subset, denoted as $U_r$ in section 4.2 and $U_{\epsilon}$  in section 4.3. This refinement process involves constraining the uncertainty set based on realistic assumptions and characteristics observed in the offline data, thereby making the set both practical and relevant to real-world applications.

Reviewer comment (Theory):
In Definition 3, do the transition probabilities need to be re-normalized if the argmin defining $Z^\pi_\sigma$ contains more than one state? If we are assuming the argmin is always unique, perhaps this should be noted in the paper.

Author response:
The problem can be easily solved by setting a rule, such as picking the state that has the smallest value in the first dimension if there are multiple choices. In our implementation, the worst state is determined using the FGSM method. Although there can be multiple states that are equally worse, the FGSM finds only one of them. We will clarify this issue in the revision.

Reviewer comment (Writing):
The definition of $p_r$ (in Lemma 3 in Appendix A.2) is given as “the probabitlity [sic] of $P \in \mathcal{U}_r$ for every $P \in \mathcal{U}$”. I am not sure how to interpret this?

Author response:
In Lemma 3, $ p_r$ represents the likelihood of each possible transition kernel $ P $ being within a robust subset $U_r $, which is taken into account by the robust policy $\pi_r$.

Reviewer comment (Reproducibility):
Could you describe more precisely how the perturbations were chosen in section 5.3?

Author response:
We defined the perturbation magnitude linearly with 20 points in the range $[0, 0.1]$. A perturbation of 0 signifies no perturbation, while a perturbation of 0.1 implies a perturbation strength of $0.1 \times | \text{ActualNextState} -  \text{State}|$, thereby ensuring that the norm of the difference between the actual state and the perturbed state is at most $0.1 \times  | \text{ActualNextState} -  \text{State}|$.

As depicted in Figure 2, we analyzed the effect of different perturbation magnitudes on agent performance. The results demonstrate that RAORL maintains robust performance even under significant state perturbations.