Paper: Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR-2024)

Reviewer comment (Novelty):
Hard constraints in safe RL have been studied in online RL settings, but several important related works are missing from the references, including:  
- Sootla, Aivar, et al. "Sauté rl: Almost surely safe reinforcement learning using state augmentation." International Conference on Machine Learning. PMLR, 2022.  
- Wang, Yixuan, et al. "Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments." International Conference on Machine Learning. PMLR, 2023.  
- Wachi, Akifumi, and Yanan Sui. "Safe reinforcement learning in constrained Markov decision processes." International Conference on Machine Learning. PMLR, 2020.

Author response:
> **W1 & Q1 & W2.1 Some of safe online RL methods can be easily extended to offline RL settings, such algorithms should be included as a baseline.**

We thank the reviewer for these works and we are happy to compare against them in our revised paper! We have added comparison with Sauté [1] and RCRL [2], since Sauté [1] is relatively versatile and RCRL [2] also adopts HJ Reachability to enforce hard constraints. Results can be found in Table 6, Appendix D.7 in our revised paper.
- Note that applying safe online RL methods in offline settings faces distributional shifts. To address this, we add a BC constraint in Sauté [1] and RCRL [2] similar to an offline RL method TD3+BC [3] to combat distributional shifts. Also, we swept the conservatism strength to find a good balance between safety constraint and behavior constraint to obtain the best possible results.
    - However, Table 6 shows that we cannot obtain good results with direct adaptations of safe online RL methods into offline settings, even with carefully tuned hyper-parameters, due to the intricate coupled training processes.
    - FISOR, however, enjoys good results thanks to its decoupled learning objectives.

Besides the experiments, we briefly discussed these safe online RL methods here:
- For Sauté[1], similar to CPQ[4], safety constraints are handled by integrating them into the state space and modifying the objective accordingly. They struggle to handle zero cost limit, thus we still consider it to be a type of soft constraint method.
- For [5], the intertwined learning process (barrier function, model, policy) is not suitable for offline scenarios.
- For [6], its online safe exploration with environments is not compatible for offline settings.

Reviewer comment (Novelty):
While the authors note little literature exists on safe **offline** RL and present Table 4 in the Appendix, there are existing studies directly related to this paper. For example, Sootla et al. (2022) can be easily extended to offline RL settings by augmenting the state and providing a large penalty for violating hard safety constraints.

Author response:
> **W1 & Q1 & W2.1 Some of safe online RL methods can be easily extended to offline RL settings, such algorithms should be included as a baseline.**

We thank the reviewer for these works and we are happy to compare against them in our revised paper! We have added comparison with Sauté [1] and RCRL [2], since Sauté [1] is relatively versatile and RCRL [2] also adopts HJ Reachability to enforce hard constraints. Results can be found in Table 6, Appendix D.7 in our revised paper.
- Note that applying safe online RL methods in offline settings faces distributional shifts. To address this, we add a BC constraint in Sauté [1] and RCRL [2] similar to an offline RL method TD3+BC [3] to combat distributional shifts. Also, we swept the conservatism strength to find a good balance between safety constraint and behavior constraint to obtain the best possible results.
    - However, Table 6 shows that we cannot obtain good results with direct adaptations of safe online RL methods into offline settings, even with carefully tuned hyper-parameters, due to the intricate coupled training processes.
    - FISOR, however, enjoys good results thanks to its decoupled learning objectives.

Besides the experiments, we briefly discussed these safe online RL methods here:
- For Sauté[1], similar to CPQ[4], safety constraints are handled by integrating them into the state space and modifying the objective accordingly. They struggle to handle zero cost limit, thus we still consider it to be a type of soft constraint method.
- For [5], the intertwined learning process (barrier function, model, policy) is not suitable for offline scenarios.
- For [6], its online safe exploration with environments is not compatible for offline settings.

Reviewer comment (Experiments):
In the experiments, all the baselines are based on the soft safety constraint, and it seems unfair to simply compare their safety performances with a method enforcing hard constraints. Although it is reasonable to compare given the lack of existing works in the same setting, if safe online RL algorithms can be extended to offline settings, such algorithms should be additionally included as baselines.

Author response:
> **W1 & Q1 & W2.1 Some of safe online RL methods can be easily extended to offline RL settings, such algorithms should be included as a baseline.**

We thank the reviewer for these works and we are happy to compare against them in our revised paper! We have added comparison with Sauté [1] and RCRL [2], since Sauté [1] is relatively versatile and RCRL [2] also adopts HJ Reachability to enforce hard constraints. Results can be found in Table 6, Appendix D.7 in our revised paper.
- Note that applying safe online RL methods in offline settings faces distributional shifts. To address this, we add a BC constraint in Sauté [1] and RCRL [2] similar to an offline RL method TD3+BC [3] to combat distributional shifts. Also, we swept the conservatism strength to find a good balance between safety constraint and behavior constraint to obtain the best possible results.
    - However, Table 6 shows that we cannot obtain good results with direct adaptations of safe online RL methods into offline settings, even with carefully tuned hyper-parameters, due to the intricate coupled training processes.
    - FISOR, however, enjoys good results thanks to its decoupled learning objectives.

Besides the experiments, we briefly discussed these safe online RL methods here:
- For Sauté[1], similar to CPQ[4], safety constraints are handled by integrating them into the state space and modifying the objective accordingly. They struggle to handle zero cost limit, thus we still consider it to be a type of soft constraint method.
- For [5], the intertwined learning process (barrier function, model, policy) is not suitable for offline scenarios.
- For [6], its online safe exploration with environments is not compatible for offline settings.

Reviewer comment (Evaluation):
The authors should discuss the experimental results on the reward performance more thoroughly. Since FISOR enforces a hard constraint, it is natural for reward performance to deteriorate. However, given that most baseline methods do not satisfy the safety constraint, it feels arbitrary to compare reward performance **within** safe agents.

Author response:
> **W2.2. Discuss the experimental results on the reward performance.**
- The comparison criteria strictly follows the DSRL benchmark [7] (Q1 in general response), where safety is prioritized over rewards. We believe this makes sense as discussing the rewards is only meaningful when safety constraints are met in safety-critical tasks.
- Previous methods may prioritize reward maximization while overlooking safety constraints. Thus, some baselines can obtain high rewards. Figure 3 shows that some baselines even encounter more safety violations than the ones trained without any safety constraint. This shows that existing safe offline RL methods are hard to strike the right balance between reward maximization and safety satisfaction. Thus, we didn't evaluate solely based on rewards, but filtered out the safe agents, and then compared the rewards, strictly following the DSRL [7]. We believe this evaluation protocol is more reasonable.

Reviewer comment (Evaluation):
The claim that "FISOR is the only method that can guarantee safety satisfaction in all tasks, while achieving top returns in most tasks" seems overstated. The authors should discuss more in the **Main Results** paragraph for future readers, for example, why CPQ satisfies the safety constraints, why FISOR does not perform well in Safety Gym tasks, etc.

Author response:
> **W2.3. Discuss more on the Main Results**
- Why does CPQ satisfy the safety constraints? 
    - CPQ is an over-conservative method that prioritizes safety rather than reward maximization. Specifically, CPQ harshly penalizes unsafe areas by pulling up the cost values on those regions using value penalty [8]. However, value penalty is proved over-conservative [9], making CPQ primarily focus on safety constraints. For example, CPQ obtains good safety results on MetaDrive tasks but performs poorly in terms of rewards.
    - In addition, the offline value regularization is hard to tune [10], making CPQ perform not very well on other tasks.
- Why FISOR does not perform well in Safety Gym tasks?
    - No. Table 1 shows that FISOR meets safety requirements (normalized cost<1 according to the DSRL evaluation protocol) and meanwhile achieves competitive rewards on all SafetyGym tasks as compared to baselines. Moreover, FISOR has the lowest average cost scores as compared to all other baselines. It would be appreciated if the reviewer could tell us why the reviewer thinks FISOR does not perform well in Safety Gym tasks.

Reviewer comment (Novelty):
[Q1] There are several safe online RL algorithms with hard constraints, and some of them can be easily extended to offline RL settings. Could the authors comment on this?

Author response:
> **W1 & Q1 & W2.1 Some of safe online RL methods can be easily extended to offline RL settings, such algorithms should be included as a baseline.**

We thank the reviewer for these works and we are happy to compare against them in our revised paper! We have added comparison with Sauté [1] and RCRL [2], since Sauté [1] is relatively versatile and RCRL [2] also adopts HJ Reachability to enforce hard constraints. Results can be found in Table 6, Appendix D.7 in our revised paper.
- Note that applying safe online RL methods in offline settings faces distributional shifts. To address this, we add a BC constraint in Sauté [1] and RCRL [2] similar to an offline RL method TD3+BC [3] to combat distributional shifts. Also, we swept the conservatism strength to find a good balance between safety constraint and behavior constraint to obtain the best possible results.
    - However, Table 6 shows that we cannot obtain good results with direct adaptations of safe online RL methods into offline settings, even with carefully tuned hyper-parameters, due to the intricate coupled training processes.
    - FISOR, however, enjoys good results thanks to its decoupled learning objectives.

Besides the experiments, we briefly discussed these safe online RL methods here:
- For Sauté[1], similar to CPQ[4], safety constraints are handled by integrating them into the state space and modifying the objective accordingly. They struggle to handle zero cost limit, thus we still consider it to be a type of soft constraint method.
- For [5], the intertwined learning process (barrier function, model, policy) is not suitable for offline scenarios.
- For [6], its online safe exploration with environments is not compatible for offline settings.

Reviewer comment (Evaluation):
[Q2] The performance of an offline RL algorithm largely depends on the data coverage of the offline dataset. Could the authors explain the (empirical) relationship between the performance of FISOR and the data coverage? Is FISOR more efficient in terms of data coverage compared to other baselines?

Author response:
> **Q2.  Could you explain the (empirical) relationship of the performance if FISOR and the data coverage? Is FISOR more efficient in terms of the data coverage compared to other baselines?**
- We have added experiments on small data settings in Figure 10, Appendix D.8 in our revised paper. FISOR shows great stability across different dataset sizes, while other methods fail miserably under small data settings. We think the robustness in the low-data regime of FISOR is attributed to its stable decoupled learning procedure and policy extraction using weighted behavior cloning.

Reviewer comment (Novelty):
The authors may need to add more references to cover "safe RL with hard constraints" to better motivate this work and claim the differences. A quick search gives me several recent papers, such as a. Wang, Yixuan, et al. "Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments." International Conference on Machine Learning. PMLR, 2023. And those in b. Zhao, Weiye, et al. "State-wise safe reinforcement learning: A survey." arXiv preprint arXiv:2302.03122 (2023).

Author response:
We really appreciate the reviewer for providing these relevant works and we are more than happy to include them in our revised paper! Here, we briefly summarize the differences and relations with these related works:

- For [1], the authors proposed a method for learning safe policy with joint soft barrier function learning, generative modeling, and policy optimization. Compared to HJ reachability, barrier functions may be overly conservative [2]. Also, the intertwined learning process is not suitable for offline RL scenarios.
- For [3], although it considers state-wise constraints, this type of problem still allows for a certain degree of constraint violation. Therefore, we classify it as a soft constraint.

Additionally, we have included more discussions of related works in Appendix A. Here, we provide a brief summarization.

- For [4], although this method can achieve zero constraint violations, it requires domain knowledge of the task to design specific safety constraints, and thus is not suitable for general and more complex tasks.
- For [5], the authors primarily focus on the expansion of safe regions in online exploration. However, interaction with the environment is not permitted in an offline setting. In contrast, we learn the feasible region directly from the dataset.

Reviewer comment (Experiments):
It seems that the baselines in the experiments all consider soft safety constraints. The reviewer would like to see the comparison with baselines belonging to safe offline RL with hard constraints if there is such work.

Author response:
To the best of our knowledge, **FISOR is the first that considers hard constraints in the offline setting**. Please see Table 4 for details. This is why we only compared soft constraints in the safe offline benchmark DSRL [6].
- The soft constraint baselines reported in our paper can be transformed into hard constraints by setting the cost limit to $0$. However, as shown in our paper (Figure 3), they suffer from extremely inaccurate estimations of the cost value and leads to increased safety violations (Table 2).
- Some safe online RL methods that consider hard constraints can be extended to the offline RL settings by introducing some regularization to combat distributional shift. We combined RCRL [7], a safe online RL method that also uses HJ reachability, with TD3+BC [8] to adapt RCRL into the offline setting. The results can be found in Table 6, Appendix D.7 in our revised paper. Table 6 shows that RCRL+TD3BC fails miserably as it is hard to achieve a good balance between reward maximization, constraint satisfaction and distributional shift avoidance. FISOR, on the other hand, obtains great performance in the offline setting due to its decoupled learning procedure.

Reviewer comment (Evaluation):
Because this work uses learning to learn feasible regions by HJ reachability, how "accurate" are the learned regions? How does the learning region error affect the optimization problem and safety violations?

Author response:
Please see Figure 1 for illustration. The learned feasible regions in FISOR are far more accurate than the ones learned by cost value functions. 
- In addition, we can select a small $0<\tau<1$ in $L_{rev}^{\tau}(u)$ of Eq.(8) to avoid infeasible states being mistakenly learned as feasible states. We have added demonstrative results in Figure 6, Appendix D.1 of the revised manuscript. Results show that a small $\tau$ will induce a smaller-sized learned feasible region, which is acceptable in safety-critical scenarios as it will induce safer policies. Figure 4(a) also supports this argument.
- Moreover, estimation errors may increase as the amount of offline data decreases. We have added experiments on small dataset settings in Figure 10, Appendix D.8. Results show that FISOR is robust to small dataset settings as compared to baselines, thanks to its great training stability.

Reviewer comment (Theory):
There seems to be an inconsistency between the goal and the actual approach. The paper aims for 100% guarantee in safety. However, the approach still need to estimate the feasible areas and then to yield the value function threshold. There could be estimation error and this estimation requires very thorough dataset provided, which may not be practical in reality.

Author response:
We agree with the reviewer that we use deep neural networks to estimate the feasible regions and approximation errors could exist. However, we must clarify the following major misunderstandings. 

- First, note that the feasible areas assessed using HJ reachability are **far more accurate and robust** as compared to those measured using cost value functions, which are widely used in previous works, as demonstrated in Figure 1. Therefore, introducing HJ reachability into safe offline RL contributes a major step towards ensuring safety in safe offline RL settings.
- Moreover, It can be quite challenging to accurately specify feasible regions in high-dimensional continuous control tasks without functional approximators such as deep neural networks. For example, MetaDrive tasks have extremely complex state spaces with over 250 dimensions. In this sense, it is **impractical** to enforce hard constraints without functional approximators. Specifically, previous safe online RL methods, that aim for 100% safety guarantee, also require to estimate the feasible regions, and approximation errors also exist [1][2][3]. None of these methods can achieve the ideal 100% safety guarantee unless the full state-action space is fully explored.

To further address the reviewer's concern, we evaluated FISOR in small dataset settings. Please see Figure 10, Appendix D.8 for details. 

- Briefly, Figure 10 shows that a small offline dataset doesn't greatly impact FISOR's safety or performance. **This shows that FISOR is robust for small-sample scenarios**.
- In addition, FISOR is the first offline RL method that considers hard constraints and proposes a stable decoupled training procedure. As shown in Table 1, FISOR is the only method that can ensure safety in all 26 benchmark tasks, while maintaining the highest reward in most of the tasks as compared to other safe baseline agents. This is a remarkable result since the safe offline RL problem with hard constraint satisfaction itself is an extremely challenging task, as it requires seeking the tricky balance among reward maximization, safety constraints satisfaction and distributional shift avoidance.

Reviewer comment (Evaluation):
The learnt infeasible states may be false negative. How do you assure that the infeasible states are really unsolvable instead of you able not able to solve it?

Author response:
Controlling $\tau$ in Eq. (8) adjusts the feasible region's conservatism. A higher $\tau$ near 1 means learning near-maximum feasible regions, while lowering $\tau$ leads to a smaller feasible region.

- We added the visualization of learned feasible regions using different $\tau$ values in Figure 6, Appendix D.1 in our revised paper. Figure 4(a) also supports this argument.
- In practice, we select $\tau=0.9<1$, from which a smaller feasible region will be learned and enforces harder safety constraints, where small errors will not likely lead to false negative but false positive. However, a false positive is acceptable since it will lead to a more conservative and safe policy. Although this may lead to some reward loss, it is acceptable in safety-critical scenarios.
- If false negative leads to unsatisfactory unsafety by chance, we can increase the algorithm's conservatism by reducing the value of $\tau$.

Reviewer comment (Theory):
How do you guarantee the gap between the ground-truth infeasible regions and the learnt infeasible regions?

Author response:
Please see the responses to W2 for details.

Reviewer comment (Evaluation):
How do you measure the normalized cost in the evaluation? Why is it not zero for cases that are 100% safe?

Author response:
For the calculation method of normalized cost, please see the general response Q1. We respond to the latter question in W1.