Paper: Uncertainty-aware Constraint Inference in Inverse Constrained Reinforcement Learning (ICLR-2024)

Reviewer comment (Presentation):
The reviewer suggests the authors should try to summarize the main questions and introduce them at the start of Section 6 to give the reader an idea of what to expect, making for an easier read.

Author response:
Thank you for your valuable suggestions. We have revised the paper by "summarizing the main questions and introducing them" at the start of Section 6 in the revised paper.

Reviewer comment (Presentation):
In the experiments, the reviewer advises that the authors should walk readers through the figures more explicitly, as readers may not easily infer the intended meaning.

Author response:
Thank you for your suggestion. We have revised our paper by directly referencing the results, thereby eliminating the need for readers to search for them. This enhances the readability and accessibility of our paper.

Reviewer comment (Presentation):
The conclusion is very short and feels rushed; the reviewer believes the authors should have more to say here.

Author response:
Thanks for mentioning this concern. Due to the page limit, we were not able to expand our discussion. In the revised version, with extra space, we have revised the conclusion section, where we have summarized the key findings, underlined the implications of our research, and clearly articulated future research directions.

Reviewer comment (Presentation):
It is unclear what the bolding in the tables signifies — whether it shows statistical significance or something else — and this should be clarified in the text.

Author response:
Thanks for mentioning this concern. In this paper, Bolding indicates the results that are most optimal among the compared results.

Reviewer comment (Writing):
The reviewer believes the paper cannot be published until the conclusion is written properly.

Author response:
Thanks for mentioning this concern. Due to the page limit, we were not able to expand our discussion. In the revised version, with extra space, we have revised the conclusion section, where we have summarized the key findings, underlined the implications of our research, and clearly articulated future research directions.

Reviewer comment (Writing):
In Section 2, the notation $\mathcal{M}^{c_{\omega}}$ is not defined anywhere.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
Figure 1 text is way too small.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
In Figure 1, it is not clear where to start reading from; the reviewer thinks one should start from $\mathcal{D}_{e}$ and suggests this should be mentioned in the caption.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Writing):
Page 3 footnote is missing a full stop.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Writing):
In Section 4.2, the notation $\mathcal{T}$ is never formally introduced.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Reproducibility):
In Algorithm 1, it is unclear whether "sample nominal trajectories" implies interacting with the environment, and if so, this should be explicitly stated.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
Table 1 text is way too small.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
Figures 3/4/5/D.3/D.4/D.5/D.6 only need one legend.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
The x-axis ticks in Figures 3/4/5/D.3/D.4/D.5/D.6 are too close.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
The plot titles in Figures 3/4/5/D.3/D.4/D.5/D.6 do not all need to say "with noise $\mathcal{N}(0,0.1)$".

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
Figure 6 is not clear, and each column refers to a grid world scenario, which should be indicated on the figure and/or mentioned in the caption.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
Figure 6 text is way too small.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
In Figure 6, the locations of the constraints are not indicated.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
In Figure 6, the agent's start and end points are not indicated.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
In Figure 7, it is unclear what each row represents; for instance, whether the top row is ICRL and the bottom row is UAICRL needs to be made clear.

Author response:
Thank you for your detailed review of this work. We have updated the paper according to your suggestions one by one as follows:  
- "*textual citations in parentheses*": we have updated the paper with true citation format.  
- "*$\mathcal{M}^{c_{\omega}}$ in Section 2*": It denotes the CMDP with constraint function $c_\omega$.  
- "*Figure 1*": We have made the text larger and added descriptions in the caption.  
- "*Page 3 footnote*": We have added the ".".  
- "*Notation in Section 4.2*": $\mathcal{T}$ denotes the trajectory set, and following [1], we slightly abuse $F$ to denote the "flow".   
 And actually, the trajectory flow is the integral of state flow.  
- "*Algorithm 1*": We have explicitly stated that "interacting with the environment". During implementation, we generate new data in each iteration, and the old augmented data is discarded.  
- "*Table 1, Figure 2's text*": We have increased the font size.  
- "*Figures 3/4/5/D.3/D.4/D.5/D.6*": We modified the figures with larger text, one legend, wider x-axis ticks, and proper titles to make the m more legible.  
- "*Figure 6 is not clear*": We have changed Figure 6 with clearer explanations, larger text, and added the constraint locations.  
- "*Figure 7 is not clear*": We have changed Figure 7 with clearer explanations and illustrations.

Reviewer comment (Presentation):
The reviewer asks whether the authors have made the figures legible.

Author response:
With your suggestions and our self-motivation to make the paper more readable and better, we have made our utmost effort to make the figures legible.

Reviewer comment (Evaluation):
Limited discussion and explanation of experiments: While the manuscript presents extensive experimental results, a more comprehensive discussion and elaboration of these findings would enhance the paper's overall quality. Moreover, a detailed examination of performance across various scales of randomness within the primary context could provide valuable insights, as noted in my question (1-3).

Author response:
We appreciate your insightful comments on the discussion and interpretation of our experimental results. We totally agree that a more exhaustive analysis would significantly enhance the quality of our paper.

However, due to constraints on page limits, we were unable to provide detailed explanations for all observed phenomena. As a result, we chose to focus on some of the most intriguing and significant findings for further elaboration.

In the revised version of our paper (see our following response and please refer to section 6 and Appendix E in our paper), we have conducted a more in-depth analysis of our results. We hope this could address your concerns.

Reviewer comment (Evaluation):
Could you please explain the reason behind the divergence trend observed in the UAICRL method in section 6.2, particularly in the Blocked Walker task? Additionally, would it be possible to provide results with an extended number of epochs? Were the curves smoothed in your analysis?

Author response:
Thank you for bringing up this concern. By "divergence trend," we assume you're referring to the observed pattern where different baseline algorithms display varying trends in their reward and constraint violation rates throughout the training curves. This divergence is most noticeable in the Blocked Walker environment. For instance, by the end of training, the rewards for UAICRL and UAICRL-NDA progressively increase, while those for other methods tend to remain unchanged or even decrease.

To understand this phenomenon, it's important to note that **both UAICRL and UAICRL-NDA implement risk-sensitive constraints**. These constraints enable the agent to more effectively manage the aleatoric uncertainty resulting from noise, thereby facilitating the learning of risk-aware policies. This could result in considerable improvement in policy updates, particularly when the agent encounters critical situations or locations.
For instance, these constraints enable the agent to initially learn to maintain a safe distance from hazardous zones. This strategy allows the agent to better understand how to accrue rewards without entering dangerous events.

**Response.** To delve deeper into this phenomenon, as suggested by the reviewer, we have extended the number of epochs in the Blocked Walker environment, subjecting it to the noise of $\mathcal{N}(0,0.1)$ (see Blocked Walker in Figure 3). In this specific task, we observed that the trend lines become smooth when the episode count reaches 60k. Furthermore, the rewards for both UAICRL and UAICRL-NDA begin to increase at a higher rate compared to the other methods. We have revised the paper by adding these updates.

Reviewer comment (Evaluation):
What factors contribute to the challenges posed by the Blocked Swimmer task? It seems that most methods struggle to learn a safe (low-cost-violation) policy for this specific task.

Author response:
Thank you for bringing up this concern. There are, in fact, several factors contributing to the complexity of the Blocked Swimmer task:

- Firstly, the Swimmer environment itself is inherently challenging to solve by RL methods. Previous works report similar difficulties in RL control [1].

- Secondly, the mechanical dynamics of the Swimmer Robot make it easier for the robot to move forward than backward. The block region is also defined in front of the robot (where the X-coordinate $>0.5$). The intrinsic robust dynamics may strongly compel the robot to proceed forward, leading to constraint violations even in deterministic settings (refer to Figure 2 in [2]).

- Thirdly, the stochastic transition dynamics further complicate the safe control, even when agents are aware of the ground-truth constraints (refer to PPO-Lag in Figure D.7). The task becomes even more difficult when the agent is required to predict constraints from provided demonstrations.

In conclusion, the Blocked Swimmer environment already presents a challenge for traditional constrained RL. The challenge is amplified when constraints need to be inferred (as in the ICRL setting) and when transition dynamics are stochastic.

**References**

[1] Franceschetti, Maël, et al. "Making Reinforcement Learning Work on Swimmer." arXiv preprint arXiv:2208.07587 (2022).

[2] Liu, Guiliang, et al. "Benchmarking constraint inference in inverse reinforcement learning." ICLR (2023).

Reviewer comment (Evaluation):
Could you elaborate on the factors leading to the relatively unsatisfactory results of the baseline methods in the tasks? Specifically, what could explain the discrepancy in the performance of GACL, which performs well in the Block Ant task but not in other tasks shown in Figures D.3 and D.4?

Author response:
Our paper primarily investigates the robustness of various methods in the face of aleatoric and epistemic uncertainties. As such, we intentionally control the scale of noise and the number of demonstrations, as detailed in our experimental settings. The baseline methods partially or completely disregard these factors (refer to Table 1). Specifically,

- Baseline methods that do not address stochastic transition dynamics fail to learn constraints that ensure the safety of the learned policy. This oversight can negatively influence control performance.

- If epistemic uncertainty in Out-of-Distribution (OoD) data points is not taken into account, it could lead to inaccurate cost predictions, subsequently affecting policy updates and overall performance.

**Response.** The observed discrepancy in GACL's performance, where it works well in the Blocked Ant task but underperforms in other environments, can be attributed to its approach to constraint learning. Unlike traditional constrained RL problems, GACL directly appends the discriminator $logD(s, a)$ to the reward function as a penalty, resulting in a modified reward function $r'(s, a) = r(s, a) + logD(s, a)$. This modification allows us to investigate the impact of simple reward shaping on constraint learning.

We find GACL performs well in the Blocked Ant since this environment has the highest dimension in all environment. In other environments with lower dimensions, the discriminator $logD(s, a)$ is prone to overfitting the imperfect data early in training. The log-probability penalty from this low-quality discriminator applies a substantial penalty to many feasible regions, rendering the algorithm incapable of learning.
However, in high-dimensional environments (like the Ant task), the discriminator is less likely to overfit. As a result, the impact of early mistakes in the discriminator is less significant, enabling continuous learning.

Reviewer comment (Experiments):
How would you describe the generalizability of your method to multi-cost settings?

Author response:
Thanks for raising this concern.
We assume the term "multi-cost settings" indicates the ground-truth CMDP has different kinds of constraints, and the question is "How well our neural constraint function can approximate these different kinds of constraints". 

We follow a recent paper [1] and study the performance of this neural constraint function in the autonomous driving environment under our UAICRL framework. The experiment results and the corresponding analyses are included in Appendix D.4. The results show that UAICRL outperforms other baselines with the highest rewards while keeping a low violation rate of both types of constraints, demonstrating its ability of uncertainty awareness with respect to multiple constraints.

[1] Liu, Guiliang, et al. "Benchmarking constraint inference in inverse reinforcement learning." ICLR (2023).

Reviewer comment (Experiments):
In practical experiment (fig 3), UAICRL actually performs similarly to UAICRL-NDA, which removes the data augmentation part.

Author response:
Thanks for raising this concern. In fact, we observe that in environments with high-dimensional state-action space (HalfCheetah, Ant, and Walker), UAICRL significantly outperforms UAICRL-NDA with higher rewards and lower constraint violation rates. However, in environments with relatively lower state-action dimensions (Swimmer and Pendulum), the difference of performance between UAICRL and UAICRL-NDA is less significant. 
This is because, in the high-dimensional environment, a training dataset of a fixed size can only encompass a limited number of data points within the input space. In contrast, **in simpler or lower-dimensional environments**, the input space is smaller. As a result, a dataset of the same size can **cover a larger proportion of data points**. This increased coverage reduces the chances of Out-of-Distribution (OoD) data points and **the impact of epistemic uncertainty**. Consequently, the incorporation of a data augmentation component does not lead to a substantial enhancement in performance. We have ensured this observation is properly conveyed in the revised paper (see Q6 in Appendix E).

Reviewer comment (Presentation):
Although the authors give more illustrations in fig.7 (I suppose the top is for MEICRL and bottom is for UAICRL), I think it's not very clear. For example, I believe the authors should at least explain what the generated trajectory is, and which parts are OOD.

Author response:
Thanks for the suggestions. We have added an example of generated trajectories in Figure 7 and Figure D.4 in the Appendix. In the paper, we have explained how the generated trajectory increases the diversity of the dataset, thus reducing the impact of epistemic uncertainty. The generated trajectory is not incorporated into the expert dataset due to the restricted size of dataset. This trajectory enhances the diversity of the dataset, thereby mitigating the redundant constraints learned by MEICRL.

Reviewer comment (Presentation):
In fig 4, the baseline should be "GACL" instead of "GAIL".

Author response:
Thanks for your correction! We have updated it in the revised paper.

Reviewer comment (Experiments):
Why are some experiments early stopped when they obviously have not converged? E.g., in fig.3&4.

Author response:
Thank you for bringing up this issue. We interpret "convergence" to mean the point (i.e., episode) at which the performance of the algorithms stabilizes or remains unchanged with further training. Yet, in a stochastic environment, the training of the algorithm is consistently influenced by noise, which makes achieving convergence more challenging. In our experiment, defining an upper bound for the convergence episode across various random seeds (since the added noise is affected by these seeds) and tested algorithms is challenging. 

To address this issue, we empirically identify such an episode at which the algorithm's performance neither significantly decreases nor increases. The plots in Figures 3 and 4 were generated according to this criterion. Even though strict convergence may not have been achieved, these training plots are particularly valuable for illustrating sample complexity. An algorithm with lower sample complexity is expected to converge more rapidly.

But to address your concerns, we conduct additional experiments with an extended number of epochs for some environments that seem not to converge (i,e, Half-cheetah, Pendulum, and Walker with noise $\mathcal{N}(0,0.1)$), and we have updated the plots in the revised paper (see Figure 3).

Reviewer comment (Evaluation):
The authors run experiments on Mujoco tasks with different scales of stochasticity in env. But many baselines have much higher constraint violation rate with smaller noise, e.g., comparing fig.3, D3&D4. My intuition is that these methods should behave better with smaller noise as they cannot model such uncertainty. Could you explain it?

Author response:
Thanks for raising this concern. In fact, we have briefly mentioned this finding in the last two lines in Section 6.2, we have updated the paper and discussed this finding in more detail (see Q2 in Appendix E). To elaborate, when noise levels are relatively low, its impact becomes challenging for the distributional value function to detect, resulting in a constraint that is less sensitive to the risk. Conversely, as the level of noise (i.e., stochasticity) increases, the **variance** of the estimated distribution becomes prominent, thereby making the constraint more responsive to the risk of unsafe behaviors. Our algorithms can thus induce stricter constraint functions and thus a lower constraint violation rate.

Reviewer comment (Experiments):
The ablation of the mutual information term in Eq. (7) results in a configuration where only the risk-sensitive constraint from Eq. (4) is utilized. This particular setup is not discussed in the paper.

Author response:
**Response.**  Thank you for your suggestion. Indeed, our ablation study includes a comparison with the UAICRL-NDA baseline, which examines the performance of the model without the consideration of epistemic uncertainty (refer to Table 1 in our paper). Since we estimate epistemic uncertainty using the mutual information term, UAICRL-NDA is structured to study the effect of removing this term. In the paper, we primarily highlight that UAICRL-NDA omits the data augmentation process. This is because, within our experiment, data augmentation has a more significant impact on model performance compared to other components in the mutual information estimation process (entropy and dropout layers as per Equation (8)). Despite this, during the implementation, we remove the entire epistemic uncertainty estimator.

Reviewer comment (Experiments):
I'm concerned about expanding the dataset by generating trajectories based on a learned flow function. It is still possible for the flow function to generate out-of-distribution data.

Author response:
**Response.** 
Thanks for your comments. Our FTG can generate a rich number of trajectories that are not included in the original dataset (out-of-distribution) due to its ability of:
- **Generating Diverse Samples.**
Intuitively, FTG is based on Generative Flow Networks (GFlowNets), which demonstrated excellent performance in generating a diverse array of samples in the realm of scientific discovery ([1]). Rather than consistently producing the highest-scoring trajectories, which correspond to the largest flows, the generation process employed by GFlowNets is characterized by sampling from a distribution represented by learned flow functions. Many of the samples are **new**, and thus they can facilitate scientific discovery. 
- **Generating OoD Trajectories.**
Conceptually, at each time step, the state-action pairs that are sampled reflect the patterns **observed in the provided dataset** (for example, molecules). However, the true strength of this approach is realized when **these pairs are concatenated**. This process enables the representation of **Out-of-Distribution (OoD) trajectories** (for example, proteins), extending the utility of the GFlowNets beyond the confines of the training data.
- **Generating OoD Data in Feature Space.** 
Many previous works ([2][3]) have demonstrated the effectiveness of generative models, such as GFlowNets, GAN, and diffusion models, in dealing with out-of-distribution data. Additionally, it's crucial to note that the superior performance of generative models (like GPT) can be attributed to leveraging high-level latent representations of raw inputs. Similarly, we believe that our FTG exhibits comparable expressive power, generating trajectories that resemble the training data in the latent space, even if they are OoD in the state-action feature space. 

In conclusion, with efficient representing capabilities and objective-driven sampling, FTG is able to generate OoD data.


**References** 

[1] Bengio, Yoshua, et al. "Gflownet foundations." Journal of Machine Learning Research (2023).

[2] Jain, Moksh, et al. "GFlowNets for AI-driven scientific discovery." Digital Discovery (2023).

[3] Marek, Petr, et al. "OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation." NAACL-HLT (2021).

Reviewer comment (Experiments):
I'm wondering if the flow functions can be substituted with other conditional generative models, or if the flow matching objective is tightly coupled with UAICRL. For example, can one use a conditional diffusion model to replace the Flow-based Trajectory Generation (FTG) algorithm?

Author response:
**Response.** We have chosen to utilize GFlowNets due to its primary design focus on **generating sequences or trajectories**. In detail, the generative process of the Flow Trajectory Generator (FTG) is **Markovian**, meaning it generates the state at time $s_t$ based on the preceding state $s_{t-1}$.

In contrast, diffusion models are primarily developed for **image generation**. Recent works have expanded these models to generate trajectories [4]. However, the process of diffusion or denoising is **inherently non-Markovian** (unless specific restrictions are imposed), resulting in the generation of an entire trajectory in one go. This model resembles a Bandit solver.

In the context of the Inverse Constrained Reinforcement Learning (ICRL) problem, the sampling of constraint-violating behaviors at any given time step should depend on the previous states and actions. Therefore, while it is possible to replace FTG with a diffusion model, we maintain that the flow-based generator is more **closely aligned with the requirements of our work.**

We have also considered the Decision Transformer (DT) [5]. However, DT is primarily proposed to generate actions instead of states, which makes it less suitable for our task.

Reviewer comment (Experiments):
While Table B.1 suggests that FTG can maintain consistent hyperparameters across various tasks, I wonder how the performance of UAICRL might be influenced by the selection of hyperparameters for the FTG network. This concern arises from the potential of FTG to either underfit or overfit, which could lead to generating out-of-distribution trajectories and potentially causing a decline in overall performance. Could you explain the process of tuning the hyperparameters for the FTG network?

Author response:
**Response.** Thanks for raising this concern. To address your concern, we have revised the paper to include the detailed fine-tuning process of FTG in Appendix B.4. Here, we provide a brief overview of it:

The key hyperparameters in the FTG are **1) sample flows $M$**: since we cannot directly calculate the continuous inflows and outflows to perform flow matching, we discretize the flows and match the discretized flows instead by sampling $M$ actions independently and uniformly from the continuous action space. It is crucial for network training with respect to underfitness or overfitness. **2) sample action size $K$**: for generating in continuous spaces, we need to sample $K$ actions from the action space and calculate their corresponding flows, and actions with larger flows will be executed with higher probabilities. Relatively large $K$ will promote exploration in the generation process, which helps to generate diverse trajectories. **3) network architecture**. **4) learning rate**. 

Following the hyperparameters in CFlowNets [6], which also study the MuJoCo environment using generative networks, we keep **the sample action size $K$, network architecture and learning rate**, and only adjust the sample flows $M$. Ideally, large $M$ may make it harder for the network to converge, while small $M$ may result in overfitting. In this work we set $M=50$ (instead of $M=100$ in [6]). The reasons are as follows: (1) Too large $M$ will not only make the network hard to converge but also result in a significant increase in GPU memory. To avoid heavy computation and underfitness, we should avoid setting $M$ too large. (2) Too small $M$ will make the network overfit and perform poorly. So we can not set $M$ too small. In addition, we utilized the annealing learning rate to avoid overfitting. (3) We have also **conducted experiments** to help us choose the hyperparameters, including analyzing the quality of the generated trajectories by FTG (check Table B.2) and evaluating the whole performance of UAICRL (check Figure B.1) with different numbers of $M$. 

Finally, to trade off the performance and computational load, we set $M=50$ and maintain its consistency without fine-tuning the hyperparameters excessively. As shown in experiments, it could work well across various tasks, demonstrating the FTG's adaptability and robustness. We acknowledge that there is always room for improvement and believe that fine-tuning will be beneficial for specific tasks. 

**References.**

[4] Zhu, Zhengbang, et al. "Diffusion Models for Reinforcement Learning: A Survey." arXiv preprint arXiv:2311.01223 (2023).

[5] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." NeurIPS (2021).

[6] Li, Yinchuan, et al. "CFlowNets: Continuous Control with Generative Flow Networks." ICLR (2023).