Paper: Learning Constraints from Offline Dataset via Inverse Dual Values Estimation (ICLR-2024)

Reviewer comment (Theory):
The biggest concern I have is the way this paper models cost values. It approximates cost value as $\lambda_d (D_f(d\| d^E)-\epsilon)$. This is very problematic since being sub-optimal does not necessarily mean it is unsafe or a cost violation. Matching with expert occupancy measures is essentially doing some sort of inverse RL on rewards rather than conventional constraint learning.

Author response:
**Response:**    We apologize for the confusion. **We have updated our paper to derive our learning objective from a constraint learning function (see Section 4) instead of the divergence between two distributions**. We hope our new derivation will serve as a better explanation for our algorithm. 

In our revised derivation, we adopt an approach to learning a cost function by introducing a penalty for undesired behavior derived from an optimal policy (objective 9). Then we address the forward optimization challenge of the learned cost function through dual optimization (objective 8). To substantiate the effectiveness of our method, we have established the equivalence condition for a feasible inverse constraint learning solution (refer to Proposition 2) and demonstrated the convergence of this learning process within finite CMDP (refer to Proposition 3). In addition to these theoretical proofs, our paper provides an analysis of our method, with both formulas and a concrete example. We illustrate how our approach leads to a learned solution that exhibits sparsity, a critical concern in constraint learning problems (refer to Section 4.1). We believe that this new derivation effectively illustrates how to transform a constraint learning problem into our bi-level optimization formula, thereby increasing the soundness of our algorithm.

Reviewer comment (Novelty):
The model construction is highly similar to RGM [1], but it is not even mentioned in the paper. RGM considers a reward correction problem using a similar DICE-based framework. Both the proposed IDVE and RGM use an expert dataset and a large unknown/sub-optimal quality dataset as inputs. Both methods model the problem as similar bi-level optimization formulations: at one level minimize $D_f(d^*\| d^E)$ and on the other level solve an offline RL problem. The only difference is that RGM learns a reward correction term and this paper learns a cost value function $V^c$. Given the similar high-level formulation, I suspect the proposed method is essentially doing some sort of reward correction rather than constraint learning.

Author response:
**Response:** We appreciate your assistance in bringing this previous work to our attention, and we apologize for our lack of awareness of it prior to your mention. After a careful review, we agree that our initial derivation shares some modeling techniques and intermediate objectives with the RGM, such as the concept of bi-level optimization.

We have updated our paper to derive our learning objective from a constraint learning function (see Section 4), replacing the divergence between two distributions. We hope this new derivation offers a more comprehensive explanation of our algorithm.

Besides, it's important to note that our work differs significantly from the RGM as we consider different problem settings and objectives. These differences create a substantial gap between our methods in the following aspects:

- In the RGM, the authors **do not provide an explicit mathematical model** for an imperfect reward. They propose four settings of imperfect rewards in Figure 1, but **none of these settings** align with the setting of our Inverse-constrained Reinforcement Learning problem.

- The RGM's aim is to outperform traditional Reinforcement Learning/Imitation Learning techniques, focusing on enhancing forward-control performance to maximize rewards. Conversely, we address an inverse inference problem where the objective is to learn the constraint in order to a) recover the safe policy across all states, and b) allow the transfer of the constraint to other environments with the same dynamics. Our primary concern is safety.

- The derivation of the RGM requires training a discriminator between the demonstration data and the policy-generated data. Our algorithm, however, does not necessitate a discriminator network. Instead, we employ a violation action buffer to better model the problem, as elaborated in section 5.3.

- While their algorithm necessitates minimizing the divergence between the learned policy and the expert policy, our algorithm also can achieves this by maximizing the cost of undesirable behavior derived from raw reward and data. However we derive it from an equivalent condition of constraint learning.

- We demonstrate the effects and dynamics of our learning process and use the chi-square divergence as an example. **This is particularly relevant to sparse constraint learning**, a point not discussed in the RGM.

Reviewer comment (Theory):
There are many designs in this paper that look not very principled. For example, although the problem is formulated in a DICE framework, the authors use techniques from in-sample learning offline RL algorithms to construct the practical algorithm. First of all, the "state-value" $V$ in DICE-based methods is not the common meaning of state-value functions in typical RL problems, they are actually Lagrangian multipliers. If the authors are familiar with the DICE-class of algorithms, they will notice that the $V$ values in DICE algorithms take very different values and in many cases behave differently from those in common RL algorithms. Hence simply learning value functions using in-sample learning algorithms like IQL or SQL and then plugging them back into a DICE formulation is inappropriate. Also, there are lots of tricks and somewhat arbitrary designs in the practical algorithm, like introducing $\lambda$ in Eq.(12), and the treatment in Eq.(15)-(17). By contrast, RGM offers a much cleaner and more rigorous derivation from theory to practical algorithm.

Author response:
**Response:** Thanks for raising the concerns. We acknowledge that the "state-value" $V$ in DICE-based methods does not have the common meaning of state-value functions in typical RL problems, and we have added a description of how to derive policy from the learned "Lagrangian multipliers" instead of treating them as the value function in an "in-sample learning algorithm."
To be more specific, we want to a policy $\pi_\psi(s,a)$ to sample from $d^*(s,a)$ according to optimal solution $V^r$, so we extract the policy by maximizing  $\max_\psi E_{s,a\sim d^*(s,a)}{\log\pi_\psi(s,a)}=\max_\psi E_{s,a\sim d^O(s,a)}{{\omega_f^*(\delta V^r(s,a))\log\pi_\psi(s,a)}}$. Thank you for correcting our mistake.

Reviewer comment (Theory):
The derivation of the proposed framework IDEV and its practical implementation introduces many unreasonable transformations, which makes the algorithm lack soundness.

Author response:
We apologize for the confusion. The primary goal of the ICRL problem is learning a cost function, and by minimizing the expected cumulative cost, the imitating agent can reproduce the expert behavior (See definition 1, Section 3). **We have updated our paper to derive our learning objective from a constraint learning function** (see Section 4) instead of the divergence between two distributions. We hope our new derivation will serve as a better explanation for our algorithm. 

In our revised derivation, we adopt an approach to learning a cost function by introducing a penalty for undesired behavior derived from an optimal policy (objective 9). Then we address the forward optimization challenge of the learned cost function through dual optimization (objective 8). To substantiate the effectiveness of our method, we have established the equivalence condition for a feasible inverse constraint learning solution (refer to Proposition 2) and demonstrated the convergence of this learning process within finite CMDP (refer to Proposition 3). 

In addition to these theoretical proofs, our paper provides an analysis of our method, with both formulas and a concrete example. We illustrate how our approach leads to a learned solution that exhibits sparsity, a critical concern in constraint learning problems (refer to Section 4.1). We believe that this new derivation effectively illustrates how to transform a constraint learning problem into our bi-level optimization formula, thereby increasing the soundness of our algorithm.

Reviewer comment (Theory):
In the first paragraph of Section 4.2, the authors approximate the expert regularizer $\lambda_d (D_f (d||d^E) - \epsilon)$ with $\mathbb{E}_d [\delta^c_V (s, a)]$. However, $\mathbb{E}_d [\delta^c_V (s, a)]$ is not a reasonable approximation of $\lambda_d (D_f (d||d^E) - \epsilon)$. There is even no connection between $\mathbb{E}_d [\delta^c_V (s, a)]$ and $\lambda_d (D_f (d||d^E) - \epsilon)$: the former is a temporal difference term w.r.t the cost function while the latter is a divergence between two distributions.

Author response:
We apologize for the confusion. The primary goal of the ICRL problem is learning a cost function, and by minimizing the expected cumulative cost, the imitating agent can reproduce the expert behavior (See definition 1, Section 3). **We have updated our paper to derive our learning objective from a constraint learning function** (see Section 4) instead of the divergence between two distributions. We hope our new derivation will serve as a better explanation for our algorithm. 

In our revised derivation, we adopt an approach to learning a cost function by introducing a penalty for undesired behavior derived from an optimal policy (objective 9). Then we address the forward optimization challenge of the learned cost function through dual optimization (objective 8). To substantiate the effectiveness of our method, we have established the equivalence condition for a feasible inverse constraint learning solution (refer to Proposition 2) and demonstrated the convergence of this learning process within finite CMDP (refer to Proposition 3). 

In addition to these theoretical proofs, our paper provides an analysis of our method, with both formulas and a concrete example. We illustrate how our approach leads to a learned solution that exhibits sparsity, a critical concern in constraint learning problems (refer to Section 4.1). We believe that this new derivation effectively illustrates how to transform a constraint learning problem into our bi-level optimization formula, thereby increasing the soundness of our algorithm.

Reviewer comment (Theory):
The introduction of the lower-level optimization in Eq.(11) is weird. The authors have replaced the expert regularizer with another term. However, they again add this regularizer in the bi-level optimization.

Author response:
We apologize for the confusion. The primary goal of the ICRL problem is learning a cost function, and by minimizing the expected cumulative cost, the imitating agent can reproduce the expert behavior (See definition 1, Section 3). **We have updated our paper to derive our learning objective from a constraint learning function** (see Section 4) instead of the divergence between two distributions. We hope our new derivation will serve as a better explanation for our algorithm. 

In our revised derivation, we adopt an approach to learning a cost function by introducing a penalty for undesired behavior derived from an optimal policy (objective 9). Then we address the forward optimization challenge of the learned cost function through dual optimization (objective 8). To substantiate the effectiveness of our method, we have established the equivalence condition for a feasible inverse constraint learning solution (refer to Proposition 2) and demonstrated the convergence of this learning process within finite CMDP (refer to Proposition 3). 

In addition to these theoretical proofs, our paper provides an analysis of our method, with both formulas and a concrete example. We illustrate how our approach leads to a learned solution that exhibits sparsity, a critical concern in constraint learning problems (refer to Section 4.1). We believe that this new derivation effectively illustrates how to transform a constraint learning problem into our bi-level optimization formula, thereby increasing the soundness of our algorithm.

Reviewer comment (Theory):
In the last line of Section 5.1, they introduce the approximation $Q^c(s, a)=\mathbb{E}_\{\left(s, a, s^{\prime}\right) \in \mathcal{D}^o}\left[\gamma V\_{\theta^c}^c\left(s^{\prime}\right)\right]$. This approximation is incorrect. The correct one is $Q^c(s, a)= c(s, a) + \gamma \mathbb{E}\_{s^\prime \sim p (\cdot|s, a)} [V^c\_{\theta^c} (s^\prime)]$.

Author response:
Sorry for this typo. We actually intended to express $Q^c(s,a)=E_{(s,a,s')\in D^o}[c(s,a)+\gamma V^c_{\theta^c}(s')]$, but omitted the term $c(s,a)$. Thank you for bringing this oversight to our attention.

Reviewer comment (Experiments):
The experiment setup is a little weird. In particular, the offline dataset in grid-world is collected by random policies, and the offline dataset in MuJoCo is collected by SAC policies. Such offline datasets may contain a large number of unsafe behaviors, contradicting the motivation of this paper. Thus, a more proper choice is to apply safe and conservative policies to collect the offline dataset.

Author response:
Thanks for the suggestion! We indeed apply safe and conservative policies to collect expert data. This approach ensures that there is no constraint-violating behavior in the expert data. However, when it comes to non-expert data, it may include risk-seeking or unsafe behaviors with larger rewards and costs than those of experts. The constraint can be inferred from the difference between expert and non-expert data.
    
A realistic example is the task of autonomous driving. We can collect observational data of driving behavior from real life, **which may contain high-risk behaviors like cutting in line, breaking the speed limit, etc**. Meanwhile, we also have access to human behavior with safe driving habits, which is safe and conservative. Therefore, it's reasonable to assume that our dataset contains unsafe behavior. 

If both expert and non-expert data were generated by conservative policies, ICRL algorithm cannot infer constraints. This is because we wouldn't have samples from constraint-violating policies, hindering our ability to infer the boundary between safe and unsafe behaviors.

Reviewer comment (Writing):
In the last paragraph of Section 3, “The demonstration dataset $\mathcal{D}_{O}$” should be “The demonstration dataset $\mathcal{D}^{O}$”.

Author response:
We apologize for the typos. They have been corrected in latest version.

Reviewer comment (Writing):
The notation of $\sum_{p_{\mathcal{T}}\left(s^{\prime} \mid s, a\right)} \gamma V^r\left(s^{\prime}\right)$ is confusing. The correct one should be $\sum_{s^\prime \in \mathcal{S}} p_{\mathcal{T}}\left(s^{\prime} \mid s, a\right) \gamma V^r\left(s^{\prime}\right)$ or $\mathbb{E}\_{s^\prime \sim p\_{\mathcal{T}}\left(\cdot \mid s, a\right) }  [\gamma V^r\left(s^{\prime}\right)]$.

Author response:
We apologize for the typos. They have been corrected in latest version.

Reviewer comment (Novelty):
Although IDVE is closely linked to the DICE family, formulated with the distribution correction estimation $w(s, a)$, there is no discussion and referencing to the DICE string of works [e.g., 1, 2]. Inclusion of OptiDICE [3] into discussion is also recommended as it also uses a closed-form solution for the inner maximization, in the DICE framework.

Author response:
Thank you for your suggestion. We have now included a citation for OptiDICE and indeed, both approaches solve a closed-form solution for the inner maximization in a dual problem.

It's crucial to clarify the distinctions between the DICE (DIstribution Correction Estimation) methods from prior work and our approach. The DICE methods:

- Are specifically designed to handle the forward control problem or imitation learning problem.
- Solve a single-level optimization problem without considering constraints.

In contrast, our method:

- Is designed to address an inverse constraint learning problem under a Constrained Markov Decision Process (CMDP).
- Solves a bi-level optimization problem (as shown in Equations 8 and 9) **which involves both updating the constraint and the control policy**.

Reviewer comment (Experiments):
While considering no costs, the comparison between IDVE w/oA and offline RL is somewhat questionable. In scenarios like limited arm halfcheetah and blocked halfcheetah, IDVE w/oA shows notably superior cumulative rewards. The inferiority of offline RL suggests potential issues with the baseline's strength or its implementation. Given that the offline RL's objective solely maximizes returns, one would anticipate it to at least match, if not surpass, IVE w/oA in terms of returns.

Author response:
Thank you for raising your concerns. We have reviewed our implementation and made necessary adjustments to the hyperparameters. The experiment has been rerun, and the updated results, presented in Table 2 and Figure 6, have been included. The updated results are marked in blue.

Reviewer comment (Presentation):
The annotation for the dashed line in Figure 6 appears to be missing. I would also recommend the authors to plot average cumulative rewards/costs for both $D^E$ and $D^{\neg E}$. It will be helpful for the audiences to better understand the numbers in Table 2.

Author response:
Thanks for pointing this out. We have added the missing annotation to enhance the clarity of Figure 6.

Reviewer comment (Evaluation):
The gap in returns between expert and sub-optimal demonstrations, as well as between offline IL and expert demonstrations, is unclear. This ambiguity arises because the environments have been modified, eliminating the availability of standardized D4RL scores for comparison. Therefore, plotting rewards/costs of both $D^E$ and $D^{\neg E}$ would be helpful to improve clarity.

Author response:
We agree including this information can improve the clarity of our work. We have included plots of average cumulative rewards/costs in Figure 6, for both the offline dataset and the expert offline dataset in our revised paper.

Reviewer comment (Theory):
Section 5.2: The choice of $\pi \propto \delta^r-\delta^c$ is somewhat odd. Such a $\pi$ won't optimize the inner equation of Eq (7), given the current value estimations. I wonder why was this chosen over $\pi \propto \exp(\delta^r-\delta^c)$?

Author response:
We apologize for this confusion. In fact, the choice of $\pi \propto \max(\delta^r - \delta^c,0)$ aligns with the use of chi-squared divergence (Section 4.1). In specific, this is the representation of optimal policy when $D_f$ is implemented by chi-squared divergence. On the other hand, $\pi \propto \exp(\delta^r-\delta^c)$ is optimal in terms of Kullback-Leibler divergence. **These choices are optimized under different f-divergences**. We have ensured this is accurately conveyed in the updated version of our paper in section 5.2.

Reviewer comment (Reproducibility):
In Figure 3, in offline RL/IL, the value functions are solely associated with rewards. How, then, were the constraints derived from these methods?

Author response:
Sorry for the confusion. We include the value functions from offline IL to study whether we can represent constraints by utilizing only the reward function. Specifically, this function should assign a low value to the constraint-violating behavior and larger value to others. In the case of offline RL, we include the value functions to better illustrate its behaviors in control.

Reviewer comment (Experiments):
In Table 5 (Appendix), the varying number of expert transitions for different tasks appears a bit random. Could the authors provide clarity on this decision?

Author response:
We apologize for creating this confusion. Across all the experiments, we control the number of expert trajectories to 100. The varying number of expert transitions is due to its dependence on the episode length in the environment. We have revised our paper to better convey this setting.