Paper: Meta Inverse Constrained Reinforcement Learning: Convergence Guarantee and Generalization Analysis (ICLR-2024)

Reviewer comment (Novelty):
**Novelty**. Liu & Zhu (2022) has proposed a bi-level optimization objective for ICRL. This paper expands upon that proposal, adapting the objective to fit within a meta-learning context. The primary innovation is the integration of a meta-prior into the initial objective (see objective (2)). The algorithm presented outlines a direct, gradient-based optimization method, which, unfortunately, is computationally infeasible under most circumstances. The core focus of this paper is on *mitigating this computational difficulty* through the implementation of three levels of approximation. Additionally, it explores the impact of these approximations on both convergence and generalization. However, it should be noted that this study deviates somewhat from the original goal of Meta ICRL. Moreover, the advancements it provides over Liu & Zhu (2022) could be considered somewhat marginal.

Author response:
**Answer**: Thanks for mentioning the focus of this paper and the advancement over ICRL [D1].

**For the comment on deviation from meta-learning**, we respectfully disagree with it. There are three major reasons.

First, let us assume that we can obtain the exact hyper-gradient and ignore the approximation of the hyper-gradient. The other parts of this paper focus on the original goal of meta-learning. In specific, the original goal of meta-learning [D7,D8,D9], including meta-RL [D11], is to learn good meta-priors such that the corresponding task-specific adaptations can perform well on new tasks even if there is only limited data in new tasks. Along the same line, we extend the problem formulation of meta-learning to meta-ICRL using the bi-level optimization problem (2)-(3). More importantly, we theoretically analyze the convergence and generalization of the algorithm. The convergence and especially the generalization are the focus of meta-learning because the goal of meta-learning is to generalize to new tasks from limited data [D7,D8,D9]. In fact, these two properties have been widely studied in meta-learning [D8,D10,D12,D13], including meta-RL [D11].

Second, the approximation part is essential to solve the problem (2)-(3) which models meta-ICRL. In specific, solving problem (2)-(3) requires to compute the hyper-gradient. However, it is challenging because the hyper-gradient includes the inverse-of-Hessian term and the gradient of the inverse-of-Hessian term (as detailed in Section 3.1). We propose novel approximation methods to address these challenges.

Third, we would like to mention that recent papers [D8,D10,D11,D12,D13] perform theoretic analysis of meta-learning algorithms subject to approximation errors of the hyper-gradient caused by practical implementation. For example, [D10] studies the convergence of MAML [D7] and studies two methods to efficiently approximate the Hessian term in the hyper-gradient. It also quantifies the approximation error and convergence.

**For the advancements over ICRL** [D1], we would like to notify that M-ICRL studies a totally different problem from ICRL, instead of advancing the methodology of ICRL. The focus of M-ICRL is the meta-learning part instead of the ICRL part. M-ICRL only utilizes ICRL as a task-specific solver, however, the focus of M-ICRL is how to learn good meta-priors and the generalization performance. In order to learn meta-priors, we propose special designs in (2)-(3) and these special designs introduce unique challenges in algorithm design and theoretical analysis. The unique challenges in algorithm design, as mentioned in Section 3.1, are the difficulties of approximating the hyper-gradients. The unique challenges in theoretical analysis have two parts. First, we study generalization while [D1] does not. Second, we need to quantify the approximation errors and their impact on the convergence while [D1] does not.

Reviewer comment (Theory):
**Theoretical Contributions** - It is surprising to note that the upper bound is raised to the power of $K$ (the number of gradient steps), rather than being linear or sublinear with respect to $K$. This implies that the approximation error could accumulate rapidly with each gradient step. Unfortunately, the paper does not provide a satisfactory explanation for this issue.

Author response:
**Answer**: The approximation error exponentially decreases rather than exponentially increases in our case. Note that $\frac{C_{\nabla^2_{\eta\eta}G}}{\lambda}\in (0,1)$, therefore the approximation error in Lemma 1 and Lemma 2 exponentially decreases when the number of $K$ increases, which is a virtue.

Reviewer comment (Theory):
**Bi-level approximation**. In Liu & Zhu's (2022), the bi-level optimization framework of ICRL is structured such that the upper-level problem was focused on reward learning, while the lower-level problem addressed constraint learning. In this current paper, the upper-level problem has been expanded to include the learning of meta-priors for both rewards and constraints ($\theta$ and $\omega$). This shift deviates from the original bi-level optimization framework of ICRL, raising some questions about its consistency.

Author response:
**Answer**: Thanks for mentioning the meta-prior and task-specific adaptation. We agree that there is a deviation from the original ICRL problem in terms of learning the cost meta-prior in the upper level, however, we still learn the task-specific cost adaptation $\eta_i^{\ast}$ in the lower level. In fact, this deviation is necessary and standard in meta-learning. Note that in our problem formulation (2)-(3), we learn the reward meta-prior $\theta$ and cost meta-prior $\omega$ in the upper level, and the task-specific cost adaptation $\eta_i^{\ast}$ in the lower-level. This is exactly the standard practice of meta-learning [D7,D8,D9] where the upper level is to learn the meta-prior and the lower level is to learn the task-specific adaptations. Here, we would like to emphasize again that meta-learning (M-ICRL) and supervised learning (ICRL) study different problems and thus it is expected that meta-learning problem has special designs. 

The task-specific reward adaptation $\varphi_i$ is computed via one-step gradient descent from the reward meta-prior $\theta$. The intuition behind this is

Reviewer comment (Theory):
**Bi-level approximation** - The paper proposes that the task-specific reward adaptation is carried out via a gradient update. However, it is unclear whether this is a well-defined adaptation for specific tasks. There is a concern that this adaptation may be inadequate. It would be beneficial to provide additional supporting evidence. This could include referencing relevant papers where similar strategies have been successfully applied.

Author response:
to avoid overfitting using early stop as a regularization (as detailed at the beginning of this response, before answering Weakness 1). This style of computing the task-specific adaptation is called MAML [D7]. It is first introduced in [D7] and then widely adopted in meta-RL [D11] and meta-IRL [D14,D15]. The convergence [D10] and generalization [D12] of MAML are both well studied. It is actually one of the two predominant meta-learning methods. The other one is iMAML [D8].

Reviewer comment (Experiments):
**Empirical Results** - The environmental parameters in the MuJoCo experiments raise some questions. The paper states (see Appendix B.2) that "The design of the reward function is such that the robots will receive a reward of +1 if they maintain the target velocity and a reward of 0 otherwise." and "For the constraint, the Swimmer experiment constrains all the states where the front tip angle exceeds a0, where a0 ∈ [0.9, 1.2]." These constraints do not appear to be significant. In particular, it's unclear why the agent would need to violate the constraint to maximize cumulative rewards, and whether maintaining the correct speed is a suitable reward for a MuJoCo task. More justification for these settings is required.

Author response:
**Answer**: Thanks for mentioning the design of reward and constraint. For the design of reward, the motivation of maintaining the target velocity is that it has wide applications in the real world, such as autonomous vehicle driving at a constant speed [D19] and robotic dog skating at a constant velocity [D20].  Moreover, maintaining target velocity in Mujoco is a standard benchmark in meta-RL [D7,D16,D17] and meta-IRL [D18]. Since meta-learning needs multiple related tasks, maintaining different target velocity can be considered as multiple related task. 

For the design of the constraint in the Swimmer case, we use this kind of constraints because we empirically find that the angle of Swimmer's front tip will typically exceed 1.2 if there is no constraint. We choose this kind of constraint to show that our algorithm can indeed learn the constraint and avoid violating the constraint.

Reviewer comment (Evaluation):
**Empirical Results** - The results for Meta-IRL are counterintuitive. One would expect that imitation learning methods like IRL, which do not model constraints, would yield higher reward and constraint violation rates. The presented results do not align with this understanding. Please provide an explanation.

Author response:
**Answer**: Meta-IRL in our experiment yields higher constraint violation rate (CVR) and lower cumulative reward (CR). It is intuitive that Meta-IRL has higher constraint violation because it does not model constraints. It has lower reward because, as mentioned in the last paragraph in Appendix B.2, the bad CVR performance will also result in bad performance of CR even if Meta-IRL may estimate the ground truth reward function well, because the episode will terminate if any constraint is violated. Since the trajectory is truncated if the constraint is violated, it has smaller cumulative reward as it has shorter length to accumulate reward.

Reviewer comment (Experiments):
**Empirical Results** - There's a considerable gap between the performance of the baselines and the expert performance. The lead of M-ICRL is substantial, suggesting that all the baselines fail in the task. Without careful design, the validity of these baselines could be called into question. More details or improvements on the baseline design are suggested.

Author response:
**Answer**: Thanks for mentioning the design of the baselines. We introduce the four baselines in detail in the first paragraph of Appendix B. Here we would like to further discuss the details and improvement. The baseline ICRL is just running the algorithm in [D1] on one demonstration. It has bad performance because it lacks data and thus suffers from overfitting. To improve this baseline, we can either use more training data or tune a regularization term to avoid overfitting. The baseline Meta-IRL directly uses the algorithm in [D14]. To improve its performance, one way is to add certain constraint signal to augment the learned reward function. The constraint signal includes certain information about the constraint so that it can help Meta-IRL to achieve better CVR performance and thus CR performance. The baseline ICLR(pre) naively learns meta-priors across all demonstrations of all the training tasks. In specific, ICRL(pre) first solves $\min_{\theta}\frac{1}{m}\sum_{i=1}^mL_i(\theta,\omega^{\ast}(\theta),D_i), \text{s.t.}\omega^{\ast}(\theta)=\arg\min_{\omega}\frac{1}{m}\sum_{i=1}^mG_i(\omega;\theta,D_i)$ where $D_i\triangleq${$D_i^{\text{tr}},D_i^{\text{eval}},D_i^{\text{h}}$}. The obtained results of this problem (i.e., $\theta$ and $\omega^{\ast}(\theta)$) are the meta-priors of ICRL(pre). ICRL(pre) then uses these meta-priors as initializations to solve the problem $\min_{\theta}L_{m+1}(\theta,\omega^{\ast}(\theta),D_{m+1}),\text{s.t.}\omega^{\ast}(\theta)=\arg\min_{\omega}G_{m+1}(\omega;\theta,D_{m+1})$ for an arbitrary new task $\mathcal{T}_{m+1}$. To improve the performance of ICRL(pre), we can make the distance between different tasks smaller. The distance between tasks is defined in Section 4.2.

Reviewer comment (Theory):
**Question** - Why does objective (1) incorporate a discounted log-likelihood? This suggests that the policy in later time steps has less impact on the likelihood objective. It raises the question: Would a Markov Decision Process (MDP) with a finite horizon be more consistent with the current objective?

Author response:
**Answer**: The reason that the objective (1) uses a discounted log-likelihood is that we study infinite horizon MDP and the log-likelihood may be infinite without the discount factor $\gamma$. It is interesting to study finite horizon MDPs. Intuitively, if the problem aimed to be solved has a finite horizon, finite horizon MDP is more suitable.

Reviewer comment (Writing):
**Question** - The $\delta$ present in gradients (6) and (7) is not defined anywhere in the paper. It would be beneficial to provide an explicit definition or reference for this term.

Author response:
**Answer**: Thanks for notifying this. We already add the explanation of $\delta$ in the newly uploaded version. The $\delta$ is just a constant, which is the magnitude of perturbation.

Reviewer comment (Novelty):
The paper is an extension of meta IRL and ICRL. Building upon these, the contribution of the study is not significant.

Author response:
Thanks for mentioning the novelty. Though our problem combines ICRL in [C1] and meta-learning in [C2,C4], our algorithm design and theoretic analysis are substantially different from those in the three references. We discuss the challenges of algorithm design in Section 3.1 and the challenges of theoretical analysis in the first paragraph of Section 4.1. Here we would like to further discuss our unique challenges and novelties in algorithm design and theoretical analysis compared to [C1,C2,C4]. 

**Algorithm design**. [C1,C2,C4] and our work all formulate a bi-level optimization problem. Note that [C2] uses MAML [C3] and MAML can be regarded as a bi-level optimization problem where the lower level is one-step gradient descent from the meta-prior and the upper level aims to optimize for the meta-prior [C4]. A critical step of solving bi-level optimization problems is to compute the hyper-gradient, i.e., the gradient of the upper-level problem. The methods of computing the hyper-gradient in [C1,C2,C4] **CANNOT** be applied to our case and thus novel algorithm designs for computing the hyper-gradient are needed. In specific, there are two unique challenges in terms of algorithm design. 

First, our hyper-gradient includes the term $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ while [C1,C2,C4] do not have this term. As mentioned in challenge (i) in Section 3.1, computing $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ requires to compute the gradient of an inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$. Since we use neural networks as the parameterized model, computing this inverse-of-Hessian term is already challenging enough and now we need to compute the gradient of this inverse-of-Hessian term, which is intractable. To solve this issue, we propose a novel approximation design where we only need to compute first-order gradients (see (6)-(7) and Algorithm 2) and we can prove that the approximation error can be arbitrarily small (Lemma 2).

Second, as mentioned in challenge (ii) in Section 3.1, our hyper-gradient includes the inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$ while [C1,C2] do not have this issue. [C4] also has this issue, however, they just use existing conjugate gradient method and assume that a good approximation can be obtained. In contrast, we propose to solve an optimization problem (10) in Algorithm 3 to approximate and more importantly, we theoretically guarantee in Lemma 1 that the approximation error can be arbitrarily small.

**Theoretical analysis**. We study **BOTH** the convergence and generalization of our algorithm while [C2] do not have theoretical analysis and [C1,C4] only study convergence. Therefore, the whole Section 4.2 is unique and novel compared to [C1,C2,C4] because it studies generalization. Even for convergence, our analysis is novel compared to [C1,C4]. In specific, since we have the unique challenges for algorithm design and we provide novel algorithm designs (i.e., approximation methods) to tackle those unique challenges, we need corresponding theoretical analysis to guarantee the performance of our unique algorithm designs, i.e., Lemma 1 and Lemma 2 quantifies the approximation error of our algorithm designs.

Reviewer comment (Novelty):
The paper only shows the convergence to $\epsilon$-FOSP, whose optimality is not discussed. Convergence of gradient-based algorithms to stationary point is not a novel contribution.

Author response:
Thanks for mentioning the convergence. We agree that convergence to global optimal solutions is very interesting, however, it is quite challenging in our case because our problem is highly non-convex. Note that our problem is still non-convex even if we use linear parameterized models instead of neural networks [C1]. We notice that some bi-level optimization works [C9,C10] have studied the convergence to the global optimal solutions, however, they all impose strong assumptions, e.g., the upper-level objective function is either convex or strongly convex. These assumptions are not satisfied in our case.

In fact, the convergence to $\epsilon$-FOSP is not trivial in our case. Convergence of SGD to stationary points has been widely studied, however, our novelty is beyond the convergence of SGD. As mentioned in the first paragraph of Section 4.1, compared to the standard SGD, the main difficulty of guaranteeing the convergence of Algorithm 1 lies in quantifying the approximation error of the hyper-gradients and how the errors affect the convergence. Note that we do not have the exact hyper-gradient, thus our algorithm is not standard SGD. The most difficult part of solving problem (2)-(3) is how to approximate the hyper-gradient and how to ensure that the approximation error does not ruin the convergence. Therefore, our major contribution of convergence is to design novel approximation methods in Section 3.2 and theoretically guarantee that the proposed approximation method can lead to convergence in Section 4.1. Besides convergence, we also study the generalization in Section 4.2.

Reviewer comment (Theory):
Assumption 1 assumes that the reward function and the cost function are bounded and smooth up to the fourth order gradient, which is a strong assumption, especially for neural networks with ReLU activation and unbounded state-action space.

Author response:
Thanks for mentioning the assumption. We agree that these assumptions could be strong when the neural networks use ReLU activation. However, the smoothness assumption can be satisfied by using some smooth activation functions such as tanh and using linear parameterized models. Moreover, as mentioned in the paragraph under Assumption 1, these assumptions are typical assumptions in RL [C5], IRL [C6,C8] and meta-RL [C7]. It is interesting to relax these assumptions, and we will explore it in the future.

Reviewer comment (Theory):
The lower level optimization (as explained in Equation 11, Appendix A.1) maximizes the reward and causal entropy objective while matching the constraint feature expectations. On the other hand, the authors of [1] formulate the optimization as a maximization of the causal entropy while matching reward feature expectations and expected cumulative costs. The slight difference is that [1] has an additional Lagrange multiplier term $\lambda$ in the dual formulation (see [1], page 4, below remark 2, in the definition of $G$). In this work, since the constraint feature expectations are directly matched, the overall form of the lower level objective $G$ (Appendix, equation 12) is not exactly the same as [1]. This is also apparent in the constrained soft Bellman policy definition adapted from [1] (no $\lambda$ in the constrained policy for this work, whereas there is a $\lambda$ in the constrained policy as described in [1], page 2, Appendix). The outcome is that the constraint function is just treated as a negative reward term that can be just added to the original reward and thus, constrained RL just amounts to running RL with reward $r-c$ (Appendix A.2 of this work also says this). I am not sure this is representative of typical constrained RL problems, since typically it is not possible to rewrite a constrained RL problem as an unconstrained RL problem with a different reward. (maybe the authors can elaborate on this)

Author response:
**Answer:** It is very insightful that you point this out. In fact, the definition of $G$ in Appendix A.1 is equivalent to the one in [B1] even if we do not have $\lambda$, and the definition of $G$ in Appendix A.1 can work for constrained RL in our case. Note that the cost function $c_{\omega}$ in our problem is not given nor fixed, but is actually learned. This point is very important for understanding why the slight difference of $\lambda$ does not matter and why the $r_{\theta}-c_{\omega}$ can work for constrained RL in our case.

First, we would like to explain why the function $G$ in Appendix A.1 is equivalent to the function $G$ in [B1]. The definition of $G$ in Appendix A.1 is $G(\omega;\theta)=H(\pi_{\omega;\theta})+J_{r_{\theta}}(\pi_{\omega;\theta})+\omega^{\top}(\mu_c(\pi_i)-\mu_c(\pi_{\omega;\theta}))$ where $\pi_{\omega;\theta}(a|s)=\frac{\exp(r_{\theta}(s,a)-\omega^{\top}\phi_c(s,a)+\gamma\int_{s'\in\mathcal{S}}P(s'|s,a)V_{\omega;\theta}^{\text{soft}}(s')ds')}{\exp(V_{\omega;\theta}^{\text{soft}}(s))}$. The definition of function G in [B1] (let us call it $\bar{G}$) is $\bar{G}(\bar{\omega};\theta,\lambda)=H(\pi_{\bar{\omega};\theta,\lambda})+J_{r_{\theta}}(\pi_{\bar{\omega};\theta,\lambda})+\lambda\bar{\omega}^{\top}(\mu_c(\pi_i)-\mu_c(\pi_{\bar{\omega};\theta,\lambda}))$ where $\pi_{\bar{\omega};\theta,\lambda}(a|s)=\frac{\exp(r_{\theta}(s,a)-\lambda\bar{\omega}^{\top}\phi_c(s,a)+\gamma\int_{s'\in\mathcal{S}}P(s'|s,a)V_{\bar{\omega};\theta,\lambda}^{\text{soft}}(s')ds')}{\exp(V_{\bar{\omega};\theta,\lambda}^{\text{soft}}(s))}$ (section 8 in Appendix of [B1]). Note that $\lambda$ is just a scalar and $\bar{\omega}$ is a vector ([B1] studies linear cost). In [B1], both $\lambda$ and $\bar{\omega}$ are learned. In our problem, $\omega$ is learned. For any learned $\lambda$ and $\bar{\omega}$, we can always learn an $\omega$ such that $\omega=\lambda\bar{\omega}$. Therefore, $G$ in our case is equivalent to $\bar{G}$ in [B1] as long as we simply replace $\lambda\bar{\omega}$ in $\bar{G}$ with $\omega$. Since the characteristics of $\lambda\bar{\omega}$ can be easily captured by a single $\omega$, there is no need to learn two parameters $\lambda$ and $\bar{\omega}$.

Second, we explain why the $r_{\theta}-c_{\omega}$ can work for the constrained RL in our case. We agree that it is not possible to rewrite a constrained RL problem as an unconstrained RL problem. However, we do not aim to use $r_{\theta}-c_{\omega}$ to solve a constrained RL problem where the reward function is $r_{\theta}$ and the cost function is $c_{\omega}$. What we do is to learn $r_{\theta}$ and $c_{\omega}$ such that the corresponding policy $\pi_{\omega;\theta}$ can imitate the expert and thus solve the original constrained RL problem where the reward function is $r_i$ and cost function is $c_i$. The learned cost function $c_{\omega}$ could be learned as the expert's cost function $c_i$ multiplying a very large positive constant so that the $r_{\theta}-c_{\omega}$ can enable $\pi_{\omega;\theta}$ to approximately satisfy the original constraint which is defined using $c_i$. The situation in [B1] is that they use $r_{\theta}-\lambda c_{\bar{\omega}}$ and they can learn both $\lambda$ and $\bar{\omega}$ to enable the policy to approximately satisfy the constraint in the original constrained RL. We have the same spirit with [B1] even if we do not have the parameter $\lambda$. Since $\lambda$ is just a scalar, as mentioned in last paragraph, we can learn a single model $c_{\omega}$ to replace $\lambda c_{\bar{\omega}}$. The $r_{\theta}-c_{\omega}$ in our case is equivalent to $r_{\theta}-\lambda c_{\bar{\omega}}$ in [B1].

In conclusion, our approach is not mis-formulated compared to the one in [B1], but is actually equivalent to the one in [B1]. More importantly, our theoretical analysis is self-contained, i.e., we can theoretically guarantee the convergence and generalization performance of our approach.

Reviewer comment (Experiments):
What if several demonstrations are available for each task? If more demonstrations are available, ICRL could perform better and the gap between M-ICRL & ICRL could be lesser. M-ICRL should still perform better, since it has a better meta-prior, but overall it would be useful to understand the empirical improvement of M-ICRL over ICRL as the number of demonstrations vary.

Author response:
**Answer**: Thanks for mentioning varying the number of demonstrations. We add an additional experiment in Appendix B.4 in the newly uploaded version to show how the performance of ICRL and M-ICRL improves when the number of demonstrations increases. Basically, ICRL improves faster than M-ICRL and the gap between ICRL and M-ICRL is decreasing. When there are more than $80$ demonstrations, the performance of ICRL and M-ICRL is very close because at this point, both M-ICRL and ICRL have enough information about the test tasks, and the meta-prior no longer plays a dominant role to help M-ICRL achieve good performance.

Reviewer comment (Theory):
While Equation 12 (Appendix A.1) is the dual of Equation 11, if the domain is extended from linear constraint functions to non-linear constraint functions, the equation would no longer behave as the dual of the original problem as formulated in Equation 11, right? Does it make sense to use this as the lower level problem, in that case?

Author response:
**Answer**: Yes, you are right! The function $G$ is no longer the dual function of (11) if we use non-linear cost functions. However, we can still use this $G$ function in the lower level.

First, the fundamental reason that the function $G$ can be used to learn cost functions is that it serves as a negative log-likelihood function of the training data (explained in Appendix A.4) instead of being the dual of (11). When the cost function is non-linear, the function $G$ is no longer the dual function of (11) but it still serves as a negative log-likelihood function of the training data. Therefore, the function $G$ does not lose its functionality when the cost function is non-linear and thus can be used as the lower-level objective function. The derivation of $G$ from (11) only serves as explaining how we come up with the function $G$.

Second, the whole theoretical analysis in this paper holds for non-linear cost functions, which provides strong justification that non-linear cost functions can work.

Reviewer comment (Writing):
(Suggestion) Notation can be slightly confusing at some places, so I would suggest mentioning in the algorithms what the inputs and the outputs are, in words (e.g., meta-priors, etc.). Implicit hyperparameters should also be mentioned in the beginning of the algorithm, e.g., $\alpha$, $B$, etc. what do these refer to, in the algorithm?

Author response:
**Answer**: Thanks for your suggestion. We have revised the algorithms accordingly. The $\alpha$ is the step size and $B$ is the number of sampled tasks at each iteration $n$.

Reviewer comment (Novelty):
In terms of novelty, it would be helpful for readers if authors can emphasize the unique novelty and challenges of solving meta inverse constrained RL beyond simply combining the techniques in the inverse constrained RL problem studied in (Liu & Zhu 2022) and the meta RL/ICRL studied in (Xu et al 2019; Rajeswaran et al 2019)?

Author response:
**Q1**: In terms of novelty, it would be helpful for readers if authors can emphasize the unique novelty and challenges of solving meta inverse constrained RL beyond simply combining the techniques in the inverse constrained RL problem studied in [A1] and the meta RL/ICRL studied in [A2,A3]?

**Answer**: Thanks for mentioning the novelty. Though our problem combines ICRL in [A1] and meta-learning in [A2,A3], our algorithm design and theoretic analysis are substantially different from those in the three references. We discuss the challenges of algorithm design in Section 3.1 and the challenges of theoretical analysis in the first paragraph of Section 4.1. Here we would like to further discuss our unique challenges and novelties in algorithm design and theoretical analysis compared to [A1,A2,A3].

**Algorithm design**. [A1,A2,A3] and our work all formulate a bi-level optimization problem. Note that [A2] uses MAML [A7] and MAML can be regarded as a bi-level optimization problem where the lower level is one-step gradient descent from the meta-prior and the upper level aims to optimize for the meta-prior [A3]. A critical step of solving bi-level optimization problems is to compute the hyper-gradient, i.e., the gradient of the upper-level problem. The methods of computing the hyper-gradient in [A1,A2,A3] **CANNOT** be applied to our case and thus novel algorithm designs for computing the hyper-gradient are needed. In specific, there are two unique challenges in terms of algorithm design.

First, our hyper-gradient includes the term $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ while [A1,A2,A3] do not have this term. As mentioned in challenge (i) in Section 3.1, computing $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ requires to compute the gradient of an inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$. Since we use neural networks as the parameterized model, computing this inverse-of-Hessian term is already challenging enough and now we need to compute the gradient of this inverse-of-Hessian term, which is intractable. To solve this issue, we propose a novel approximation design where we only need to compute first-order gradients (see (6)-(7) and Algorithm 2) and we can prove that the approximation error can be arbitrarily small (Lemma 2).

Second, as mentioned in challenge (ii) in Section 3.1, our hyper-gradient includes the inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$ while [A1,A2] do not have this issue. [A3] also has this issue, however, they just use existing conjugate gradient method and assume that a good approximation can be obtained. In contrast, we propose to solve an optimization problem (10) in Algorithm 3 to approximate and more importantly, we theoretically guarantee in Lemma 1 that the approximation error can be arbitrarily small.

**Theoretical analysis**. We study **BOTH** the convergence and generalization of our algorithm while [A2] do not have theoretical analysis and [A1,A3] only study convergence. Therefore, the whole Section 4.2 is unique and novel compared to [A1,A2,A3] because it studies generalization. Even for convergence, our analysis is novel compared to [A1,A3]. In specific, since we have the unique challenges for algorithm design and we provide novel algorithm designs (i.e., approximation methods) to tackle those unique challenges, we need corresponding theoretical analysis to guarantee the performance of our unique algorithm designs, i.e., Lemma 1 and Lemma 2 quantifies the approximation error of our algorithm designs.

Reviewer comment (Novelty):
There is a related work "A CMDP-within-online framework for Meta-Safe Reinforcement Learning, Vanshaj Khattar, et al, ICLR, 2023" which studies the meta learning for the constrained RL. What is the similarity and differences between authors' methodology compared with their works in terms of how to deal with constraints and meta-learning? It is important to add such comparisons in the paper.

Author response:
**Q2**: There is a related work [A4] which studies the meta learning for the constrained RL. What is the similarity and differences between authors' methodology compared with their works in terms of how to deal with constraints and meta-learning? It is important to add such comparisons in the paper.

**Answer**: Thanks for mentioning this valuable reference, we have added the comparisons in Appendix A.15 in our newly updated version. First, we would like to mention that [A4] studies a different problem. [A4] studies meta constrained RL where the constraint (cost function) is given while we study meta ICRL where the cost function needs to be learned.

For dealing with the constraint, [A4] has a constrained RL problem to solve for each task. It uses a constrained RL algorithm called CRPO [A5] to solve the constrained RL problem and get a corresponding task-specific policy. We have a cost learning problem in the lower level and we use gradient descent to obtain a corresponding task-specific cost adaptation. One similarity between [A4] and our work is that we both do not require the exact task-specific adaptation and this makes the theoretical analysis of the meta learning performance more challenging.

For dealing with meta-learning, [A4] studies an online setting where at each online iteration, a new task is input and a corresponding task-specific policy adaptation is computed. At each online iteration, it updates the policy meta-prior by minimizing the KL divergence between the policy meta-prior and the current task-specific policy adaptation via one or multiple online gradient descent steps. In contrast, we utilize a bi-level optimization framework where we learn the meta-priors in the upper level such that the corresponding task-specific adaptations can maximize the likelihood of the demonstrations of each task. In order to optimize for the meta-priors, we need to compute the hyper-gradient which is very challenging in our case. We propose several novel approximation methods and algorithm designs to approximate the hyper-gradient. In conclusion, the meta-prior in our case is learned such that the task-specific adaptations adapted from the meta-prior have good performance on each specific task while the meta-prior in [A4] is learned such that the meta-prior is close to the task-specific adaptations according to the metric of KL divergence.

Reviewer comment (Reproducibility):
The formulated M-ICRL is a complicated formulation, and the proposed approximate methods further increases the complexity of the algorithm. Can authors summarize the tricks in the implementation level to achieve the reported good results? This is helpful for the future researchers who want to extend this paper.

Author response:
**Q4**: The formulated M-ICRL is a complicated formulation, and the proposed approximate methods further increases the complexity of the algorithm. Can authors summarize the tricks in the implementation level to achieve the reported good results? This is helpful for the future researchers who want to extend this paper. 

**Answer**: Thanks for mentioning the implementation tricks and we have added the details in Appendix B.3 in the newly uploaded version. We have two major tricks, the first trick is to reduce the sample complexity of computing all the gradient and second-order terms in our algorithm, and the second trick is to reduce the iteration complexity of solving the lower-level problem and computing the partial gradients.

**Reducing the sample complexity**. Note that the expressions of all the gradient and second-order terms are included in Appendix A.3. These terms all have expectation terms which need to be estimated. Take $\nabla_{\omega}G_i(\omega;\theta)=\nabla_{\omega}J_{c_{\omega}}(\pi_i)-E_{\zeta\sim P_{\pi_{\omega;\theta}}}[\sum_{t=0}^{\infty}\gamma^t\nabla_{\omega}c_{\omega}(s_t,a_t)]$ as an example, we need to roll out $\pi_{\omega;\theta}$ to get a set of trajectories in order to estimate $E_{\zeta\sim P_{\pi_{\omega;\theta}}}[\sum_{t=0}^{\infty}\gamma^t\nabla_{\omega}c_{\omega}(s_t,a_t)]$. The trick of reducing sample complexity is that for each task, we use the same sampled trajectory set to estimate the expectation terms in all gradient and second-order terms, e.g., $\nabla_{\omega}L$, $\nabla_{\omega\omega}^2G_i$, etc. By doing so, we only need to conduct the sample operation once when a new task is input, and then we just use these samples to estimate the expectation terms of all the gradient and second-order terms within this task.

**Reducing the iteration complexity**. The trick we use is warm start. For solving the lower level problem, we do not initialize $\eta$ randomly. In fact, we use the obtained task-specific adaptation of last task to be the initialization of this task. Since the tasks are relevant in our case, The corresponding task-specific adaptations are not too far from each other. Therefore, this kind of initialization can significantly reduce the iteration numbers needed to compute the task-specific adaptations. We use the same trick for Algorithm 3 and it can significantly reduce the needed number of iterations, especially for computing the partial gradients of the perturbed variables in Algorithm 2. Since $\theta+\delta\Delta_{\theta,i}$ and $\theta-\delta\Delta_{\theta,i}$ are very close given that $\delta$ is chosen small, empirically, fewer than ten gradient descent steps from the warm start initialization can provide a good result.