Paper: Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? (ICLR-2024)

Reviewer comment (Writing):
Given that this paper proposes new metrics for IRL, I found the discussion / comparison with previous work a bit vague. At times, the language and writing was a bit informal, and sometimes confusing to interpret. For example, in the appendix you write that "our method that considers is greater than theirs". What does it mean for a method to be greater than another method? Did you mean your metric? You write in C.3 that the "metric can't capture transitions". What does this actually mean?

Author response:
We appreciate the reviewer for bringing up this question. We have made extensive revisions and polished Appendix C significantly in the latest version. We have addressed the issues you mentioned, and you can review the latest version.  For example, regarding "our method that considers is greater than theirs", detailed explanations are provided in the discussion above Lemma C.1 in the most recent version. Concerning "metric can't capture transitions",  comprehensive explanations can be found in the discussion above Proposition C.4 in the latest version.

Reviewer comment (Novelty):
I would urge the authors to rewrite their comparison with prior work in Appendix C, focusing on cases where one subsumes the other, and giving concrete examples when your algorithm(s) can achieve guarantees while previous algorithms cannot. What is currently lacking is a distinction in the writing between (1) comparing the quantities \( d^\pi \) and \( d^\mathrm{all} \) themselves to prior work; (2) comparing guarantees that your algorithms can achieve to guarantees that algorithms from prior work can achieve.

Author response:
Thank you very much for your suggestions. We have made substantial revisions in the latest version of Appendix C as per your requests (see Appendix C in the latest version).

Reviewer comment (Theory):
(minor) weakness: the upper/lower bounds seem to be loose. In particular, it would be good if the authors commented on the fact that the lower bound has a \( \min \{S, A\} \) term - where does this come from? can it be improved? In general, the regime that we care about is when \( S \ge A \), so the fact that the rate is only sharp when \( S \le A \) is not very meaningful.

Author response:
About $\textrm{min}\\{S,A\\}$ factor: Regarding $\mathrm{min}\\{S,A\\}$ term, the introduction of $\textrm{min}\\{S,A\\}$ is introduced as follows. In constructing hard instances, there is a step where we need to ensure $\mathbb{P}_h ( s_i| s_i,a_i)=1$ for $i\leq \textrm{min}\\{S,A\\}$. This particular step is crucial for our proof. Exploring ways to improve the lower bound on $\textrm{min}\\{S, A\\}$ could be a promising avenue for future research. We believe our results are still meaningful, and the condition $S < O(A)$ also holds significance. Regarding the lower bound in the online setting, we have made improvements (Theorem G.2), improving the original $\mathcal{O}(H^2SA\textrm{min}\\{S,A\\})$ to $\mathcal{O}(H^3SA\textrm{min}\\{S,A\\})$. Although it does not exactly match the upper bound, it has been significantly tightened. We appreciate the reviewer for bringing up this question.

About $H$ factor: Regarding the lower bound in the online setting, we have made improvements (Theorem G.2), improving the original $\mathcal{O}(H^2SA\textrm{min}\\{S,A\\})$ to $\mathcal{O}(H^3SA\textrm{min}\\{S,A\\})$. Although it does not exactly match the upper bound, it has been significantly tightened. We appreciate the reviewer for bringing up this question. As for the lower bound in the offline setting, even though we employed the same MDP construction as in proving the lower bound in the online setting, we did not fully exploit the property of our MDP construction, where transitions are different for each h, because of the construction of the behavior policy. This results in our lower bound having a dependence on $H$ of $\mathcal{O}(H^2)$. How to prove a tighter lower bound for the $H$-factor would be a highly intriguing avenue for future research.

Reviewer comment (Writing):
Many typos throughout, especially in the appendix. Some examples: "Broaderly" in the first paragraph; at bottom of page 4, "Given a policy \( \pi \), We" -> lower case the "we"; a missing citation before Corollary 4.3? I see an "()"; "week transferability" should be "weak transferability"?; some typo in the equation (C.6); at the beginning of page 23, in the first display equation, should it be \( a \in \mathrm{supp} \) instead of \( a \notin \mathrm{supp} \)?

Author response:
We thank the reviewer for this feedback. We have addressed and corrected the majority of the typos as suggested.

Reviewer comment (Theory):
I'm a bit confused about the discussion of the Hausdorff metric used by Metelli et al. The quantity that they propose seems to be some notion of diameter. I'm not sure why it should go to zero as you collect more samples, as even when \( \mathcal{R} = \widehat{\mathcal{R}} \) the quantity doesn't seem like it should be zero since the set of possible rewards could be large. But your results imply that an upper bound on this quantity goes to zero, so what am I missing here?

Author response:
If we understood correctly, the reviewer is asking why the Hausdorff metric appears to be a kind of diameter for sets and why it tends to zero? The Hausdorff metric $D^{\sf H}$ (see Section C.2) is indeed a measure of distance between sets, not a diameter. For instance, when two sets are equal, the Hausdorff metric becomes zero. And so the upper bound goes to zero.

Reviewer comment (Theory):
While the definition of the metric seems mathematically well defined, it seems a bit weird: to define the reward mapping, you need a value and advantage function (V,A) (e.g., Eq. 3.2). However, inside the definition of the metric, you define a new value function of a particular policy and reward function (e.g., in Eq. 3.1). How do these two value functions relate?

Author response:
$(V, A) \in \mathcal{V} \times \mathcal{A}$ represents a set of parameters, not a value function. Each pair $(V, A)$ determines a solution to an IRL problem (Lemma 2.2). Let $r^\theta = \mathscr{R}(V, A)$, where $\mathscr{R}$ is a reward mapping induced by an IRL problem $(\mathcal{M}, \pi^E)$. The connection between the parameter $\theta = (V, A)$ and the value function defined in Eq. 3.1 on Page 3 is as follows: $V^{\pi^E}_h(s; r^\theta) = V_h(s)$ and $A^{\pi^E}_h(s, a; r^\theta)=A_h(s,a)$.

Reviewer comment (Writing):
The main paper writing needs to be improved. Yes, the soundness of this paper maybe good, considering similar pessimism and exploration ideas from past works. But this submission looks like a hurried submission with many typos like `Theoretical low bound in offline setting.`, ` $\pi_b = \pi_E ().,$`, `RLP utilizes empirical MDP and Pessimism frameworks`, `for all offline IRL problems with probability at least $1 − \delta$, has to take at least ... samples in expectation.` (both h.p. and in expectation?!), and so on. In addition to typos, the out-of-margin equation formatting makes a strenuous reading experience. To be honest, I am not sure if the authors can fix this writing issue of 56 pages during the rebuttal period, but I will welcome some attempts since proceedings require good quality.

Author response:
We sincerely apologize for the confusion caused by writing issues. Thank you very much for pointing out the concrete writing issues, and we have corrected most of them in the latest version.

Reviewer comment (Theory):
The closest work I can think of is [1] that provides theoretical guarantees for imitation learning ($\approx$ reward-free IRL+RL) in both online and offline data setting. The current work stops at reward learning, that is, the IRL problem. But without the extra RL step using the learned reward, is an incomplete story. The paper talks about similarities with RLHF; yes, there is a connection but one needs to eventually learn the optimal policy. Yes, one can just do planning with the learned reward and learned transition model, but equipping it with traditional model-based guarantees is important for making connections with other relevant works. _Model based guarantee_ will be unsatisfactory since [1] gives results for general function approximation.

Author response:
In the latest version, we have included the sample complexity for extra RL steps with learned rewards (see Corollary I.6 & I.7). We appreciate the reviewer for bringing up this question.

Reviewer comment (Theory):
Moreover, this manuscript subsumes many results from (Li et al., 2023) which is a non-peer reviewed work. This makes it hard to check soundness since one needs to evaluate both (at least the relevant parts required for this submission). I am mentioning this due to my emergency review. My score reflects the fact that generative model setting (samples from every state-action pair $\approx$ uniform concentrability) in Lindner et al. (2023), is equipped with pessimism and optimism terms to account for partial concentrability using the usual techniques in offline RL (Rashidinejad, et al. 2021) and thereafter.

Author response:
While the algorithm in  Lindner et al. (2023) also incorporates pessimism, we emphasize a fundamental difference in the purpose of pessimism between  Lindner et al. (2023) and our work. In  Lindner et al. (2023), the $E^h_k(s,a)$ defined in Eq (FBI) serves as an uncertainty measure, and the algorithm minimizes this $E^h_k(s,a)$ at each step to determine the sampling strategy (see Section 6.1 in Lindner et al. (2023)). On the other hand, our introduction of pessimism aims to impose a high penalty on state-action pairs that are rarely visited. This is done to ensure that the recovered rewards satisfy the monotonicity condition: $\widehat{\mathscr{R}}_h(s,a)\leq \mathscr{R}^\star_h(s,a)$. This condition is crucial for obtaining guarantees in performing RL algorithms with learned rewards, as detailed in our latest version (see Proposition C.6 and Section E.4).

Reviewer comment (Theory):
Why do you use $d^\pi$ (Definition 3.1) as a (pre)metric between reward functions? This is not a metric, and therefore the problem that you highlight in Lemma C.4 for the Hausdorff distance between sets sussists also for your (pre)metric $D_\Theta^\pi$ (Definition 3.2). It might be that the proposed (pre)metric $D_\Theta^\pi$ is zero even if the reward sets do not coincide.

Author response:
We first clarify that Lemma C.4 (Lemma C.3  in the latest version) is intended to compare the Hausdorff metric and mapping-based metric, rather than metrics between rewards. The reason for “the proposed (pre)metric $D^\pi_{\Theta}$is zero even if the reward sets do not coincide” is due to the selection of metric between rewards: $d^{\pi}$ (it is easy to see that $d^\pi(r,r')=0$ does not imply $r=r'$). Lemma C.4 (Lemma C.3  in the latest version) aims to demonstrate that $D^{\mathsf{M}}$ is strictly stronger than $D^{\mathsf{H}}$ for some $d$, but lacks a discussion of the mapping-based metric in the first version. Therefore, we have modified Lemma C.4 (Lemma C.3  in the latest version) to include a discussion of both the Hausdorff metric and the mapping-based metric (See Lemma C.3 in in the latest version).

Reviewer comment (Theory):
What is the rationale behind choosing the (pre)metric in Definition 3.1? In the usual pipeline, the reward recovered by IRL is then used for training RL agents. It does not seem to me that the (pre)metric in Definition 3.1 guarantees anything on how close the performance of the trained RL agent with the learned reward. In "Towards Theoretical Understanding of Inverse Reinforcement Learning" the authors focus on the actual distance between reward functions, not induced value functions. Can the author elaborate?

Author response:
Regarding the rationale behind choosing the (pre)metric in Definition 3.1 and why we do not use the actual distance between reward functions, please refer to the answer to “ Why do you use $d^\pi$ (Definition 3.1) as a (pre)metric between reward functions?”.

Regarding the lack of guarantees for the performance of the trained RL agent with the learned reward, in fact, we do have guarantees for the performance of the trained RL agent with the learned rewards. We apologize for any confusion; our previous versions did not emphasize this part. In the latest version, we specifically discuss guarantees for performing RL algorithms with learned rewards (see Section C.4 and Corollary I.6, I.7). We appreciate the reviewer for bringing up this question.

Reviewer comment (Theory):
Why do your lower bounds not depend on $\delta$? This is quite significant especially when $\delta$ is small. Inspecting the proofs in comparison with "Towards Theoretical Understanding of Inverse Reinforcement Learning", it seems that the authors have adapted one construction only (the one that provides the part of the lower bound that does not depend on $\delta$). Why this choice?

Author response:
We apologize for any confusion caused. We failed to clarify that our lower bounds also hold for small $\delta$, such as $\delta \leq 1/3$ in the first version.  This error has been corrected in our latest version. We appreciate the reviewer for bringing up this question. We believe that our results hold true when $\delta$ is very small, and thus, they are highly meaningful.

Reviewer comment (Theory):
How did you manage in the proof of the upper bound the fact that rewards defined as in the ground truth reward mapping are not bounded in $[-1,+1]$ even though the parameters $(V,A)\in\mathcal{V}\times\mathcal{A}$? In "Towards Theoretical Understanding of Inverse Reinforcement Learning", Lemma B.1 (appendix), they normalize the reward functions, but you don't. What allows you to avoid this step?

Author response:
Although the reward is not bounded within $[-1, 1]$, we emphasize that $\mathcal{V}\times \mathcal{A}$ is bounded (see the Definition of $\mathcal{V}\times \mathcal{A}$ on page 4). Specifically when $(V,A)\in \mathcal{V}\times \mathcal{A})$, we have $|V_h|_{\infty},|A_h|_{\infty}\leq H-h+1$, which is sufficient for our proof (see the proof of Theorem 4.2).

Reviewer comment (Theory):
The lower bounds do not match the upper bounds, especially for what concerns the dependence on the horizon H. What is the reason for this gap?

Author response:
Regarding the lower bound in the online setting, we have made improvements (Theorem G.2), improving the original $\mathcal{O}(H^2SA\textrm{min}\\{S,A\\})$ to $\mathcal{O}(H^3SA\textrm{min}\\{S,A\\})$. Although it does not exactly match the upper bound, it has been significantly tightened. We appreciate the reviewer for bringing up this question.

As for the lower bound in the offline setting, even though we employed the same MDP construction as in proving the lower bound in the online setting, we did not fully exploit the property of our MDP construction, where transitions are different for each h, because of the construction of the behavior policy. This results in our lower bound having a dependence on $H$ of $\mathcal{O}(H^2)$. How to prove a tighter lower bound for the $H$-factor would be a highly intriguing avenue for future research.

Regarding the $\mathrm{min}\\{S,A\\}$ term, the $\textrm{min}\\{S,A\\}$ factor is introduced as follows. In constructing hard instances, there is a step where we need to ensure $\mathbb{P}_h ( s_i| s_i,a_i)=1$ for $i\leq \textrm{min}\\{S,A\\}$. This particular step is crucial for our proof. Exploring ways to improve the lower bound on $\textrm{min}\\{S, A\\}$ could be a promising avenue for future research. We believe our results are still meaningful, and the condition $S < O(A)$ also holds significance.

Reviewer comment (Novelty):
the title has nothing to do with the paper; the paper concerns a sample complexity analysis in the IRL setting, stop. Nothing in the paper gives novel results on whether IRL is harder than RL, so the title must be changed

Author response:
Our paper's title “Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning?” aims to convey that we can apply standard RL techniques to address IRL problems, including techniques such as pessimism and some exploring strategies. I apologize for any confusion. In the latest version, we provide the framework for our IRL algorithms in Section E.4. Here, we give a condition (Condition E.4) that is sufficient to do IRL. Then, we illustrate that pessimism can fulfill Condition E.4. We appreciate the reviewer for bringing up this question.

Reviewer comment (Theory):
the proof of the second part of Lemma C.1, although easy, is missing

Author response:
Thank you for pointing these out. We have added the proofs in Section C.5.

Reviewer comment (Theory):
the proof of Proposition C.2 is missing

Author response:
Thank you for pointing these out. We have added the proofs in Section C.5.

Reviewer comment (Novelty):
in Section 1, Introduction, when listing the contributions, authors state that this work contributes at "providing an answer to the longstanding non-uniqueness issue in IRL". This statement is factually false. Indeed, the authors investigate IRL as the reconstruction of the feasible reward set (and this formulation is not introduced in the paper), providing novel analysis for the offline and online (with forward model)

Author response:
We appreciate the reviewer's feedback. We want to emphasize that, to the best of our knowledge, we are the first to consider IRL using reward mapping instead of just feasible sets. Additionally, we introduce a mapping-based performance metric to address the no-uniqueness problem. We believe our approach differs from previous work (see Section C).

Reviewer comment (Theory):
the use of O-tilde is incorrect for what concerns the dependence on $\delta$. Conventionally, O-tilde does not hide dependences on $\log(1/\delta)$. This is not just a cosmetic comment, but seems to hide an additional term present in the upper bound and not present in the lower bound, spotting an additional term that is not matched

Author response:
For our upper bounds, we were indeed using $\tilde{O}(\cdot)$ to suppress $\log(1/\delta)$ and the other polylog factors, as we explained in each occurrence of it in the upper bounds.

For our lower bounds, indeed, the statements are only for $\delta=1/3$ (which then implies the same statement for all $\delta\le 1/3$) for which $\log(1/\delta)$ is a constant. We have slightly tweaked the statements to clarify this, but we agree with the reviewer that this is looser than what we want.

Upon initial inspection, we believe it is **possible to improve our lower bounds to have an additional $\log(1/\delta)$ factor**, by using the inequality $1-{\rm TV}\ge \frac{1}{2}e^{-\rm KL}\ge \delta$ whenever ${\rm KL}\le \log(1/(2\delta))$ instead of Pinsker’s inequality, and arguing that $n\le O(\log(1/\delta))$ (times the other factors) will result in ${\rm KL}\le \log(1/(2\delta))$ (as KL scales linearly in the sample size $n$ due to its additivity). However, we need to go through the full argument more carefully. We will include this in our final version if it works out.