Paper: Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification (ICLR-2024)

Reviewer comment (Theory):
The inclusion of concrete examples of the transformation would be beneficial. Currently, it is challenging to image the types of transformations that would satisfy Proposition 3, and more importantly, how to interpret these transformations requires clarification.

Author response:
We appreciate that the transformations described in Proposition 3 are quite difficult to visualise. The proof of Proposition 3 includes a geometric description of these transformations, but they are still not easy to understand intuitively. To some extent, this is just a consequece of the mathematics --- we wanted to derive necessary and sufficient conditions, and in this case, those conditions turned out to be somewhat messy. However, we can certainly include a few examples to make Proposition 3 more intuitive. First of all, Proposition 3 is satisfied by any transformation that can be expressed as a combination of potential shaping, positive linear scaling, and S'-redistribution, and these transformations are reasonably easy to understand (see Section 1.2). More generally, a transformation satisfies Proposition 3 if it never changes the policy order of the reward function by a large amount. For example, suppose we have a Bandit MDP with three actions, and that the reward obtained from these actions is $\langle 1, 1-\epsilon, 0\rangle$. A reward transformation that satisfies Proposition 3 might change this reward to $\langle 1-\epsilon, 1, 0\rangle$, for example. We can include a few examples like this, to help make Proposition 3 more intelligible.

Reviewer comment (Theory):
As per the reviewer's understanding, an MCE (Maximum Causal Entropy) policy corresponds to a Boltzmann-rational policy. Specifically, the optimal policy representation in soft-Q learning complies with Boltzmann-rationality. This point may warrant further verification.

Author response:
The MCE policy and the Boltzmann-rational policy are not identical, though they are similar in some ways. The Boltzmann-rational policy is given by softmaxing the optimal Q-function, whereas the MCE policy is given by softmaxing the soft Q-function. The optimal Q-function and the soft Q-function are in turn not identical, which means that these two behavioural models are slightly different.

Reviewer comment (Experiments):
It would be beneficial to present some empirical results, which could illuminate the consequences of failing to achieve robustness under the misspecification of behavioural models.

Author response:
We agree that more extensive empirical investigation would be interesting, but we consider this to be out of scope for this paper. It would be redundant to verify our positive results (i.e. Theorem 1 and 2) experimentally, as our proofs already tell us what the outcome of such experiments would be. It would be more interesting to verify our negative results (i.e. Theorem 3, 4, and 5). However, the outcome of these experiments would be heavily dependent on the choice of prior probability distribution over the true reward function $R^\star$, and on the choice of inductive bias of the learning algorithm --- our proofs tell us that there for any inductive bias is a prior probability distribution over $R^\star$ under which our bounds are attained with high probability. For Theorem 4 and 5, the converse also holds. Nonetheless, more extensive empirical investigation would certainly be interesting, but we consider this to be out of scope for this paper.

Reviewer comment (Writing):
In proposition 3, what's the meaning of $\|R,t_n(R)\|_2$? Should it be |R-t_n(R)\|_2?

Author response:
Yes, it should be $||R - t_n(R)||_2$. In this paper, we have used the convention where $||x, y||_2$ denotes the metric induced by the $\ell_2$-norm, ie $||x - y||_2$. Since several reviewers found this confusing, we will replace it with the more typical notation.

Reviewer comment (Theory):
According to the results of this paper, all the commonly applied behavior models are not $\epsilon/\delta$-separating, are there any behavior model is $\epsilon/\delta$, maybe under the mild relaxation on the distance under the reward and policy space?

Author response:
The result that all the common behavioural models fail to be $\epsilon/\delta$-separating does depend on the choice of metric(s). Specifically, it is a consequence of the fact that two reward functions can have a small $\ell_2$-norm, even though they have a large STARC-distance (see Appendix D). As such, the proof would not go through if we measure the distance between rewards using the $\ell_2$-norm. However, if we do this, then we would not obtain informative regret bounds (as discussed in Section 2.2). There may be other metrics under which the standard behavioural models are $\epsilon/\delta$-separating, and under which some form of regret bounds or other theoretical guarantees can be obtained. However, no such metric has been proposed in the existing literature. We believe that further exploration of this question is an interesting topic for further work.

Reviewer comment (Theory):
In this paper, the authors consider a general IRL procedure that outputs an arbitrary reward function $R$ from the class $\tilde{\mathcal{R}} = \\{R_1: f (R_1) = \pi = g(R_2) \\}$. However, in practice, we often run a specific IRL algorithm such as MaxEnt IRL [Ziebart, 2010], which performs reward selection and outputs the chosen reward. Therefore, it could be more meaningful to analyze the robustness of a specific IRL algorithm to misspecification.

Author response:
We would like to thank reviewer jn4g for their review and feedback!

We agree that it could be interesting to provide a more detailed analysis of specific IRL algorithms, such as MaxEnt IRL. However, note that all of our results apply to any algorithm that is based on the MCE model, including MaxEnt IRL. In this sense, our analysis is simply more general than an analysis which is restricted to a particular algorithm. 

Of course, in general, the inductive bias of an algorithm can be important for understanding its behaviour --- this is discussed in some more detail in Appendix A.4. However, note that Proposition 1 and 2 together imply that if two reward functions have the same MCE policy, then their STARC distance is 0. This means that if an algorithm is based on the MCE model, then its inductive bias will not affect the quality of the inferred reward function in the limit of infinite data. As such, we would not be able to draw stronger conclusions by incorporating more details about the MaxEnt IRL algorithm.

Reviewer comment (Theory):
In Proposition 3, $\\| R, t_n (R) \\|_2$ should be $\\| R - t_n (R) \\|_2$?

Author response:
Yes, in Proposition 3, it should be $||R - t_n(R)||_2$. In this paper, we have used the convention where $||x, y||_2$ denotes the metric induced by the $\ell_2$-norm, ie $||x - y||_2$. Since several reviewers found this confusing, we will replace it with the more typical notation.

Reviewer comment (Novelty):
I am not sure about the extent of novelty in this contribution. The problem of robustness of IRL to behavior model misspecification has been studied in the past as mentioned in the literature survey in the paper. The new pseudometric used by the authors has also been introduced in an earlier paper cited by the authors. If this is correct, and the analysis is the main novel contribution, then I would expect a more detailed comparison of the analysis with the other works in literature.

Author response:
The novelty of our contribution does indeed lie in our analysis (i.e., the results we present in Section 3). While misspecification in IRL has been studied by a handful of recent papers, our analysis is more comprehensive, and is done in a richer formal framework, which lets us infer many novel insights. We have attempted to explain precisely how our work extends earlier work in Section 1.1 and Appendix G, but we can certainly make these sections more detailed. To make this more clear, we have included a more detailed comparison between our paper and earlier work in a separate comment.

The relationship between our work and previous work in the same area can be summarised as follows:

Freedman et al, 2020 and Viano et al, 2021 study two specific forms of misspecification in IRL, namely choice set misspecification and environment dynamics misspecification. By contrast, our analysis covers many different forms of misspecification --- our Theorem 1 and Corollary 1 provides necessary and sufficient conditions that cover all types of misspecification, and we also investigate perturbation robustness and misspecified discount functions in more depth. Neither Freedman et al, 2020 nor Viano et al, 2021 cover these forms of misspecification.

Skalse & Abate, 2023 study many different kinds of misspecification, and like us, they also aim to derive necessary and sufficient conditions that describe all forms of permissible misspecification. However, they measure the distance between reward functions using equivalence classes. This is a serious limitation, because it means that their their analysis is unable to distinguish between large and small errors in the inferred reward. By contrast, we use metrics, which allows for a more fine-grained analysis. 

Hong et al, 2022 measure the distance between policies using the $\ell_2$ norm, assume that the behavioural model satisfies strong log-concavity, and assume that there is a unique reward function that maximises fit to the training data. Using the $\ell_2$-norm is problematic, as we briefly discuss in Secton 2.2, and the latter two conditions are often not satisfied in practice. These factors limit the extent to which their conclusions are applicable to real-world cases. By contrast, we use a metric with stronger theoretical guarantees, and we carry out our analysis in terms of the behavioural models that actually are common in contemporary IRL. Moreover, we provide necessary and sufficient conditions for misspecification robustness (in Section 3.1), which is not done by Hong et al, 2022.

Reviewer comment (Theory):
The main contribution of this paper is its theoretical results. The authors hhave stated the theorems, propositions and corollaries in the main paper without any proofs or even proof sketches. Furthermore, there are no numerical experiments conducted to demonstrate the impact of the theoretical findings.

Author response:
All proofs are provided in Section F of the appendix, on pages 20-26. We also provide a high-level intuitive explanation of some of these proofs in Section D and E of the appendix, on pages 17-19. There would not be enough space to fit this in the main part of the text. However, note that it is very common to put long proofs in the appendix. As for experiments, we note that this is a theoretical paper, rather than an empirical paper. It would be redundant to verify our positive results (i.e. Theorem 1 and 2) experimentally, as our proofs already tell us what the outcome of such experiments would be. It would be more interesting to verify our negative results (i.e. Theorem 3, 4, and 5). However, the outcome of these experiments would be heavily dependent on the choice of prior probability distribution over the true reward function $R^\star$, and on the choice of inductive bias of the learning algorithm --- our proofs tell us that there for any inductive bias is a prior probability distribution over $R^\star$ under which our bounds are attained with high probability. For Theorem 4 and 5, the converse also holds. Nonetheless, more extensive empirical investigation would certainly be interesting, but we consider this to be out of scope for this paper.

Reviewer comment (Theory):
Can't the difference in observed versus predicted behavior of a human assuming a particular behavior model be explained by incorrect or incomplete optimization rather than incorrectness of the behavior model?

Author response:
Yes, there are many potential sources of error in the IRL problem. Our work focuses on misspecification of the behavioural model, as we believe this issue to be particularly important and difficult to overcome. However, IRL certainly has many other difficulties as well.

Reviewer comment (Theory):
For a given dataset, can't two different behavioral models yield the same reward function? If so, isn't accuracy of reward prediction an insufficient criterion for estimating the validity of the behavioral model? In this paper, do the authors wish to learn a true reward function, albeit woith an incorrect behavior model, or do they wish to establish correctness of the behavior model as well?

Author response:
Yes, two behavioural models can produce the same reward function(s) for a given data distribution. However, the aim of our work is not to determine the vailidity of the behavioural model, but rather, to quantify how sensitive IRL is to misspecification of the behavioural model. In other words, we want to know when it is possible to infer a reward function that is "close enough" to the true reward function using a behavioural model that is misspecified.

Reviewer comment (Writing):
Are the brackets correctly places in the definition of S' redistribution in the third paragraph of Sec 1.2? If so, I do not understand this definition. Can the authors please clarify?

Author response:
The brackets are indeed incorrect, thank you for spotting this. The correct statement should read $E[R_1(s,a,S')] = E[R_2(s,a,S')]$.

Reviewer comment (Theory):
In Point 3 of Defintion 1, should $R_1$ and $R_2$ be necessarily different? If so, why? What happens in the case when $f = g$?

Author response:
$R_1$ and $R_2$ do not have to be different; the purpose of Point 3 is to ensure that $\mathrm{Im}(g) \subseteq \mathrm{Im}(f)$.

Reviewer comment (Theory):
Is the orderting of policies mentioned in Proposition 1 a partial ordering?

Author response:
No, it is the linear ordering induced by the policy evaluation function $J$. That is, $R_1$ and $R_2$ are considered to have the same ordering of policies if $J_1(\pi_1) \leq J_1(\pi_2)$ if and only if $J_2(\pi_1) \leq J_2(\pi_2)$.

Reviewer comment (Theory):
I did not understand the 2-norm of the first term in the equation written in Proposition 3. Can the authors please explain this?

Author response:
Any reward function can be viewed as an $|S|^2|A|$-dimensional vector. We use $||x,y||_2$ to denote the metric induced by the $\ell_2$-norm. This means that $||R, t_n(R)||_2$ is the $\ell_2$-distance between the two reward functions $R$ and $t_n(R)$. Since several reviewers found this notation confusing, we will replace it with more typical notation.

Reviewer comment (Experiments):
What happens to IRL robustness when the true behavior model is Cumulative Prospect Theory and the assumed one is Boltzmann-rationality?

Author response:
Cumulative Prospect Theory is not covered by our analysis in this paper, but we consider it (along with other models from the behavioural sciences, such as hyperbolic discounting) to be an important topic for further work. We would however expect the worst-case error to be very large in this case, and proof strategies similar to those used in Section 3.3 could probably be used. Note that Cumulative Prospect Theory can lead to non-stationary policies.

Reviewer comment (Experiments):
What if we have two IRL algorithms that differ in implementation but assume the same underlying behavioural model? Will this affect the results?

Author response:
Our results apply to any algorithm which uses a particular behavioural model, so our results are directly applicable in this case. However, note that if we make additional assumptions about both the true reward function, *and* about the inductive bias of the learning algorithm, then we may be able to derive stronger results. This is discussed in Appendix A.4.

Reviewer comment (Experiments):
Does the IRL reward parameterization affect the results?

Author response:
This might affect the results if the reward is parameterised in a way that cannot express all possible reward functions -- this is discussed in the third paragraph of Section 2.1, and in Appendix A.3. If the parameterisation can express all possible reward functions, then it will not affect the results.

Reviewer comment (Theory):
Are there more ways in which reward functions can be different but yield the same optimal policy, except for potential shaping, S' redistribution, and positive linear scaling?

Author response:
Yes. Potential shaping, S'-redistribution, and positive linear scaling together completely characterise all ways in which two reward functions can have the same *policy order*. However, two reward functions can have different policy orderings, and still have the same optimal policy. To see this, consider a Bandit MDP with three actions, and two reward functions $\langle 2, 1, 0\rangle$, $\langle 2, 0, 1\rangle$. These reward functions have the same optimal policy, but different policy orderings (and so they do not differ by potential shaping, S'-redistribution, and positive linear scaling).

Reviewer comment (Theory):
Shouldn't a reward change of the type \( R' = R + b \) where \( b \) is a constant also lead to the same optimal policy? Is this discussed somewhere (may have missed it)?

Author response:
These types of transformations can be expressed as potential shaping, just let $\Phi(s) = -b/(1-\gamma)$ for all states $s$.

Reviewer comment (Writing):
Typo on Page 2, Paragraph 2, Line 6: "misspecifiaction" should be "misspecification".

Author response:
We also thank the reviewer for notifying us of the typos!

Reviewer comment (Writing):
Typo in Section 3 title: "rubustness" should be "robustness".

Author response:
We also thank the reviewer for notifying us of the typos!