Paper: Characterising Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting (ICLR-2024)

Reviewer comment (Theory):
It's challenging to comprehend the concept of sophisticated policy as delineated in Definition 7. For instance, it's unclear why the policy, $\pi(\xi)$, is not dependent on the time step and how it correlates with step-wise policies. Similarly, it's puzzling why the Q function $Q^\pi(\xi,a)$ is also independent of the time step. Given that the optimal policy can vary at each time step, it becomes complex to determine which strategy exhibits more "sophistication". In many Markov Decision Processes (MDPs), the so-called sophisticated policy is not singular. The paper states that "$\pi$ is sophisticated if it only takes actions that are optimal given that all subsequent actions are sampled from $\pi$." Could you clarify this definition? Specifically, I'm interested in understanding how one would define optimality in a non-stationary MDP that spans across different (or all) time steps.

Author response:
To say that a policy is sophisticated is essentially to say that it satisfies a certain stability criterion --- namely, if $\pi$ is sophisticated, then the agent never has any (local) incentive to deviate from $\pi$. One way to understand this is that a sophisticated policy is somewhat analogous to a Nash equilibrium, if the agent at each time step is thought of as a separate decision maker (which is justified by the fact that its preferences are not temporally consistent). Another way to understand it is that if a policy $\pi$ is updated using policy gradients, then a sophisticated policy will be a sort of local optimum (though it may not necessarily be an attractor point, in the same way as a Nash equilibrium may not be an attractor point). This also means that it is not always clear how to compare sophisticated policies (again, in the same way as how it is not always clear to select between Nash equilibria); a policy is simply either sophisticated or not. Thus, sophisticated policies do not give us an unambiguous notion of "optimality" (though they are nonetheless a solution concept).

Reviewer comment (Theory):
The definition of identifiability appears to be founded on an exponentially discounted MDP, even though the paper focuses on a non-exponentially discounted setting. The paper attempts to provide some intuitive explanations, but they fall short in terms of persuasiveness. If the term 'optimality' has a clear definition under different behavior models, then the term 'identifiability' should also exhibit the capacity to characterize these models.

Author response:
We have choosen to quantify identifiability in terms of exponentially discounted optimal policies, because even if humans discount hyperbolically, we typically want to create systems that discount exponentially --- this is why exponential discounting is vastly more common in the RL literature. Therefore, we think the most relevant formalisation is to assume that the observed demonstrator discounts hyperbolically, but that the policy which will be computed with the learnt reward function discounts exponentially. Other choices could be made instead, which might lead to different results. Of course, some alternative formalisations would end up being quite trivial. For example, if $P$ is the equivalence relation under which $R_1 \equiv_P R_2$ iff $R_1$ and $R_2$ have the same sophisticated policies, and $f : \mathcal{R} \to \Pi$ returns a maximally supportive sophisticated policy, then it is immediate from the definition that $f$ is $P$-identifiable, etc. This setup would thus not allow for the derivation of any "deep" results.

Reviewer comment (Theory):
Why the Q function $Q^\pi(\xi,a)$ is also independent of the time step?

Author response:
Again, note that $\xi$ is a trajectory, rather than a state.

Reviewer comment (Evaluation):
How the existence of non-identifiability impacts empirical performance?

Author response:
Note that the empirical performance of an IRL algorithm for hyperbolic discounting will be heavily dependent on the prior distribution from which the ground truth reward function is sampled, which limits the generalisability of such results. In particular, for any inductive bias of the learning algorithm there exists a prior distribution for the ground truth reward such that the counterexamples identified by our theorems will occur with high probability.

Reviewer comment (Theory):
What potential solutions could address the issue of non-identifiability in IRL?

Author response:
We think that the most promising solutions will rely on incorporating prior information, or data from other data sources. However, exploring these options is beyond the scope of this paper.

Reviewer comment (Novelty):
The main weakness of the work is the motivation of it. The authors do not provide enough reasons why we need to consider a different discounted setting with respect to the exponential discounted one. In literature, when is it used the hyperbolic setting? Why is it relevant in practice? Moreover, if the setting is more general and reasonable, I think it would be better to present it directly in the background section rather than presenting the standard exponential discounted ones and then the new setting.

Author response:
First of all, we would like to note that the main motivation behind our work is the usage of IRL in the context of *preference elicitation*. That is, the setting where we wish to use IRL to learn a representation of the actual preferences of a human subject. In this setting, it is important that we use a behavioural model that actually represents the decision making process of a human as closely as possible. Now, a vast amount of research in behavioural psychology, and related fields, shows that humans are better modelled as using hyperbolic discounting rather than exponential discounting --- we provide references to some of this research in the introduction to the paper. This means that a behavioural model with hyperbolic discounting will be more accurate than (and hence preferable to) a behavioural model with exponential discounting, all other things being equal. Note also that this is not necessarily the case when IRL is used in the context of *imitation learning*, since it in this context is not fundamentally important that the learnt reward function represents the actual preferences of the demonstrator, as long as it helps the imitation learning process.

Reviewer comment (Presentation):
The main focus of the paper is (reading the abstract) on Inverse Reinforcement Learning, but, in the end, the IRL contribution of the paper is condensed into only one page and a half.

Author response:
Yes, the main results on IRL are given in Section 5, but all results in Section 3 and 4 build towards these results. It would not be possible to derive (or even state) the results in Section 5 without first presenting the results given in Section 3 and 4.

Reviewer comment (Experiments):
There are no experimental or numerical evaluations of the proposed approach at least to show why the proposed setting is relevant.

Author response:
We agree that experimental evaluations could be interesting, but we consider this to be out of scope for this paper. Similar recent works on identifiability in IRL are also typically theoretical, rather than experimental (see Dvijotham & Todorov, 2010; Cao et al., 2021; Kim et al., 2021; Skalse et al., 2022; Schlaginhaufen & Kamgarpour, 2023; Metelli et al., 2023).

Reviewer comment (Theory):
A reward function is optimal under more than one policy. Then, why is the behavioral model defined as a mapping between $\mathcal{R} \rightarrow \Pi$ and not $\mathcal{R} \rightarrow P^\Pi$?

Author response:
This choice was made to make our analysis more tractable. For example, if an optimality criterion corresponds to more than one policy, then we may assume that the demonstrator has some fixed criterion for breaking ties between them. This assumption is not fundamentally important, and can be lifted, at the cost of making some of the theorem statements and proofs more messy. For example, both Theorem 6 and 7 would go through without much modification.

Reviewer comment (Theory):
Proposition 1 seems to be not easy to verify. How can we understand if an MDP satisfies it?

Author response:
To say that an MDP is episodic is simply to say that it cannot run forever; most MDPs used in practice satisfy this criterion. For example, note that any MDP with a bounded time horizon is episodic. The definition used in Proposition 1 is simply somewhat more general.

Reviewer comment (Theory):
Why is it relevant to choose discount factors that are not temporally consistent? Can the change in preference of an agent be described with a change in the reward function?

Author response:
As noted before, the current literature in the behavioural sciences suggests that humans are better modelled as using hyperbolic discounting, rather than exponential discounting. For this reason, we consider the setting with non-exponential discounting in IRL to be both very relevant and under-explored.

Reviewer comment (Theory):
If in the end, we are using exponential discounting to find our optimal policy why do we need to study a different setting before?

Author response:
To quantify the "size" of $\mathrm{Am}(f)$, we need to determine if all reward functions in $\mathrm{Am}(f)$ share some relevant property. There are multiple choices that could be made here, but we have choosen to use (exponentially discounted) optimal policies, for the reasons outlined in Section 5. In short, even if humans discount hyperbolically, we typically want to create systems that discount exponentially --- this is why exponential discounting is vastly more common in the RL literature. Therefore, we should assume that the observed demonstrator discounts hyperbolically, but that the policy which will be computed with the learnt reward function discounts exponentially.

Reviewer comment (Theory):
From R we could get to different f(R), which is denoted Am(f), a set of rational models follows R. Rather than knowing this set is singleton, I think a more important question maybe how small the set is, and whether it is contiguous.

Author response:
As for the last point, we would expect $\mathrm{Am}(f)$ to be contiguous in the non-exponential setting, but we do not have a proof of this fact. We should also note that we agree that it is not very important whether or not $\mathrm{Am}(f)$ is a singleton. Indeed, in both the exponential and the non-exponential setting, $\mathrm{Am}(f)$ is infinitely large (both in the sense that it has an infinite cardinality, and in the sense that it has an infinite Lebesgue measure). Therefore, we think it is more important to know whether or not all reward functions in $\mathrm{Am}(f)$ are "equivalent" in some relevant sense. This is why we use the notion of $P$-identifiability to express our results in Section 5.

Reviewer comment (Theory):
Do you think non-exponential discounting effects contiguity?

Author response:
As for the last point, we would expect $\mathrm{Am}(f)$ to be contiguous in the non-exponential setting, but we do not have a proof of this fact. We should also note that we agree that it is not very important whether or not $\mathrm{Am}(f)$ is a singleton. Indeed, in both the exponential and the non-exponential setting, $\mathrm{Am}(f)$ is infinitely large (both in the sense that it has an infinite cardinality, and in the sense that it has an infinite Lebesgue measure). Therefore, we think it is more important to know whether or not all reward functions in $\mathrm{Am}(f)$ are "equivalent" in some relevant sense. This is why we use the notion of $P$-identifiability to express our results in Section 5.