Identiﬁability and Generalizability from Multiple Experts in Inverse Reinforcement Learning
Abstract
While Reinforcement Learning (RL) aims to train an agent from a reward function in a given environment, Inverse Reinforcement Learning (IRL) seeks to recover the reward function from observing an expert’s behavior.
It is well known that, in general, various reward functions can lead to the same optimal policy, and hence,
IRL is ill-deﬁned.
However, [1] showed that, if we observe two or more experts with different discount factors or acting in different environments, the reward function can under certain conditions be identiﬁed up to a constant.
This work starts by showing an equivalent identiﬁability statement from multiple experts in tabular MDPs based on a rank condition, which is easily veriﬁable and is shown to be also necessary.
We then extend our result to various different scenarios, i.e., we characterize reward identiﬁability in the case where the reward function can be represented as a linear combination of given features, making it more interpretable, or when we have access to approximate transition matrices.
Even when the reward is not identiﬁable, we provide conditions characterizing when data on multiple experts in a given environment allows to generalize and train an optimal agent in a new environment.
Our theoretical results on reward identiﬁability and generalizability are validated in various numerical experiments.
Introduction
Engineering a reward function in Reinforcement Learning can be troublesome in certain scenarios like driving [2], robotics [3] and economics/ﬁnance [4].
In economics and ﬁnance, the reward or objective/utility function of the agent are of fundamental importance but are not known a priori [5–8].
In such cases, it may be easier to get demonstrations from an expert policy.
Therefore, multiple algorithms have been developed to learn from demonstrations, e.g., in inverse reinforcement learning (IRL) and imitation learning (IL).
In IRL, the goal is to recover the reward function maximized by the agent, while in IL the expert demonstrations are used solely to learn a nearly optimal policy.
In economics/ﬁnance, inference on the reward function is the focus of a large literature on estimation, testing, and policy analysis of structural models [9–11].
However, the reward function is often highly parameterized and represented by a low-dimensional set of parameters, or the literature focuses on estimating reduced-form causal relationships but not the true reward function [12, 13].
The attractiveness of IRL relies on the fact that the reward function is the most “succinct” representation of a task [14].
Indeed, identifying the 36th Conference on Neural Information Processing Systems ( reward function for each state-action pair allows generalizing the task to different transition dynamics and environments, which is not possible when using IL or highly parameterized structural models.
However, the IRL problem is unfortunately ill-posed since there always exist inﬁnitely many reward functions for which the observed expert policy is optimal [15, 16].
The problem is known as reward shaping, and it is intuitively explained with the fact that, in the long term, the optimal policy is not affected by inﬂating the reward in the current period and decreasing the one in the next.
This difﬁculty originated a long debate on advantages and disadvantages of IL and IRL [17–20].
When multiple experts are available, differing in the transition matrices of the environments they each act in, and/or their discount factors, IRL can in certain cases infer the true reward function, up to a constant [21–23, 1].
Inspired by [1], we derive an equivalent necessary and sufﬁcient condition on the expert environments, which is easily veriﬁable, ensuring that the true reward can be identiﬁed up to a constant shift.
When this identiﬁability condition holds, the state-action dependent rewards can be recovered from expert demonstrations.
We then derive identiﬁability results in various alternative scenarios, e.g., when we only have access to approximate transition matrices and, alternatively, when the reward function is known to be a linear combination of given features [24, 25].
However, full reward identiﬁability remains a strong requirement, and we provide a negative result of non-identiﬁability from any number of experts, in the presence of exogenous variables in the MDP.
Nonetheless, even when the identiﬁability condition does not hold, the recovered reward function could still be used to train an optimal expert for a different environment.
To this end, we characterize situations where observing multiple experts in given environments allows to train an optimal agent in a new environment.
Related work
Since its introduction in [15, 16], the IRL problem has been known to be ill-posed, since the observed expert policy can be optimal with respect to various reward functions.
The set of reward transformations that preserve policy optimality are studied in [26, 16, 1, 27, 28].
[29] studied the unidentiﬁability related to suboptimal experts.
In this paper, we assume access to the optimal entropy regularized policies of multiple experts.
Signiﬁcant progress has been made to construct heuristics that select a single reward function from the set of IRL solutions (often called the feasible set), such as feature-based matching [30], maximum margin IRL [31], maximum causal entropy IRL [32, 33], maximum relative entropy IRL [34], Bayesian IRL [35–37], ﬁrst-order optimality conditions [38, 39] or second-order optimality conditions [40, 41].
Popular IL algorithms implicitly select a feasible reward function via a convex reward regularizer [19, 42, 43] or using preference/ranking based algorithms [44, 45].
However, none of these approaches guarantee the identiﬁcation of the true reward function.
The problem of identiﬁability in IRL has been investigated ﬁrst in [21, 22] that study a setting where the learner can actively select optimal experts in multiple environments.
The main result in [21, 22] is that interactively querying environments outputs a reward within a constant shift from the true one.
The multiple experts setting has also been studied in [46] but in the context of value alignment veriﬁcation where the aim is not to recover the reward function but rather verify that the value function of the agent is close to a target value.
IRL from multiple MDPs also appears in [23] where the authors consider the problem of learning a reward function compatible with a dataset of demonstrations collected by multiple experts.
In addition, [47] study structural conditions on the MDP for reward identiﬁcation in the ﬁnite horizon setting and [48] study identiﬁability in linearly solvable MDPs.
Our work is inspired by [1].
Our ﬁrst identiﬁability result provides an equivalent statement as their value distinguishability condition, but can be easily checked in practice, and allows to derive other identiﬁability results in alternative scenarios.
Finally, the motivation for IRL is often predicting the expert behavior under new transitions dynamics [49, 50, 20].
We show that for this goal, it is not necessary to identify the exact reward, hence we give a condition on the observed experts’ environments and the test environment under which an optimal expert can be trained in the test environment.
This perspective has also been taken in [51].
However, this work requires stronger assumptions on the transfer environment that we avoid in this paper, only requiring access to multiple experts.
Moreover, our work contributes to AI safety [52–54] alleviating the reward hacking and side effects problems [53].
Indeed, by restricting the reward to linear combinations of a set of chosen features, we can provably recover an interpretable reward function inducing the optimal behavior, which is particularly desirable in medical applications [55, 56].
An important consideration for IRL comes from [57] that formalizes the fact that there exist tasks that can not be induced by optimizing a reward function.
In this work and in IRL in general, we bypass this difﬁculty assuming that the expert is optimizing a reward function.
2.1 Related works in the economics literature
The economics/ﬁnance literature differentiates between axiomatic and revealed preference theory.
In axiomatic preference theory, the reward function is posited or derived from basic axioms.
In empirical and experimental work, however, simple reward function speciﬁcations are often rejected and agents have been shown to exhibit behavioral biases and/or non-standard preferences.
Differently, our work relates to the literature on revealed preference.
Revealed preference theory, initiated by [58, 59], provides an approach to analyze actions (e.g., consumer’s demand or investors’ trading) by assuming that observed choices provide information about the underlying preferences, or reward function.
Revealed preference theory is, hence, similar in spirit to IRL.
But IRL has not widely been used in revealed preference analysis.
We refer to [60, 61] for excellent reviews of recent advances in revealed preference theory.
The goal of revealed preference theory is to recover the agents’ preferences.
This task is important because knowledge of the reward function is required to conduct counterfactual policy analysis.
Notice that for this task, knowing only the policy function is insufﬁcient.
In ﬁnancial applications, for instance, the impact of a Tobin tax can be assessed only knowing investors’ preferences for trading (see, e.g. [62]).
Preliminaries
A typical RL environment is characterised by a Markov Decision Process M = {S , A , T, γ, r, P 0 } , where S , A are the sets of states and actions respectively, T : S ×
A × S → [0 , 1] is the state transition probability, i.e., T ( s (cid:48) | s, a ) denotes the probability of arriving in state s (cid:48) when taking action a in state s .
R : S ×
A → R denotes the reward function, γ the discount factor and P 0 is the initial state distribution.
At each time step t , an agent observes the current state s t ∈ S and takes an action a t ∼ π ( ·| s t ) where π is the agent’s policy which determines a distribution over all actions in A at every state.
The agent gets a reward r t = r ( s t , a t ) and transitions to a new state s t +1 sampled according to the transition probability T .
An agent acting optimally in M seeks to maximize its cumulative sum of rewards.
In addition, we assume that the agent seeks to diversify its possible actions, and hence that it maximizes the following entropy regularized sum of discounted rewards:
[Equation]
where E πs denotes the expectation over trajectories { ( s t , a t } t ≥ 0 starting from state s
= s and following policy π and H ( π )
= − (cid:80) a ∈A π ( a ) log π ( a ) is the entropy of π .
The function V πλ is called the (entropy regularized) value function of π .
In Inverse RL, the reward function r is unknown, but we observe an agent acting optimally with respect to some reward function, and we wish to recover the reward function that the agent optimizes.
We now recall some results from [1].
Theorem 1.
For a ﬁxed policy π ( a | s ) > 0 , discount factor γ ∈ [0 , 1) , and an arbitrary choice of function v : S → R , there is a unique corresponding reward function
[Equation]
such that the MDP with reward r yields an entropy-regularized optimal policy π ∗ λ
= π and V πλ = v .
By observing a single expert, it is hence possible to design a reward that yields any arbitrary value function, and there are hence |S| degrees of freedom remaining in the recovered reward function.
An idea explored in [1] is to assume that we observe two experts in two different MDPs with different transition dynamics and discount rates, but acting optimally with respect to the same reward function.
The authors show that the reward can be identiﬁed up to a constant from observing the expert policies provided that the MDPs of the experts satisfy the following value-distinguishing assumption.
Deﬁnition 2.
Consider a pair of Markov decision problems on the same state and action spaces, but with respective discount rates γ 1 , γ 2 and transition probabilities T 1 , T 2 .
We say that this pair is value-distinguishing if, for any function v 1 , v 2 : S → R , the statement
[Equation] [Equation]
−
[Equation]
a
[Equation]
for all a ∈ A
[Equation]
s ∈ S (2) implies at least one of v 1 and v 2 is a constant function.
The way this assumption is stated makes it difﬁcult to verify in practice, and the authors of [1] do not attempt to verify it in their experiments.
Reward identiﬁcation and generalization
In this section, we present our main theoretical results on reward identiﬁability and generalizability.
In the ﬁrst part, we show an equivalent condition to Deﬁnition 2 for reward identiﬁcation from two experts (Theorem 3).
The simplicity of our condition makes it easily veriﬁable and extendable to various scenarios, in particular to the cases where we observe more than two experts (Corollary 5), when the class of rewards is linearly parameterized with a set of given features (Theorem 7), or when we have access to approximated transition matrices (Theorem 8).
We also provide a negative result on reward non-identiﬁability in MDPs with exogenous variables, which are common in many real world scenarios.
In the second part, we analyse reward generalizability.
Here, we provide a condition guaranteeing that a reward compatible with two experts leads to an optimal policy in a third environment (Theorem 11).
The proofs of the results are all postponed to Appendix A.
4.1 Reward identiﬁability
Consider two Markov decision problems on the same set of states and actions S and A respectively, but with different transition dynamics T 1 , T 2 and discount factors γ 1 , γ 2 .
Let r ∈ R |S|×|A| be the reward function common to the two environments, and let v 1 , v 2 ∈ R |S| be the entropy regularized values functions associated expert policies π 1 and π 2 in each environment respectively.
According to
Theorem 1, we have that ∀ ( s, a ) ∈ S × A ,
[Equation]
We hence deduce that ∀ a ∈ A ,
[Equation]
where ∀ a ∈ A , T ia ∈ R S×S is the transition matrix for action a and expert i = 1 , 2 , i.e., T ia ( s, s (cid:48) ) =
T i ( s (cid:48) | s, a ) .
By including all available actions to the experts, we can write 
[Equation]
In order to identify a unique reward function, we need to identify a unique associated value function.
We hence want the linear system (4) to yield a unique solution, i.e., the |A||S|× 2 |S| matrix on the left hand side to be full rank, i.e., to have rank 2 |S| .
However, it is well known that, for any MDP, adding a constant to the reward would not change the associated optimal policy.
Hence, there is an intrinsic degree of freedom in reward identiﬁability which is impossible to get rid of from only observing expert policies.
In order to identify the reward up to a constant, we need this degree of freedom to be the only one in the linear system (4), i.e., the associated matrix to have rank 2 |S| − 1 .
This result is summarized in the following theorem, and its complete proof can be found in Appendix A.1.
Theorem 3.
Consider two Markov decision problems on the same set of states and actions, but with different transition dynamics T 1 , T 2 and discount factors γ 1 , γ 2 .
Suppose that we observe two experts acting each in one of these environments, optimally with respect to the same reward function, in the sense that their policies maximize the entropy regularized reward in their respective environments.
Then, the reward function can be recovered up to the addition of a constant if and only if
[Equation]
This condition turns out to be equivalent to Deﬁnition 2, as shown at the end of Appendix A.1, but is stated in a way that is easier to check in practice and allows us to further characterize identiﬁability in various scenarios.
First of all, this result naturally extends to the case where we observe any number of experts.
We provide hereafter the result in the case of three experts.
Corollary 4.
Consider three Markov decision problems on the same set of states and actions, but with different transition dynamics T 1 , T 2 , T 3 and discount factors γ 1 , γ 2 , γ 3 .
Suppose that we observe three experts acting each in one of these environments, optimally with respect to the same reward function.
Then, the reward function can be recovered up to the addition of a constant if and only if
[Equation]
An interesting scenario is the one where the two experts act in the same environment, and only the discount rate is varied.
Corollary 5.
Consider two Markov decision problems on the same set of states and actions, with the same transition matrix T and reward function but different discount factors γ 1 (cid:54) =
γ 2 .
Then, the reward function is identiﬁable up to a constant by observing two experts in ( T, γ 1 ) , ( T, γ 2 ) iff
[Equation]
Remark 1.
Interestingly, condition (7) is equivalent to the condition for identiﬁcation of a actionindependent reward from a single expert, assuming such a reward exists ([1], Corollary 3).
Next, we provide a negative result concerning MDPs with exogenous variables, i.e., a variable whose dynamics are independent of the agent’s action.
This MDP class is common in economics/ﬁnance and has been studied in many real world scenarios including inventory control problems [63], variable weather conditions and customer demands [64], wildﬁre management [65], and stock market ﬂuctuations [66].
We also provide examples involving such variables in the experimental section.
Corollary 6.
Suppose that the state space is constructed as a set of variables each taking a ﬁnite number of values, i.e., S = { s ∈ R
d : s i ∈ S i } .
The transition matrices for each action a can be deﬁned by specifying the evolution of each state variable s t
+1 i depending on ( s t , a ) .
Suppose that there exists a state variable whose evolution only depends on its previous value, but neither on the other state variables nor the action taken: such a variable is called an exogenous variable.
Note that this variable can still affect the evolution of all other variables, and its evolution can vary across the environment of the observed experts.
Then, the reward function is not identiﬁable (even up to a constant) using any number of experts.
Such a negative result motivates the search for milder requirements than arbitrary reward identiﬁcation, which is too hard of a goal to achieve in certain scenarios.
A possible way to improve reward identiﬁability is to restrict the class of possible rewards, e.g., by constraining it to be a linear combination of a set of chosen features.
This is known as Feature matching IRL [49, 67–70].
The smaller the set of features, the easier to identify the reward, as described in the following theorem.
This method also allows to recover a more interpretable reward function, since the recovered parameters are associated with speciﬁc features.
Theorem 7.
Suppose that we restrict the class of possible reward functions to the one parameterized as r w ( s, a )
=
w T f s,a ∀ a ∈ A , s ∈ S where f : S ×
A → R d is a given feature function, and w ∈ R d denotes the reward parameters.
Suppose that the d chosen features are linearly independent, i.e., that f Ts,a v = 0 ∀ s, a ⇒ v = 0 .
Then, if 1 ∈ Im   f a 1 .
f a |A|   , the reward is identiﬁable up to constant by observing experts acting in ( T 1 , γ 1 ) , ( T 2 , γ 2 )
if and only if rank
[Equation]

[Equation]
(8) where f a =
( f s 1 ,a . . .
f s |S| ,a ) T ∈ R |S|× d .
On the other hand, if 1 / ∈ Im   f a 1 .
f a |A|   , then the reward can be exactly recovered provided that the rank of the matrix on the left hand side of equation (8) , which augments equation (5) by the features being matched, is 2 |S| + d .
Finally, it usually happens that the exact transition matrices { T a } a ∈A are not known exactly and must be estimated, e.g., from samples.
Verifying condition (5) on the approximated matrices may be misleading since the rank is very sensitive to small perturbations.
Hence, we provide hereafter an identiﬁability condition in the case where we only have access to approximated transition matrices.
Theorem 8.
Suppose that we approximate the transition matrices { T ia } a ∈A as { ˆ T ia } a ∈A such that (cid:107) T ia − ˆ T ia (cid:107) 2 ≤ (cid:15) ∀ a ∈ A , i = 1 , 2 .
Suppose that we verify condition (5) using the approximated matrices, i.e., we compute the second smallest eigenvalue σ of the following matrix:
[Equation]
Then, condition (5) on the true transition matrices { T a } a ∈A holds provided that
[Equation]
Remark 2.
The matrix estimator ˆ T a can be obtained from samples.
For example, [71][Lemma 5] shows that a high probability bound on the max norm (cid:107) T a − ˆ T a (cid:107) max ≤ (cid:15) requires O ( (cid:15) − 4 ) samples from a generative model [72].
This would imply the following bound on the spectral norm: (cid:107) T a − ˆ T a (cid:107) 2 ≤ |S|(cid:107) T a − ˆ T a (cid:107) max ≤ |S| (cid:15) .
However, the dependence on (cid:15) can be improved as we show next applying the matrix Bernstein bound [73, 74].
Theorem 9.
Let ˆ T a be the empirical estimator for T a .
Then with probability greater than 1 − δ ,
[Equation]
Therefore, we can obtain (cid:107) T a − ˆ T a (cid:107) 2 ≤ (cid:15) with O ( (cid:15) − 2 ) samples.
4.2 Generalization to unknown environments
We now focus on reward generalizability, i.e., the ability to recover a reward function that would allow us to train an optimal policy in a new environment.
Suppose that we recover a reward function that is compatible with two experts acting in two MDPs M 1 , M 2 , and that we use this reward to train an expert in a third environment M 3 , assuming all environments share the same true reward function but possibly different transition dynamics and discount factors.
What condition guarantees that the trained expert will be optimal in M 3 ?
This generalization requirement is milder than full reward identiﬁcation.
Indeed, being able to identify the reward (even up to a constant) naturally allows to train an optimal policy in any other environment sharing the same reward.
However, even in the presence of non-trivial degrees of freedom, it may be the case that any recovered reward sufﬁces to train an optimal policy in a given other environment.
Intuitively, the third training environment should not vary too much from the observed environments
M 1 , M 2 .
More precisely, if observing a third expert in environment 3 does not provide any further identiﬁcation of the reward than with environments 1 and 2 , then any reward compatible with environments 1 and 2 leads to an optimal policy in environment 3 .
The condition is made precise in the following theorem.
Deﬁnition 10.
Consider three Markov decision problems on the same set of states and actions, but with different transition matrices T 1 , T 2 , T 3 and discount factors γ 1 , γ 2 , γ 3 .
Suppose that we observe two optimal entropy regularized experts with respect to the same reward function in environmentsand 2 .
We say that ( T 1 , γ 1 ) , ( T 2 , γ 2 ) generalize to ( T 3 , γ 3 ) if any reward compatible with the two experts in environments 1 and 2 leads to an optimal expert in environment 3 .
The deﬁnition naturally extends to more than two observed experts.
Theorem 11.
( T 1 , γ 1 ) , ( T 2 , γ 2 ) generalize to ( T 3 , γ 3 ) if and only if rank
[Equation]
This condition is also necessary, in the sense that, if it does not hold, then there exists a reward function compatible with experts 1 and 2 but which leads to a sub-optimal policy in environment 3 .
One interesting question is whether observing two experts in the same environment with different discount factors allows to generalize to any other expert with arbitrary discount factor.
It turns out to be the case under some commutativity constraint on the transition matrices.
Corollary 12.
Consider a single environment with transitions T .
Suppose that there exists an action a 0 ∈
A such that T a 0 commutes with T a for all a ∈ A .
Then for any 0 < γ 1 , γ 2 , γ 3 < 1 with γ 1 (cid:54) =
γ 2 , ( T, γ 1 ) , ( T, γ 2 ) generalize to ( T, γ 3 ) .
Remark 3.
The commutativity condition cannot simply be removed.
Indeed, we provide in Appendix A.9 an example with two actions with non-commutative transition matrices for which condition (12) is not satisﬁed.
Experiments
We now present empirical validations of our claims 1 .
In particular, we verify the identiﬁability requirement given by Theorem 3 in the context of randomly generated transition matrices and different gridworlds with uniform additive noise in the dynamics.
In addition, we study a Windy-Gridworld and a ﬁnancial model that we term Strebulaev-Whited both involving exogenous variables in their state spaces.
In agreement with Corollary 6, the reward
(a
[Figure]
r α       (b) r γ       (c) r true       (d) r α − r true       (e) r γ − r true function is not identiﬁable in these environments, highlighting the necessity of imposing milder requirements than full reward recovery.
For example, in Windy-Gridworld , we show that by observing multiple experts acting in environments with different wind distributions, we can generalize, i.e., train an optimal expert in environments with arbitrary other wind distribution, in accordance with
Theorem 11.
On the other hand, in Strebulaev-Whited , given the additional information that the reward function can be represented as a linear combination of some known features, we can identify the reward, validating the condition of Theorem 7.
The algorithms are described in Appendix B.
5.1 Identiﬁability experiments
Experiments on Random-Matrices The ﬁrst experiment involves randomly generated transition matrices and reward function with |S| = 18 , |A| = 5 .
This setting matches the numerical evidence in [1].
Their algorithm recovers the reward function but the connection with their theoretical contribution is not highlighted.
On the contrary, we have no theory practice mismatch, since we verify exactly the condition in Theorem 3.
In particular, for the 100 random seed we tried the rank of the matrix A is 2 |S| − 1 = 35 , then invoking Theorem 3 we can conclude that the reward function is identiﬁable up to a constant shift.
We provide a visual example of the recovered reward in Figure 4 in Appendix C.
Experiments on Gridworld As a second example of identiﬁability, we consider Gridworld , where the state space is a squared grid with 100 states while the action set is given by A = { up , down , left , right } with dynamics given by T α ( s (cid:48) | s, a ) =
(1 − α ) T det ( s (cid:48) | s, a ) + αU ( s (cid:48) | s, a ) where T det ( s (cid:48) | s, a ) represents deterministic transition dynamics where for example the action right leads to the state on the right with probability 1 .
If an action would lead outside the grid, then the agent stays in the current state with probability 1 .
The dynamics U ( s (cid:48) | s, a ) are instead uniform over the states that are ﬁrst adjacent to the current state.
In other words, U ( ·| s, a ) =
Unif( N ( s )) ∀ a ∈ A where N ( s ) denotes the set of ﬁrst neighbors of the state s .
We generate two different environments changing the value of α , choosing α 1 = 0 .
4 and α 2 = 0 .
We notice that, even using the same discount factor γ = 0 .
9 , the condition of Theorem 3 holds.
When α is kept ﬁxed, we also notice that the condition of Corollary 5 holds, and hence the reward can be recovered by just varying the discount factor γ of the experts.
We numerically verify that the reward can indeed be identiﬁed up to a constant shift in these two settings (see Figure 1).
5.2 Generalizability experiments
In this section, we present cases where identiﬁability is not possible due to the presence of exogenous variables.
However, we notice that the generalizability condition in Theorem 11 is often satisﬁed, even for a test environment with parameters rather different than the environments of the observed experts.
We start brieﬂy describing the environments to later comment on the results.
Experiments on
WindyGridworld The WindyGridworld environment augments the Gridworld state representation by including a wind direction.
The wind impacts the position transitions by making the agent move one step in the direction of the wind in addition to the action taken.
The wind
(a) r      (b) r true      (c) r − r true 0.
(d) π T test r true 0.
(e) π T test r true − π T test r variable.
While the reward is not identiﬁable a
    1XPEHURIH[SHUWV     5 DQ N  G L II H U HQ F H 
[Figure]
    1XPEHURIH[SHUWV    
[Figure]
V
[Figure]
R II U
[Figure]
In Figure 2b, we see that we can obtain better identiﬁability (although never full identiﬁability) when increasing the number of experts.
Once we have observed 4 experts, we do not get further identiﬁability by observing more experts, hence leading to generalizability as shown in Figure 2a and Figure 3.
We conjecture that this number of experts is linked to the number of values that the exogenous variable, i.e. the wind direction, can take.
Furthermore, although the actions in Gridworld do not exactly commute (because of the boundary), observing two experts in the same environment with different discount factors enables generalizing to a different discount factor (see Figure 6 in Appendix C).
The condition of Corollary 12 is hence sufﬁcient but not necessary.
Experiments on Strebulaev-Whited
The Strebulaev-Whited environment is the neoclassical investment model in which a ﬁrm has a Cobb-Douglas production function with decreasing returns to scale, as in [75].
The goal of the agent is to maximize proﬁts discounted at rate 0 < γ < 1 .
The state of the agent is deﬁned by the capital level k ≥ 0 and an exogenously given persistent stochastic productivity shock z .
We can summarize the state by s =
( k, z ) .
The next state s (cid:48)
=
( k (cid:48) , z (cid:48) ) is determined separately for k (cid:48) and z (cid:48) .
We have that k (cid:48)
=
(1 − δ ) k + ak, where δ is the depreciation rate of physical capital and a is today’s rate of investment which is the action in the model.
The variable z (cid:48) evolves according to ln z (cid:48)
=
ρ ln z + (cid:15) where (cid:15) ∼ N (0 , σ (cid:15) ) .
The continuous variables k and z are discretized according to the scheme proposed in [76].
Hence, we obtain a discrete process with K 2 possible values for the state variable s =
( k, z )
(so |S| = K 2 ) and K values for the action a .
In the experiments in Figure 7 in Appendix C , we choose K = 20 and consider two environments with different values of σ (cid:15) set to 0 .
02 and 0 . 04 , respectively.
We observe that the rank of the identiﬁability matrix is 552 .
Since 552 < 2 |S| − 1 = 799 , the reward function is not identiﬁable up to a constant as expected in MDPs with exogenous states.
Nonetheless, when we consider a third environment with σ (cid:15) = 0 .
6 , the generalizability condition in Theorem 11 is satisﬁed.
Hence, the expert behavior can be predicted in the third environment (see Figure 7e in Appendix C ).
5.3 Identiﬁability experiments with a restricted reward class
The ﬁnal result presents a numerical validation of Theorem 7 in the environment
Strebulaev-Whited with exogenous state variable.
In this model, the true reward function can be expressed as a linear combination of the three features given by f s,a = [ z ((1 − δ ) k + ak ) θ , (1 − δ ) k, ak ] T , where s = ( k, z ) and the parameter ρ ∈ (0 , 1) captures the curvature of the production function.
We set ρ = 0 .
The ﬁrst feature corresponds to the ﬁrm’s output or sales which is available from the ﬁrm’s income statement, the second feature is the ﬁrm’s current capital stock net of depreciation which is available from the balance sheet, and the third feature is the level of investment that determines the future level of capital stock.
The true reward function can be written as r ( s, a ) =
w T f s,a with w =
[1 , 1 , − 1] T .
It can be interpreted as follows: the agent’s reward of investment is an increase in output/sales, w 1 = 1 > 0 , while the cost of capital is 1 and, hence, investment is costly, w 3 = − 1 < 0 .
At the same time, the capital stock is valuable and can be liquidated at a price of w 2 = 1 > 0 .
Knowing these features, we can verify that the rank of the matrix in Equation (8), is 803 which is equal to 2 | S | + d in this environment ( | S | = 400 and d = 3 ).
Invoking Theorem 7, we can conclude that the reward function is identiﬁable exactly, which is veriﬁed numerically in Figure 8 in
Appendix C. Expressing the reward in terms of features hence helps identiﬁability and interpretability.
Conclusion
In this paper, we analyze conditions that guarantee identiﬁability of the reward function (up to an additive constant) from multiple observed experts maximizing the same reward and facing different transition dynamics.
This allows us to train optimal policies in any other environment sharing the same reward with the environments of the observed experts.
On the other hand, in order to generalize to unknown environments, such strong reward identiﬁcation is not required, and we provide a milder necessary and sufﬁcient condition for generalizability
We also provide identiﬁability results in a variety of settings, i e
, linearly parameterized reward, approximated transition matrices, observation of any number of experts, as well as a non-identiﬁability result in the presence of exogenous variables
In the following, we list the main limitations of our work that will be the subject of future studies.
Observing experts in different environments.
We saw that observing a single expert in one environment cannot lead to reward identiﬁcation in our setting.
We hence need to observe at least two experts acting in different enough environments.
To motivate this assumption, note that varying environments are ubiquitous in RL, in particular in Robust RL which deals with the training of experts that perform well in different environments, where the transition dynamics can vary to some extent.
It is hence rather common to consider that the transition dynamics of a given environment can change.
This was studied, e.g., in [77–80], where the authors considered different Mujoco environments with varying friction coefﬁcients, or object masses, which inﬂuence the dynamics.
Also, instead of observing different experts in different environments, we could imagine that we observe a single expert in a single environment that varies over time (but with ﬁxed reward), and that the expert adapts to these changes.
Such observations would provide us optimal actions in environments with different transition dynamics, and thus our results would apply.
This is of particular interest in economics/ﬁnance where the environment is in constant evolution.
Assuming entropy regularized experts.
When observing real world data, we have to face the fact that humans do not follow this idealized mathematical model.
However, it turns out that our results still hold for the more general class of regularized MDPs [81] where we replace the entropy with any strongly convex function (see Appendix D).
Whether the ﬂexibility in the choice of the strongly convex regularizer allows to better capture real-world behaviors is an open question.
Acknowledgements
This work has received ﬁnancial support from the Enterprise for Society Center (E4S) and SNF project 100018_192584.
This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021_205011.
This project has received funding from the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement n° 725594 - time-data).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the supplementary.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proofs
We provide hereafter the proofs of the statements made in the main body.
A.1 Proof of Theorem
Let r ∈ R |S|×|A| be the reward function common to the two experts, and let v 1 , v 2 ∈ R |S| be the entropy regularized values functions associated experts 1 and 2 respectively and the reward function r .
Then, according to Theorem
[Equation]
we have that ∀ ( s, a ) ∈ S × A ,
[Equation]
where π
[Equation]
π 2 denote the policies of experts 1 and 2 respectively.
We hence deduce that ∀ a ∈ A ,
[Equation]
By including all available actions to the experts
[Equation]
we can write
[Equation]
Reward identiﬁability is directly related to the size of the solution space of the linear system (14).
Since we assume that both experts are optimal with respect to a true reward function r , we know that the associated value function solves equation (14), and hence that this system is feasible.
The solution space then depends on the rank of the matrix in the left hand side of (14), which we denote by A .
We ﬁrst show that there always exists an eigenvector of A associated with eigenvalue 0 .
Indeed, since the matrices T a are transition matrices, their rows must sum to 1 , which can be written as T a 1 =where 1 is a S dimensional column vector of 1 ’s. Hence,
[Equation]
Hence, the vector
[Equation]
is an eigenvector of A with eigenvalue 0 , and corresponds to the invariance of the optimal policy under addition of a constant to the reward function.
Suppose now that rank ( A ) = 2 |S|− 1 .
Since A has 2 S columns, this implies that the only eigenvector with eigenvalue 0 is
[Equation]
, and thus that we can recover the value function v 1 (or v 2 equivalently) up to an additive constant.
Using Theorem 1 again, it implies that we can also recover the reward function up to a constant.
On the other hand, suppose that rank ( A ) < 2 |S| − 1 .
Then, there exists another vector in Ker ( A ) which is linearly independent of (cid:18)
[Equation]
, and whose addition to the value function would not change the optimal policy.
However, it is easy to check that the only eigenvector of A with eigenvalue 0 of the form
[Equation]
with c 1 , c 2 ∈ R is proportional to
[Equation]
Hence, any other vector in Ker ( A ) would induce a modiﬁcation of the value and reward functions more complex than just adding a constant.
The provided condition is hence also necessary.
Equivalence with Deﬁnition 2.
It turns out that our rank condition (5) is equivalent to the valuedistinguishing assumption of Deﬁnition 2.
To show this, we ﬁrst notice that, if v 1 , v 2 satisfy equation (2), and if v 1 is a constant vector, then v 2 must also be a constant vector, and vice versa.
Indeed, equation (2) can be written as
[Equation]
Since, ∀ a ∈ A , i = 1 , 2 , 1 is an eigenvector of T ia with eigenvalue 1, then 1 is also an eigenvector of ( I − γ 2 T 2 a ) − 1 with eigenvalue 1 1 − γ 2 .
Hence, if v 1 =
c 1 is a constant vector, then v
= ( I − γ 2 T 2 a ) − 1 ( I − γ 1 T 1 a ) v
= c 1 − γ 1 1 − γ 2 1 is also a constant vector, and the associated constant is determined by the constant of v 1 .
Thus, the condition of Deﬁnition 2 can be rewritten as
[Equation]
This is hence equivalent to
[Equation]
which is equivalent to equation (5).
A.2 Proof of Corollary
Let v 1 , v 2 , v 3 ∈ R |S| be the entropy regularized value functions associated with experts 1 , 2 andrespectively.
Following the proof of Theorem 3, these vectors must satisfy  
[Equation]
I − γ 1 T 1 a |A| − ( I − γ 2 T 2 a |A| )
I − γ 1 T 1 a − ( I − γ
[Equation]
T 3 a 1 ) .
I − γ 1 T 1 a |A| − (
[Equation]
a |A| )
   
[Equation]

[Equation] [Equation]
[Equation] [Equation]
A| )   .
Similarly as previously, we can easily show that the vector  
[Equation]
−
[Equation]
−
[Equation]
  ∈
Ker ( A (cid:48) ) , where A (cid:48) denotes the matrix on the left of equation (15)
In order for the reward to be recovered up to a constant, we hence need that there is no other linearly independent vector in Ker ( A (cid:48) ) , i.e., that rank ( A (cid:48) )
=
3 |S|− 1 .
A.3 Proof of Corollary
We want to show that dim
[Equation]
Suppose that
[Equation] [Equation]
or equivalently
[Equation]
Subtracting equation (18) for a = a 1 and a = a i , we get
[Equation]
[Equation]
Using equation (7) and the fact that the vector 1 ∈ R |S| always belongs to Ker  
[Equation] [Equation]
T
[Equation] [Equation]
T a |A
[Equation]
  , we have that Ker
[Equation]
Span ( 1 ) .
Thus, we deduce from equation (19) that
[Equation] [Equation]
for some c ∈ R .
Using the fact that for any a ∈ A , 1 is an eigenvector of T a with eigenvalue 1 , we deduce from (18) and (20) that
[Equation]
Solving equations (20) and (21) for v 1 and v 2 , we ﬁnd v 1 =
c (1 − γ 2 ) γ 1 − γ 2 1 and v 2 =
c (1 − γ 1 ) γ 2 − γ 1 1 .
Therefore,
Ker
[Equation]
which shows that condition (16) holds.
On the other hand, if condition (7) does not hold, then
[Equation]
T a 1 − T a |A| 
 contains another vector v 0 which is not a constant vector, so the reward cannot be recovered up to a constant.
A.4 Proof of Theorem
Suppose that 1 ∈ Im   f a 1 .
f a |A|   , i.e., ∃ w ∈ R d such that
[Equation]
This implies that
[Equation]
Suppose that condition (8) holds, i.e., that dim
[Equation]
Equations (22) and (23) thus imply that
[Equation]
This means that for any v 1 , v 2 satisfying ( I − γ 1 T 1 a ) v
[Equation]
=
[Equation]
T 2 a ) v 2 ∀ a ∈ A and such that ∃ w ∈ R d , ( I − γ 1 T 1 a ) v 1 = f a w ∀ a ∈ A , then v 1 ∝ 1 .
Now suppose that we recover a reward function r ( s, a )
=
w T f s,a compatible with the two experts, i.e.,
[Equation] [Equation]
where r ∗ ( s, a ) =
w ∗ T f s,a denotes the true reward and v 1 satisﬁes ( I − γ 1 T 1 a ) v 1 =
( I − γ 2 T 2 a ) v 2 ∀ a ∈
A .
Then, ( I − γ 1 T 1 a ) v 1 = r ( · , a ) − r ∗ ( · , a ) =
f a ( w − w ∗ ) , and hence ∃ ˜ w ∈ R d such that ( I − γ 1 T 1 a ) v 1 = f a ˜ w ∀ a ∈ A .
Thus, v 1 ∝ 1 and the reward is recovered up to a constant.
Suppose now that 1 / ∈
Im   f a 1 .
f a |A|   .
Then, the condition rank  
I − γ 1 T 1 a
I − γ 2 T 2 a ...
[Equation]
means that this matrix is full rank and hence that its kernel is { 0 } .
Thus, if we recover a reward of the form (25), following the same argument as previously, this means that v 1 = 0 , and thus that the reward function is recovered exactly.
A.5 Proof of Corollary
Without loss of generality, let us assume that the exogenous variable can only take two possible values, i.e., the state space is deﬁned as S
=
{ ( s, e ) : s ∈ S 0 , e ∈ { e 1 , e 2 }} , where e denotes the exogenous variable and S 0 contains all other variables.
Exogenity of variable e implies that ∀ e ∈ { e 1 , e 2 } , p ( e t +1 =
e 1 | s t =
s, e t
=
e, a t = a )
= p ( e t +1 =
e 1 | e t = e ) does not depend on s nor a .
Suppose that we order the states as {{ ( e 1 , s ) } s ∈S 0 , { ( e 2 , s ) } s ∈S 0 } .
Then, the transition matrix for each expert i associated with action a has the following form:
[Equation]
27) where for each expert i and exogenous variable e j , j = 1 , 2 , p ij = p
i ( e t +1 =
e j | e t = e j ) and T ia,j ∈
R |S 0 |×|S 0 | denotes the transition matrix of expert i for state variables in S 0 knowing that the current value of state variable e is e j , i.e.
[Equation]
where p i denotes the state transition probability in environment i .
We ﬁrst show the result in the case of two experts.
The matrix A =  
I − γ 1 T 1 a
I − γ 2 T 2 a 1 .
I − γ 1 T 1 a |A |
I − γ 2 T 2 a |A| 
 has the following form:
[Equation]
a 1 ,
[Equation]
I
[Equation]
γ 2 p 22 T 2 a 1 ,
[Equation]
I
[Equation] [Equation]
[Equation]
|A|
[Equation]
− γ
[Equation]
p
[Equation]
T 1 a |A| ,
I − γ 2 p 21 T 2 a |A| ,− γ 2 (1 − p 21 ) T 2 a |A| ,
[Equation]
|
[Equation]
|
[Equation]
I
[Equation]
γ
[Equation] [Equation]
|A| ,
[Equation]
γ
[Equation] [Equation]
T 2 a |A| ,
I − γ 2 p 22 T 2 a |A| , 
We know that v 0 =
[Equation]

[Equation]
  is an eigenvector of A with eigenvalue 0 , corresponding to an addition of a constant to the reward.
In order to show that the reward is not identiﬁable, we need to ﬁnd another vector in Ker ( A ) linearly independent of v 0 .
We search for such a vector of the form v 1 = 
[Equation]
 .
Using the fact that 1 is an eigenvector of any transition matrix with eigenvalue 1 , the condition v 1 ∈ Ker ( A ) is equivalent to
[Equation]
This system of equations turns out to have a unique solution for ( c 1 , c 2 ) since det
[Equation]
since 0 ≤ γ 2 < 1 .
Hence, Ker ( A ) contains at least two linearly independent vector, and thus rank ( A ) < 2 |S| − 1 .
So, according to Theorem 3, the reward function is not identiﬁable up to a constant.
This means that, in addition to a global constant that we can add to the reward, we can also add a constant only to the rewards associated with a speciﬁc value of the exogenous variable.
The proof naturally extends to the case of multiple experts, and when the exogenous variable can take more than two values.
Actually, in the latter case, we can ﬁnd even more linearly independent vectors in
Ker ( A ) , corresponding to adding a constant to the rewards associated with each possible value of the exogenous variable.
A.6 Proof of Theorem
Deﬁne A = 
[Equation]
[Equation] [Equation]

[Equation]
and ˆ A
[Equation]
 
[Equation]
I − γ 1 ˆ T 1 a |A |
I − γ 2 ˆ T 2 a |A|   .
For an arbitrary matrix M , let σ 2 ( M ) denote the second smallest singular value of M .
Note that the condition (5) for A is equivalent to σ 2 ( A ) > 0 .
From Weyl’s inequality for singular values[82], we have that
[Equation]
Moreover,
(cid:107)
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
max
[Table]
  ≤ (cid:112) 2 |A| max( γ 1 , γ 2 ) (cid:15).
Therefore, σ 2 ( A ) ≥ σ 2 ( ˆ A ) − (cid:112) 2 |A| max( γ 1 , γ 2 ) (cid:15) , and hence σ 2 ( A ) > 0 provided that σ 2 ( ˆ A ) > (cid:112) 2 |A| max( γ 1 , γ 2 ) (cid:15) .
A.7 Proof of Theorem
Proof.
ˆ T a can be constructed as follows.
Sample N |S| states { s (cid:48) i } N |S| i =1 from the distribution T ( ·| s, a ) for every state s ∈ S .
Let N ( s ) denote the number of times state s has been sampled, i.e. N ( s )
=
N |S| .
Form the matrix ˜ T i =
[ 1 ( s i =
s,s (cid:48)
i = s (cid:48) )
N N ( s ) ] s,s (cid:48)
It holds that ∀ i , E [ ˜ T i ]
= T a , λ max ( ˜ T i − T a ) ≤ |S| + 1 , λ max ( E [( ˜ T i − T a ) 2 ]) ≤ |S| 2 and Trace( E [( ˜ T i − T a ) 2 ]) ≤ |S| 2 .
Then, the result follows applying
Lemma 10 in [73] and assuming δ < 1 /e .
Finally, we conclude with a covering argument over the set A .
A.8 Proof of Theorem
Lemma 13.
The condition of equation 12 holds if and only if ∀ v 1 , v 2 ∈ R |S| satisfying ( I − γ 1 T 1 a ) v 1 =
( I − γ 2 T 2 a ) v 2 , ∀ a ∈ A , there exists v 3 ∈ R |S| such that
( I − γ 3 T 3 a ) v 3 =
( I − γ 1 T 1 a ) v 1 , ∀ a ∈ A .
Proof.
Denote by A 1 , A 2 the matrices shown and the left and right hand side of equation (12) respectively, so that the equation reads rank ( A 1 ) =
rank ( A 2 ) −|S| , or equivalently 2 |S|− rank ( A 1 ) = 3 |S| − rank ( A 2 ) .
Using the rank theorem, it follows that dim ( Ker ( A 1 ))
=
dim ( Ker ( A 2 )) , i.e., dim ( { ( v 1 , v 2 ) ∈ R 2 |S| : ( I − γ 1 T 1 a ) v 1 =
( I − γ 2 T 2 a ) v 2 ∀ a ∈
[Equation]
Since all matrices I − γ 3 T 3 a are invertible for any a ∈ A
[Equation]
it follows that for any ( v 1 , v 2 ) ∈ R 2 |S|
[Equation]
there can exist at most one vector v 3 ∈ R |S| such that ( I − γ 1 T 1 a
[Equation]
v
[Equation]
A .
We hence deduce that
[Equation]
[Equation]
Plugging equation (29) in (28)
[Equation]
we have
[Equation]
Moreover, we can clearly see that
[Equation]
Thus, together with equation (30), we can conclude that
[Equation] [Equation]
which shows the result.
Suppose now that condition 12 does not hold,
i
[Equation]
e.,
[Equation]
This directly implies that there must exist a pair ( v 1 , v 2 ) , such that there exists no v 3 ∈ R |S| satisfying ( I −
[Equation]
[Equation] [Equation]
) v 3 =
[Equation]
I − γ
[Equation] [Equation]
) v 1 ∀ a ∈
A hence ﬁnalizing the proof.
We now turn to the proof of Theorem 11.
Let r ∗ be the ground truth reward, and suppose that we recover some reward function r from policies π 1 , π 2 , i.e., π 1 , π 2 are optimal with respect to both rewards r and r ∗ on ( T 1 , γ 1 ) , ( T 2 , γ 2 ) respectively.
Suppose that we train a policy π 3 optimally with respect to r on ( T 3 , γ 3 ) .
We want to show that π 3 is also optimal with respect to the true reward r ∗ .
Let v i , v i ∗ be the value vectors associated to expert i = 1 , 2 with respect to rewards r and r ∗ respectively, i.e., such that
[Equation]
Let v 3 be the value vector associated with expert 3 with respect to reward r , i.e., such that ∀ a ∈
A
[Equation]
We need to show that there exists a vector v 3 ∗ ∈ R |S| such that ∀ a ∈
A
[Equation]
Using equations (32), (33) and (34), we have ∀ a ∈
A
[Equation] [Equation]
[Equation]
Moreover, subtracting equations (32) and (33), we have
[Equation]
Therefore, using our assumption and Lemma 13, there exists a vector ˜ v 3 ∈ R |S| such that
( I − γ 1 T 1 a )( v 1 − v 1 ∗ ) =
( I − γ 3 T 3 a )˜ v 3 .
Hence, combined with equation (38), we conclude that there exists v 3 ∗ ∈ R |S| such that ∀ a ∈
A
[Equation]
Using Theorem 1, we conclude that r ∗ belongs to the set rewards compatible with π 3 , and hence that π 3 , which has been optimized for r , is also optimal for the ground truth reward r ∗ .
On the other hand, if condition 12 does not hold, according to Lemma 13, we can construct a reward function r compatible with experts 1 and 2 that cannot be written in the form r ( · , a ) =
λ log π 3 ( a |· ) + ( I − γ 3 T 3 a ) v 3 for some v 3 ∈ R |S| .
Hence, thanks to Theorem 1, the policy π 3 cannot be optimal for such a reward function.
Hence, there will necessarily exist some recovered reward functions that would lead to a sub-optimal policy in environment 3 .
A.9 Proof of Corollary
For the setup describe in this corollary, we need to verify condition (12).
By the rank theorem, this condition is equivalent to dim
[Equation]
To this end, we will show that any element ( v 1 , v 2 ) ∈ R 2 |S| of the kernel space of the left hand side is associated a single element ( v 1 , v 2 , v 3 ) ∈ R 3 |S| of the kernel space of the right hand side.
More precisely, we need to show that for any v 1 , v 2 satisfying
[Equation]
there exists a unique v 3 ∈ R |S| such that
[Equation]
Consider the action a 0 satisfying by assumption
that T a 0 commutes with all other matrices T a , a ∈ A .
Deﬁne v 3 =
( I − γ 3 T a 0 ) − 1 ( I − γ 1 T a 0 ) v 1 .
Notice that for any a ∈ A , we can write
I − γ 3 T a = α
( I − γ 1 T a ) + (1 − α )
( I − γ 2 T a ) where α
= γ 3 − γ 2 γ 1 − γ 2 .
Moreover, recall that, if any two invertible matrices A and B commute, then A and B − 1 also commute.
Using these properties, we then have for any a ∈ A ,
[Equation]
Uniqueness of v 3 is trivial since the matrices ( I − γ 3 T a ) are invertible, which shows that condition (40) holds.
Counter-example when the commutativity constraint does not hold.
We now provide a simple example showing that the required generalizability condition (12) does not always hold in the case where the commutativity condition breaks.
Suppose |S| = 3 , |A| = 2 and
T a 1 =
[Equation]
These matrices do not commute and we have for any discount factors γ 1 , γ 2 , γ 3 all different, 4 = rank
[Equation]
[Equation]
B Algorithms details
This section provides the detailed pseudocode of the procedures we introduced for reward identiﬁcation (Algorithm 1), for generalizability (Algorithm 3) and identiﬁcation when the reward function can be expressed as linear combination of known features (Algorithm 2).
Algorithm
Input: Expert transition matrices T 1 , T 2 , entropy-regularized optimal policies π 1 , π 2 .
Compute matrix
[Equation]
  (43) if rank ( A ) = 2 |S| − 1 then
Identiﬁable =
True
Form vector b ∈ R |S||A| such that b ( s, a )
=
λ log π 1 ( a | s ) π 2 ( a | s ) (ordered by states ﬁrst)
Recover value vectors (cid:18) vv(cid:19) =
( A T A ) − 1 A T b
Recover the reward function as r ( s, a ) = λ log π 1 ( a | s ) − γ (cid:80) s (cid:48)
T 1 ( s (cid:48) | s, a ) v 1 ( s (cid:48) )
+ v 1 ( s ) or equivalently r ( s, a )
= λ log π 2 ( a | s ) − γ (cid:80) s (cid:48)
T 2 ( s (cid:48) | s, a ) v 2 ( s (cid:48) )
+ v 2 ( s ) else
Identiﬁable =
False end if
Output: Identiﬁable and recovered reward r .
Algorithm
Input: Expert transition matrices T 1 , T 2 , entropy-regularized optimal policies π 1 , π 2 , features set { f a } a .
Compute matrix
A :=   − ( I − γ 1 T 1 a 1 )
I − γ 2 T 2 a ...
− ( I − γ 1 T 1 a |A| )
I − γ 2 T 2 a |A| − ( I − γ 1 T 1 a 1 ) f a 1 .
− ( I − γ 1 T 1 a |A| )
f a |A|   (44) if rank ( A ) = 2 |S| + d then
Identiﬁable =
True
Form vectors b 1 , b 2 ∈ R |S||A| deﬁned as b 1 ( s, a ) = λ log π 1 ( a | s ) π 2 ( a | s ) , b 2 ( s, a ) =
λ log π 1 ( a | s ) and b ∈ R 2 |S||A| as b = (cid:18) b 1 b 2 (cid:19)
Recover value vectors and reward weights
[Equation]
Recover the reward function as r ( s, a ) =
w T f s,a else
Identiﬁable =
False end if
Output: Identiﬁable and recovered reward r .
Algorithm
Input: Expert transition matrices T 1 , T 2 , transfer transition matrix T 3 , entropy-regularized optimal policies π 1 , π 2 .
Compute matrix
A :=   − ( I − γ 1 T 1 a 1 )
I − γ 2 T 2 a 1 .
− ( I − γ 1 T 1 a |A| )
I − γ 2 T 2 a |A| 
 if the condition in Equation (12) holds then
Generalizable =
True
Form vector b ∈ R |S||A| such that b ( s, a )
=
λ log π 1 ( a | s ) π 2 ( a | s )
Recover the value vectors (cid:18) vv(cid:19) =
( A T A ) − 1 A T b
Recover the reward function as r ( s, a ) = λ log π 1 ( a | s ) − γ (cid:80) s (cid:48)
T 1 ( s (cid:48) | s, a ) v 1 ( s (cid:48) )
+ v 1 ( s )
Recover the optimal entropy regularized policy π 3 in T 3 using the recovered reward r with any
RL algorithm.
else
Generalizable =
False end if
Output: Generalizable and recovered policy π 3 .
Algorithms 1 and 3 can be generalized to an arbitrary number of experts.
Indeed, denoting the matrix in Equation (43) as A 2 , we can construct the matrix A n for n experts recursively as follows:
[Equation]
Similarly, we can construct the vector b n as
[Equation]
where b 1 denotes the vector deﬁned in the algorithms for 2 experts.
The rest of the procedures remain unchanged.
C Additional experiments
This section provides the experimental results and environment details omitted from the main text.
Additional details for Gridworld
In the main text, we omitted the description of the reward function.
We provide it hereafter for completeness.
The reward function is obtained assigning a value at every state according to the grid shown in Figure 5.
This reward function would depend only on states.
To obtain a state-action dependent reward function, we add a penalty of − 30 for moving right, − 20 for moving down, − 10 for moving left and 0 for a step upwards.
Additional details for WindyGridworld In WindyGridworld , the agent moves of one step according to the next state sampled from T α ( s (cid:48) | s, a ) =
(1 − α ) T det ( s (cid:48) | s, a ) + αU ( s (cid:48) | s, a ) where
T det ( s (cid:48) | s, a ) as in Gridworld .
In addition to that the agent takes an additional step according to the wind direction.
The wind direction w is sampled from the wind distribution generated by sampling each entry of the non normalized P wind from a normal distribution and normalizing the obtained vector.
After sampling the wind direction we sample the corresponding next state from T det ( s (cid:48) | s, w ) .
The reward function is the same used for the environment Gridworld .
Results on Random-Matrices We report in Figure 4, the results omitted from the main text.
In
Figure 4, we show the reward recovered with Algorithm 1 and the difference with respect to the true reward.
It clearly emerges that the recovered reward is within a constant shift from the true reward function.
Results on Gridworld with state only reward
We provide an additional result on Gridworld where we do not consider the penalty assigned to the different actions.
In this case, the reward depends only on states but the learner is not informed about this feature.
In Figure 5, we show the recovered reward.
Given that the reward depends only on the states we show the 2D representation of the state space suppressing the action dimension.
We used the Gridworld implementation released with [83].
Results on WindyGridworld with different discount factors In the main text we showed that we need to observe 4 experts to generalize to a new wind distribution.
Hereafter, we provide experiments on the generalization to a new environment with a different discount factor.
We veriﬁed that in this case observing two experts is enough to generalize.
The comparison between recovered rewards and policies can be found in Figure 6.
Computational resources The experiment can be reproduced with a standard laptop.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
D Extension to Regularized MDPs
It turns out that our identiﬁability result is valid more generally for regularized MDPs [81] where the entropy term in equation (1) is replaced by any other strongly convex differentiable function of the policy Ω( π ) .
Indeed, we can use Proposition 1 and Deﬁnition 1 in [81] to establish that for any value vector any v and reward r , there exists a unique policy that satisﬁes
[Equation]
where Ω ∗ denotes the Fenchel conjugate of Ω .
By the distributivity property (iii) in Proposition 1 of [5], we can subtract a function dependent only on state in the argument without affecting the equality.
This gives that for any v and r , there exists a unique π such that
[Equation]
Using the convexity of Ω , we have that ∇ Ω is the inverse map of ∇ Ω ∗ .
Hence we obtain
[Equation]
which is the equivalent of our Theorem 1 for general strongly convex regularizers.
The only part changing is the left hand side.
However, we saw in the analysis that reward identiﬁability was not depending on this part of the equation.
When using a different regularizer, the recovered reward given observed expert policies will be different, but the identiﬁability condition remains the same.
This extension relax the assumption of entropy regularized experts but, unfortunately, epsilon-greedy or deterministic greedy policies would not ﬁt this setting.
Identiﬁability is more challenging with these kinds of experts because the knowledge of such policies only informs us with the action yielding the highest expected value, but no information about the relative difference with respect to other actions, in contrast with regularized stochastic policies.