I NVARIANCE IN P OLICY O PTIMISATION AND P ARTIAL I DENTIFIABILITY IN R EWARD L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Designing reward functions for complex, real-world tasks is challenging.
Reward learning lets one instead infer reward functions from data.
However, multiple reward functions often ﬁt the data equally well, even in the inﬁnite-data limit.
Prior work often considers reward functions to be uniquely recoverable, by imposing additional assumptions on data sources.
By contrast, we formally characterise the partial identiﬁability of popular data sources, including demonstrations and trajectory preferences, under multiple standard sets of assumptions.
We analyse the impact of this partial identiﬁability on downstream tasks such as policy optimisation, including under shifts in environment dynamics.
We unify our results in a framework for comparing data sources and downstream tasks by their invariances, with implications for the design and selection of data sources for reward learning.
I NTRODUCTION
A wide range of problems can be represented as sequential decision-making tasks, where the goal is to maximise some numerical reward (Sutton & Barto, 2018).
However, designing an appropriate reward function remains a challenge in complex real-world tasks (Amodei et al., 2016; Leike et al., 2018; Dulac-Arnold et al., 2019).
Reward learning algorithms infer task reward functions from data sources such as expert demonstrations (Ng & Russell, 2000), preferences over trajectories (Christiano et al., 2017), and many others (Jeon et al., 2020).
This approach has extended the applicability of sequential decision-making techniques to more complex tasks (e.g. Abbeel et al., 2010; Christiano et al., 2017; Singh et al., 2019; Stiennon et al., 2020).
Multiple reward functions are often consistent with the data source, even in the inﬁnite-data limit.
For most data sources, this fundamental ambiguity has been acknowledged, but its extent has not been characterised.
In section 3, we formally characterise the ambiguity of several popular data sources including expert demonstrations and trajectory preferences.
These inﬁnite-data limits bound the information recoverable from ﬁnite data sets using any algorithm, so they are useful for evaluating algorithms relative to their limits, and data sources relative to each other.
Uniquely identifying a reward function is unnecessary when all plausible reward functions lead to the same downstream outcome in a given application, such as policy optimisation. Characterising this ambiguity tolerance for various applications allow us to evaluate the ambiguity of a data source relative to a given application. Learnt reward functions are often used for policy optimisation, for example via reinforcement learning (RL) 1 .
In section 3 we formally characterise the ambiguity tolerance of policy optimisation under arbitrary dynamics.
Ambiguity and ambiguity tolerance are formally related.
Both concern invariances – of data sources or downstream outcomes – to reward function transformations .
Thus, our main contribution is to catalogue the invariances of various mathematical objects derived from the reward function.
In section 4, we explore a partial order on these invariances, and its implications for the selection and evaluation of data sources, addressing an open problem in reward learning (Leike et al., 2018, §3.1).
R
Q π β
[Figure]
G ζ
G ξ ĺ ζβ ĺ ξβ ĺ
[Figure]
ĺ
[Figure]
increasingambiguity (tolerance)
I R L p r e f .
c m p .
(a) id mask of supp p τ q
S 1 -R. k - Φ Φ
PLS
ZPMT
OPT (b)
1. R ELATED WORK
Inverse reinforcement learning (IRL; Russell, 1998) is the prototypical example of reward learning.
IRL infers a reward function from the behavioural data of a task expert by inverting a model of the expert’s planning algorithm (Armstrong & Mindermann, 2017; Shah et al., 2019).
Existing work partially characterises the inherent ambiguity of behaviour for certain planning algorithms (Ng &
Russell, 2000; Cao et al., 2021) and classes of tasks (Dvijotham & Todorov, 2010; Kim et al., 2021).
We extend these results to more planning algorithms and arbitrary time-unbounded, stochastic tasks, using a more expressive space of reward functions that reveals novel ambiguity.
Reward learning models have been proposed for many other data sources (Jeon et al., 2020).
A popular and effective data source is preferences over behavioural trajectories (Akrour et al., 2012;
Christiano et al., 2017).
Unlike for IRL, the ambiguity arising from these data sources has not been formally characterised.
We contribute a formal characterisation of the ambiguity for central models of evaluative feedback including trajectory preferences.
Several studies have explored learning from expert behaviour and preferences (Ibarz et al., 2018;
Palan et al., 2019;
Bıyık et al., 2020; Koppol et al., 2020), or other multi-modal data sources (Tung et al., 2018; Jeon et al., 2020).
One motivation is that different data sources may provide complementary reward information (Koppol et al., 2020), eliminating some ambiguity.
Similarly, Amin et al.
(2017) and Cao et al.
(2021) observe reduced ambiguity by combining behavioural data across multiple tasks .
Our partial order provides a general framework for understanding these results.
Computing an optimal behavioural policy is a primary application of learnt reward functions (Abbeel & Ng, 2004; Wirth et al., 2017).
Ng et al.
(1999) proved that potential shaping transformations always preserve the set of optimal policies, and so are always tolerable for this application.
We extend this result, characterising the full set of transformations that preserve optimal policies in each task, including for additional policy optimisation techniques such as maximum entropy RL.
Ambiguity corresponds to the partial identiﬁability (Lewbel, 2019) of the reward function modelled as a latent parameter.
The prevailing response to partial identiﬁability in reward learning has been to impose additional constraints or assumptions until the data identiﬁes the reward function uniquely (or, at least, sufﬁciently for policy optimisation).
Following Manski (1995; 2003) and Tamer (2010), we instead describe ambiguity given various constraints and assumptions.
This gives practitioners results appropriate for their real data (and the ambiguity tolerance of their actual application).
IRL is related to dynamic discrete choice (Rust, 1994; Aguirregabiria & Mira, 2010), a problem where identiﬁability has been extensively studied (e.g., Aguirregabiria, 2005; Srisuma, 2015; Arcidiacono & Miller, 2020).
We study a simpler setting with known tasks.
IRL also relates to preference elicitation (Rothkopf & Dimitrakakis, 2011) and inverse optimal control (Ab Azar et al., 2020).
Preferences over sequential trajectories are not typically considered as a data source in other ﬁelds.
1. P RELIMINARIES
We consider an idealised setting with ﬁnite, observable, inﬁnite-horizon sequential decision-making environments, formalised as Markov Decision Processes (MDPs; Sutton & Barto, 2018, §3).
An
MDP is a tuple p S , A , τ, µ 0 , R, γ q where S and A are ﬁnite sets of environment states and agent actions ; τ : S ˆ A Ñ ∆ p S q encodes the transition distributions governing the environment dynamics; µ 0 P ∆ p S q is an initial state distribution; R : S
ˆ A ˆ S Ñ R is a deterministic reward function 2 ; and γ P p 0 , 1 q is a reward discount rate .
We distinguish states in the support of µ 0 as initial states , and states s with τ p s | s, a q “ 1 and R p s, a, s q “ 0 for all a as terminal states .
We represent the transition from state s to state s 1 using action a as the tuple x “ p s, a, s 1 q .
We classify p s, a, s 1 q as possible in an MDP if s 1 is in the support of τ p s, a q , otherwise it is impossible .
A trajectory is an inﬁnite sequence of concatenate transitions ξ “ p s 0 , a 0 , s 1 , a 1 , s 2 , . . .
q , and a trajectory fragment of length n is a ﬁnite sequence of n concatenate transitions ζ “ p s 0 , a 0 , s 1 , . . .
, a n ´ 1 , s n q .
A trajectory or fragment is possible if all of its transitions are possible, and is impossible otherwise.
A trajectory or fragment is initial if its ﬁrst state is initial.
A state or transition is reachable if it is part of some possible and initial trajectory.
Given an MDP, we deﬁne the return function G as the cumulative discounted reward of entire trajectories and trajectory fragments: G p ζ q “ ř n ´ 1 t “ 0 γ t R p s t , a t , s t ` 1 q for a trajectory fragment ζ of length n , and similarly for trajectories.
We primarily consider this return function with various restricted domains , such as only possible or initial trajectories or trajectory fragments.
A policy π : S Ñ ∆ p
A q encodes an agent’s behaviour as a state-conditional action distribution.
Together with an MDP’s transition distribution τ , a policy π induces a distribution of trajectories starting from each state.
We denote such a trajectory starting from s with the random variable Ξ s , and its remaining components with random variables A 0 , S 1 , A 1 , S 2 , and so on.
Given an MDP and a policy π , and the value function encodes the expected return from states,
V π p s q “ E Ξ s „ π,τ “
G p Ξ s q ‰ ; and the Q -function of π encodes the expected return
given an initial action, Q π p s, a q “ E Ξ s „ π,τ “
G
[Equation]
Q π and V π satisfy a Bellman equation :
[Equation]
for all s P S and a P A .
Their difference, A π p s, a q “ Q π p s, a q ´ V π p s q , is the advantage function .
We further deﬁne a policy evaluation function , J , encoding the expected return from following a particular policy in an MDP, J p π q “ E S 0 „ µ“
V π p S
[Equation]
q ‰ .
J induces an order over policies.
A policy maximising J is an optimal policy , denoted π ‹ .
Similarly, Q ‹ , V ‹ , and A ‹ denote the Q -, value, and advantage functions of an optimal policy.
Since J may be multimodal, we often discuss the set of optimal policies.
However, Q ‹ , V ‹ , and A ‹ are unique.
Moreover, we consider several policies resulting from alternative planning algorithms .
Given a base policy π 0 , and an inverse temperature parameter β ą 0 , we deﬁne the Boltzmann policy with respect to π 0 , denoted π π 0 β , using the softmax function:
[Equation]
The Boltzmann-rational policy, π ‹ β , is the Boltzmann policy with respect to optimal policies, as used for IRL by Ramachandran & Amir (2007).
The popular Maximum Entropy policy, π β , is deﬁned using a modiﬁed policy evaluation function with an entropy regularisation term (Haarnoja et al., 2017), and solves the recurrence π β “ π π β β (i.e., π β is a Boltzmann policy with respect to itself).
R EWARD F UNCTION T RANSFORMATIONS
In this section, we discuss how invariance to reward function transformations relates to inﬁnite-data ambiguity in reward learning and ambiguity tolerance in applications.
Deﬁnition 2.1 (Transformations and invariances) .
A transformation is a map between reward functions.
The invariances of an object X derived from reward R via function f are all transformations t that preserve f : X “ f p R q “ f p t p R qq for all R .
We say that X determines R up to its invariances.
A set of transformations carves out a partition of the space of reward functions by grouping together those reward functions reachable from one another using the transformations.
The partition carved out by the invariances of an object is the equivalence kernel of the object’s derivation function – grouping the reward functions from which identical objects are derived into partition blocks.
Given a reward learning data source, consider the object encoding the information available from the data source in the inﬁnite-data limit (Lewbel, 2019, §3.1).
The invariances of this object represent the inﬁnite-data ambiguity of the data source – it is impossible to recover the reward function beyond the corresponding partition block, as the remaining functions imply indistinguishable data.
Similarly, consider a downstream application of learnt reward functions involving the computation of an object.
The object’s invariances capture the ambiguity tolerance of this computation, as by definition all reward functions in each block of the corresponding partition lead to identical outcomes.
2. F UNDAMENTAL R EWARD T RANSFORMATIONS
We introduce several fundamental sets of reward function transformations, forming a basis for the invariances we study in section 3.
Before considering several novel transformations, we recall potential shaping , introduced by Ng et al.
(1999) and widely known to preserve optimal policies in all MDPs.
We further distinguish special potential shaping transformations with constant potential over an MDP’s initial states.
We explore some properties of potential shaping in appendix A.
Deﬁnition 2.2 (Potential Shaping) .
A potential function is a function Φ : S Ñ R , where Φ p s q “if s is a terminal state.
If Φ p s q “ k for all initial states then we say that Φ is k -initial .
Let R and Rbe reward functions.
Given a discount γ , we say R 1 is produced by ( k -initial) potential shaping of
R if R 1 p s, a, s 1 q “ R p s, a, s 1 q ` γ ¨ Φ p s 1 q ´ Φ p s q for some ( k -initial) potential function Φ .
Deﬁnition 2.3 ( S 1 -Redistribution) .
Let R and R 1 be reward functions.
Given transition dynamics τ , say R 1 is produced by S 1 -redistribution of R if E S 1 „ τ p s,a
[Equation]
S 1 -redistribution allows changing R arbitrarily for impossible transitions.
Moreover, if at least two states s 1 1 , s 1 2 are in the support of τ p s, a q then S 1 -redistribution lets us increase R p s, a, s 1 1 q and decrease R p s, a, s 1 2 q by a proportionate amount.
Note that S 1 -redistribution depends crucially on the reward function’s dependence on the successor state.
This set of transformations collapses to the identity for simpler spaces of reward functions, as we explore in appendix C.
Deﬁnition 2.4 (Monotonic Transformations) .
Let R and R 1 be reward functions.
Say R 1 is produced by a zero-preserving monotonic transformation of R if for all pairs of transitions x, x 1 P S ˆ A ˆ S ,
R p x q ď R p x 1 q if and only if R 1 p x q ď R 1 p x 1 q , and R p x q “ 0 if and only if R 1 p x q “ 0 .
Moreover, say R 1 is produced by positive linear scaling of R if R 1 “ c ¨ R for some positive constant c .
A zero-preserving monotonic transformation is simply a monotonic transformation that maps zero to itself.
Positive linear scaling is a special case.
Deﬁnition 2.5 (Optimality-Preserving Transformation) .
Let R and R 1 be reward functions.
Given a function O : S Ñ P p
A q ´ t ∅ u , transition dynamics τ , and discount rate γ , we say R 1 is produced from R by an optimality-preserving transformation with O if there is a function Ψ : S Ñ R such that E S 1 „ τ p s,a q “
R 1 p s, a, S 1 q ` γ ¨ Ψ p S 1 q ‰ ď Ψ p s q for all s, a , with equality if and only if a P O p s q .
This transformation gives the reward functions with optimal actions from O ( Ψ determines the new value function).
Note that in practice an implicit dependence on R is introduced through the deﬁnition of O .
Also, we can reach arbitrary R 1 if O is unconstrained (in practice, we constrain O ).
Finally, we consider transformations allowing the reward to vary freely for a given set of transitions.
Deﬁnition 2.6 (Masking) .
Let R and R 1 be reward functions.
Given a transition set X Ď S ˆ A ˆ S , say R 1 is produced by a mask of X from R if R p x q “ R 1 p x q for all x R X .
I NVARIANCES OF R EWARD -R ELATED O BJECTS
In this section we catalogue the invariances of various central objects derived from reward functions, including expert trajectory distributions, the ranking of trajectories induced by the return function, and the set of optimal policies.
Some of these objects correspond to the information available in the inﬁnite-data limit of a reward learning data source, while others correspond to the outcome of a downstream application.
If an object X can be derived from another object Y without further reference to the reward function, then X inherits Y ’s invariances.
For example, the optimal Q -function’s invariances are inherited by various expert policies.
Accordingly, we organise this section by incrementally deriving our objects of interest starting from the reward function, cataloguing the invariances introduced in each step.
This also mirrors the structure of ﬁgure 1a.
We defer all proofs until appendix B.
3. I NVARIANCES OF E XPERT B EHAVIOUR
Inverse reinforcement learning (IRL) algorithms infer a task’s reward function from the behaviour of task experts.
Formally, this behaviour is represented as an expert’s policy or a sample of trajectories .
To characterise the corresponding invariances, we begin with Q -functions – instrumental to deriving many policies.
Q -functions are invariant to S 1 -redistribution since they are deﬁned as an expectation over the successor state S 1 , We show that this is the only invariance for Q -functions.
Theorem 3.1.
Given an MDP and a policy π , the Q -function for π , Q π , determines R up to S 1redistribution.
The optimal Q -function, Q ‹ , has precisely the same invariances.
This invariance is inherited by any object that can be derived from a Q -function.
However, note that
S 1 -redistribution vanishes in simpler spaces of reward functions, as we explore in appendix C.
We now turn to policies derived using various planning algorithms.
These policies are instrumental in constructing the trajectories studied in IRL.
For example, Ramachandran & Amir (2007) and Ziebart et al.
(2008) assume that expert behaviour is drawn from a Boltzmann-rational policy , and Ziebart et al.
(2010) assume a Maximum Entropy policy We catalogue the invariances of arbitrary Boltzmann policies , of which these other policies are special cases.
As these policies can be derived from Q -functions, they inherit invariance to S 1 -redistribution.
We show they are also invariant to potential shaping, but not to any other transformations.
Theorem 3.2.
Given an MDP, an inverse temperature parameter β , and a base policy π 0 , the
Boltzmann policy π π 0 β determines R up to S 1 -redistribution and potential shaping.
The Boltzmannrational policy, π ‹ β , and the Maximum Entropy policy, π β , have precisely the same invariances.
By contrast, Ng & Russell (2000) and Abbeel & Ng (2004) assume that experts follow an optimal policy .
Optimal policies inherit S 1 -redistribution invariance from the optimal Q -function, and are also known to be invariant to potential shaping (Ng et al., 1999).
Under an additional assumption that a given policy is maximally supportive , in that it takes all optimal actions with positive probability, we show that these invariances and any additional invariances are captured in a class of optimalitypreserving transformations (Deﬁnition 2.5) based on the set of optimal actions in each state.
Theorem 3.3.
Given an MDP, let O p s q “ arg max a A ‹ p s, a q .
A maximally supportive optimal policy determines R up to optimality-preserving transformations with O .
Additional invariances arise if we consider optimal policies that may lack support for optimal actions.
As a well-known example, the zero-reward is consistent with any policy in this sense.
In the inﬁnite-data limit, a data source of trajectories sampled from a policy reveals the distribution of trajectories induced by the policy, and therefore the policy itself for all states reachable via its supported actions.
A Boltzmann policy supports all actions, so in the inﬁnite data limit, samples of trajectories determine the policy for all reachable states.
We show this introduces invariance precisely to changes in the reward of unreachable transitions.
Theorem 3.4.
Given an MDP, an inverse temperature parameter β , and a base policy π 0 , the distribution of trajectories induced by the Boltzmann policy π π 0 β from all initial states determines R up to S 1 -redistribution, potential shaping, and a mask of unreachable transitions.
The distributions of trajectories induced by the Boltzmann-rational policy, π ‹ β , and the Maximum Entropy policy, π β , from all initial states, have precisely the same invariances.
Similarly, trajectories sampled from an optimal policy reveal the policy in those states that its actions reach.
This again introduces additional invariance to transformations of reward in other states.
Theorem 3.5.
Given an MDP, consider the distribution of trajectories induced by a maximally supportive optimal policy.
Let S be the set of states in supported trajectories.
Let O be the set of functions O deﬁned on S such that O p s q “ arg max a A ‹ p s, a q for all s P S .
The induced distribution of trajectories determines R up to optimality-preserving transformations with O P O .
Note that a mask of the complement of S is not permitted.
However, the fact that O is unconstrained outside S leaves reward effectively unconstrained in those states, except that the reward of transitions out of S may have to “compensate” for the value of their successor states, to prevent new actions that lead out of S from becoming optimal.
3. I NVARIANCES OF T RAJECTORY E VALUATION
The return function, capturing the reward accumulated over a trajectory, is instrumental in deriving data for evaluative feedback such as reward labels and trajectory preference comparisons.
We consider the invariances of the return function for various restricted domains.
Theorem 3.6.
Given an MDP, the return function restricted to possible trajectory fragments, G ζ , determines R up to a mask of impossible transitions;
Theorem 3.7.
Given an MDP, the return function restricted to possible and initial trajectories, G ξ , determines R up to zero-initial potential shaping and a mask of unreachable transitions.
The limited invariance of the return of fragments arises because this restricted domain still includes individual (possible) transitions.
Additional invariances will arise from additional restrictions, such as a minimum or maximum fragment length, or a restriction to initial trajectory fragments.
Pairwise comparisons between trajectories are studied as a data source for reward learning (Akrour et al., 2012; Christiano et al., 2017).
It is common to model the comparisons as based on the return of trajectories, but with accompanying decision noise following a Boltzmann distribution .
Under this assumption, in the limit of inﬁnite noisy comparisons for each pair of trajectories, the data source reveals the Boltzmann distributions.
Boltzmann noise encodes relative cardinal information about the return of trajectories and fragments, so little invariance is introduced.
Formally, given an MDP and an inverse temperature parameter β ą 0 , let ĺ ζβ be a distribution over each pair of possible trajectory fragments, ζ 1 , ζ 2 , such that
[Equation]
and let ĺ ξβ be the analogous distribution over each pair of possible and initial trajectories.
Theorem 3.8.
Given an MDP, the distribution of comparisons of possible trajectory fragments, ĺ ζβ , determines R up to a mask of impossible transitions.
Theorem 3.9.
Given an MDP, the distribution of comparisons of possible and initial trajectories, ĺ ξβ , determines R up to k -initial potential shaping and a mask of unreachable transitions.
The limited invariance of Boltzmann comparisons of fragments arises from the very ﬂexible comparisons permitted, including, for example, comparisons between individual transitions and empty trajectories.
Additional invariances will arise from additional restrictions, such as permitting comparisons only between fragments of a ﬁxed length.
Moreover, it is worth reiterating that these invariances rely heavily on the precise structure of the decision noise revealing cardinal information in the inﬁnite-data limit.
It is also possible to model trajectory comparisons as noiseless comparisons based on the return.
The inﬁnite data limit then corresponds to the order induced by the return functions.
Formally, deﬁne the noiseless order of possible trajectory fragments as a relation, ĺ ζ ‹ , on possible trajectory fragments:
[Equation]
Similarly, deﬁne the noiseless order of possible and initial trajectories as the analogous relation, ĺ ξ ‹ , for pairs of possible and initial trajectories.
These relations omit cardinal information about pairwise comparisons, and so invariances to certain monotonic transformations are introduced.
The precise monotonic invariances depend on the MDP (for example, see the proof in appendix B.3).
Theorem 3.10.
We have the following bounds on the invariances of the noiseless order of possible trajectory fragments, ĺ ζ ‹ .
In all MDPs:
(1) ĺ ζ ‹ is invariant to positive linear scaling and a mask of impossible transitions; and (2) ĺ ζ ‹ is not invariant to transformations other than zero-preserving monotonic transformations or masks of impossible transitions.
Moreover, there exist MDPs attaining each of these bounds.
We give a lower bound on the invariances of the noiseless order of possible and initial trajectories, ĺ ξ ‹ .
Since ĺ ξ ‹ can be derived from ĺ ξβ , it inherits the latter’s invariances.
Moreover, like ĺ ζ ‹ , ĺ ξ ‹ is always invariant to positive linear scaling.
Theorem 3.11.
Given an MDP, the noiseless order of possible and initial trajectories, ĺ ξ ‹ , is invariant to k -initial potential shaping, positive linear scaling, and a mask of unreachable transitions.
3. I NVARIANCES OF P OLICY O PTIMISATION
The primary application of learnt reward functions is to compute optimal policies, using techniques such as RL.
Policy optimisation procedures typically compute a single optimal policy.
However, in terms of invariances, one may desire to preserve the whole set of optimal policies, so as not to tolerate any sub-optimal policies becoming optimal through a reward transformation.
The set of optimal policies inherits S 1 -redistribution invariance from the optimal Q -function, and is also known to be invariant to potential shaping (Ng et al., 1999).
In fact, because a maximally supportive optimal policy can be derived from the set of optimal policies and vice versa , the set shares the same invariances as a maximally supportive optimal policy (Theorem 3.3).
Theorem 3.12.
Given an MDP, let O p s q “ arg max a A ‹ p s, a q .
Then the set of optimal policies determines R up to optimality-preserving transformations with O .
Moreover, if one uses an algorithm not guaranteed to ﬁnd a globally optimal policy, one may desire to preserve the entire order induced on the space of policies by the policy evaluation function, rather than just the set of maximising policies.
Future work could investigate the invariances of the ordinal information in the policy evaluation function.
Note that since the set of optimal policies can be derived from this order, the order has at most the invariances of the set of optimal policies.
Finally, we sketch some bounds on the invariances of the set of optimal policies across all MDPs .
Potential shaping and linear scaling preserve optimal policies in each MDP, and hence in all MDPs.
S 1 -redistribution and optimality-preserving transformations for a given MDP might not.
Moreover,
Theorem 3.12 implies that any transformation that is not an optimality-preserving transformation in a given MDP cannot preserve optimal policies in that MDP, let alone all MDPs.
I MPLICATIONS FOR R EWARD L EARNING
So far we have catalogued the invariances of transformations to the reward function of various reward function derived objects.
These invariances characterise the inﬁnite-data ambiguity of several reward learning data sources, and the ambiguity tolerance of policy optimisation.
In this section, we discuss the implications for the practical evaluation of reward learning data sources.
We begin by deﬁning a mathematical framework for comparisons between data sources and applications in terms of their ambiguity.
The characterisation of ambiguity and tolerance as invariances to reward transformations suggests a natural partial order on data sources and applications.
Recall that the invariances of an object correspond to a partition of the space of reward functions (section 2).
We lift the reﬁnement relation for partitions (Aigner, 1979, §I.2.B) to data sources and applications as follows.
Deﬁnition 4.1 (Ambiguity reﬁnement) .
Consider two reward learning data sources (or applications), X and Y .
Let Π X
and Π Y be the partitions of the space of reward functions corresponding to their respective invariances (deﬁnition 2.1).
If Π X is a partition reﬁnement of Π Y , we write X ĺ Y , and we say X is no more ambiguous than Y (or X is tolerable for application Y ).
If X ĺ Y but not
Y ĺ X , then we write
Given two data sources X and Y , X ĺ Y corresponds to X conﬂating no additional reward functions compared to Y in the inﬁnite-data limit.
This is the sense in which we say X is no more ambiguous than Y .
Moreover, given a downstream application Z , X ĺ Z is precisely the condition of Z tolerating the inﬁnite-data ambiguity of data source X : X ĺ Z if and only if the reward functions conﬂated by X in the inﬁnite-data limit all lead to the same outcome in Z .
Of our fundamental reward functions transformations, there are several clear instances of ambiguity reﬁnement in a given MDP, as summarised in ﬁgure 1b.
Invariance to k -initial potential shaping ( k - Φ ) corresponds to less ambiguity than general potential shaping ( Φ ).
Likewise positive linear scaling (PLS) is less ambiguous than zero-preserving monotonic transformations (ZPMT), and a mask of impossible transitions is less ambiguous than S ’-redistribution ( S 1 -R).
All of these transformations other than zero-preserving monotonic transformations are less ambiguous than the optimality-preserving transformations we have encountered.
More concretely, we can compare the ambiguity of speciﬁc data sources.
Some of these comparisons are indicated in ﬁgure 1a.
For example, the ambiguity tolerance of the set of optimal policies is a class of optimality-preserving transformations.
Each of the data sources that are less ambiguous than this tolerance are sufﬁcient for policy optimisation.
Notably, this excludes noiseless comparisons between trajectory fragments in some MDPs.
Specifically, policy optimisation does not, in general, tolerate zero-preserving monotonic transformations (ZPMT), while noiseless comparisons are invariant to this transformation in some MDPs (Theorem 3.10).
Policy optimisation also does not tolerate data sources based on possible and initial trajectories, which are invariant to a mask of unreachable transitions.
However, these sources are tolerable if the application only requires optimal behaviour in reachable states.
Moreover, we can compare data sources drawn from one MDP to applications in another MDP, such as under a shift in transition dynamics or initial state distribution.
This captures the common sim-to-real setting where learning takes place in a simulated or otherwise restricted environment that differs from the ﬁnal deployment environment.
The simplest transformations to consider are masks of possible or reachable transitions.
These are parametrised by transition dynamics.
In general, the ambiguity corresponding to a mask of X is less than for a mask of X 1 Ą X .
For example, if the new dynamics supports transitions that were previously impossible, then sources with invariance to a mask from the original MDP may not be tolerable for applications in the new MDP.
A similar results holds for S 1 -redistribution, which involves an expectation over MDP dynamics.
As an extreme example, we prove that when the transition dynamics are changed for every state and action, S 1 -redistribution under the original dynamics permits an arbitrary Q -function under the new dynamics.
Naturally, data sources derived from Q -functions may also be affected by shifts in dynamics.
Note that this strong result relies on the formulation of rewards as depending on the successor-state (cf. appendix C).
Theorem 4.1.
Consider an MDP p S , A , τ, µ 0 , R, γ q , a policy π , and alternative transition dynamics τ 1 with τ p s, a q ‰ τ 1 p s, a q for all s P S , a P A .
Given a function Q 1 : S ˆ A Ñ R , there exists a reward function R 1 , produced from R by S 1 -redistribution under τ , such that Q 1 is the Q -function for π under R 1 and τ 1 .
Ambiguity reﬁnement is a partial order, and some data sources are indeed incomparable .
In consolation, we observe that such incomparable ambiguity is complementary ambiguity, in that by combining the associated data sources, we reduce overall ambiguity about the latent reward.
Theorem 4.2.
Given data sources X and Y , let p X, Y q denote the combined data source formed from X and Y .
If X and Y are incomparable, then p X, Y q ă X and p X, Y q ă Y .
This perspective highlights promising directions for the design of reward learning data sources.
In particular, this suggests developing reward learning algorithms for mixtures of data sources with complementary ambiguity.
Unfortunately, most popular data sources actually appear to have similar kinds of ambiguity given one MDP.
However, ambiguity could be reduced by incorporating data from multiple MDPs , along the lines of Amin et al.
(2017) and Cao et al.
L IMITATIONS AND F UTURE W ORK
Our results give an upper bound on the amount of information that can be extracted from a given data source.
However, in practice, these bounds may not be reached.
In particular, our results are for the limit of inﬁnite data.
But in practice data sets are ﬁnite and, when data collection is expensive, may be fairly small.
An important direction for future work is to characterise how much information is contained in data sets of varying sizes and data sources.
This would enable practitioners to determine the most sample efﬁcient data source for a ﬁxed data collection budget.
Furthermore, our results rely on the data being generated according to the process assumed by the reward learning algorithm.
However, most popular approaches are a poor ﬁt for human data (Orsini et al., 2021).
For example, human demonstrations are rarely perfectly optimal or Boltzmannrational.
Moreover, there is often a trade-off between how informative a data source is and how easy it is for a user to provide data.
As an extreme example, a user directly specifying the target reward function is maximally informative – if users could complete such a task correctly.
We expect the maximum informativeness of a data source to be a useful metric, but it should be considered alongside the cost and tractability of collecting different kinds of data.
C ONCLUSION
Substantial effort has been invested to develop reward learning algorithms for a variety of data sources.
A fundamental question to ask is how effective are these algorithms relative to an optimal algorithm for that data source?
Our results characterise the information available in different data sources, enabling algorithms to be compared to this theoretical upper bound.
Moreover, our framework enables direct comparisons between different data sources.
We ﬁnd that some data sources are strictly less informative than others, such as noiseless preference comparisons vs. return labels.
By contrast, others are incomparable and have complementary ambiguity, such as
Q -values (invariant to S 1 -redistribution but not potential shaping) vs. episode return G ξ (invariant to some potential shaping, but not to S 1 -redistribution).
In particular, we have characterised the invariances of various reward-related objects to transformations such as potential shaping.
We have shown that these objects form a partial order under ambiguity reﬁnement .
These results, summarised in ﬁgure 1, allow us to predict the ambiguity of data sources generated from these objects.
While practitioners could simply collect data from the least ambiguous source, this might be very expensive.
Our framework also identiﬁes the ambiguity tolerance of downstream applications (such as policy optimisation) that need to compute these objects.
This enables practitioners to identify reward learning data sources with low ambiguity in the areas their application is sensitive to, enabling higher performance without unnecessary costs.
E THICS S TATEMENT
It is important that AI systems are aligned with the interests of users and other stakeholders.
In openended problems, directly specifying how the AI system should behave is intractable.
Prior work has identiﬁed reward learning as an essential building block for AI systems that can cooperate with humans (Dafoe et al., 2020, §4.1.3), especially for powerful AI systems (Bostrom, 2014, chapter 12).
We hope that our work provides greater clarity on both the limits and potential of various reward learning data sources.
However, given the importance of the domain, we should stress that our work provides only one useful angle by which to evaluate data sources.
In particular, we do not consider sample efﬁciency, robustness to misspeciﬁcation, or the cost of data collection.
Moreover, even if a theoretically optimal and practically robust reward learning algorithm were to be developed, there would still remain important normative questions.
In particular, what kinds of values we are aligning the AI system to – stated preferences, revealed preferences, instructions, or something else (Gabriel, 2020)?
Additionally, it is important that all relevant stakeholders are able to provide input into the system.
This may constrain the kinds of data we can collect.
For example, while only task experts might be able to provide demonstrations, a wider variety of stakeholders might be able to provide preference comparisons.
While a thorough evaluation of these considerations are beyond the scope of this paper, we would encourage practitioners to evaluate reward learning data sources holistically, including but not wholly relying on our results.
R EPRODUCIBILITY S TATEMENT
Our results are all theoretical in nature.
We introduce notation and other background material in sections 1.2 and 2.
Necessary assumptions are listed there and in each theorem statement.
Proofs for some fundamental lemmas are provided in appendix A and all other proofs are in appendix B
A P ROPERTIES OF F UNDAMENTAL R EWARD T RANSFORMATIONS
We begin with some supporting results concerning the basic reward transformations, used in section 3 to characterise the invariances of various objects derived from the reward function.
The following result captures how potential shaping affects various reward-related functions.
Lemma A.1.
Consider M and M 1 , two MDPs differing only in their reward functions, respectively
R and R 1 .
Denote the return, Q -, value, and policy evaluation functions of M 1 by G 1 , Q 1 π , V 1 π , and
J 1 .
If R 1 is produced by potential shaping of R with a potential function Φ , then:
(1) for a trajectory fragment ζ “ p s 0 , a 0 , s 1 , . .
, s n q , G 1 p ζ q “ G p ζ q ` γ n Φ p s n q ´ Φ p s 0 q ; (2) for a trajectory ξ “ p s 0 , a 0 , . . .
q , G 1 p ξ q “ G p ξ q ´ Φ p s 0 q ; (3) for a state s P S , and action a P A , Q 1 π p s, a q “ Q π p s, a q ´ Φ p s q ; (4) for a state s P S , V
π p s q “ V π p s q ´ Φ p s q ; and (5) for a policy π , J 1 p π q “ J p π q ´ E S 0 „ µ“ Φ p S 0 q ‰ .
Proof.
(1) is given by straightforward induction on the length of ζ .
For (2), take the limit as the length of a preﬁx goes to inﬁnity, whereupon γ n Φ p s n q goes to zero ( γ ă 1 by deﬁnition, and Φ p s n q is bounded since its domain is ﬁnite).
(3) and (4) were proved for optimal policies by Ng et al.
and they also observed that the extension to arbitrary policies is straightforward (following immediately from (2), for example).
(5) is immediate from (4).
The following results explain how k -initial potential shaping and linear scaling of R correspond to afﬁne transformations of G .
Lemma A.2.
Let p S , A , τ, µ 0 , R, γ q be an MDP, R 1 a reward function, and k P R a constant.
Then we have that G 1 p
ξ q “ G p ξ q´ k for all possible and initial trajectories ξ , if and only if R 1 is produced from R by k -initial potential shaping and a mask of unreachable transitions.
Proof.
The converse follows from Lemma A.1 and that varying the reward for unreachable transitions does not affect the return of any possible, initial trajectories, by deﬁnition.
In the forward direction, consider an arbitrary reachable state s P S .
Let ζ be a possible, initial, length n trajectory fragment ending in s (at least one exists, since s is reachable).
Let ξ s be some possible trajectory starting in s , and let ∆ ξ s “ G p ξ s q ´ G 1 p ξ s q .
Let ζ ` ξ s denote the concatenation of ζ and ξ s .
Since ζ ` ξ s is an initial trajectory, we have by assumption G p ζ ` ξ s q´ G 1 p ζ ` ξ s q “ k .
Moreover, note that G p ζ ` ξ s q “ G p ζ q ` γ n G p ξ s q , (and likewise for G 1 ), by deﬁnition of return.
Then (recalling that we have deﬁned γ ą 0 ),
[Equation]
Therefore ∆ ξ s is independent of ξ s , except for possible dependence on ξ s ’s starting state s .
We associate a unique P p s q “ ∆ ξ s with each reachable s .
Now consider a reachable transition p s, a, s 1 q .
Further straightforward algebraic manipulation shows that R 1 p s, a, s
[Equation]
[Equation] [Equation]
P p s q .
Moreover, from the deﬁnition of terminal states we must have that P p s q “ ∆ ξ s “ 0 for terminal s , and by assumption we have that P p s q “ ∆ ξ s “ k if s is initial.
So R 1 is given by k -initial potential shaping of R with Φ p s q “ P p s q for reachable transitions.
Any variation in reward for unreachable transitions can be accounted for by the mask.
Lemma A.3.
Let p S , A , τ, µ 0 , R, γ q be an MDP, R 1 a reward function, and c P R a constant.
Then
G 1 p ξ q “ c ¨ G p ξ q for all possible initial trajectories ξ , if and only if R 1 is produced from R by zeroinitial potential shaping, linear scaling by a factor of c , and a mask of all unreachable transitions.
Proof.
From Lemma A.1, we have that 0-initial potential shaping leaves G unchanged.
Likewise, a mask of unreachable transitions does not affect G for possible initial trajectories.
Moreover, it is straightforward that linear scaling of R by a factor of c leads to a linear scaling of G by the same factor of c .
Hence the converse is established.
For the forward direction, consider an arbitrary reachable state s P S .
Let ξ s be some possible trajectory starting in s , and let ∆ ξ s “ c ¨ G p ξ s q ´ G 1 p ξ s q .
Let ζ be a possible, initial, length n trajectory fragment ending in s .
Since the concatenation ζ ` ξ s is a possible initial trajectory, we have that
[Equation]
Therefore ∆ ξ s is independent of ξ s , except for possible dependence on ξ s ’s starting state s .
We associate a unique P p s q “ ∆ ξ s with each reachable s .
Now consider a reachable transition p s, a, s 1 q .
Further straightforward algebraic manipulation shows that R 1 p s, a, s 1 q “ c ¨ R p s, a, s 1 q` γ ¨ P p s 1 q´ P p s q .
Moreover, from the deﬁnition of terminal states we must have that P p s q “ ∆ s “ 0 for terminal s , and by assumption we have that P p s q “ 0 if s is initial.
Thus R 1 is given by ﬁrst potential shaping R with Φ p s q “ 1 c P p s q , and then linear scaling with c .
B P ROOFS
We provide proofs for the theoretical results presented in the main paper.
B. P ROOFS FOR S ECTION 3.1 R ESULTS N OT C ONCERNING O PTIMAL P OLICIES
We provide proofs for some of the results in section 3.1.
The proofs relating to optimal policies (theorems 3.3 and 3.5) are given in the next subsection (appendix B.2).
Theorem 3.1.
Given an MDP and a policy π , the Q -function for π , Q π , determines R up to S 1redistribution.
The optimal Q -function, Q ‹ , has precisely the same invariances.
Proof.
Q π satisﬁes the Bellman equation for all s P S , a P A :
[Equation]
This equation can be rewritten as
[Equation]
From this, we can see that Q π is invariant to S 1 -redistribution of R , and no other transformations.
Q ‹ “ Q π ‹ where π ‹ is any optimal policy derived from Q ‹ , so the invariances of the optimal
Q -function follow as a special case.
Theorem 3.2.
Given an MDP, an inverse temperature parameter β , and a base policy π 0 , the
Boltzmann policy π π 0 β determines R up to S 1 -redistribution and potential shaping.
The Boltzmannrational policy, π ‹ β , and the Maximum Entropy policy, π β , have precisely the same invariances.
Proof.
By equation (2), π π 0 β can be derived from A π 0 .
A π 0 itself can be derived from Q π 0 , given π 0 (by equation (1), A π 0 p s, a q “ Q
[Equation]
s,
[Equation]
Q π 0 p s, A q ‰ ).
Thus π π 0 β is invariant to S 1redistribution by Theorem 3.1.
Moreover, by Lemma A.1, potential shaping causes a state-dependent shift of Q π 0 .
This shift does not affect A π 0 .
Therefore π π 0 β is also invariant to potential shaping.
Conversely, recall (or see Lemma B.1, below) that the softmax function is invariant only to constant shifts.
This means that π π 0 β is invariant to all and only those transformations of R that produce statedependent shifts in Q π .
Let B : S Ñ R , and suppose R and R 1 are two reward functions such that the corresponding Q -functions (for π ) satisfy Q 1 p s, a q “ Q p s, a q ` B p s q .
Then
[Equation]
where the expectations are over S 1 „ τ p s, a q .
Now set Φ p s q “ ´ B p s q , and we can see that the difference between R and R 1 is described by potential shaping and S 1 -redistribution.
The Boltzmann-rational policy determines its own base policy (it is enough to determine the optimal
Q -function up to a state-dependent shift), and the Maximum Entropy policy is its own base policy, so the invariances of these policies follow as a special case of this result.
Lemma B.1.
Consider two functions f : X Ñ R and g : X Ñ R deﬁned on a ﬁnite set X .
Then g is constant if, for all x P X ,
[Equation]
Proof.
This is an elementary property of the softmax function.
It can be seen as follows:
[Equation]
which is constant in x .
(The converse is true, but we do not make use of it, so we omit a proof.)
Theorem 3.4.
Given an MDP, an inverse temperature parameter β , and a base policy π 0 , the distribution of trajectories induced by the Boltzmann policy π π 0 β from all initial states determines R up to S 1 -redistribution, potential shaping, and a mask of unreachable transitions.
The distributions of trajectories induced by the Boltzmann-rational policy, π ‹ β , and the Maximum Entropy policy, π β , from all initial states, have precisely the same invariances.
Proof.
That the distribution is invariant to S 1 -redistribution and potential shaping follows from Theorem 3.2.
The distribution is also invariant to changes in the reward for transitions out of unreachable states, since these rewards cannot affect the policy for reachable states.
As a result the distribution is additionally invariant to a mask of unreachable transitions.
The trajectory distribution can be factored into the separate distributions π π 0 β p s q P ∆ p
A q for each reachable state s , by conditioning on a supported preﬁx trajectory fragment that leads to s and marginalising over subsequent states and actions.
Via a similar argument to the proof of Theorem 3.2, the distribution determines the reward function for transitions (out of these reachable states) up to potential shaping and transformations and S 1 -redistribution (as they affect reachable states).
As for Theorem 3.2, the invariances for the Boltzmann-rational policy and the Maximum Entropy policy arise as special cases.
B. P ROOFS FOR S ECTION 3.1 R ESULTS C ONCERNING O PTIMAL P OLICIES
Our results concerning the invariance of optimal policies and their trajectories follow from the following general result:
Lemma B.2.
Given an MDP M , suppose we have access to the set of optimal actions for each state in a subset of states S Ď S .
Assume that this subset is closed under optimal actions given the transition dynamics of M (that is, taking optimal actions from states in S never leads to states outside of S ).
Let O be the set of functions O deﬁned on S such that O p s q “ arg max a P A A ‹ p s, a q for all s P S (but O is unconstrained for states outside S ).
Then these optimal action sets determine
R up to optimality-preserving transformations with O P O .
Proof.
Suppose R 1 is obtained from M ’s reward R via an optimality-preserving transformation with some O P O .
Let Ψ be the corresponding value-bounding function, that is, a function Ψ : S Ñ R satisfying, for all s P S and a P A ,
[Equation]
τ p s
[Equation]
with equality if and only if a P O p s q .
Since O p s q is non-empty (by deﬁnition), we have for all s P S
[Equation]
This recursive condition on Ψ is the Bellman optimality equation for the unique optimal value function, V 1 ‹ , of the MDP with transformed reward R 1 .
Therefore, Ψ p s q “ V 1 ‹ p s q for all s P S , and we can rewrite equation (3) as
[Equation]
with equality only for a P O p s q .
Now, consider a state s P S .
By assumption, for this s , O p s q “ arg max a P A A ‹ p s, a q .
Then for this state, the actions that attain the optimal value bound in equation (4) are these same optimal actions.
Therefore, R 1 induces the same sets of optimal actions from states in S .
Conversely, consider a second MDP M 1 , differing from M only in its reward function, R 1 .
Assume the set of optimal actions in states in S agrees with the optimal actions in M for those states.
Let V 1‹ and A 1‹ denote the optimal value and advantage functions for M 1 .
The Bellman optimality equation for M 1 ensures that, for s P S ,
[Equation]
with the maximum attained precisely by the actions a P arg max a P A p
A 1‹ p s, a qq .
Setting O p s q “ arg max a P A p
A 1‹ p s, a qq , equation (5) can be rewritten as
[Equation]
for all s P S and a P A , with equality if and only if a P O p s q .
Now, for s P S , we have arg max a P A p
A 1‹ p s, a qq “ arg max a P A p
A ‹ p s, a qq , because M and M 1 have matching sets of optimal actions for these states (by assumption).
Then equation (6) shows that R 1 is produced from R by an optimality-preserving transformation with O p s q “ arg max a P A p
A 1‹ p s, a qq (and Ψ p s q “ V 1‹ p s q ).
We are now in a position to prove the results from the main text:
Theorem 3.3.
Given an MDP, let O p s q “ arg max a A ‹ p s, a q .
A maximally supportive optimal policy determines R up to optimality-preserving transformations with O .
Proof.
An arbitrary maximally supportive optimal policy determines the set of optimal actions from all states in the MDP.
Its invariances follow as a special case of Lemma B.2, with S “ S .
Theorem 3.5.
Given an MDP, consider the distribution of trajectories induced by a maximally supportive optimal policy.
Let S be the set of states in supported trajectories.
Let O be the set of functions O deﬁned on S such that O p s q “ arg max a A ‹ p s, a q for all s P S .
The induced distribution of trajectories determines R up to optimality-preserving transformations with O P O .
Proof.
The distribution of trajectories can be factored into separate distributions π ‹ p s q P ∆ p
A q for each state s P S (in a manner similar to Theorem 3.4, as proved above).
These individual distributions determine the set of optimal actions within those states.
Noting that S is clearly closed under optimal actions in the MPD since all optimal actions are supported by the policy, the invariance result follows from Lemma B.2.
B. P ROOFS FOR S ECTION 3.2 R ESULTS
Theorem 3.6.
Given an MDP, the return function restricted to possible trajectory
fragments, G ζ , determines R up to a mask of impossible transitions
Proof.
The result is immediate, since the restricted domain still includes all possible transitions (as length one trajectory fragments with return equal to the reward of the transition), and no fragments with impossible transitions.
Theorem 3.7.
Given an MDP, the return function restricted to possible and initial trajectories, G ξ , determines R up to zero-initial potential shaping and a mask of unreachable transitions.
Proof.
The result follows from Lemma A.2 with k “ 0 .
Theorem 3.8.
Given an MDP, the distribution of comparisons of possible trajectory fragments, ĺ ζβ , determines R up to a mask of impossible transitions.
Proof.
Since ĺ ζβ can be derived from G ζ , it is invariant to a mask of impossible transitions by
Theorem 3.6.
Conversely, ĺ ζβ determines R for all possible transitions.
This is because R p s, a, s 1 q is encoded in the Boltzmann distribution of comparisons between the length zero trajectory fragment ζ 0 “ p s q and the length one trajectory fragment ζ 1 “ p s, a, s 1 q , and can be recovered as follows:
[Equation]
Therefore ĺ ζβ is invariant to precisely a mask of impossible transitions.
Theorem 3.9.
Given an MDP, the distribution of comparisons of possible and initial trajectories, ĺ ξβ , determines R up to k -initial potential shaping and a mask of unreachable transitions.
Proof.
Note that as ĺ ξβ can be derived from G ξ , by Theorem 3.7, ĺ ξβ is invariant to 0 -initial potential shaping and a mask of unreachable transitions.
It is additionally invariant to k -initial potential shaping for arbitrary constants k P R , and no other transformations: G ξ can be recovered from ĺ ξβ up to a constant (we can compare all possible initial trajectories to an arbitrary reference trajectory and recover their relative return using a similar manipulation as above, but we can’t determine the return of the reference trajectory).
From there, the precise invariance follows from Lemma A.2.
Theorem 3.10.
We have the following bounds on the invariances of the noiseless order of possible trajectory fragments, ĺ ζ ‹ .
In all MDPs:
(1) ĺ ζ ‹ is invariant to positive linear scaling and a mask of impossible transitions; and (2) ĺ ζ ‹ is not invariant to transformations other than zero-preserving monotonic transformations or masks of impossible transitions.
Moreover, there exist MDPs attaining each of these bounds.
Proof.
For (1), positive linear scaling of reward by a constant c leads to the same scaling of the return of each trajectory fragment, and this always preserves the relation ĺ ζ ‹ , since for any c ą 0 , c
¨ G p ζ 1 q ď c ¨ G p ζ 2 q ô G p ζ 1 q ď G p ζ 2 q for all pairs of trajectory fragments ζ 1 , ζ 2 .
Moreover, ĺ ζ ‹ inherits invariance to a mask of impossible transitions from G ζ (Theorem 3.6).
For (2), let R 1 be produced from R via some transformation that is neither a mask of impossible transitions nor a zero-preserving monotonic transformation.
It must be that either R 1 fails to preserve the ordinal comparison of two possible transitions, or that it fails to preserve the set of zero-reward possible transitions, compared to R .
In the ﬁrst case, consider two possible transitions whose rewards are not preserved, x 1 and x 2 .
Without loss of generality suppose R p x 1 q ď R p x 2 q but R 1 p x 1 q ą R 1 p x 2 q .
This corresponds to a change in ĺ ζ ‹ ’s comparison of the length one trajectories formed from x 1 and x 2 , namely x 1 ĺ ζ ‹ x 2 from true to false.
Similarly, in the second case, the comparisons between the transition whose reward became or ceased to be zero and a length one trajectory (with return 0) will have changed.
Therefore, ĺ ζ ‹ is not invariant to such transformations.
The bound (1) is attained by the following MDP invariant precisely to positive linear scaling and a mask of impossible transitions.
Let S “ t s u , A “ t a 1 , a 2 u , R p s, a 1 , s q “ 1 , and R p s, a 2 , s q “ 1 ` γ .
Since R p s, a 2 , s q “ R p s, a 1 , s q ` γR p s, a 1 , s q , the corresponding order relation will contain both p s, a 2 , s q ĺ ζ ‹ p s, a 1 , s, a 1 , s q and p s, a 1 , s, a 1 , s q ĺ ζ ‹ p s, a 2 , s q .
This property requires that
R p s, a 1 , s q “ p 1 ` γ q ¨ R p s, a 2 , s q , which is preserved only by linear scaling of R .
(Non-positive linear scaling is already ruled out by (2)).
The bound (2) is attained by the following MDP invariant to arbitrary zero-preserving monotonic transformations.
Let S “ t s 1 , s 2 u , A “ t a u , with possible transitions p s 1 , a, s 2 q
and p s 2 , a, s 2 q , and R p s 1 , a, s 2 q ą R p s 2 , a, s 2 q “ 0 .
Any zero-preserving monotonic transformation of R preserves the ordering of all possible trajectory fragments, namely that all nonempty trajectories starting in shave positive return and all other possible trajectories have zero return.
Theorem 3.11.
Given an MDP, the noiseless order of possible and initial trajectories, ĺ ξ ‹ , is invariant to k -initial potential shaping, positive linear scaling, and a mask of unreachable transitions.
Proof.
The pairwise Boltzmann distributions of ĺ ξβ can be used to derive the noiseless comparisons of ĺ ξ ‹ .
Therefore, ĺ ξ ‹ is invariant to k -initial potential shaping and a mask of unreachable transitions.
That ĺ ξ ‹ is also invariant to positive linear scaling follows from a similar argument as for the ﬁrst bound in Theorem 3.10, proved above.
B. P ROOFS FOR S ECTION 3.3 R ESULTS
Theorem 3.12.
Given an MDP, let O p s q “ arg max a A ‹ p s, a q .
Then the set of optimal policies determines R up to optimality-preserving transformations with O .
Proof.
The set of all optimal policies determines a maximally supportive policy, for example by constructing a policy that supports all actions supported by any policy in the set.
Likewise, a maximally supportive policy determines the set of optimal policies, namely as the set of all policies whose support is a subset of the maximally supportive policy.
Therefore, these objects share precisely the same invariances.
B. P ROOFS FOR S ECTION 4 R ESULTS
Theorem 4.1.
Consider an MDP p S , A , τ, µ 0 , R, γ q , a policy π , and alternative transition dynamics τ 1 with τ p s, a q ‰ τ 1 p s, a q for all s P S , a P A .
Given a function Q 1 : S ˆ A Ñ R , there exists a reward function R 1 , produced from R by S 1 -redistribution under τ , such that Q 1 is the Q -function for π under R 1 and τ 1 .
Proof.
Per deﬁnition 2.3, that R 1 is produced from R by S 1 -redistribution under τ requires, for all s P S and a P A ,
[Equation]
Per equation (1), that Q 1 is the Q -function for π under R 1 and τ 1 requires, for all s P S and a P A ,
[Equation]
or, equivalently,
[Equation]
For brevity, denote the right-hand sides of equations (7) and (8) as R p s, a q and Q p s, a q respectively.
Let s P S and a P A .
By assumption, τ 1 p s, a q ‰ τ p s, a q .
Therefore, for at least two s 1 , denoted s 1 1 and s 1 2 , we have τ 1 p s 1 1 | s, a q ă τ p s 1
| s, a q and τ 1 p s 1 2 | s, a q ą τ p s 1 2 | s, a q .
For the remaining s 1 , set
R p s, a, s 1 q “ 0 .
This reduces equations (7) and (8) to the following:
[Equation]
This system has a solution as its determinant is τ 1 p s 1
| s, a q ¨ τ p s
[Equation]
1 | s
[Equation]
a q ´ τ
[Equation]
| s
[Equation]
a q ¨ τ
[Equation]
p s
[Equation]
1 | s
[Equation]
a q
[Equation]
Theorem 4.2.
Given data sources X and Y , let p X, Y q denote the combined data source formed from X and Y .
If X and Y are incomparable, then p X, Y q ă X and p X, Y q ă Y .
Proof.
Transformations that preserve p X, Y q necessarily preserve X , therefore p X, Y q ĺ X .
But since X and Y are incomparable, there is some transformation that preserves X and not Y .
This transformation does not preserve p X, Y q .
Therefore, p X, Y q ă X .
Similarly, p X, Y q ă Y .
We note that the above result is also an elementary consequence of the lattice structure of the partial order of partition reﬁnement (Aigner, 1979, §I.2.B), since the combined data source corresponds to the meet of the original data sources.
C O THER S PACES OF R EWARD F UNCTIONS
Hitherto we have assumed reward functions are members of S
ˆ A
ˆ S Ñ R .
That is, they are deterministic functions of transitions, depending on the state, action and the successor state.
In this appendix, we discuss several alternative spaces of reward functions, and their implications for the invariance properties of various objects derived from the reward function.
C. R ESTRICTED - DOMAIN R EWARD F UNCTIONS
It is common in both reinforcement learning and reward learning to consider less expressive spaces of reward functions.
In particular, the domain of the reward function is often restricted to S or S ˆ A .
When modelling a task, the choice of reward function domain is usually a formality: An MDP taking full advantage of the domain S
ˆ A ˆ S has an ‘equivalent’ MDP with a restricted domain, and some added auxiliary states (Russell & Norvig, 2009, §17).
Conversely, reward functions with restricted domains can be viewed as a special case of functions from S ˆ A ˆ S where the functions are constant in the ﬁnal argument(s).
Restricting the domain can be an appealing simpliﬁcation when modelling a task, hence the popularity of these formulations.
When modelling a data source, this equivalence may not apply: We may not have access to data regarding auxiliary states, and so assuming a restricted domain effectively assumes the latent reward is indeed constant with respect to the successor state (and possibly the action) of each transition.
This assumption may or may not be warranted.
If a restricted domain of S or S ˆ A is preferred, then our invariance results can be adapted in a straightforward manner.
In general, since we are effectively considering a subspace of candidate reward functions for transformations, ambiguity can only decrease.
In particular, these restrictions have two main consequences.
Firstly, the reward function transformation of S 1 -redistribution vanishes to the identity transformation, since it allows variation only in the successor state argument of the reward function, which is now impossible.
This reduces the effective ambiguity of the Q -function and all derivative objects.
Notably, the Q -function uniquely identiﬁes the reward function, and Boltzmann policies have the same invariances as Boltzmann comparisons between trajectories.
Restricting the domain to S means the (state) value function for an arbitrary known policy also uniquely identiﬁes the reward function, but doesn’t otherwise alter the invariances we have explored.
Secondly, for most MDPs, the available potential shaping transformations are restricted, but not eliminated.
The function added in a potential shaping transformation ( γ ¨ Φ p s 1 q ´ Φ p s q ) nominally depends on the successor state of the transition.
Some transformed reward functions may rely on this dependence, falling outside of the restricted domain.
However, some non-zero transformations will usually remain.
For example, in a discounted MDP without terminal states, a non-zero constant potential function Φ p s q “ k does not effectively depend on s , and the reward transformation of adding γ ¨ Φ p s 1 q´
Φ p s q “ p γ ´ 1 q¨ k to a reward function does not introduce a dependence on s 1 .
In general, the set of remaining potential shaping transformations will depend on the network structure of the MDP.
At the extreme, in a deterministic MDP with state-action rewards, all potential shaping transformations are permitted, since a dependence on s 1 can be satisﬁed by a .
C. S TOCHASTIC R EWARD F UNCTIONS
Certain tasks are naturally modelled as providing rewards drawn stochastically from some distribution upon each transition.
An even more expressive space of reward functions than we consider is the space of transition-conditional reward distributions 3 .
Identifying the reward function in this case is more challenging in general because the latent parameter contains a full distribution of information for each input, rather than a single point.
In the spirit of this paper, we sketch a characterisation of this additional ambiguity.
A deterministic reward function can be viewed as the conditional expectation of a reward distribution function.
Taking the expectation of the reward distribution for each transition introduces invariance, since the expectation operation is not injective (except in certain restricted cases such as for parametric families of distributions that can be parametrised by their mean).
The invariance introduced is akin to S 1 -redistribution, but with an expectation over the support of the reward distribution rather than the successor state of each transition.
In the extension of the RL formalism to account for stochastic rewards, this expectation is effectively the ﬁrst step in the derivation of each of the objects we have studied.
Therefore, all of these objects inherit this new invariance.
As a consequence, all data sources are effectively more ambiguous with respect to this new latent parameter.
For example, if optimal comparisons between trajectories are understood to be performed based on the pairwise comparison of the expected return of each individual trajectory, then these comparisons are also invariant to transformations of the reward distributions that preserve their means.
Fortunately, much of reinforcement learning also focuses on expected return and reward in application .
Accordingly, most downstream tasks are tolerant to any ambiguity in the exact distribution of stochastic rewards, beyond identifying the mean.
Since this is the same kind of ambiguity that is introduced by considering the latent parameter of reward learning as a conditional distribution rather than a deterministic function, our results are still informative for these situations.
C. F URTHER S PACES AND F UTURE W ORK
For certain applications, including risk-sensitive RL where non-mean objectives are pursued (Morimura et al., 2010a;b; Dabney et al., 2018), the distribution of stochastic rewards can be consequential.
Moreover, the introduction of stochastic rewards suggests considering data sources based on samples rather than expectations, such as a data source of trajectory comparisons based on sampled trajectory returns.
Characterising the invariances of these objectives to transformations of the reward distribution, and thereby their ambiguity tolerance , is left to future work.
In future extensions of this work to handle continuous MDPs, there will be an opportunity to study the effect of restricting to various parametrised spaces of reward functions.
For example, it is common in reinforcement learning and reward learning to study MDPs with reward functions that are linear in a feature vector associated with each transition.
This kind of restriction may reduce the available reward transformations compared to those available to a non-parametric reward function, in a similar manner to restricting the domain of a ﬁnite reward function as discussed above.
The relaxation of the Markovian assumption also introduces a broader space of reward functions, and with it, new dimensions for transformations and invariance.
As one example, related to potential shaping, the non-Markovian additive transformations studied by Wiewiora et al.
(2003) will amount to new invariances of the optimal policy and other related objects.