Q UANTIFYING D IFFERENCES IN R EWARD F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
For many tasks, the reward function is too complex to be speciﬁed procedurally, and must instead be learned from user data.
Prior work has evaluated learned reward functions by examining rollouts from a policy optimized for the learned reward.
However, this method cannot distinguish between the learned reward function failing to reﬂect user preferences, and the reinforcement learning algorithm failing to optimize the learned reward.
Moreover, the rollout method is highly sensitive to details of the environment the learned reward is evaluated in, which often differ in the deployment environment.
To address these problems, we introduce the
Equivalent-Policy Invariant Comparison (EPIC) distance to quantify the difference between two reward functions directly, without training a policy.
We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy.
Furthermore, we ﬁnd EPIC can be precisely approximated and is more robust than baselines to the choice of visitation distribution.
Finally, we show that EPIC distance bounds the regret of optimal policies even under different transition dynamics, and conﬁrm empirically that it predicts policy training success.
Our source code is available at — double blind: supplementary material —.
I NTRODUCTION
Reinforcement learning (RL) has reached or surpassed human performance in many domains with clearly-deﬁned reward functions, such as games [19; 14; 22] and narrowly-scoped robotic manipulation tasks [15].
Unfortunately, the reward functions for most real-world tasks are difﬁcult or impossible to procedurally specify.
Even a task as simple as peg insertion from pixels has a non-trivial reward function that must usually be learned [21, IV.A].
Most real-world tasks have far more complex reward functions than this.
In particular, tasks involving human interaction depend on complex and user-dependent preferences.
These challenges have inspired work on learning a reward function, whether from demonstrations [12; 16; 25; 8; 3], preferences [1; 24; 6; 17; 26] or both [10; 4].
Prior work usually evaluates the learned reward function ˆ R using the “rollout method”: training a policy π ˆ R to optimize ˆ R and then examining rollouts from π ˆ R .
Unfortunately, this method is computationally expensive because it requires us to solve an RL problem.
Furthermore, the rollout method produces false negatives when the reward ˆ R matches user preferences, but the RL algorithm fails to maximize ˆ R .
The rollout method also produces false positives : of the many reward functions inducing the desired rollout in a given environment, only a small subset align with the user’s preferences.
If the initial state distribution or transition dynamics change, misaligned rewards may induce undesirable policies.
Reinforcement learning is founded on the observation that it is usually easier and more robust to specify a reward function, rather than a policy maximizing that reward function.
Applying this insight to reward function analysis, we develop methods to compare reward functions directly , without training a policy.
We summarize our desiderata for reward function distances in Table 1.
We introduce the Equivalent-Policy Invariant Comparison (EPIC) pseudometric that meets all ﬁve desiderata.
EPIC (section 4) canonicalizes the reward functions’ potential-based shaping, then computes the correlation between the canonical rewards over a visitation distribution D of transitions.
For comparison, we also propose two baselines (section 5), Episode Return Correlation (ERC) and
Nearest Point in Equivalence Class (NPEC) , which partially satisfy the desiderata.
[Table]
EPIC works best when D has support on all realistic transitions.
In our experiments, we achieve this by using uninformative priors, such as a uniform distribution over transitions.
Moreover, we ﬁnd EPIC is robust to the exact choice of distribution D , producing similar results across a range of distributions, whereas ERC and especially NPEC are highly sensitive to the choice of D (section 6.3).
Reward learning algorithms are typically benchmarked on tasks with a known ground-truth reward function R .
When using the rollout method, it is common to report the regret : how much less true reward R is obtained by a policy π ˆ R optimized for the learned reward ˆ R versus a policy π R optimized for R .
Theorem 4.9 shows that reward functions with low EPIC distance to the true reward R induce optimal policies with low regret even in unseen environments.
We also conﬁrm empirically that this result holds in practice (section 6.2).
However, while reward functions with high EPIC distance tend to fail in the test environment, they sometimes still work in the training environment.
EPIC therefore has a lower false positive rate than the rollout method, making it particularly attractive in safety-critical applications where reliability is a key design goal.
R ELATED WORK
There exists a variety of methods to learn reward functions.
One prominent family is inverse reinforcement learning (IRL; 12), which infers a reward function from demonstrations.
The IRL problem is inherently underconstrained: many different reward functions can lead to the same demonstrations.
Bayesian IRL [16] handles this ambiguity by inferring a posterior over reward functions.
By contrast, Maximum Entropy IRL [25] selects the highest entropy reward function consistent with the demonstrations; this method has scaled to high-dimensional environments [7; 8].
An alternative approach is to learn from preference comparisons between two trajectories [1; 24; 6; 17].
T-REX [4] is a hybrid approach, learning from a ranked set of demonstrations.
More directly, Cabi et al.
[5] learn from “sketches” of cumulative reward over an episode.
To the best of our knowledge, there is no prior work that focuses on evaluating reward functions directly.
The most closely related work is Ng et al.
[13], identifying reward transformations guaranteed not to change the optimal policy.
However, a variety of ad-hoc methods have been developed to evaluate reward functions.
The rollout method – evaluating rollouts of a policy trained on the learned reward – is evident in the earliest work on IRL [12].
Fu et al.
[8] reﬁned the rollout method by testing on a transfer environment, inspiring our experiment in section 6.2.
Recent work has compared reward functions by scatterplotting returns [10; 4], inspiring our ERC baseline (section 5.1).
B ACKGROUND
This section introduces material needed for the distances deﬁned in subsequent sections.
We start by deﬁning a distance metric , then introduce the Markov Decision Process (MDP) formalism, and ﬁnally describe when reward functions induce the same optimal policy in any compatible MDP.
Deﬁnition 3.1.
Let X be a set and d : X × X → [0 , ∞ ) a function.
d is a premetric if d ( x, x ) =for all x ∈ X .
d is a pseudometric if, furthermore, for all x, y, z ∈ X , d ( x, y ) =
d ( y, x ) and d ( x, z ) ≤ d ( x, y ) + d ( y, z ) .
d is a metric if, furthermore, for all x, y ∈ X , d ( x, y ) =
⇐⇒ x = y .
We wish for d ( R A , R B ) = 0 when reward functions R A and R B are in the same equivalence class, even if R A (cid:54) =
R B .
This is forbidden in a metric but permitted in a pseudometric, while retaining other guarantees such as symmetry and triangle inequality that a metric provides.
Accordingly, a pseudometric is usually the best choice for a distance d over reward functions.
Deﬁnition 3.2.
A Markov Decision Process (MDP) M = ( S , A , γ, µ, T , R ) consists of a set of states S and a set of actions A ; a discount factor γ ∈ [0 , 1] ; an initial state distribution µ ( s ) ; a transition distribution T ( s (cid:48) | s, a ) specifying the probability of transitioning to s (cid:48) from s after taking action a ; and a reward function R ( s, a, s (cid:48) ) specifying the reward upon taking action a in state s and transitioning to state s (cid:48) .
A trajectory τ consists of a sequence of states and actions, τ = ( s 0 , a 0 , s 1 , a 1 , · · · ) , where each s i ∈ S and a i ∈ A .
The return on a trajectory is deﬁned as the sum of discounted rewards, g ( τ ; R ) =
(cid:80) | τ | t =
γ t R ( s t , a t , s t +1 ) , where the length of the trajectory | τ | may be inﬁnite.
In the following, we assume a discounted ( γ < 1 ) inﬁnite-horizon MDP.
The results can be generalized to undiscounted ( γ = 1 ) MDPs subject to regularity conditions needed for convergence.
A stochastic policy π ( a | s ) assigns probabilities to taking action a ∈ A in state s ∈ S .
The objective of an MDP is to ﬁnd a policy π that maximizes the expected return, G ( π ) =
E τ ( π ) [ g ( τ ; R )] , where τ ( π ) is a trajectory generated by sampling the initial state s 0 from µ , each action a t from the policy π ( a t
| s t ) and successor states s t +1 from the transition distribution T ( s t +1 | s t , a t ) .
An MDP M has a set of optimal policies π ∗ ( M ) that maximize the expected return, π ∗ ( M )
=
arg max π G ( π ) .
In this paper, we consider the setting where we only have access to an MDP\R, M − =
( S , A , γ, µ, T ) .
The unknown reward function R must be learned from human data.
Typically, only the state space
S , action space A and discount γ are known exactly, with the initial state µ and transition dynamics
T only observable from interacting with the environment M − .
In the following, we describe an equivalence class whose members are guaranteed to have the same set of optimal policies in any
MDP\R M − with ﬁxed S , A and γ (allowing the unknown T and µ to take arbitrary values).
Deﬁnition 3.3.
A potential shaping reward is deﬁned as R ( s, a, s (cid:48) )
= γ Φ( s (cid:48) ) − Φ( s ) , given a potential Φ : S → R and where γ is the MDP discount rate.
Deﬁnition 3.4 (Reward Equivalence) .
We deﬁne two bounded reward functions R A and R B to be equivalent , R A ≡ R B , for a ﬁxed ( S , A , γ ) if and only if there exists a constant λ > 0 and a bounded potential function Φ : S → R such that for all s, s (cid:48) ∈ S and a ∈ A :
[Equation]
Note R
A − R B ≡ Zero (where Zero is the all-zero reward)
if and only if R A ≡ R B with λ = 1 .
Proposition 3.5.
The binary relation ≡ is an equivalence relation.
Let R A , R B , R C : S×A×S → R be bounded reward functions.
Then ≡ is reﬂexive, R A ≡ R A ; symmetric, R A ≡ R B implies
R B ≡ R A ; and transitive, ( R A ≡ R B )
∧ ( R B ≡ R C ) implies R A ≡ R C .
Proof.
See section A.3.1 in supplementary material.
The expected return of potential shaping γ Φ( s (cid:48) ) − Φ( s ) on a trajectory segment ( s 0 , · · · , s T ) is γ T Φ( s T ) − Φ( s 0 ) .
The ﬁrst term γ T Φ( s T ) → 0 as T → ∞ , while the second term Φ( s 0 ) only depends on the initial state, and so potential shaping does not change the set of optimal policies [13].
Scaling a reward function by a positive factor λ > 0 scales the expected return of all trajectories by λ , leaving the set of optimal policies unchanged.
The set of optimal policies is also invariant to a constant shift c ∈ R of the reward, however this can already be obtained by shifting Φ by cγ − 1 .
If R A ≡ R B , for a ﬁxed ( S , A , γ ) , then for any MDP\R M − = ( S , A , γ, µ, T ) we have π ∗ (( M − , R A ))
=
π ∗ (( M − , R B )) , where ( M − , R ) denotes the MDP speciﬁed by M − with reward function R .
In other words, R A and R B induce the same optimal policies for all initial state distributions µ and transition dynamics T .
C OMPARING REWARD FUNCTIONS WITH EPIC
In this section we introduce the Equivalent-Policy Invariant Comparison (EPIC) pseudometric.
This novel distance canonicalizes the reward functions’ potential-based shaping, then compares the canonical representatives using Pearson distance, which is invariant to scale.
Together, this construction makes EPIC invariant on reward equivalence classes.
See section A.3.2 for proofs.
We deﬁne the canonically shaped reward C D S , D A ( R ) as an expectation over some arbitrary distributions D S and D A over states S and actions A respectively.
This construction means C D S , D A ( R ) only depends on ( S , A , γ ) , and not on the initial state distribution µ or transition dynamics T .
In particular, we may evaluate R on transitions that are impossible in the training environment, since these may become possible in a deployment environment with a different µ or T .
Deﬁnition 4.1 (Canonically Shaped Reward) .
Let R : S ×
A × S → R be a reward function.
Given distributions D S ∈ ∆( S ) and D A ∈ ∆( A ) over states and actions, let S and S (cid:48) be random variables independently sampled from D S and A sampled from D A .
We deﬁne the canonically shaped R to be:
[Equation]
[Equation]
Informally, if R (cid:48) is shaped by potential Φ , then increasing Φ( s ) decreases R (cid:48)
( s, a, s (cid:48) ) but increases E [ − R (cid:48)
( s, A, S (cid:48) )] , canceling.
Similarly, increasing Φ( s (cid:48) ) increases R (cid:48)
( s, a, s (cid:48) )
but decreases E [ γR (cid:48)
( s (cid:48) , A, S (cid:48) )] .
Finally, E [ R ( S, A, S (cid:48) )] centers the reward, canceling constant shift.
Proposition 4.2 (The Canonically Shaped Reward is Invariant to Shaping) .
Let R : S × A × S → R be a reward function and Φ : S → R a potential function.
Let γ ∈ [0 , 1] be a discount rate, and
D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions.
Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s (cid:48) )
=
R ( s, a, s (cid:48) )
+ γ Φ( s (cid:48) )
− Φ( s ) .
Then the canonically shaped R (cid:48) and R are equal:
C D S , D A
( R (cid:48) )
=
C D S , D A
( R ) .
Proposition 4.2 holds for arbitrary distributions D S and D A .
However, in the following Proposition we show that the potential shaping introduced by the canonicalization C D S , D A ( R ) is more inﬂuenced by perturbations to R of higher joint probability transitions ( s, a, s (cid:48) ) .
This suggests choosing D S and
D A to have broad support, making C D S , D A ( R ) more robust to perturbations of any given transition.
Proposition 4.3.
Let S and A be discrete, with |S| ≥ 2 .
Let D S ∈ ∆( S ) and D A ∈ ∆( A ) .
Let
R, ν : S ×
A × S → R be reward functions, with ν ( s, a, s (cid:48) )
=
λ I [( s, a, s (cid:48) )
= ( x, u, x (cid:48) )] , λ ∈ R , x, x (cid:48) ∈ S and u ∈ A .
Let Φ D S , D A ( R )( s, a, s (cid:48) ) =
C D S , D A
( R ) ( s, a, s (cid:48) )
− R ( s, a, s (cid:48) ) .
Then:
[Equation]
We have canonicalized potential shaping; next, we compare the rewards in a scale-invariant manner.
Deﬁnition 4.4.
The Pearson distance between random variables X and Y is deﬁned by the expression
D ρ ( X, Y ) = 1 √ 2 (cid:112) 1 − ρ ( X, Y ) , where ρ ( X, Y ) is the Pearson correlation between X and Y .
Lemma 4.5.
The Pearson distance D ρ is a pseudometric.
Moreover, let a, b ∈ (0 , ∞ ) , c, d ∈ R and
X, Y be random variables.
Then it follows that 0 ≤ D ρ ( aX + c, bY + d ) =
D ρ ( X, Y ) ≤ 1 .
We can now deﬁne EPIC in terms of the Pearson distance between canonically shaped rewards.
Deﬁnition 4.6 (Equivalent-Policy Invariant Comparison (EPIC) pseudometric) .
Let D be some visitation distribution over transitions s a → s (cid:48) .
Let S, A, S (cid:48) be random variables jointly sampled from
D .
Let D S and D A be some distributions over states S and A respectively.
The Equivalent-Policy
Invariant Comparison (EPIC) distance between reward functions R A and R B is:
[Equation]
Theorem 4.7.
The Equivalent-Policy Invariant Comparison distance is a pseudometric.
The triangle inequality is particularly important.
For example, consider an environment with an expensive to evaluate ground-truth reward R .
Directly comparing many learned rewards ˆ R to R might be prohibitively expensive.
We can instead pay a one-off cost: query R a ﬁnite number of times and infer a proxy reward R P with D EPIC ( R, R P ) ≤ (cid:15) .
The triangle inequality allows us to evaluate ˆ R via comparison to R P , since D EPIC (ˆ R, R ) ≤ D EPIC (ˆ R, R P ) + (cid:15) .
This is particularly useful for benchmarks, which may be expensive to build but must be cheap to use.
Theorem 4.8.
Let R A , R (cid:48)
A , R B , R (cid:48) B : S ×
A × S → R be reward functions such that R (cid:48)
A ≡ R A and R (cid:48)
B ≡ R B .
Then 0 ≤ D EPIC ( R (cid:48)
A , R (cid:48) B ) =
D EPIC ( R A , R B ) ≤ 1 .
Sparse   - - - - - -  - - - - -  - - - - - -  -
Dense   -   -  -   - -  -   - -  -    -   -  -    -  -
Path   -   -  -   - -  -   - -  -    -   -  -    -  -
Cliff - -
The following is our main theoretical result, showing EPIC distance gives a bounds on the regret between policies optimizing for one of the two reward functions relative to the other.
In other words,
EPIC bounds the difference in returns for policies optimal for either reward function.
Theorem 4.9.
Let M be a γ -discounted MDP\R with discrete state and action spaces S and A .
Let
R A , R B : S ×
A × S → R be bounded rewards, and π ∗ A , π ∗ B be respective optimal policies.
Let
D π ( t, s t , a t , s t +1 )
denote the distribution over transitions S ×
A × S induced by policy π at time t , and D ( s, a, s (cid:48) ) be the visitation distribution used to compute D EPIC .
Suppose there exists K > 0 such that K D ( s t , a t , s (cid:48) t +1 ) ≥ D π ( t, s t , a t , s (cid:48) t +1 ) for all times t ∈ N , triples ( s t , a t , s t +1 ) ∈ S
× A × S and policies π ∈ { π ∗ A , π ∗ B } .
Then the regret under R A from executing π ∗ B instead of π ∗ A is at most
G R A ( π ∗ A ) − G R A ( π ∗ B ) ≤ 16 K (cid:107) R A (cid:107) 2 (1 − γ ) − 1 D EPIC
( R A , R B ) .
The key assumption is that the visitation distribution D spends at least 1 /K ’
th as much time at each transition as the rollouts of π ∗ A and π ∗ B .
In ﬁnite cases, a uniform D guarantees K ≤ |S| 2 |A| .
We generalize the regret bound to continuous spaces in theorem A.15 via a Lipschitz assumption.
Importantly, the returns of π ∗ A and π ∗ B converge as D EPIC
( R A , R B ) → 0 in both cases, no matter which reward function you evaluate on.
While EPIC upper bounds policy regret, it does not lower bound it.
In fact, no reward distance can lower bound regret in arbitrary environments.
For example, suppose the deployment environment transitions to a randomly chosen state independent of the action taken.
In this case, all policies obtain the same expected return, so the policy regret is always zero, regardless of the reward functions.
As a pedagogical example, we compute the EPIC distance between the reward functions in ﬁgure 1 for a deterministic 3 × 3 gridworld.
Despite assigning different rewards to each transition, Sparse and
Dense are equivalent and have zero EPIC distance.
By contrast, D EPIC ( Path , Cliff ) = 0 .
27 , almost as much as D EPIC
( Sparse , Cliff ) = 0 .
Although Path and Cliff have identical optimal policies in deterministic settings, the rewards induce very different optimal policies under stochastic dynamics.
See ﬁgure A.2 for the distances between all reward pairs.
For this example, we used state and action distributions D S and D A uniform over S and A , and visitation distribution D uniform over state-action pairs ( s, a ) , with s (cid:48) deterministically computed.
It is important these distributions have adequate support.
As an extreme example, if D S and D have no support for a particular state then the reward of that state has no effect on the distance.
We can compute
EPIC exactly in a tabular setting, but in general use a sample-based approximation (section A.1.1).
B ASELINE APPROACHES FOR COMPARING REWARD FUNCTIONS
To the best of our knowledge, EPIC is the ﬁrst method to quantitatively evaluate reward functions without training a policy.
Given the lack of established methods, we develop two alternatives as baselines: Episode Return Correlation (ERC) and Nearest Point in Equivalence Class (NPEC).
5. E PISODE R ETURN C ORRELATION (ERC)
The goal of an MDP is to maximize expected episode return, so it is natural to compare reward functions by the returns they induce.
If the return of a reward function R A is a positive afﬁne transformation of another reward R B , then R A and R B have the same set of optimal policies.
This suggests using Pearson distance, which is invariant to positive afﬁne transformations.
Deﬁnition 5.1 (Episode Return Correlation (ERC) pseudometric) .
Let D be some distribution over trajectories.
Let E be a random variable sampled from D .
The Episode Return Correlation distance between reward functions R A and R B is the Pearson distance between their episode returns on D ,
D ERC ( R A , R B ) =
D ρ ( g ( E ; R A ) , g ( E ; R B )) .
Prior work has scatterplot the return of R A against R B over episodes [4, ﬁgure 3] and ﬁxed-length segments [10, section D].
ERC is the Pearson distance of such plots, so is a natural baseline.
We approximate ERC by the correlation of episode returns on a ﬁnite collection of rollouts.
Under special conditions, ERC is invariant to shaping.
Let R be a reward function and Φ a potential function, and deﬁne the shaped reward R (cid:48)
( s, a, s (cid:48) )
=
R ( s, a, s (cid:48) )
+ γ Φ( s (cid:48) )
− Φ( s ) .
The return under the shaped reward on a trajectory τ = ( s 0 , a 0 , · · · , s T ) is g ( τ ; R (cid:48) )
=
g ( τ ; R ) + γ T Φ( s T ) − Φ( s 0 ) .
If the initial state s 0 and terminal state s T are ﬁxed, then γ T Φ( s T ) − Φ( s 0 ) is constant.
Since Pearson distance is invariant to constant shifts, ERC is invariant to shaping in this case.
For inﬁnite-horizon discounted MDPs, only the initial state s 0 need be ﬁxed, since γ T Φ( s T ) → 0 as T → ∞ .
However, if the initial state s 0 is stochastic, the ERC distance can take on arbitrary values under shaping.
Let R A and R B be two arbitrary reward functions.
Suppose that there are at least two distinct initial states, s A and s B , with non-zero measure in D .
Choose potential Φ( s ) = 0 everywhere except Φ( s A ) =
Φ( s B )
=
c , and let R (cid:48)
A and R (cid:48)
B denote R A and R B shaped by Φ .
As c → ∞ , the correlation ρ ( g ( E ; R (cid:48) A ) , g ( E ; R (cid:48) B )) tends to one.
This is since the relative difference tends to zero,
even though g ( E ; R (cid:48) A ) and g ( E ; R (cid:48) B ) continue to have the same absolute difference as c varies.
Consequently, the ERC pseudometric D ERC ( R (cid:48)
A , R (cid:48) B ) → 0 as c → ∞ .
By an analogous argument, setting Φ( s A ) =
c and Φ( s B ) = − c gives D ERC ( R (cid:48)
A , R (cid:48) B ) → 1 as c → ∞ .
5. N EAREST P OINT IN E QUIVALENCE C LASS (NPEC)
NPEC takes the minimum L p distance between equivalence classes.
See section A.3.3 for proofs.
Deﬁnition 5.2 ( L p distance) .
Let D be a visitation distribution over transitions s a → s (cid:48) and let p ≥be a power.
The L p distance between reward functions R A and R B is the L p norm of their difference:
[Equation]
Proposition 5.3.
(1) D L p , D is a metric in L p space, where functions f and g are identiﬁed if they agree almost everywhere on D . (2) D L p , D is a pseudometric if functions are identiﬁed only if they agree at all points.
The L p distance is affected by shaping and positive rescaling despite not changing the optimal policy.
A natural solution is to take the distance from the nearest point in the equivalence class:
D U NPEC ( R A , R B ) =
inf R (cid:48)
A ≡ R A D L p , D ( R (cid:48)
A , R B ) .
Unfortunately, D U NPEC is sensitive to R B ’s scale.
It is tempting to instead take the inﬁmum over both arguments of D L p , D .
However, inf R (cid:48)
A ≡ R A ,R (cid:48)
B ≡ R B D L p , D ( R (cid:48)
A , R B )
=
since all equivalence classes come arbitrarily close to the origin in L p space.
Instead, we ﬁx this by normalizing D U NPEC .
Deﬁnition 5.4.
The Nearest Point in Equivalence Class (NPEC) premetric is deﬁned by:
D NPEC ( R A , R B ) =
D U NPEC
( R A ,R B ) D U NPEC ( Zero ,R B ) when D U NPEC ( Zero , R B ) (cid:54) = 0 and 0 otherwise.
If D U NPEC ( Zero , R B )
=
then D U NPEC ( R A , R B ) = 0 since R A can be scaled arbitrarily close to
Zero .
Since all policies are optimal for R ≡ Zero , we choose D NPEC ( R A , R B )
=
in this case.
Theorem 5.5.
D NPEC is a premetric on the space of bounded reward functions.
Moreover, let R A , R A (cid:48) , R B , R B (cid:48) :
S ×
A × S → R be bounded reward functions such that R
A ≡ R A (cid:48) and R B ≡ R B (cid:48) .
Then 0 ≤ D NPEC ( R A (cid:48) , R B (cid:48)
= D NPEC ( R , R ) ≤ 1 .
S S
D D
M
R B
S S
D D
M
R A 0.00 0.51 0.00 0.51 0.
[Figure]
EPIC
S S
D D
M
R B 0.00 0.88 0.22 0.86 0.
[Figure]
S S
D D
M
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
quadratic control penalty, no control penalty.
S is Sparse( x ) = 1 [ | x | < 0 . 05] , D is shaped Dense( x, x (cid:48) )
=
Sparse( x ) + | x (cid:48) | − | x | , while M is Magnitude( x ) =
−| x | .
Width of 95% conﬁdence interval (see Figure A.3) is less than 0 .
02 for EPIC and ERC but as large as 0 .
3 for NPEC.
Note that D NPEC may not be symmetric and so is not, in general, a pseudometric: see proposition A.2.
The inﬁmum in D U NPEC can be computed exactly in a tabular setting, but in general we must approximate it using gradient descent.
This gives an upper bound for D U NPEC , but the quotient of upper bounds D NPEC may be too low or too high.
See section A.1.2 for details of the approximation.
E XPERIMENTS
We evaluate EPIC and the baselines ERC and NPEC in a variety of continuous control tasks.
First, we compute the distance between hand-designed reward functions, ﬁnding EPIC to be the most reliable distance.
Although NPEC produces qualitatively similar results, it has a high degree of approximation error.
Moreover, ERC sometimes suffers from pathological failures, such as assigning a high distance to equivalent rewards.
Second, we ﬁnd the distance of learned reward functions to a ground-truth reward predicts the return obtained by policy training, even in an unseen test environment.
Finally, we show EPIC is robust to the exact choice of visitation distribution D , whereas ERC and especially
NPEC are highly sensitive to the choice of D .
6. C OMPARING HAND - DESIGNED REWARD FUNCTIONS
We compare procedurally speciﬁed reward functions in four tasks.
Figure 2 presents results in the proof-of-concept PointMass task.
The results for Gridworld , HalfCheetah and Hopper , in section A.2.4, are qualitatively similar.
In PointMass the agent can accelerate left or right on a line.
The reward functions include ( ) or exclude ( ) a quadratic penalty (cid:107) a (cid:107) 22 .
The sparse reward ( S ) gives a reward of 1 in the region around the goal state.
The dense reward ( D ) is a shaped version of the sparse reward.
The magnitude reward ( M ) is the negative distance of the agent from the goal.
We ﬁnd that EPIC correctly identiﬁes the equivalent reward pairs ( S - D and S - D ) with estimated distance < 1 × 10 − 3 .
By contrast, NPEC has substantial approximation error:
D NPEC ( D , S ) = 0 .
Similarly, D ERC ( D , S ) = 0 .
56 due to ERC’s erroneous handling of stochastic initial states.
Moreover, NPEC is computationally inefﬁcient: Figure 2(b) took 31 hours to compute.
By contrast, the ﬁgures for EPIC and ERC were each generated in under an hour.
See section A.2.6 for more information on runtimes.
6. P REDICTING POLICY PERFORMANCE FROM REWARD DISTANCE
We train reward models on the PointMaze task from Fu et al.
[8], and evaluate the ground-truth ( GT ) return of a policy optimized for the learned reward.
Table 2 shows that rewards with low distance from GT achieve high returns.
High distance rewards sometimes work but are sensitive to dynamics.
[Table]
PointMaze is a MuJoCo environment where a point mass agent must navigate around a wall to reach a goal.
The train and test variants differ only in the position of the wall.
We evaluate four reward learning algorithms: Regress ion onto reward labels [ target method from 6, section 3.3],
Pref erence comparisons on trajectories [6]
[Table]
and adversarial IRL with a state-only ( AIRL SO ) and state-action ( AIRL SA ) reward model [8].
All models are trained using synthetic data from an oracle with access to the ground-truth; see section A.2.2 for details.
Both Regress and Pref achieve very low distances, producing near-expert policy performance in both the train and test variants.
The AIRL SO and AIRL SA models have distances an order of magnitude greater.
The more expressive AIRL SA achieves near-expert performance in train but fails to transfer to test .
The less-expressive AIRL SO has poor performance in both variants, although the generator policy (trained simultaneously with the reward) performs reasonably in train .
Due to reward ambiguity, rewards such as AIRL * that are distant from the ground-truth GT can still produce a good policy.
For example, a “memorized” reward function that assigns reward only to states visited by an expert will induce the expert policy in the train variant.
Nonetheless, it will have a large distance from GT , even if the visitation distribution D only contains transitions from train .
This is appropriate since in test the “expert” policy produced by this reward runs straight into the wall.
6. S ENSITIVITY OF REWARD DISTANCE TO VISITATION STATE DISTRIBUTION
We would like the reward distances to be robust to the exact choice of visitation distribution D .
In
Table 2, we report distances calculated under distributions induced by rollouts from three different policies.
π unif takes actions uniformly at random, producing broad support over transitions; π ∗ is an expert policy, yielding a distribution concentrated at or near the goal; and Mix is a mixture of the two.
In EPIC, D S and D A are marginalized from D and so also vary between conditions.
We ﬁnd EPIC is robust to varying D : the distance varies by less than 2 × , and the ranking between the reward models is the same across visitation distributions, except for Mix favoring AIRL SA over
AIRL SO .
By contrast, NPEC is highly sensitive to the choice of D : the distance of Pref varies by over 500 × between π unif and π ∗ .
ERC lies somewhere in the middle: the distances vary by as much as 25 × .
Overall, EPIC is clearly the least sensitive to choice of D in this environment.
Nonetheless, even with EPIC some care must be taken when choosing D .
Typically, D is collected via rollouts of some exploration policy in an environment.
This works well when the deployment environment has a similar set of reachable states to the rollout environment, even if some details of the dynamics – such as the position of the wall in PointMaze – differ.
However, when the deployment environment allows a transition ( s, a, s (cid:48) ) that is not physically attainable in the rollout environment, then D will place no support on this transition and the reward R ( s, a, s (cid:48) ) can take arbitrary values without affecting the distance.
In general, any black-box method for assessing reward models – including the rollout method – only has predictive power on transitions visited during testing.
C ONCLUSION
Our novel EPIC distance compares reward functions directly, without training a policy.
We have proved it satisﬁes the axioms of a pseudometric, is bounded and invariant to equivalent rewards and bounds the regret of optimal policies.
Empirically, we ﬁnd the EPIC distance between procedurally speciﬁed reward functions is more reliable than the NPEC and ERC baselines.
Furthermore, we ﬁnd the distance of learned reward functions to the ground-truth reward predicts the return of policies optimized for the learned reward, in both the train and unseen test environments
A S UPPLEMENTARY MATERIAL A. A PPROXIMATION P ROCEDURES A.1. S AMPLE - BASED APPROXIMATION FOR EPIC DISTANCE
We approximate EPIC distance (deﬁnition 4.6) by estimating Pearson distance on a set of samples, canonicalizing the reward on-demand.
Speciﬁcally, we sample a batch B V of N V samples from the visitation distribution D , and a batch B M of N M samples from the joint state and action distributions D
S × D A .
For each ( s, a, s (cid:48) )
∈ B V , we approximate the canonically shaped rewards (deﬁnition 4.1) by taking the mean over B M :
[Equation]
We drop the constant c from the approximation since it does not affect the Pearson distance; it can also be estimated in O ( N 2 M ) time by c =
γN 2 M (cid:80) ( x, · ) ∈ B M (cid:80) ( x (cid:48) ,u ) ∈
B M R ( x, u, x (cid:48) ) .
Finally, we compute the Pearson distance between the approximate canonically shaped rewards on the batch of samples
B V , yielding an O ( N V N M ) time algorithm.
A.1. O PTIMIZATION - BASED APPROXIMATION FOR NPEC DISTANCE
D NPEC ( R A , R B ) (section 5.2) is deﬁned as the inﬁmum of L p distance over an inﬁnite set of equivalent reward functions R ≡ R A .
We approximate this using gradient descent on the reward model
[Equation]
where ν, c ∈ R are scalar weights and w is a vector of weights parameterizing a deep neural network Φ w .
The constant c ∈ R is unnecessary if Φ w has a bias term, but its inclusion simpliﬁes the optimization problem.
We optimize ν, c, w to minimize the mean of the cost
[Equation]
on samples ( s, a, s (cid:48) ) from a visitation distribution D .
Note
[Equation]
) upper bounds the true NPEC distance since R ν,c,w ≡ R A .
We found empirically that ν and c need to be initialized close to their optimal values for gradient descent to reliably converge.
To resolve this problem, we initialize the afﬁne parameters to ν ← log λ and c found by:
[Equation]
We use the active set method of Lawson & Hanson [11] to solve this constrained least-squares problem.
These initial afﬁne parameters minimize the L p distance D L p , D ( R ν,c, 0 ( s, a, s (cid:48) ) ,
R B ( s, a, s (cid:48) ))
when p = 2 with the potential ﬁxed at Φ 0 ( s ) = 0 .
A.1. C ONFIDENCE I NTERVALS
We report conﬁdence intervals to help measure the degree of error introduced by the approximations.
Since approximate distances may not be normally distributed, we use bootstrapping to produce a distribution-free conﬁdence interval.
For EPIC and NPEC, we compute independent approximate distances over different seeds, and then compute a bootstrapped conﬁdence interval on the distances for each seed.
We use 30 seeds for EPIC but only 3 seeds for NPEC due to its greater computational requirements.
In ERC, computing the distance is very fast, so we instead apply bootstrapping to the collected episodes , computing the ERC distance for each bootstrapped episode sample.
[Table]
[Table]
All
[Table]
All
[Table]
All
[Table]
All
[Table]
All
[Table]
All
[Table]
All
[Table]
All
[Table]
All
[Table]
All
[Table]
A. E XPERIMENTS A.2. H YPERPARAMETERS FOR A PPROXIMATE D ISTANCES
Table A.1 summarizes the hyperparameters and distributions used to compute the distances between reward functions.
Most parameters are the same across all environments.
We use a visitation distribution of uniform random transitions D unif in the simple GridWorld environment with known determinstic dynamics.
In other environments, the visitation distribution is sampled from rollouts of a policy.
We use a random policy π unif for PointMass , HalfCheetah and Hopper in the hand-designed reward experiments (section 6.1).
In PointMaze , we compare three visitation distributions (section 6.3) induced by rollouts of π unif , an expert policy π ∗ and a Mix ture of the two policies, sampling actions from either π unif or π ∗ and switching between them with probability 0 .per timestep.
A.2. T RAINING L EARNED R EWARD M ODELS
For the experiments on learned reward functions (sections 6.2 and 6.3), we trained reward models using adversarial inverse reinforcement learning (AIRL; 8), preference comparison [6] and by regression onto the ground-truth reward [ target method from 6, section 3.3].
For AIRL, we use an existing open-source implementation [23].
We developed new implementations for preference comparison and regression, available at — double blind: supplementary material —.
We also use the RL algorithm proximal policy optimization (PPO; 18) on the ground-truth reward to train expert policies to provide demonstrations for AIRL, and on learned reward models to evaluate their performance.
[Table]
[Table] [Table]
target method from section 3.3]
[Table]
For PPO and AIRL we used the default hyperparameters in tables A.2 and A.3, ﬁnding them adequate and so performing no further tuning.
For preference comparison we performed a sweep over batch size, trajectory length and learning rate to decide on the hyperparameters in table A.4.
Total timesteps was selected once diminishing returns were observed in loss curves.
The exact value of the regularization weight was found to be unimportant, largely controlling the scale of the output at convergence.
Finally, for regression we performed a sweep over batch size, learning rate and total timesteps to decide on the hyperparameters in table A.5.
We found batch size and learning rate to be relatively unimportant with many combinations performing well, but regression was found to converge slowly but steadily requiring a relatively large 10 × 10 6 timesteps for good performance in our environments.
All algorithms are trained on synthetic data generated from the ground-truth reward function.
AIRL is provided with a large demonstration dataset of 100 000 timesteps from an expert policy trained on the ground-truth reward, similar in size to the total number of timesteps AIRL is trained for (see table A.3).
In preference comparison and regression, each batch is sampled afresh from the visitation distribution speciﬁed in table A.1 and labeled according to the ground-truth reward.
A.2. C OMPUTING INFRASTRUCTURE
Experiments were conducted on a small number of n1-standard-96 Google Cloud Platform VM instances, with 48 CPU cores on an Intel Skylake processor and 360 GB of RAM.
It takes less than a week of compute on a single n1-standard-96 instance to run all the experiments described in this paper.
A.2. C OMPARING HAND - DESIGNED REWARD FUNCTIONS
We compute distances between hand-designed reward functions in four environments: GridWorld ,
PointMass , HalfCheetah and Hopper .
The reward functions for GridWorld are described in Figure A.1, and the distances are reported in Figure A.2.
We report the approximate distances and conﬁdence intervals between reward functions in the other environments in Figures A.3, A.4 and A.5.
We ﬁnd the (approximate) EPIC distance closely matches our intuitions for similarity between the reward functions.
NPEC often produces similar results to EPIC, but unfortunately is dogged by optimization error.
This is particularly notable in higher-dimensional environments like HalfCheetah and Hopper , where the NPEC distance often exceeds the theoretical upper bound of 1 .
and the conﬁdence interval width is frequently larger than 0 . 2 .
By contrast, ERC distance generally has a tight conﬁdence interval, but systematically fails in the presence of shaping.
For example, it conﬁdently assigns large distances between equivalent reward pairs in PointMass such as S - D .
However, ERC produces reasonable results in
HalfCheetah and Hopper where rewards are all similarly shaped.
In fact, ERC picks up on a detail in Hopper that EPIC misses: whereas EPIC assigns a distance of around 0 .
71 between all rewards of different types (running vs backﬂipping), ERC assigns lower distances when the rewards are in the same direction (forward or backward).
Given this, ERC may be attractive in some circumstances, especially given the ease of implementation.
However, we would caution against using it in isolation due to the likelihood of misleading results in the presence of shaping.
A.2. C OMPARING LEARNED REWARD FUNCTIONS
Previously, we reported the mean approximate distance from a ground-truth reward of four learned reward models in PointMaze (Table 2).
Since these distances are approximate, we report 95% lower and upper bounds computed via bootstrapping in Table A.7.
We also include the relative difference of the upper and lower bounds from the mean, ﬁnding the relative difference to be fairly consistent across reward models for a given algorithm and visitation distribution pair.
The relative difference is less than 1% for all EPIC and ERC distances.
However, NPEC conﬁdence intervals can be as wide as 50%: this is due to the method’s high variance, and the small number of seeds we were able to run because of the method’s computational expense.
[Table]
A.2. R UNTIME OF D ISTANCE M ETRICS
We report the empirical runtime for EPIC and baselines in Table A.6, performing 25 pairwise comparisons across 5 reward functions in PointMass .
These comparisons were run on an unloaded machine running Ubuntu 20.04 (kernel 5.4.0-52) with an Intel i9-7920X CPU and 64 GB of RAM.
We ﬁnd that in just 17 seconds EPIC can provide results with a 95% conﬁdence interval < 0 . 023 , an order of magnitude tighter than NPEC running for over 8 hours.
Training policies for all learned rewards in this environment using PPO would also be slow, taking over 4 hours even with onlyseeds.
While ERC is relatively fast, it takes a large number of samples to achieve tight conﬁdence intervals, and moreover the distance comparisons are systematically erroneous in the presence of stochastic initial states (see sections 5.1 and 6.2).
Sparse - - - - - -  - - - - -  - - - - - -  -
Dense         -      -       -
Penalty
Center      -   -  -   - -  -   - -  -    -   -  -    -  -
Path   -   -  -   - -  -   - -  -    -   -  -    -  -
Cliff - - - -      - - - -      - - - -
Sparse
Dense
Center
Penalty
Path
Cliff
[Table]
Sparse
Dense
Center
Penalty
Path
Cliff
[Table]
[Table] [Table]
[Table] [Table]
D ( R
A , R
[Table] [Table]
Sparse
Dense
Center
Penalty
Path
Cliff
[Table]
Sparse
Dense
Center
[Table]
n alty
Path
Cliff
[Table]
D ( R
A , R
[Figure] [Figure]
S S
D D
M
R B
S S
D D
M
R A
[Table]
S S
D D
M
R B
S S
D D
M
R A 6e-5 2e-3 9e-5 2e-3 9e2e-3 6e-5 2e-
5e-5 2e9e-5 2e-3 7e-5 2e-3 9e2e-3 5e-5 2e-
 5e-5 2e9e-4 2e-3 9e-4 2e-3 5e-
[Table] [Table]
S S
D D
M
R B 2e-5 9e-
3e-2 2e-2 2e9e-2 9e-7 2e-1 4e-3 1e1e-
1e-2 1e-5 6e-3 1e6e-
2e-3 9e-2 2e-6 6e1e-
1e-1 3e-1 1e-2 4e-
(e) NPEC CI Width
S S
D D
M
R B 7e-9 1e-2 4e-3 4e-3 3e1e-2 7e-9 6e-3 5e-
4e4e-3 6e-3 1e-8 1e-3 4e4e-
5e-3 1e-3 1e-8 4e3e-
4e-3 4e-3 4e-3 7e0 .0 .1 .
D W ( R A , R
B ) (f) ERC CI Width no
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
5e
[Figure]
6e-
[Figure]
1e
[Figure]
6e
[Figure]
1e-
[Figure]
1e
[Figure]
1e
[Figure]
5e
[Figure]
7e
[Figure]
1e
[Figure]
2 1e
[Figure]
2 1e
[Figure]
1 3e
[Figure]
2e
[Figure]
2 1e
[Figure]
2 1e
[Figure]
2 1e
[Figure]
2e-1 2e-1 1e-2 1e
[Figure]
3e
[Figure]
8e
[Figure]
9e
[Figure]
1e
[Figure]
2e
[Figure]
1e
[Figure]
1e
[Figure]
3e
[Figure]
1e
[Figure]
1e
[Figure]
quadratic control penalty, no control penalty.
Conﬁdence Interval (CI) : 95% CI computed by bootstraping over 10 000 samples.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Table]
AIRL SO
[Table]
AIRL SA
[Table] [Table]
[Table] [Table]
SO
[Table]
AIRL SA
[Table] [Table]
[Table]
A. P ROOFS A.3. B ACKGROUND
Proposition 3.5.
The binary relation ≡ is an equivalence relation.
Let R A , R B , R C : S×A×S → R be bounded reward functions.
Then ≡ is reﬂexive, R A ≡ R A ; symmetric, R A ≡ R B implies
R B ≡ R A ; and transitive, ( R A ≡ R B )
∧ ( R B ≡ R C ) implies R A ≡ R C .
Proof.
R A ≡ R A since choosing λ = 1 > 0 and Φ( s ) = 0 , a bounded potential function, we have
R A ( s, a, s (cid:48) )
=
λR A ( s, a, s (cid:48) )
+ γ Φ( s (cid:48) )
− Φ( s ) for all s, s (cid:48) ∈ S and a ∈ A .
Suppose R A ≡ R B .
Then there exists some λ > 0 and a bounded potential function Φ : S → R such that R B ( s, a, s (cid:48) )
=
λR A ( s, a, s (cid:48) )
+ γ Φ( s (cid:48) )
− Φ( s ) for all s, s (cid:48) ∈ S and a ∈ A .
Rearranging:
[Equation]
Since 1 λ > 0 and Φ (cid:48) ( s ) = − 1 λ Φ( s ) is a bounded potential function, it follows that R B ≡ R A .
Finally, suppose R A ≡ R B and R B ≡ R C .
Then there exists some λ 1 , λ 2 > 0 and bounded potential functions
Φ 1 , Φ 2 : S → R such that for all s, s (cid:48) ∈ S and a ∈ A :
[Equation]
Substituting the expression for R B into the expression for R C :
[Equation]
where λ = λ 1 λ 2 > 0 and Φ( s ) =
λ 2 Φ 1 ( s ) + Φ 2 ( s ) is bounded.
Thus R A ≡ R C .
A.3. E QUIVALENT -P OLICY I NVARIANT C OMPARISON (EPIC) PSEUDOMETRIC
Proposition 4.2 (The Canonically Shaped Reward is Invariant to Shaping) .
Let R : S × A × S → R be a reward function and Φ : S → R a potential function.
Let γ ∈ [0 , 1] be a discount rate, and
D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions.
Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s (cid:48) )
=
R ( s, a, s (cid:48) )
+ γ Φ( s (cid:48) )
− Φ( s ) .
Then the canonically shaped R (cid:48) and R are equal:
[Equation]
Proof.
Let s
[Equation]
a, s (cid:48)
∈ S ×
A × S .
Then by substituting in the deﬁnition of R (cid:48) and using linearity of expectations:
[Equation]
where the penultimate step uses E
[Φ( S (cid:48) )]
=
E [Φ( S )] since S and S (cid:48) are identically distributed.
Proposition 4.3.
Let S and A be discrete, with |S| ≥ 2 .
Let D S ∈ ∆( S ) and D A ∈ ∆( A ) .
Let
R, ν : S ×
A × S → R be reward functions, with ν ( s, a, s (cid:48) )
=
λ I [( s, a, s (cid:48) )
= ( x, u, x (cid:48) )] , λ ∈ R , x, x (cid:48) ∈ S and u ∈ A .
Let Φ D S , D A ( R )( s, a, s (cid:48) ) =
C D S , D A
( R ) ( s, a, s (cid:48) )
− R ( s, a, s (cid:48) ) .
Then:
[Equation]
Proof.
Observe that:
[Equation]
where S and S (cid:48) are random variables independently sampled from D S , and A independently sampled from D A .
Then:
[Equation]
Now:
[Equation]
[Equation]
where the ﬁnal step follows by substituting s = x and s (cid:48) (cid:54)
= x (using |S| ≥ 2 ).
Lemma 4.5.
The Pearson distance D ρ is a pseudometric.
Moreover, let a, b ∈ (0 , ∞ ) , c, d ∈ R and
X, Y be random variables.
Then it follows that 0 ≤ D ρ ( aX + c, bY + d ) =
D ρ ( X, Y ) ≤ 1 .
Proof.
For a non-constant random variable V , deﬁne a standardized (zero mean and unit variance) version:
[Equation]
The Pearson correlation coefﬁcient on random variables A and B is equal to the expected product of these standardized random variables:
[Equation]
Let W , X , Y be random variables.
Identity .
Have ρ ( X, X ) = 1 , so D
ρ ( X, X ) = 0 .
Symmetry .
Have ρ ( X, Y )
=
ρ ( Y, X ) by commutativity of multiplication, so D
ρ ( X, Y ) =
D ρ ( Y, X ) .
Triangle Inequality .
For any random variables A, B :
[Equation]
[Equation]
Since (cid:104) A, B (cid:105) =
E [ AB ] is an inner product over R , it follows by the Cauchy-Schwarz inequality that
E [ AB ] ≤ (cid:112) E
[ A 2 ] E [ B 2 ] .
So:
[Equation]
Taking the square root of both sides:
[Equation]
) as required.
Positive Afﬁne Invariant and Bounded D ρ ( aX + c, bY + d )
=
D ρ ( X, Y ) is immediate from ρ ( X, Y ) invariant to positive afﬁne transformations.
Have − 1 ≤ ρ ( X, Y ) ≤ 1 , so 0 ≤ 1 − ρ ( X, Y ) ≤ 2 thus 0 ≤ D ρ ( X, Y ) ≤ 1 .
Theorem 4.7.
The Equivalent-Policy Invariant Comparison distance is a pseudometric.
Proof.
The result follows from D ρ being a pseudometric.
Let R A , R B and R C be reward functions mapping from transitions S ×
A × S to real numbers R .
Identity .
Have:
[Equation] [Equation]
since D ρ ( X, X ) = 0 .
Symmetry .
Have:
[Equation]
since D ρ ( X, Y ) =
D ρ ( Y, X ) .
Triangle Inequality .
Have:
[Equation]
since D ρ ( X, Z ) ≤ D ρ ( X, Y )
+ D ρ ( Y, Z ) .
Theorem 4.8.
Let R A , R (cid:48)
A , R B , R (cid:48) B : S ×
A × S → R be reward functions such that R (cid:48)
A ≡ R A and R (cid:48)
B ≡ R B .
Then 0 ≤ D EPIC ( R (cid:48)
A , R (cid:48) B ) =
D EPIC ( R A , R B ) ≤ 1 .
Proof.
Since D EPIC is deﬁned in terms of D ρ , the bounds 0 ≤ D EPIC ( R (cid:48)
A , R (cid:48) B ) and
D EPIC ( R A , R B ) ≤ 1 are immediate from the bounds in lemma 4.5.
Since R (cid:48)
A ≡ R A and R (cid:48)
B ≡ R B , we can write for X ∈ { A, B } :
[Equation]
for some scaling factor λ X > 0 and potential function Φ X : S → R .
By proposition 4.2:
[Equation]
Moreover, since C D S , D A ( R ) is deﬁned as an expectation over R and expectations are linear:
[Equation]
Unrolling the deﬁnition of D EPIC and applying this result gives:
[Equation] [Equation]
lemma 4.
[Equation]
A.3. N EAREST P OINT IN E QUIVALENCE C LASS (NPEC) PREMETRIC
Proposition 5.3.
(1) D L p , D is a metric in L p space, where functions f and g are identiﬁed if they agree almost everywhere on D . (2) D L p , D is a pseudometric if functions are identiﬁed only if they agree at all points.
Proof.
(1) D L p , D is a metric in the L p space since L p is a norm in the L p space, and d ( x, y )
= (cid:107) x − y (cid:107) is always a metric.
As f = g at all points implies f = g almost everywhere, certainly D L p , D ( R, R ) = 0 .
Symmetry and triangle inequality do not depend on identity so still hold.
Proposition A.1 (Properties of D U NPEC ) .
Let R A , R B , R C : S ×
A × S → R be bounded reward functions, and λ ≥ 0 .
Then D U NPEC :
• Is invariant under ≡ in source :
[Equation]
Invariant under scale-preserving ≡ in target :
[Equation]
• Scalable in target :
[Equation]
• Bounded :
[Equation]
Proof.
We will show each case in turn.
Invariance under ≡ in source
If R A ≡ R B , then:
[Equation]
[Equation]
61) (62) since R ≡ R A if and only if R ≡ R B as ≡ is an equivalence relation.
Invariance under scale-preserving ≡ in target
If R B − R C ≡ Zero , then we can write R B ( s, a, s (cid:48) )
− R C ( s, a, s (cid:48) )
= γ Φ( s (cid:48) ) − Φ( s ) for some potential function Φ : S → R .
Deﬁne f ( R )( s, a, s (cid:48) )
=
R ( s, a, s (cid:48) ) − γ Φ( s (cid:48) )
+ Φ( s ) .
Then for any reward function R : S ×
A × S → R :
[Equation]
Crucially, note f ( R ) is a bijection on the equivalence class
[ R ] .
Now, substituting this into the expression for the NPEC premetric:
[Equation] [Equation]
Scalable in target
First, note that D L p , D is absolutely scalable in both arguments:
[Equation]
E linear
[Equation]
Now
[Equation]
for λ >
[Equation]
applying this to D U NPEC :
[Equation] [Equation]
In the case λ = 0 , then:
[Equation]
Rearranging, we have:
[Equation]
Bounded
Let d (cid:44) D NPEC ( Zero , R B ) .
Then for any (cid:15) > 0 , there exists some potential function Φ : S → R such that the L p distance of the potential shaping R ( s, a, s (cid:48) )
(cid:44) γ Φ( s ) − Φ( s ) from R B satisﬁes:
[Equation]
Let λ ∈
Deﬁne:
[Equation]
Now:
[Equation]
Since R A is bounded, D L p , D ( R A , Zero ) must be ﬁnite, so:
[Equation]
It follows that for any (cid:15) > 0 there exists some λ > 0 such that:
[Equation]
Note that R
A ≡ R (cid:48) λ for all λ > 0 .
So:
[Equation]
prop.
[Equation]
eq.
77 and eq.
[Equation]
[Equation]
[Equation]
Since (cid:15) > 0 can be made arbitrarily small, it follows:
[Equation]
completing the proof.
Theorem 5.5.
D NPEC is a premetric on the space of bounded reward functions.
Moreover, let R A , R A (cid:48) , R B , R B (cid:48) :
S ×
A × S → R be bounded reward functions such that R
A ≡ R A (cid:48) and R B ≡ R B (cid:48) .
Then 0 ≤ D NPEC ( R A (cid:48) , R B (cid:48) ) =
D NPEC ( R A , R B ) ≤ 1 .
Proof.
We will ﬁrst prove D NPEC is a premetric, and then prove it is invariant and bounded.
Premetric
First, we will show that D NPEC is a premetric.
Respects identity
[Equation]
If D U NPEC ( Zero , R A ) =
then D NPEC
( R A , R A )
= 0 as required.
Suppose from now on that
D U NPEC ( R A , R A ) (cid:54) = 0 .
It follows from prop 5.3 that D L p , D ( R A , R A )
= 0 .
Since X ≡ X , 0 is an upper bound for D U
NPEC
( R A , R A ) .
By prop 5.3 D L p , D is non-negative, so this is also a lower bound for D U
NPEC
( R A , R A ) .
So D U NPEC ( R A , R A )
=
and:
[Equation]
Well-deﬁned: D NPEC ( R A , R B ) ≥
By prop 5.3, it follows that D L p , D ( R, R B ) ≥ 0 for all reward functions R : S ×
A × S .
Thus 0 is a lower bound for { D L p , D ( R, R B ) | R : S ×
A × S} , and thus certainly a lower bound for { D L p , D ( R, Y ) | R ≡ X } for any reward function X .
Since the inﬁmum is the greatest lower bound, it follows that for any reward function X :
[Equation]
In the case that D U NPEC ( Zero , R B ) = 0 , then D NPEC ( R A , R B ) = 0 which is non-negative.
From now on, suppose that D U NPEC ( Zero , R B ) (cid:54) = 0 .
The quotient of a non-negative value with a positive value is non-negative, so:
[Equation]
[Equation]
Invariant and Bounded
Since R (cid:48)
B ≡ R B , we have R (cid:48)
B − λR B ≡ Zero for some λ > 0 .
By proposition A.1, D U NPEC is invariant under scale-preserving ≡ in target and scalable in target.
That is, for any reward R :
[Equation]
In particular, D U NPEC ( Zero , R B (cid:48) )
=
λD U NPEC
( Zero , R B ) .
As λ > 0 , it follows that
[Equation]
Suppose D U NPEC ( Zero , R B ) = 0 .
Then D NPEC ( R, R B ) =
= D NPEC ( R, R B (cid:48) ) for any reward
R , so the result trivially holds.
From now on, suppose D U NPEC ( Zero , R B ) (cid:54)
= 0 .
By proposition A.1, D U NPEC is invariant to ≡ in source.
That is, D U NPEC ( R A , R B
[Equation]
=
[Equation]
, so:
[Equation]
By eq.
[Equation]
Since D NPEC is a premetric it is non-negative.
By the boundedness property of proposition A.1,
[Equation]
so:
[Equation]
which completes the proof.
Note when D L p , D is a metric, then D NPEC ( X, Y )
=
if and only if X = Y .
Proposition A.2.
D NPEC is not symmetric in the undiscounted case.
Proof.
We will provide a counterexample showing that D NPEC is not symmetric.
Choose the state space S to be binary { 0 , 1 } and the actions A to be the singleton { 0 } .
Choose the visitation distribution D to be uniform on s 0 → s for s ∈ S .
Take γ = 1 , i.e. undiscounted.
Note that as the successor state is always the same as the start state, potential shaping has no effect on D L p , D , so WLOG we will assume potential shaping is always zero.
Now, take R A ( s )
= 2 s and R B ( s )
= 1 .
Take p = 1 for the L p distance.
Observe that
D L p , D ( Zero , R A ) = 12 ( | 0 | + | 2 | )
= 1 and D L p , D ( Zero , R B ) =
12 ( | 1 | + | 1 | )
= 1 .
Since potential shaping has no effect, D U NPEC ( Zero , R ) =
D L p , D ( Zero , R ) and so D NPEC ( Zero , R A ) =
and D NPEC ( Zero , R B ) = 1 .
Now:
[Equation]
with the inﬁmum attained at λ = 12 .
But:
[Equation]
where:
[Equation]
[Equation]
[Equation]
Note that:
[Equation]
So f ( λ ) ≥ 2 on all of its domain, thus:
[Equation]
Consequently:
[Equation]
so D NPEC is not symmetric.
A. F ULL N ORMALIZATION V ARIANT OF EPIC
Previously, we used the Pearson distance D ρ to compare the canonicalized rewards.
Pearson distance is naturally invariant to scaling.
An alternative is to explicitly normalize the canonicalized rewards, and then compare them using any metric over functions.
Deﬁnition A.3 (Normalized Reward) .
Let R : S × A × S → R be a bounded reward function.
Let (cid:107)·(cid:107) be a norm on the vector space of reward functions over the real ﬁeld.
Then the normalized R is:
[Equation]
Note that ( λR ) N = R N for any λ > 0 as norms are absolutely homogeneous.
We say a reward is standardized if it has been canonicalized and then normalized.
Deﬁnition A.
(Standardized Reward) .
Let R : S × A × S → R be a bounded reward function.
Then the standardized R is:
[Equation]
Now, we can deﬁne a pseudometric based on the direct distance between the standardized rewards.
Deﬁnition A.5 (Direct Distance Standardized Reward) .
Let D be some visitation distribution over transitions s a → s (cid:48) .
Let D S and D A be some distributions over states S and A respectively.
Let
S, A, S (cid:48) be random variables jointly sampled from D .
The Direct Distance Standardized Reward pseudometric between two reward functions R A and R B is the L p distance between their standardized versions over D :
[Equation]
where the normalization step, R N , uses the L p norm.
For brevity, we omit the proof that D DDSR is a pseudometric, but this follows from D L p , D being a pseudometric in a similar fashion to theorem 4.7.
Note it additionally is invariant to equivalence classes, similarly to EPIC.
Theorem A.6.
Let R A , R A (cid:48) , R B and R B (cid:48) be reward functions mapping from transitions
S ×
A × S to real numbers R such that R
A ≡ R A (cid:48) and R B ≡ R B (cid:48) .
Then:
[Equation]
Proof.
The invariance under the equivalence class follows from R S being invariant to potential shaping and scale in R .
The non-negativity follows from D L p , D being a pseudometric.
The upper bound follows from the rewards being normalized to norm 1 and the triangle inequality:
[Equation]
Since both DDSR and EPIC are pseudometrics and invariant on equivalent rewards, it is interesting to consider the connection between them.
In fact, under the L 2 norm, then DDSR recovers EPIC.
First, we will show that canonical shaping centers the reward functions.
Lemma A.7 (The Canonically Shaped Reward is Mean Zero) .
Let R be a reward function mapping from transitions S ×
A × S to real numbers R .
Then:
[Equation]
Proof.
Let X , U and X (cid:48) be random variables that are independent of S , A and S (cid:48) but identically distributed. [Equation] [Equation] =
[Equation]
[Equation]
[Equation]
where the penultimate step follows since A is identically distributed to U , and S (cid:48) is identically distributed to X (cid:48) and therefore to X .
Theorem A.8.
D DDSR with p = 2 is equivalent to D EPIC .
Let R A and R B be reward functions mapping from transitions S ×
A × S to real numbers R .
Then:
[Equation]
[Equation]
Proof.
Recall from the proof of lemma 4.5 that:
[Equation]
where (cid:107)·(cid:107) 2 is the L 2 norm (treating the random variables as functions on a measure space) and Z ( U ) is a centered (zero-mean) and rescaled (unit variance) random variable.
By lemma A. [Equation] the canonically shaped reward functions are already centered under the joint distribution D S
× D
A × D S , and normalization by the L 2 norm also ensures they have unit variance.
Consequently:
[Equation]
completing the proof
[Equation]
A. R EGRET B OUND
In this section, we derive an upper bound on the regret in terms of the EPIC distance.
Speciﬁcally, given two reward functions R A and R B with optimal policies π ∗ A and π ∗ B , we show that the regret (under reward R A ) of using policy π ∗ B instead of a policy π ∗ A is bounded by a function of D EPIC ( R A , R B ) .
First, in section A.5.1 we derive a bound for MDPs with discrete state and action spaces.
In section A.6 we then present an alternative bound for MDPs with arbitrary state and action spaces and Lipschitz reward functions.
Finally, in section A.7 we show that in both cases the regret tends to 0 as D EPIC
( R A , R B ) → 0 .
A.5. D ISCRETE MDP S
We start in lemma A.9 by showing that L 2 distance upper bounds L 1 distance.
Next, in lemma A. we show regret is bounded by the L 1 distance between reward functions using an argument similar to [20].
Then in lemma A.11 we relate regret bounds for standardized rewards R S to the original reward R .
Finally, in theorem 4.9 we use section A.4 to express D EPIC in terms of the L 2 distance on standardized rewards, deriving a bound on regret in terms of the EPIC distance.
Lemma A.9.
Let (Ω , F , p ) be a probability space and f : Ω → R a measurable function whose absolute value raised to the p -th power for p ∈ { 1 , 2 } has a ﬁnite expectation.
Then the L 1 norm of f is bounded above by the L 2 norm:
[Equation]
Proof.
Let X be a random variable sampled from µ , and consider the variance of f ( X ) : [Equation] [Equation] 132) ≥
[Equation]
Rearranging terms, we have
[Equation]
Taking the square root of both sides gives:
[Equation]
135) as required.
Lemma A.10.
Let M be an MDP\R with discrete state and action spaces S and A .
Let R A , R B :
S ×
A × S → R be rewards.
Let π ∗ A and π ∗ B be policies optimal for rewards R A and R B in M .
Let
D π ( t, s t , a t , s t +1 )
denote the distribution over trajectories that policy π induces in M at time step t .
Let D ( s, a, s (cid:48) ) be the (stationary) visitation distribution over transitions
S ×
A × S used to compute
D EPIC .
Suppose that there exists some K > 0 such that K D ( s t , a t , s (cid:48)
t +1 ) ≥ D π ( t, s t , a t , s (cid:48) t +1 ) for all time steps t ∈ N , triples s t , a t , s t +1 ∈ S
× A × S and policies π ∈ { π ∗ A , π ∗ B } .
Then the regret under R A from executing π ∗ B optimal for R B instead of π ∗ A is at most:
[Equation]
Proof.
Noting G R A ( π ) is maximized when π = π ∗ A , it is immediate that
[Equation]
[Equation]
[Equation]
We will show that both these terms are bounded above by K 1 − γ D L 1 , D ( R A , R B ) , from which the result follows.
First, we will show that for policy
[Equation]
Writing R ( τ ) =
(cid:80) Tt =
γ t R ( s t , a t , s [Equation] t ) for convenience, we have that for π ∈ { π ∗ A , π ∗ B } :
[Equation]
By assumption, D π ( t, s t , a t , s
[Equation]
, so:
[Equation]
as required.
In particular, substituting π = π ∗ B gives:
[Equation]
Rearranging gives:
[Equation]
So certainly:
[Equation]
By a symmetric argument, substituting π = π ∗ A gives:
[Equation]
It follows that
[Equation]
Substituting inequalities 149 and 153 into eq.
139 yields the required result.
Note that if D = D unif , uniform over S ×
A × S , then K ≤ |S| 2 |A| .
Lemma A.11.
Let M be an MDP\R with state and action spaces S and A .
Let R A , R B : S×A×S → R be bounded rewards.
Let π ∗ A and π ∗ B be policies optimal for rewards R A and R B in M .
Suppose the regret under the standardized reward R S A from executing π ∗ B instead of π ∗ A is upper bounded by some U ∈ R :
[Equation]
Then the regret under the original reward R A is bounded by:
[Equation]
Proof.
Recall that
[Equation]
where C D S , D A
( R ) is simply R shaped with some (bounded) potential Φ .
It follows that:
[Equation]
where s 0 depends only on the initial state distribution µ .
(In the ﬁnite-horizon case, there is also a term γ T Φ( s T ) , where s T is the ﬁxed terminal state; this term can be neglected in the discounted inﬁnite-horizon case as γ T Φ( s T ) → 0 as T → ∞ for any bounded Φ .)
Since s 0 does not depend on π , the terms cancel when taking the difference in returns:
[Equation]
Combining this with eq 154 gives
[Equation]
Finally, we will bound (cid:107) C D S , D A
( R A ) (cid:107) 2 in terms of (cid:107) R A (cid:107) 2 , completing the proof.
Recall:
[Equation]
[Equation]
where S and S (cid:48) are random variables independently sampled from D S and A sampled from D A .
By the triangle inequality on the L 2 norm and linearity of expectations, we have:
[Equation]
f [Equation] [Equation] 162) where f ( s, a, s (cid:48) ) =
E [ R ( s (cid:48) , A, S (cid:48) )] , g ( s, a, s (cid:48) )
=
E [ R ( s, A, S (cid:48) )] and c = E [ R ( S, A, S (cid:48) )] .
Letting
X (cid:48) be a random variable sampled from D S independently from S and S (cid:48) , have
[Equation]
So (cid:107) f (cid:107) 2 ≤ (cid:107) R (cid:107) 2 and, by an analogous argument, (cid:107) g (cid:107) 2 ≤ (cid:107) R (cid:107) 2 .
Similarly
[Equation]
Combining these results, we have
[Equation]
Substituting eq.
171 into eq.
160 yields:
[Equation]
as required.
Theorem 4.9.
Let M be a γ -discounted MDP\R with discrete state and action spaces S and A .
Let
R A , R B : S ×
A × S → R be bounded rewards, and π ∗ A , π ∗ B be respective optimal policies.
Let
D π ( t, s t , a t , s t +1 )
denote the distribution over transitions S ×
A × S induced by policy π at time t , and D ( s, a, s (cid:48) ) be the visitation distribution used to compute D EPIC .
Suppose there exists K > 0 such that K D ( s t , a t , s (cid:48) t +1 ) ≥ D π ( t, s t , a t , s (cid:48) t +1 ) for all times t ∈ N , triples ( s t , a t , s t +1 ) ∈ S
× A × S and policies π ∈ { π ∗ A , π ∗ B } .
Then the regret under R A from executing π ∗ B instead of π ∗ A is at most
[Equation]
Proof.
Recall from section A.4 that:
[Equation]
Applying lemma A.9 we obtain:
[Equation]
Note that π ∗ A is optimal for R SA and π ∗ B is optimal for R SB since the set of optimal policies for R S is the same as for R .
Applying lemma A.10 and eq.
174 gives
[Equation]
Applying lemma A.11 yields the required result:
[Equation]
A. L IPSCHITZ R EWARD F UNCTIONS
In this section, we generalize the previous results to MDPs with continuous state and action spaces.
The challenge is that even though the spaces may be continuous, the distribution D π ∗ induced by an optimal policy π ∗ may still be a discrete distribution on some ﬁnite set of transitions T ..
However, the expectation over a continuous distribution D is unaffected by the reward at any ﬁnite subset of points.
Accordingly, the reward can be varied arbitrarily on transitions T – causing arbitrarily small or large regret – while leaving the EPIC distance ﬁxed.
To rule out this pathological case, we assume the rewards are Lipschitz smooth.
This guarantees that if the expected difference between rewards is small on a given region, then all points in this region have bounded reward difference.
We start by deﬁning a relaxation of the Wasserstein distance W α in deﬁnition A.12.
In lemma A. we then bound the expected value under distribution µ in terms of the expected value under alternative distribution ν plus W α ( µ, ν ) .
Next, in lemma A.14 we bound the regret in terms of the L 1 distance between the rewards plus W α ; this is analogous to lemma A.10 in the discrete case.
Finally, in theorem A.15 we use the previous results to bound the regret in terms of the EPIC distance plus W α .
Deﬁnition A.12.
Let S be some set and let µ, ν ∈ ∆( S ) be probability distributions on S .
We deﬁne the relaxed Wasserstein distance between µ and ν by:
[Equation]
where Γ α ( µ, ν ) ⊆
∆( S × S ) is the set of probability distributions on S × S satisfying for all x, y ∈ S :
[Equation]
Note that W 1 is equal to the (unrelaxed) Wasserstein distance (in the (cid:96) 1 norm).
Lemma A.13.
Let S be some set and let µ, ν ∈ ∆( S ) be probability distributions on S .
Let f : S
→ R be an L -Lipschitz function on norm (cid:107)·(cid:107) .
Then, for any α ≥ 1 :
[Equation]
Proof.
Let p ∈
Γ α ( µ, ν ) .
Then:
[Equation]
deﬁnition of E
[Equation]
[Equation]
µ is a marginal of
[Equation]
f
[Equation]
Lipschitz
[Equation]
[Equation]
deﬁnition of E
[Equation]
Since this holds for all choices of p , we can take the inﬁmum of both sides, giving:
[Equation]
Lemma A.14.
Let M be an MDP\R with state and action spaces S and A .
Let R A , R B : S×A×S → R be L -Lipschitz rewards on some norm (cid:107)·(cid:107) on S ×
A × S .
Let π ∗ A and π ∗ B be policies optimal for rewards R A and R B in M .
Let D π,t
( s t , a t , s t +1 )
denote the distribution over trajectories that policy π induces in M at time step t .
Let D ( s, a, s (cid:48) ) be the (stationary) visitation distribution over transitions
S ×
A × S used to compute D EPIC .
Let α ≥ 1 , and let B α ( t ) =
max π ∈ π ∗ A ,π ∗ B W α ( D π,t , D ) .
Then the regret under R
A from executing π ∗ B optimal for R B instead of π ∗ A is at most:
[Equation]
Proof.
By the same argument as lemma A.10 we have for any policy π :
[Equation]
Let f ( s, a, s (cid:48) )
=
R A ( s, a, s (cid:48) )
− R B ( s, a, s (cid:48) ) , and note f is 2 L -Lipschitz since R A and R B are both
L -Lipschitz.
Now, by lemma A.13, letting µ = D π,t and ν = D , we have:
[Equation]
1 , D π
[Equation]
So, for π ∈ { π ∗ A , π ∗ B } , it follows that
[Equation]
By the same argument as for lemma A.10, it follows that
[Equation]
completing the proof.
Theorem A.15.
Let M be an MDP\R with state and action spaces S and A .
Let R A , R B : S ×
A × S → R be bounded, L -Lipschitz rewards on some norm (cid:107)·(cid:107) on S ×
A × S .
Let π ∗ A and π ∗ B be policies optimal for rewards R A and R B in M .
Let D π ( t, s t , a t , s t +1 ) denote the distribution over trajectories that policy π induces in M at time step t .
Let D ( s, a, s (cid:48) ) be the (stationary) visitation distribution over transitions
S ×
A × S used to compute D EPIC .
Let α ≥ 1 , and let
[Equation] ,t , D ) .
Then the regret under R A from executing π ∗ B optimal for R B instead of π ∗
A is at most:
[Equation]
Proof.
By a similar argument [Equation] applying theorem A.14 and eq.
174 gives
[Equation]
Applying lemma A.11 yields
[Equation]
as required.
A. L IMITING B EHAVIOR OF R EGRET
Theorem 4.9 – the regret bound for discrete MDPs – directly implies that, as EPIC distance tends to 0 , the regret also tends to 0 .
By contrast, our regret bound in theorem A.15 for (possibly continuous)
MDPs with Lipschitz reward functions includes the relaxed Wasserstein distance W α as an additive term.
At ﬁrst glance, it might therefore appear possible for the regret to be positive even with a zero EPIC distance.
However, in this section we will show that in fact the regret tends to 0 as
D EPIC ( R A , R B ) → 0 in the Lipschitz case as well as the discrete case.
We show in lemma A.16 that if the expectation of a non-negative function over a totally bounded measurable metric space S tends to zero under one distribution with adequate support, then it also tends to zero under all other distributions.
For example, taking S to be a hypercube in Euclidean space with the Lebesque measure satisﬁes these assumptions.
We conclude in theorem A.17 by showing the regret tends to 0 as the EPIC distance tends to 0 .
Lemma A.16.
Let M =
( S, d ) be a totally bounded metric space, where d ( x, y )
= (cid:107) x − y (cid:107) .
Let ( S, A, µ ) be a measure space on S with the Borel σ -algebra A and measure µ .
Let p, q ∈ ∆( S ) be probability density functions on S .
Let δ > 0 such that p ( s ) ≥ δ for all s ∈ S .
Let f n : S → R be a sequence of L -Lipschitz functions on norm (cid:107)·(cid:107) .
Suppose lim n →∞ E X ∼ p
| f n ( X ) | ] = 0 .
Then lim n →∞ E Y ∼ q [ | f n ( Y ) | ] = 0 .
Proof.
Since M is totally bounded, for each r > 0 there exists a ﬁnite collection of open balls in S of radius r whose union contains M .
Let B r ( c )
=
{ s ∈ S | (cid:107) s − c (cid:107) < r } , the open ball of radius r centered at c .
Let C ( r ) denote some ﬁnite collection of Q ( r ) open balls:
[Equation]
such that (cid:83) B ∈ C ( r ) B = S .
It is possible for some balls B r ( c n ) to have measure zero, µ ( B r ( c n ))
= 0 , such as if S contains an isolated point c n .
Deﬁne P ( r ) to be the subset of C ( r ) with positive measure:
[Equation]
and let p r, 1 , · · · , p r,Q (cid:48) ( r ) denote the centers of the balls in P .
Since P ( r ) is a ﬁnite collection, it must have a minimum measure:
[Equation]
Moreover, by construction of P , α ( r ) > 0 .
Let S (cid:48)
( r ) be the union only over balls of positive measure:
[Equation]
Now, let D ( r ) =
S \ S (cid:48)
( r ) , comprising the (ﬁnite number of) measure zero balls in C ( r ) .
Since measures are countably additive, it follows that D ( r ) is itself measure zero: µ ( D ( r ))
= 0 .
Consequently:
[Equation]
for any measurable function g : S → R .
Since lim n →∞ E X ∼ p
[ | f n ( X ) | ] = 0 , for all r > 0 there exists some N r ∈ N such that for all n ≥ N r :
[Equation]
By Lipschitz continuity, for any s, s (cid:48) ∈ S :
[Equation]
[Equation] In particular, since any point s ∈ S (cid:48) ( r ) is at most r distance from some ball center p r,i , then | f n ( p r,i )
| ≥ |
f n ( s ) | − Lr .
So if there exists s ∈ S (cid:48) ( r ) such that | f n ( s ) | ≥ 3 Lr , then there must exist a ball center p r,i with | f n ( p r,i ) | ≥ 2 Lr .
Then for any point s (cid:48)
∈ B r ( p r,i ) :
[Equation]
Now, we have:
[Equation] [Equation]
eq.
[Equation] [Equation]
non-negativity of | f
[Equation]
[Equation]
eq.
[Equation] [Equation]
integrating w.
[Equation]
[Equation]
But this contradicts eq. 203, and so can only hold if n < N r .
It follows that for all n ≥ N r and s ∈ S (cid:48) ( r ) , we have | f n ( s ) | < 3 Lr , and so in particular:
[Equation]
Let (cid:15) > 0 .
Choose r = (cid:15) 3 L .
Then for all n ≥ M r , E Y ∼ q [ | f n ( Y ) | ] < (cid:15) .
It follows that:
[Equation]
completing the proof.
Theorem A.17.
Let M be an MDP\R with state and action spaces S and A .
Let R A , R B : S ×
A ×
S → R be bounded rewards on some norm (cid:107)·(cid:107) on S ×
A × S .
Let π ∗ A and π ∗ B be policies optimal for rewards R A and R B in M .
Let D π ( t, s t , a t , s t +1 ) denote the distribution over trajectories that policy π induces in M at time step t .
Let D ( s, a, s (cid:48) ) be the (stationary) visitation distribution over transitions
S ×
A × S used to compute D EPIC .
Suppose that either:
Discrete : S and A are discrete .
Moreover, suppose that there exists some K > 0 such that K D ( s t , a t , s (cid:48)
t +1 ) ≥ D π ( t, s t , a t , s (cid:48) t +1 ) for all time steps t ∈ N , triples s t , a t , s t +1 ∈ S
× A × S and policies π ∈ { π ∗ A , π ∗ B } . 2.
Lipschitz : ( S ×
A × S , d ) is a totally bounded measurable metric space where d ( x, y )
= (cid:107) x − y (cid:107) .
Moreover, R A and R B are L -Lipschitz on (cid:107)·(cid:107) .
Furthermore, suppose there exists some δ > 0 such that D ( s, a, s (cid:48) ) ≥ δ for all s, a, s (cid:48) ∈ S×A×S , and that D π ( t, s t , a t , s t +1 ) is a non-degenerate probability density function (
i.e. no single point has positive measure).
Then as D EPIC ( R A , R B ) → 0 , G R A ( π ∗ A ) − G R A ( π
[Equation]
B
[Equation]
→ 0 .
Proof.
In case (1) Discrete , by theorem 4.9:
[Equation]
Moreover, by optimality of π ∗ A we have 0 ≤ G R A ( π ∗ A ) − G R A ( π ∗ B ) .
So by the squeeze theorem, as
D EPIC
[Equation]
R
[Equation]
[Equation] → [Equation] , G R A
[Equation]
From now on, suppose we are in case (2) Lipschitz .
By the same argument as lemma A.10, we have for any policy π :
[Equation]
Applying lemma A.11 we have:
[Equation]
By equation 174, we know that D L 1 , D
( R S A , R S B ) {+→ 
+}
as D EPIC ( R A , R B ) → 0 .
By lemma A.16, we know that D L 1 , D π,t ( R AS , R BS )
→ 0 as D L 1 , D ( R SA , R SB )
→ 0 .
So we can conclude that as
D EPIC ( R A , R B ) → 0 :
[Equation]
(218) as required.