Original: D YNAMICS -A WARE C OMPARISON OF L EARNED R EWARD F UNCTIONS
Revised: D YNAMICS -A WARE C OMPARISON OF L EARNED R EWARD F UNCTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world.
Revised: The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world.

Original: However, comparing reward functions, for example as a means of evaluating reward learning methods, presents a challenge.
Revised: However, comparing reward functions, for example as a means of evaluating reward learning methods, presents a challenge.

Original: Reward functions are typically compared by considering the behavior of optimized policies, but this approach conﬂates deﬁciencies in the reward function with those of the policy search algorithm used to optimize it.
Revised: Reward functions are typically compared by considering the behavior of optimized policies, but this approach conﬂates deﬁciencies in the reward function with those of the policy search algorithm used to optimize it.

Original: To address this challenge, Gleave et al.
Revised: To address this challenge, Gleave et al.

Original: (2020) propose the Equivalent-Policy Invariant Comparison (EPIC) distance.
Revised: (2020) propose the Equivalent-Policy Invariant Comparison (EPIC) distance.

Original: EPIC avoids policy optimization, but in doing so requires computing reward values at transitions that may be impossible under the system dynamics.
Revised: EPIC avoids policy optimization, but in doing so requires computing reward values at transitions that may be impossible under the system dynamics.

Original: This is problematic for learned reward functions because it entails evaluating them outside of their training distribution, resulting in inaccurate reward values that we show can render EPIC ineffective at comparing rewards.
Revised: This is problematic for learned reward functions because it entails evaluating them outside of their training distribution, resulting in inaccurate reward values that we show can render EPIC ineffective at comparing rewards.

Original: To address this problem, we propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric.
Revised: To address this problem, we propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric.

Original: DARD uses an approximate transition model of the environment to transform reward functions into a form that allows for comparisons that are invariant to reward shaping while only evaluating reward functions on transitions close to their training distribution.
Revised: DARD uses an approximate transition model of the environment to transform reward functions into a form that allows for comparisons that are invariant to reward shaping while only evaluating reward functions on transitions close to their training distribution.

Original: Experiments in simulated physical domains demonstrate that DARD enables reliable reward comparisons without policy optimization and is signiﬁcantly more predictive than baseline methods of downstream policy performance when dealing with learned reward functions.
Revised: Experiments in simulated physical domains demonstrate that DARD enables reliable reward comparisons without policy optimization and is signiﬁcantly more predictive than baseline methods of downstream policy performance when dealing with learned reward functions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reward functions provide a general way to specify objectives for reinforcement learning.
Revised: Reward functions provide a general way to specify objectives for reinforcement learning.

Original: Tasks such as game playing have simple, natural reward functions that enable learning of effective policies (Tesauro, 1995; Mnih et al., 2013; Silver et al., 2017); however, many tasks, such as those involving human interaction, multiple competing objectives, or high-dimensional state spaces, do not have easy to deﬁne reward functions (Knox et al., 2021; Holland et al., 2013; Cabi et al., 2019).
Revised: Tasks such as game playing have simple, natural reward functions that enable learning of effective policies (Tesauro, 1995; Mnih et al., 2013; Silver et al., 2017); however, many tasks, such as those involving human interaction, multiple competing objectives, or high-dimensional state spaces, do not have easy to deﬁne reward functions (Knox et al., 2021; Holland et al., 2013; Cabi et al., 2019).

Original: In these cases, reward functions can be learned from demonstrations, explicit preferences, or other forms of reward supervision (Abbeel & Ng, 2004; Christiano et al., 2017; Cabi et al., 2019).
Revised: In these cases, reward functions can be learned from demonstrations, explicit preferences, or other forms of reward supervision (Abbeel & Ng, 2004; Christiano et al., 2017; Cabi et al., 2019).

Original: However, evaluating the quality of learned reward functions, which may be too complicated to manually inspect, remains an open challenge.
Revised: However, evaluating the quality of learned reward functions, which may be too complicated to manually inspect, remains an open challenge.

Original: Prior work on comparing reward functions typically involves a computationally expensive and indirect process of (i) learning a reward function, (ii) learning a policy on that reward function, and (iii) evaluating that policy on a known, ground-truth reward function.
Revised: Prior work on comparing reward functions typically involves a computationally expensive and indirect process of (i) learning a reward function, (ii) learning a policy on that reward function, and (iii) evaluating that policy on a known, ground-truth reward function.

Original: Gleave et al.
Revised: Gleave et al.

Original: (2020) refer to this as the “rollout method” for evaluating methods for learning reward functions.
Revised: (2020) refer to this as the “rollout method” for evaluating methods for learning reward functions.

Original: Furthermore, the rollout method can be potentially dangerous in safety critical environments because (i) it cannot distinguish deﬁciencies between the reward function and the policy optimization procedure, and (ii) rewards are evaluated on the distribution of transitions induced by an optimized policy, which may not capture all the transitions required to accurately quantify reward differences.
Revised: Furthermore, the rollout method can be potentially dangerous in safety critical environments because (i) it cannot distinguish deﬁciencies between the reward function and the policy optimization procedure, and (ii) rewards are evaluated on the distribution of transitions induced by an optimized policy, which may not capture all the transitions required to accurately quantify reward differences.

Original: To address these challenges, Gleave et al.
Revised: To address these challenges, Gleave et al.

Original: (2020) introduce Equivalent-Policy Invariant Comparison (EPIC).
Revised: (2020) introduce Equivalent-Policy Invariant Comparison (EPIC).

Original: EPIC quantiﬁes differences in reward functions directly, without learning policies.
Revised: EPIC quantiﬁes differences in reward functions directly, without learning policies.

Original: EPIC uses an ofﬂine dataset of transitions, and consists of two stages.
Revised: EPIC uses an ofﬂine dataset of transitions, and consists of two stages.

Original: First, transitions from this dataset are used to convert reward functions to a canonical form that is invariant to reward transformations that do not affect the optimal policy.
Revised: First, transitions from this dataset are used to convert reward functions to a canonical form that is invariant to reward transformations that do not affect the optimal policy.

Original: Second, the correlation between reward values is computed on transition samples, yielding a pseudometric capturing reward function similarity.
Revised: Second, the correlation between reward values is computed on transition samples, yielding a pseudometric capturing reward function similarity.

Original: However, when computing this canonical form, EPIC evaluates rewards on transitions between all state-state combinations, regardless of whether those state-state transitions are possible or not.
Revised: However, when computing this canonical form, EPIC evaluates rewards on transitions between all state-state combinations, regardless of whether those state-state transitions are possible or not.

Original: While this makes it possible for EPIC to quantify reward distances that are robust to arbitrary changes in transition dynamics, in practice evaluating learned reward functions on physically impossible transitions leads to unreliable reward values, as these are outside the distribution of the transitions on which the reward functions are learned.
Revised: While this makes it possible for EPIC to quantify reward distances that are robust to arbitrary changes in transition dynamics, in practice evaluating learned reward functions on physically impossible transitions leads to unreliable reward values, as these are outside the distribution of the transitions on which the reward functions are learned.

Original: We show empirically in Section 5 that these out-of-distribution reward evaluations can render EPIC an ineffective reward comparison metric.
Revised: We show empirically in Section 5 that these out-of-distribution reward evaluations can render EPIC an ineffective reward comparison metric.

Original: To address this issue, we propose a new method for quantifying differences in reward functions,
Revised: To address this issue, we propose a new method for quantifying differences in reward functions,

Original: Dynamics-Aware Reward Distance (DARD), that transforms reward functions into a form allowing for reward-shaping-invariant comparisons while ensuring that learned reward functions are only evaluated on transitions that are physically realizable.
Revised: Dynamics-Aware Reward Distance (DARD), that transforms reward functions into a form allowing for reward-shaping-invariant comparisons while ensuring that learned reward functions are only evaluated on transitions that are physically realizable.

Original: We achieve this by assuming access to a transition model for the environment of interest, and evaluate reward functions on transitions sampled from this transition model.
Revised: We achieve this by assuming access to a transition model for the environment of interest, and evaluate reward functions on transitions sampled from this transition model.

Original: This ensures that reward functions are not evaluated on impossible transitions, which will necessarily be out-of-distribution, and is well motivated in a variety of physical environments in which transition models are unlikely to change signiﬁcantly between learning and deployment.
Revised: This ensures that reward functions are not evaluated on impossible transitions, which will necessarily be out-of-distribution, and is well motivated in a variety of physical environments in which transition models are unlikely to change signiﬁcantly between learning and deployment.

Original: Experiments suggest that DARD better reﬂects reward function similarity than does
Revised: Experiments suggest that DARD better reﬂects reward function similarity than does

Original: EPIC in comparing learned reward functions in two simulated, physics-based environments: the ﬁrst involving multi-agent navigation, and the second involving manipulating a robotic arm.
Revised: EPIC in comparing learned reward functions in two simulated, physics-based environments: the ﬁrst involving multi-agent navigation, and the second involving manipulating a robotic arm.

Original: Videos available at https://sites.google.com/view/dard-paper .
Revised: Videos available at https://sites.google.com/view/dard-paper .

Original:  R ELATED W ORK 2. L EARNING R EWARD F UNCTIONS
Revised:  R ELATED W ORK 2. L EARNING R EWARD F UNCTIONS

Original: A variety of methods exist for learning reward functions, which can be categorized based on the form of supervision used.
Revised: A variety of methods exist for learning reward functions, which can be categorized based on the form of supervision used.

Original: Rewards can be learned from supervision provided by explicit labels.
Revised: Rewards can be learned from supervision provided by explicit labels.

Original: For example, human-labeled preferences between demonstrations can be used to learn reward functions (Christiano et al., 2017).
Revised: For example, human-labeled preferences between demonstrations can be used to learn reward functions (Christiano et al., 2017).

Original: Rewards can also be labeled on a per-timestep basis, allowing for learning via supervised regression (Cabi et al., 2019).
Revised: Rewards can also be labeled on a per-timestep basis, allowing for learning via supervised regression (Cabi et al., 2019).

Original: Alternatively, inverse reinforcement learning (IRL) methods typically assume access to expert demonstrations, and attempt to ﬁnd a reward function under which the expert is uniquely optimal (Ziebart et al., 2008; Abbeel & Ng, 2004).
Revised: Alternatively, inverse reinforcement learning (IRL) methods typically assume access to expert demonstrations, and attempt to ﬁnd a reward function under which the expert is uniquely optimal (Ziebart et al., 2008; Abbeel & Ng, 2004).

Original: IRL is underconstrained in that many reward functions can make the expert policy appear optimal.
Revised: IRL is underconstrained in that many reward functions can make the expert policy appear optimal.

Original: This problem is often addressed by favoring reward functions that induce high-entropy optimal policies (Ziebart et al., 2008).
Revised: This problem is often addressed by favoring reward functions that induce high-entropy optimal policies (Ziebart et al., 2008).

Original: Recent work in IRL focuses on learning reward functions with expressive function approximators such as neural networks, particularly in an adversarial formulation (Fu et al., 2017).
Revised: Recent work in IRL focuses on learning reward functions with expressive function approximators such as neural networks, particularly in an adversarial formulation (Fu et al., 2017).

Original: There has also been recent interest in ofﬂine IRL methods (Lee et al., 2019).
Revised: There has also been recent interest in ofﬂine IRL methods (Lee et al., 2019).

Original: In this work, we study how these various reward learning methods can be evaluated efﬁciently and accurately in practice.
Revised: In this work, we study how these various reward learning methods can be evaluated efﬁciently and accurately in practice.

Original: 2. E VALUATING M ETHODS OF L EARNING R EWARD F UNCTIONS
Revised: 2. E VALUATING M ETHODS OF L EARNING R EWARD F UNCTIONS

Original: Reward functions are typically evaluated by ﬁrst learning an optimal policy under the reward function.
Revised: Reward functions are typically evaluated by ﬁrst learning an optimal policy under the reward function.

Original: The value of the induced policy is found by aggregating the returns from online trajectory rollouts under the known, ground-truth reward function.
Revised: The value of the induced policy is found by aggregating the returns from online trajectory rollouts under the known, ground-truth reward function.

Original: This requirement of online rollouts from experimental policies is often prohibitive in safety-critical domains such as autonomous driving or decision making in healthcare (Thapa et al., 2005).
Revised: This requirement of online rollouts from experimental policies is often prohibitive in safety-critical domains such as autonomous driving or decision making in healthcare (Thapa et al., 2005).

Original: Off-policy evaluation (OPE) methods attempt to address these issues by estimating the value of a policy in a purely ofﬂine setting by using previously collected data.
Revised: Off-policy evaluation (OPE) methods attempt to address these issues by estimating the value of a policy in a purely ofﬂine setting by using previously collected data.

Original: Traditional OPE methods address policy mismatch by using a form of importance sampling (Precup et al., 2000).
Revised: Traditional OPE methods address policy mismatch by using a form of importance sampling (Precup et al., 2000).

Original: While this approach can be effective, it is unbiased only if the underlying expert policy is known, and can often be subject to large variance.
Revised: While this approach can be effective, it is unbiased only if the underlying expert policy is known, and can often be subject to large variance.

Original: Direct OPE methods (Dudik et al.
Revised: Direct OPE methods (Dudik et al.

Original: ; Voloshin et al.
Revised: ; Voloshin et al.

Original: (2020)), apply regression-based techniques in either a model-based (Paduraru, 2013) or model-free setting (Le et al., 2019).
Revised: (2020)), apply regression-based techniques in either a model-based (Paduraru, 2013) or model-free setting (Le et al., 2019).

Original: Contemporary works combine these two methods using Doubly Robust
Revised: Contemporary works combine these two methods using Doubly Robust

Original: Estimation, which originated in a contextual bandits setting (Dudik et al., 2011), and was later extended to reinforcement learning by Jiang & Li ( and importance sampling entirely, such as Irpan et al.
Revised: Estimation, which originated in a contextual bandits setting (Dudik et al., 2011), and was later extended to reinforcement learning by Jiang & Li ( and importance sampling entirely, such as Irpan et al.

Original: (2019), which recasts OPE as a positiveunlabeled classiﬁcation problem with the Q-function as the decision function.
Revised: (2019), which recasts OPE as a positiveunlabeled classiﬁcation problem with the Q-function as the decision function.

Original: An overview of all of these methods and empirical analysis of their performance can be found in Voloshin et al.
Revised: An overview of all of these methods and empirical analysis of their performance can be found in Voloshin et al.

Original: All OPE methods indirectly evaluate the reward by estimating the value of the induced optimal policy.
Revised: All OPE methods indirectly evaluate the reward by estimating the value of the induced optimal policy.

Original: This introduces additional complexity into the evaluation procedure and can obfuscate sources of error as resulting either from aspects of policy learning or due to the underlying reward function.
Revised: This introduces additional complexity into the evaluation procedure and can obfuscate sources of error as resulting either from aspects of policy learning or due to the underlying reward function.

Original: EPIC (Gleave et al., 2020) seeks to circumvent these shortcomings by comparing reward functions directly.
Revised: EPIC (Gleave et al., 2020) seeks to circumvent these shortcomings by comparing reward functions directly.

Original: Our work builds on EPIC as covered in Section 4, but uses a transition model of the environment to signiﬁcantly improve the accuracy of comparisons involving learned reward functions.
Revised: Our work builds on EPIC as covered in Section 4, but uses a transition model of the environment to signiﬁcantly improve the accuracy of comparisons involving learned reward functions.

Original:  P RELIMINARIES 3. M ARKOV D ECISION P ROCESSES
Revised:  P RELIMINARIES 3. M ARKOV D ECISION P ROCESSES

Original: A Markov decision process (MDP) is a tuple ( S , A , T , R , γ , d 0 ) where S and A are the state and action spaces.
Revised: A Markov decision process (MDP) is a tuple ( S , A , T , R , γ , d 0 ) where S and A are the state and action spaces.

Original: The transition model T : S ×
Revised: The transition model T : S ×

Original: A × S → [0 , 1] maps a state and action to a probability distribution over next states, the reward R : S ×
Revised: A × S → [0 , 1] maps a state and action to a probability distribution over next states, the reward R : S ×

Original: A × S → R measures the quality of a transition.
Revised: A × S → R measures the quality of a transition.

Original: Finally, γ ∈ [0 , 1] and d 0 are respectively the discount factor and initial state distribution.
Revised: Finally, γ ∈ [0 , 1] and d 0 are respectively the discount factor and initial state distribution.

Original: A trajectory τ consists of a sequence of state-action pairs: τ = { ( s 0 , a 0 ) , ( s 1 , a 1 ) , ... } , and the return g of a trajectory is deﬁned as the sum of discounted rewards along that trajectory: g ( τ )
Revised: A trajectory τ consists of a sequence of state-action pairs: τ = { ( s 0 , a 0 ) , ( s 1 , a 1 ) , ... } , and the return g of a trajectory is deﬁned as the sum of discounted rewards along that trajectory: g ( τ )

Original: = (cid:80) ∞ t =

Revised: = (cid:80) ∞ t =

Original: γ t R ( s t , a t , s t +1 ) .
Revised: γ t R ( s t , a t , s t +1 ) .

Original: The goal in an MDP is to ﬁnd a policy π : S ×
Revised: The goal in an MDP is to ﬁnd a policy π : S ×

Original: A → [0 , 1] that maximizes the expected return, E [ g ( τ )] , where s 0 ∼ d 0 , a t ∼ π ( a t | s t ) , and s t +1 ∼ T ( s t +1 | s t , a t ) .
Revised: A → [0 , 1] that maximizes the expected return, E [ g ( τ )] , where s 0 ∼ d 0 , a t ∼ π ( a t | s t ) , and s t +1 ∼ T ( s t +1 | s t , a t ) .

Original: We denote a distribution over a space S as D S , and the set of distributions over a space S as ∆( S ) .
Revised: We denote a distribution over a space S as D S , and the set of distributions over a space S as ∆( S ) .

Original: 3. R EWARD F UNCTION E QUIVALENCE
Revised: 3. R EWARD F UNCTION E QUIVALENCE

Original: Our goal is to deﬁne a metric for comparing reward functions without learning policies.
Revised: Our goal is to deﬁne a metric for comparing reward functions without learning policies.

Original: Nevertheless, the measure of similarity we are interested in is the extent to which the optimal policies induced by two reward functions are the same.
Revised: Nevertheless, the measure of similarity we are interested in is the extent to which the optimal policies induced by two reward functions are the same.

Original: Because different reward functions can induce the same set of optimal policies, we in fact seek a pseudometric as formalized in the following deﬁnition.
Revised: Because different reward functions can induce the same set of optimal policies, we in fact seek a pseudometric as formalized in the following deﬁnition.

Original: Deﬁnition 3.1 (Pseudometric (Deﬁnition 3.5 from Gleave et al.
Revised: Deﬁnition 3.1 (Pseudometric (Deﬁnition 3.5 from Gleave et al.

Original: Let X be a set and d :
Revised: Let X be a set and d :

Original: X × X → [0 , ∞ ] a function.
Revised: X × X → [0 , ∞ ] a function.

Original: d is a premetric if d ( x, x ) = 0 for all x ∈ X .
Revised: d is a premetric if d ( x, x ) = 0 for all x ∈ X .

Original: d is a pseudometric if, furthermore, it is symmetric, d ( x, y )
Revised: d is a pseudometric if, furthermore, it is symmetric, d ( x, y )

Original: =
Revised: =

Original: d ( y, x ) for all x, y ∈ X ; and satisﬁes the triangle inequality, d ( x, z ) ≤ d ( x, y )
Revised: d ( y, x ) for all x, y ∈ X ; and satisﬁes the triangle inequality, d ( x, z ) ≤ d ( x, y )

Original: +
Revised: +

Original: d ( y, z ) for all x, y, z ∈ X .
Revised: d ( y, z ) for all x, y, z ∈ X .

Original: d is a metric if, furthermore, for all x, y ∈ X , d ( x, y )
Revised: d is a metric if, furthermore, for all x, y ∈ X , d ( x, y )

Original: = 

Revised: =

Original: =
Revised: =

Original: ⇒ x = y.
Revised: ⇒ x = y.

Original: Which rewards induce the same set of optimal policies?
Revised: Which rewards induce the same set of optimal policies?

Original: Ng et al.
Revised: Ng et al.

Original: (1999) showed that without additional prior knowledge about the MDP only reward functions that are related through a difference in state potentials are equivalent.
Revised: (1999) showed that without additional prior knowledge about the MDP only reward functions that are related through a difference in state potentials are equivalent.

Original: This “reward shaping” refers to an additive transformation F applied to an initial reward function R to compute a shaped reward R (cid:48) :
Revised: This “reward shaping” refers to an additive transformation F applied to an initial reward function R to compute a shaped reward R (cid:48) :

Original: R (cid:48)
Revised: R (cid:48)

Original: ( s, a, s (cid:48) ) =
Revised: ( s, a, s (cid:48) ) =

Original: R ( s, a, s (cid:48) )
Revised: R ( s, a, s (cid:48) )

Original: + F ( s, a, s (cid:48) ) .
Revised: + F ( s, a, s (cid:48) ) .

Original: Reward shaping is typically applied in a reward design setting to produce denser reward functions that make policy optimization easier in practice (Ng et al., 2003; Schulman et al., 2015).
Revised: Reward shaping is typically applied in a reward design setting to produce denser reward functions that make policy optimization easier in practice (Ng et al., 2003; Schulman et al., 2015).

Original: Ng et al.
Revised: Ng et al.

Original: (1999) showed that if F is of the form F ( s, a, s (cid:48) )
Revised: (1999) showed that if F is of the form F ( s, a, s (cid:48) )

Original: = γ Φ( s (cid:48) ) − Φ( s ) for arbitrary state potential function Φ( s ) , the set of optimal policies under R and R (cid:48) are the same.
Revised: = γ Φ( s (cid:48) ) − Φ( s ) for arbitrary state potential function Φ( s ) , the set of optimal policies under R and R (cid:48) are the same.

Original: This provides a deﬁnition for reward equivalence:
Revised: This provides a deﬁnition for reward equivalence:

Original: Deﬁnition 3.2 (Reward Equivalence, Deﬁnition 3.3 from Gleave et al.
Revised: Deﬁnition 3.2 (Reward Equivalence, Deﬁnition 3.3 from Gleave et al.

Original: Bounded rewards
Revised: Bounded rewards

Original: R A and R B , are equivalent if and only if ∃ λ > 0 , and Φ
Revised: R A and R B , are equivalent if and only if ∃ λ > 0 , and Φ

Original: [Equation]
Revised: [Equation]

Original: S → R bounded s.t.
Revised: S → R bounded s.t.

Original: [Equation]
Revised: [Equation]

Original: This form of equivalence is only possible in reward functions that depend on ( s, a, s (cid:48) ) , which motivates our focus on these types of rewards in this paper.
Revised: This form of equivalence is only possible in reward functions that depend on ( s, a, s (cid:48) ) , which motivates our focus on these types of rewards in this paper.

Original: This choice is further motivated by the fact that there are cases where conditioning on the next state s (cid:48) can make reward learning simpler, for example in highly stochastic MDPs, or in cases where rewards are a simple function of ( s, a, s (cid:48) ) , but not of ( s, a ) .
Revised: This choice is further motivated by the fact that there are cases where conditioning on the next state s (cid:48) can make reward learning simpler, for example in highly stochastic MDPs, or in cases where rewards are a simple function of ( s, a, s (cid:48) ) , but not of ( s, a ) .

Original: We next describe an existing pseudometric that in theory is capable of identifying these reward equivalence classes.
Revised: We next describe an existing pseudometric that in theory is capable of identifying these reward equivalence classes.

Original: 3. E QUIVALENT -P OLICY I NVARIANT C OMPARISON (EPIC)
Revised: 3. E QUIVALENT -P OLICY I NVARIANT C OMPARISON (EPIC)

Original: EPIC is a pseudometric for comparing reward functions and involves two stages.
Revised: EPIC is a pseudometric for comparing reward functions and involves two stages.

Original: The ﬁrst stage converts reward functions into a canonical form that removes the effect of reward shaping, and the second stage computes the correlation between the canonicalized rewards.
Revised: The ﬁrst stage converts reward functions into a canonical form that removes the effect of reward shaping, and the second stage computes the correlation between the canonicalized rewards.

Original: The following deﬁnition describes the canonicalization stage.
Revised: The following deﬁnition describes the canonicalization stage.

Original: Deﬁnition 3.3 (Canonically Shaped Reward (Deﬁnition 4.1 from Gleave et al.
Revised: Deﬁnition 3.3 (Canonically Shaped Reward (Deﬁnition 4.1 from Gleave et al.

Original: Let R :
Revised: Let R :

Original: S ×A×S → R be a reward function.
Revised: S ×A×S → R be a reward function.

Original: Given distributions D S ∈ ∆( S ) and D A ∈ ∆( A ) over states and actions, let S and S (cid:48) be independent random variables distributed as D S and A distributed as
Revised: Given distributions D S ∈ ∆( S ) and D A ∈ ∆( A ) over states and actions, let S and S (cid:48) be independent random variables distributed as D S and A distributed as

Original: D A .
Revised: D A .

Original: Deﬁne the canonically shaped R to be:
Revised: Deﬁne the canonically shaped R to be:

Original: [Equation]
Revised: [Equation]

Original: Fig.
Revised: Fig.

Original: 1 visualizes this process.
Revised: 1 visualizes this process.

Original: Informally, this operation computes the mean of the potentials of s and s (cid:48) with respect to S (cid:48) using only evaluations of the reward R , and subtracts/adds those averages from/to the original, shaped reward, thereby canceling out their effects.
Revised: Informally, this operation computes the mean of the potentials of s and s (cid:48) with respect to S (cid:48) using only evaluations of the reward R , and subtracts/adds those averages from/to the original, shaped reward, thereby canceling out their effects.

Original: This cancellation behavior is formalized in the following proposition.
Revised: This cancellation behavior is formalized in the following proposition.

Original: Proposition 1 (The Canonically Shaped Reward is Invariant to Shaping (Proposition 4.2 from
Revised: Proposition 1 (The Canonically Shaped Reward is Invariant to Shaping (Proposition 4.2 from

Original: Gleave et al.
Revised: Gleave et al.

Original: Let R : S ×
Revised: Let R : S ×

Original: A × S → R be a reward function, and Φ : S → R a potential function.
Revised: A × S → R be a reward function, and Φ : S → R a potential function.

Original: Let γ ∈ [0 , 1] be a discount rate, and D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions.
Revised: Let γ ∈ [0 , 1] be a discount rate, and D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions.

Original: Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s
Revised: Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s

Original: [Equation]
Revised: [Equation]

Original: Then the canonically shaped R (cid:48) and R are equal: C D S , D A ( R (cid:48) )
Revised: Then the canonically shaped R (cid:48) and R are equal: C D S , D A ( R (cid:48) )

Original: =
Revised: =

Original: C D S , D A
Revised: C D S , D A

Original: ( R ) .
Revised: ( R ) .

Original: This invariance means that canonicalized rewards can be compared without reward shaping impacting the comparison.
Revised: This invariance means that canonicalized rewards can be compared without reward shaping impacting the comparison.

Original: In practice, canonicalization is performed over an ofﬂine dataset of transitions.
Revised: In practice, canonicalization is performed over an ofﬂine dataset of transitions.

Original: The rewards for ( s, a, s (cid:48) ) transitions are computed, and then canonicalized using (1).
Revised: The rewards for ( s, a, s (cid:48) ) transitions are computed, and then canonicalized using (1).

Original: The distributions D S and D A are the marginal distributions of states and actions in the dataset on which evaluation proceeds.
Revised: The distributions D S and D A are the marginal distributions of states and actions in the dataset on which evaluation proceeds.

Original: As we will discuss in the next section, this results in evaluating learned reward functions outside their training distribution, leading to inaccurate reward values.
Revised: As we will discuss in the next section, this results in evaluating learned reward functions outside their training distribution, leading to inaccurate reward values.

Original: EPIC’s second stage computes the Pearson distance between two canonicalized reward functions.
Revised: EPIC’s second stage computes the Pearson distance between two canonicalized reward functions.

Original: Deﬁnition 3.4 (Pearson Distance (Deﬁnition 4.4 from Gleave et al.
Revised: Deﬁnition 3.4 (Pearson Distance (Deﬁnition 4.4 from Gleave et al.

Original: The Pearson distance between random variables X and Y is deﬁned by the expression D ρ ( X, Y ) =
Revised: The Pearson distance between random variables X and Y is deﬁned by the expression D ρ ( X, Y ) =

Original: (cid:112) 1 − ρ ( X, Y ) / (cid:112) (2) , where ρ ( X, Y ) is the Pearson correlation between X and Y.
Revised: (cid:112) 1 − ρ ( X, Y ) / (cid:112) (2) , where ρ ( X, Y ) is the Pearson correlation between X and Y.

Original: In combining these two steps, EPIC is capable of quantifying the similarity of two reward functions while being invariant to shaping (due to canonicalization), and invariant to changes in shift and scale (due to the use of the Pearson distance).
Revised: In combining these two steps, EPIC is capable of quantifying the similarity of two reward functions while being invariant to shaping (due to canonicalization), and invariant to changes in shift and scale (due to the use of the Pearson distance).

Original: Deﬁnition 3.5 (EPIC pseudometric (Deﬁnition 4.6 from Gleave et al.
Revised: Deﬁnition 3.5 (EPIC pseudometric (Deﬁnition 4.6 from Gleave et al.

Original: Let S , A , S (cid:48) be random variables jointly following some coverage transition distribution.
Revised: Let S , A , S (cid:48) be random variables jointly following some coverage transition distribution.

Original: Let D S , and D A respectively be the state and action distributions.
Revised: Let D S , and D A respectively be the state and action distributions.

Original: The EPIC pseudometric between rewards R A and R B is:
Revised: The EPIC pseudometric between rewards R A and R B is:

Original: [Equation]
Revised: [Equation]

Original: Note that during the canonicalization stage, S , A , S (cid:48) are independent while in the stage computing the Pearson correlation between canonical rewards, the joint distribution of S , A , S (cid:48) is used.
Revised: Note that during the canonicalization stage, S , A , S (cid:48) are independent while in the stage computing the Pearson correlation between canonical rewards, the joint distribution of S , A , S (cid:48) is used.

Original:  DARD: D YNAMICS -A WARE R EWARD D ISTANCE
Revised:  DARD: D YNAMICS -A WARE R EWARD D ISTANCE

Original: Here we present DARD, a new distance measure between learned reward functions that leverages information about system dynamics to prevent errors due to out-of-distribution queries of learned rewards.
Revised: Here we present DARD, a new distance measure between learned reward functions that leverages information about system dynamics to prevent errors due to out-of-distribution queries of learned rewards.

Original: As described in 3.3, the EPIC distance measure ﬁrst transforms reward functions as shown in equation (1).
Revised: As described in 3.3, the EPIC distance measure ﬁrst transforms reward functions as shown in equation (1).

Original: This transformation procedure involves computing expectations E
Revised: This transformation procedure involves computing expectations E

Original: [ R ( s (cid:48) , A, S (cid:48) )] , E
Revised: [ R ( s (cid:48) , A, S (cid:48) )] , E

Original: [ R ( s, A, S (cid:48) )] , and E [ R ( S, A, S (cid:48) )] with independent random variables S and S (cid:48) distributed as D S , which in practice is the marginal distribution of states in the dataset.
Revised: [ R ( s, A, S (cid:48) )] , and E [ R ( S, A, S (cid:48) )] with independent random variables S and S (cid:48) distributed as D S , which in practice is the marginal distribution of states in the dataset.

Original: Thus, the resulting transitions may not actually be possible under the transition model of the environment of interest.
Revised: Thus, the resulting transitions may not actually be possible under the transition model of the environment of interest.

Original: Computing transformed rewards with these out-of-distribution transitions allows EPIC to be robust to (arbitrary) changes in the transition model.
Revised: Computing transformed rewards with these out-of-distribution transitions allows EPIC to be robust to (arbitrary) changes in the transition model.

Original: Moreover, without additional information about the transition
Revised: Moreover, without additional information about the transition

Original: S − 5 − 4 − 3 − 2 − 1 02 

Revised: S − 5 − 4 − 3 − 2 − 1 02

Original: S (cid:48) − 5 − 4 − 3 −−
Revised: S (cid:48) − 5 − 4 − 3 −−

Original: R ( S ,
Revised: R ( S ,

Original: S (cid:48) ) − 4 −− 2 −
Revised: S (cid:48) ) − 4 −− 2 −

Original: R ( s,s (cid:48) )
Revised: R ( s,s (cid:48) )

Original: E [ R ( s,S (cid:48) )] E [ R ( s (cid:48) ,S (cid:48) )]
Revised: E [ R ( s,S (cid:48) )] E [ R ( s (cid:48) ,S (cid:48) )]

Original: E [ R ( S,S (cid:48) )]
Revised: E [ R ( S,S (cid:48) )]

Original: C D S , D A
Revised: C D S , D A

Original: ( R )( s,s (cid:48) )
Revised: ( R )( s,s (cid:48) )

Original: S − 5 − 4 − 3 − 2 − 1 02 

Revised: S − 5 − 4 − 3 − 2 − 1 02

Original: S (cid:48) − 5 − 4 − 3 −−
Revised: S (cid:48) − 5 − 4 − 3 −−

Original: R ( S ,
Revised: R ( S ,

Original: S (cid:48) ) − 4 −− 2 −
Revised: S (cid:48) ) − 4 −− 2 −

Original: R ( s,s (cid:48) )
Revised: R ( s,s (cid:48) )

Original: E [ R ( s,S (cid:48) )]
Revised: E [ R ( s,S (cid:48) )]

Original: E [ R ( s (cid:48) ,S (cid:48)(cid:48) )] E [ R ( S (cid:48) ,S (cid:48)(cid:48) )]
Revised: E [ R ( s (cid:48) ,S (cid:48)(cid:48) )] E [ R ( S (cid:48) ,S (cid:48)(cid:48) )]

Original: C T ( R )( s,s (cid:48) )
Revised: C T ( R )( s,s (cid:48) )

Original: (cid:48) in the visualization by samples comprising the different expectations being dispersed globally over S and/or S (cid:48) .
Revised: (cid:48) in the visualization by samples comprising the different expectations being dispersed globally over S and/or S (cid:48) .

Original: Our approach in contrast uses a transition model to transform the reward function using approximately in-distribution transitions as illustrated visually by its local sampling in computing the expectations.
Revised: Our approach in contrast uses a transition model to transform the reward function using approximately in-distribution transitions as illustrated visually by its local sampling in computing the expectations.

Original: Both approaches arrive at the unshaped value of zero depicted by the red triangle.
Revised: Both approaches arrive at the unshaped value of zero depicted by the red triangle.

Original: model, two reward functions that behave identically on in-distribution transitions, but differently on out-of-distribution transitions, could justiﬁably be regarded as different.
Revised: model, two reward functions that behave identically on in-distribution transitions, but differently on out-of-distribution transitions, could justiﬁably be regarded as different.

Original: However, if these reward functions are learned from data, evaluating them on out-of-distribution transitions can lead to unreliable reward values, introducing errors into the reward distance computed by EPIC.
Revised: However, if these reward functions are learned from data, evaluating them on out-of-distribution transitions can lead to unreliable reward values, introducing errors into the reward distance computed by EPIC.

Original: For example, the reward value of a transition in which a home robot teleports across a room is likely to be arbitrarily incorrect, since no similar transition would be observed when the robot interacts with its environment.
Revised: For example, the reward value of a transition in which a home robot teleports across a room is likely to be arbitrarily incorrect, since no similar transition would be observed when the robot interacts with its environment.

Original: Because EPIC necessarily evaluates learned reward functions on out-of-distribution data when computing reward distances, these distances can have signiﬁcant inaccuracies as we show empirically in Section 5.
Revised: Because EPIC necessarily evaluates learned reward functions on out-of-distribution data when computing reward distances, these distances can have signiﬁcant inaccuracies as we show empirically in Section 5.

Original: To address this challenge, we assume access to a transition model of the environment (which may also be learned in practice), and propose Dynamics-Aware Reward Distance (DARD), a new reward distance measure, which builds on EPIC.
Revised: To address this challenge, we assume access to a transition model of the environment (which may also be learned in practice), and propose Dynamics-Aware Reward Distance (DARD), a new reward distance measure, which builds on EPIC.

Original: DARD computes an alternative reward transformation that evaluates reward functions on transitions that are much closer to those observed in the environment.
Revised: DARD computes an alternative reward transformation that evaluates reward functions on transitions that are much closer to those observed in the environment.

Original: This helps to encourage evaluation of reward functions closer to the distribution of transitions on which they were trained while still providing a distance metric between reward functions that is invariant to potential shaping.
Revised: This helps to encourage evaluation of reward functions closer to the distribution of transitions on which they were trained while still providing a distance metric between reward functions that is invariant to potential shaping.

Original: Deﬁnition 4.1 (Dynamics-Aware Transformation) .
Revised: Deﬁnition 4.1 (Dynamics-Aware Transformation) .

Original: Let R : S ×
Revised: Let R : S ×

Original: A × S → R be a reward function.
Revised: A × S → R be a reward function.

Original: Given distributions D S ∈ ∆( S ) and D A ∈ ∆( A ) over states and actions, let S be a random variable distributed as D S , and A be a random variable distributed as D A .
Revised: Given distributions D S ∈ ∆( S ) and D A ∈ ∆( A ) over states and actions, let S be a random variable distributed as D S , and A be a random variable distributed as D A .

Original: Furthermore, given a probabilistic transition model deﬁning a conditional distribution over next states T ( S (cid:48) | S, A ) , let
Revised: Furthermore, given a probabilistic transition model deﬁning a conditional distribution over next states T ( S (cid:48) | S, A ) , let

Original: S (cid:48) and S (cid:48)(cid:48) be random variables distributed as T ( S (cid:48) | s, A ) and T ( S (cid:48)(cid:48) | s (cid:48) , A ) respectively.
Revised: S (cid:48) and S (cid:48)(cid:48) be random variables distributed as T ( S (cid:48) | s, A ) and T ( S (cid:48)(cid:48) | s (cid:48) , A ) respectively.

Original: Deﬁne the dynamics-aware transformation of R as follows:
Revised: Deﬁne the dynamics-aware transformation of R as follows:

Original: [Equation]
Revised: [Equation]

Original: By sampling next states from the transition model conditioning on the current state (either s or s (cid:48) ), this transformation is able to evaluate reward models at ( s, a, s (cid:48) )
Revised: By sampling next states from the transition model conditioning on the current state (either s or s (cid:48) ), this transformation is able to evaluate reward models at ( s, a, s (cid:48) )

Original: transitions that are closer to their training distribution.
Revised: transitions that are closer to their training distribution.

Original: In the case of the term E [ R ( S (cid:48) , A, S (cid:48)(cid:48) )] , the transitions evaluated are not perfectly in-distribution with respect to T .
Revised: In the case of the term E [ R ( S (cid:48) , A, S (cid:48)(cid:48) )] , the transitions evaluated are not perfectly in-distribution with respect to T .

Original: This is because S (cid:48)(cid:48) is distributed conditionally on the original s (cid:48) as opposed to the random variable S (cid:48) .
Revised: This is because S (cid:48)(cid:48) is distributed conditionally on the original s (cid:48) as opposed to the random variable S (cid:48) .

Original: These transitions are nevertheless much closer to physically realizable than those sampled in EPIC.
Revised: These transitions are nevertheless much closer to physically realizable than those sampled in EPIC.

Original: In Section 5 we show that learned reward functions are relatively insensitive to these slight errors in practice.
Revised: In Section 5 we show that learned reward functions are relatively insensitive to these slight errors in practice.

Original: We select D A to densely cover the space of possible actions.
Revised: We select D A to densely cover the space of possible actions.

Original: We do this because during policy optimization, policies may select a wide range of possible actions, resulting in evaluating learned reward functions on this same distribution of actions.
Revised: We do this because during policy optimization, policies may select a wide range of possible actions, resulting in evaluating learned reward functions on this same distribution of actions.

Original: See Figure 1 for a visualization of our dynamics-aware transformation.
Revised: See Figure 1 for a visualization of our dynamics-aware transformation.

Original: Informally, this operation normalizes the reward values of a transition ( s, a, s (cid:48) )
Revised: Informally, this operation normalizes the reward values of a transition ( s, a, s (cid:48) )

Original: based on the reward values of nearby transitions sampled from the transition model.
Revised: based on the reward values of nearby transitions sampled from the transition model.

Original: This normalization allows for a comparison between reward functions that is invariant to potential shaping.
Revised: This normalization allows for a comparison between reward functions that is invariant to potential shaping.

Original: Proposition 2 (The dynamics-aware transformed reward is invariant to shaping) .
Revised: Proposition 2 (The dynamics-aware transformed reward is invariant to shaping) .

Original: Let R : S ×
Revised: Let R : S ×

Original: A ×
Revised: A ×

Original: S → R be a reward function, and Φ : S → R a state potential function.
Revised: S → R be a reward function, and Φ : S → R a state potential function.

Original: Let γ ∈ [0 , 1] be a discount factor, and D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions, and
Revised: Let γ ∈ [0 , 1] be a discount factor, and D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions, and

Original: T : S ×
Revised: T : S ×

Original: A → ∆( S ) be a conditional distribution over next states.
Revised: A → ∆( S ) be a conditional distribution over next states.

Original: Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s (cid:48) )
Revised: Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s (cid:48) )

Original: =
Revised: =

Original: R ( s, a, s (cid:48) )
Revised: R ( s, a, s (cid:48) )

Original: + γ Φ( s (cid:48) )
Revised: + γ Φ( s (cid:48) )

Original: − Φ( s ) .
Revised: − Φ( s ) .

Original: Then the transition-relative transformations of R (cid:48) and R are equal: C T ( R (cid:48) )
Revised: Then the transition-relative transformations of R (cid:48) and R are equal: C T ( R (cid:48) )

Original: =
Revised: =

Original: C T ( R ) .
Revised: C T ( R ) .

Original: Proof: See Section A.4.
Revised: Proof: See Section A.4.

Original: Deﬁnition 4.2 (DARD Distance Measure) .
Revised: Deﬁnition 4.2 (DARD Distance Measure) .

Original: Let S , A , S (cid:48) be random variables jointly following some coverage transition distribution, and let T be the transition model of the corresponding environment.
Revised: Let S , A , S (cid:48) be random variables jointly following some coverage transition distribution, and let T be the transition model of the corresponding environment.

Original: We deﬁne the DARD pseudometric between rewards R A and R B as follows:
Revised: We deﬁne the DARD pseudometric between rewards R A and R B as follows:

Original: [Equation]
Revised: [Equation]

Original: We prove that the above distance measure is a pseudometric in Section A.4.
Revised: We prove that the above distance measure is a pseudometric in Section A.4.

Original: The key insight behind DARD is that we can use knowledge of transition dynamics to deﬁne a distance measure between reward functions that only evaluates those functions at realizable transitions.
Revised: The key insight behind DARD is that we can use knowledge of transition dynamics to deﬁne a distance measure between reward functions that only evaluates those functions at realizable transitions.

Original: Though, unlike EPIC, DARD assumes knowledge of transition dynamics and that these dynamics will remain similar during deployment, DARD uses these additional assumptions to produce significantly more accurate and predictive distances between learned reward functions in practice.
Revised: Though, unlike EPIC, DARD assumes knowledge of transition dynamics and that these dynamics will remain similar during deployment, DARD uses these additional assumptions to produce significantly more accurate and predictive distances between learned reward functions in practice.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In experiments we evaluate the following hypotheses: H1 : EPIC is a poor predictor of policy return for learned reward functions due to its out-of-distribution evaluation of these reward functions.
Revised: In experiments we evaluate the following hypotheses: H1 : EPIC is a poor predictor of policy return for learned reward functions due to its out-of-distribution evaluation of these reward functions.

Original: H2 :
Revised: H2 :

Original: DARD is predictive of policy return even for learned reward models, while maintaining EPIC’s invariance to reward shaping.
Revised: DARD is predictive of policy return even for learned reward models, while maintaining EPIC’s invariance to reward shaping.

Original: H3 :
Revised: H3 :

Original: DARD is predictive of policy performance across diverse coverage distributions.
Revised: DARD is predictive of policy performance across diverse coverage distributions.

Original: H4 :
Revised: H4 :

Original: DARD is predictive of policy return when using learned, high-ﬁdelity transition models.
Revised: DARD is predictive of policy return when using learned, high-ﬁdelity transition models.

Original: We test these hypotheses by computing each metric across a set of reward models, and then evaluate a policy trained on each reward model against a ground truth reward function through on-policy evaluation.
Revised: We test these hypotheses by computing each metric across a set of reward models, and then evaluate a policy trained on each reward model against a ground truth reward function through on-policy evaluation.

Original: We perform these steps across two environments as described next.
Revised: We perform these steps across two environments as described next.

Original: 5. E XPERIMENTAL S ETUP
Revised: 5. E XPERIMENTAL S ETUP

Original: Environments We evaluate DARD in two environments.
Revised: Environments We evaluate DARD in two environments.

Original: We ﬁrst study the Bouncing Balls environment, which requires an agent to navigate from a random initial position to a random goal position while avoiding a set of balls randomly bouncing around the scene.
Revised: We ﬁrst study the Bouncing Balls environment, which requires an agent to navigate from a random initial position to a random goal position while avoiding a set of balls randomly bouncing around the scene.

Original: This environment captures elements of real-world, physical environments with multiple interacting agents, as is the case, for example, in autonomous driving.
Revised: This environment captures elements of real-world, physical environments with multiple interacting agents, as is the case, for example, in autonomous driving.

Original: In these types of environments, EPIC samples transitions in a manner such that agents teleport around the scene, causing issues for learned reward models.
Revised: In these types of environments, EPIC samples transitions in a manner such that agents teleport around the scene, causing issues for learned reward models.

Original: The second environment is adapted from the Reacher environment from OpenAI Gym (Brockman et al., 2016).
Revised: The second environment is adapted from the Reacher environment from OpenAI Gym (Brockman et al., 2016).

Original: This environment captures aspects of robotic manipulation tasks where arbitrary changes in joint conﬁguration between s and s (cid:48) can cause issues for learned reward models.
Revised: This environment captures aspects of robotic manipulation tasks where arbitrary changes in joint conﬁguration between s and s (cid:48) can cause issues for learned reward models.

Original: Videos illustrating both environments can be found at https://sites.google.com/view/dard-paper .
Revised: Videos illustrating both environments can be found at https://sites.google.com/view/dard-paper .

Original: Coverage Distributions
Revised: Coverage Distributions

Original: In order to compute reward distances for DARD, we must sample transitions from the environments.
Revised: In order to compute reward distances for DARD, we must sample transitions from the environments.

Original: In the experiments we consider two coverage distributions.
Revised: In the experiments we consider two coverage distributions.

Original: The ﬁrst is the distribution induced by a policy that takes actions uniformly at random across magnitude bounds (denoted by π uni ).
Revised: The ﬁrst is the distribution induced by a policy that takes actions uniformly at random across magnitude bounds (denoted by π uni ).

Original: The second is the distribution over transitions of an expert policy π ∗ , which we learn in both environments using Proximal Policy Optimization (PPO)
Revised: The second is the distribution over transitions of an expert policy π ∗ , which we learn in both environments using Proximal Policy Optimization (PPO)

Original: Schulman et al.
Revised: Schulman et al.

Original: Action Sampling and Transition Models
Revised: Action Sampling and Transition Models

Original: Both environments have a 2D, continuous action space.
Revised: Both environments have a 2D, continuous action space.

Original: DARD samples actions from these spaces (i.e., deﬁnes D A ) by deﬁning an uniformly-spaced, 2D grid of action values within magnitude bounds.
Revised: DARD samples actions from these spaces (i.e., deﬁnes D A ) by deﬁning an uniformly-spaced, 2D grid of action values within magnitude bounds.

Original: We consider two versions of DARD in the experiments.
Revised: We consider two versions of DARD in the experiments.

Original: The ﬁrst uses (approximate) ground-truth transition models, and the second uses learned transition models, which we refer to as DARD-L in the results.
Revised: The ﬁrst uses (approximate) ground-truth transition models, and the second uses learned transition models, which we refer to as DARD-L in the results.

Original: For the ground-truth transition model in the Bouncing Balls environment we use a constant velocity model that deterministically samples next states conditional on each sampled action.
Revised: For the ground-truth transition model in the Bouncing Balls environment we use a constant velocity model that deterministically samples next states conditional on each sampled action.

Original: For the Reacher environment, we use the underlying MuJoCo simulator (Todorov et al., 2012) as the transition model, again deterministically sampling next states.
Revised: For the Reacher environment, we use the underlying MuJoCo simulator (Todorov et al., 2012) as the transition model, again deterministically sampling next states.

Original: We additionally learn neural network transition models for each environment.
Revised: We additionally learn neural network transition models for each environment.

Original: In the Bouncing
Revised: In the Bouncing

Original: Balls environment, we learn a transition model that independently predicts position deltas for each ball based on its physical state and the action (in the case of the controlled agent).
Revised: Balls environment, we learn a transition model that independently predicts position deltas for each ball based on its physical state and the action (in the case of the controlled agent).

Original: In the Reacher environment, we learn a transition model that takes as input the state and action, and predicts a state delta used to produce the next state.
Revised: In the Reacher environment, we learn a transition model that takes as input the state and action, and predicts a state delta used to produce the next state.

Original: See Appendix A.1.4 for model architecture and training details.
Revised: See Appendix A.1.4 for model architecture and training details.

Original: Reward Models For each environment, we deﬁne or learn the following reward models:
Revised: Reward Models For each environment, we deﬁne or learn the following reward models:

Original: GROUND TRUTH:
Revised: GROUND TRUTH:

Original: The reward function against which we perform on-policy evaluation.
Revised: The reward function against which we perform on-policy evaluation.

Original: In both environments, this is a goal-based reward given when the agent position is within some range of the goal location.
Revised: In both environments, this is a goal-based reward given when the agent position is within some range of the goal location.

Original: The Reacher environment also contains additive terms penalizing distance from the goal and large action magnitudes.
Revised: The Reacher environment also contains additive terms penalizing distance from the goal and large action magnitudes.

Original: SHAPED:
Revised: SHAPED:

Original: The ground truth reward function, shaped by a potential function.
Revised: The ground truth reward function, shaped by a potential function.

Original: In both environments, we use a potential shaping based on the distance to the goal location.
Revised: In both environments, we use a potential shaping based on the distance to the goal location.

Original: In the Bouncing
Revised: In the Bouncing

Original: Balls environment, we take the difference in the square root of the distance to goal (due to the large distance values).
Revised: Balls environment, we take the difference in the square root of the distance to goal (due to the large distance values).

Original: In the Reacher environment we use the difference of distances from the end effector to goal directly.
Revised: In the Reacher environment we use the difference of distances from the end effector to goal directly.

Original: FEASIBILITY:
Revised: FEASIBILITY:

Original: A manually-deﬁned reward model for the Bouncing Balls environment, which returns the shaped reward function when evaluated with a dynamically-feasible transition, and returns unit Gaussian noise otherwise.
Revised: A manually-deﬁned reward model that returns the shaped reward function when evaluated with a dynamically-feasible transition, and returns unit Gaussian noise otherwise.

Original: This serves as a sanity check to illustrate that EPIC performs poorly on models that exhibit inaccurate reward values on OOD transitions.
Revised: This serves as a sanity check to illustrate that EPIC performs poorly on models that exhibit inaccurate reward values on OOD transitions.

Original: REGRESS:
Revised: REGRESS:

Original: A learned model trained via regression to the shaped reward model.
Revised: A learned model trained via regression to the shaped reward model.

Original: We regress the shaped reward function because otherwise the metrics do not need to account for shaping (see Appendix A.2.1 for an example).
Revised: We regress the shaped reward function because otherwise the metrics do not need to account for shaping (see Appendix A.2.1 for an example).

Original: Although simple, this method is used in practice in cases where people manually label reward functions (Cabi et al., 2019), and was used as a baseline in Christiano et al.
Revised: Although simple, this method is used in practice in cases where people manually label reward functions (Cabi et al., 2019), and was used as a baseline in Christiano et al.

Original: (“target” method, section 3.3).
Revised: (“target” method, section 3.3).

Original: REGRESS OUT-OF-DISTRIBUTION:
Revised: REGRESS OUT-OF-DISTRIBUTION:

Original: A model trained similarly to REGRESS, except that it is trained “out-of-distribution” in the sense that during training it samples ( a , s (cid:48) ) pairs randomly from the dataset such that its training distribution matches the evaluation distribution of EPIC.
Revised: A model trained similarly to REGRESS, except that it is trained “out-of-distribution” in the sense that during training it samples ( a , s (cid:48) ) pairs randomly from the dataset such that its training distribution matches the evaluation distribution of EPIC.

Original: PREFERENCES: A reward model learned using preference comparisons, similar to the approach from Christiano et al.
Revised: PREFERENCES: A reward model learned using preference comparisons, similar to the approach from Christiano et al.

Original: We synthesize preference labels based on ground-truth return.
Revised: We synthesize preference labels based on ground-truth return.

Original: All learned models are fully-connected neural networks.
Revised: All learned models are fully-connected neural networks.

Original: In the Bouncing Balls environment, the input to the network is a speciﬁc set of features designed to be susceptible to OOD issues (a squared distance is computed between s and s (cid:48) as one of the features).
Revised: In the Bouncing Balls environment, the input to the network is a speciﬁc set of features designed to be susceptible to OOD issues (a squared distance is computed between s and s (cid:48) as one of the features).

Original: For the Reacher environment, we concatenate s , a , and s (cid:48) as in the learned-reward experiments in Gleave et al.
Revised: For the Reacher environment, we concatenate s , a , and s (cid:48) as in the learned-reward experiments in Gleave et al.

Original: Baselines We compare with two baseline methods.
Revised: Baselines We compare with two baseline methods.

Original: First, we compare with EPIC.
Revised: First, we compare with EPIC.

Original: Second, we compare with a simple calculation of the Pearson distance ( D ρ ).
Revised: Second, we compare with a simple calculation of the Pearson distance ( D ρ ).

Original: We include this baseline in order to assess the extent to which reward shaping is an issue that needs to be accounted for in comparing rewards (see Appendix A.2.
Revised: We include this baseline in order to assess the extent to which reward shaping is an issue that needs to be accounted for in comparing rewards (see Appendix A.2.1 for additional motivation).

Original: 5. R ESULTS AND D ISCUSSION
Revised: 5. R ESULTS AND D ISCUSSION

Original: [Table]
Revised: [Table]

Original: Table 1 summarizes the experimental results, which provide support for H1 across both environments.
Revised: Table 1 summarizes the experimental results, which provide support for H1 across both environments.

Original: We ﬁrst note as a sanity check that EPIC assigns large distances between GROUND TRUTH and FEASIBILITY.
Revised: We ﬁrst note as a sanity check that EPIC assigns large distances between GROUND TRUTH and FEASIBILITY.

Original: This demonstrates EPIC’s tendency to assign large distances to reward functions that are equivalent when assuming a ﬁxed transition model [Table] but different under an arbitrary transition model.
Revised: This demonstrates EPIC’s tendency to assign large distances to reward functions that are equivalent when assuming a ﬁxed transition model, but different under an arbitrary transition model.

Original: This behavior manifests with learned models in practice due to EPIC’s
Revised: This behavior manifests with learned models in practice due to EPIC’s

Original: OOD evaluation of those models.
Revised: OOD evaluation of those models.

Original: For example, in both environments EPIC assigns large distances to the REGRESS model when, based on policy performance, we see that it is quite similar to the ground truth (disregarding shaping).
Revised: For example, in both environments EPIC assigns large distances to the REGRESS model when, based on policy performance, we see that it is quite similar to the ground truth (disregarding shaping).

Original: This results from EPIC’s evaluation of these reward models with OOD transitions, which is illustrated by the fact that the metric is more much predictive of on-policy performance on the REGRESS OOD model (which does not suffer from OOD evaluation issues).
Revised: This results from EPIC’s evaluation of these reward models with OOD transitions, which is illustrated by the fact that the metric is more much predictive of on-policy performance on the REGRESS OOD model (which does not suffer from OOD evaluation issues).

Original: Perhaps most importantly, EPIC fails to be predictive of episode return when comparing learned models.
Revised: Perhaps most importantly, EPIC fails to be predictive of episode return when comparing learned models.

Original: For example, in the Reacher environment, the metric assigns a lower distance to PREFERENCES (PREF) than REGRESS despite the lower episode returns of PREF.
Revised: For example, in the Reacher environment, the metric assigns a lower distance to PREFERENCES (PREF) than REGRESS despite the lower episode returns of PREF.

Original: The results from both environments also support H2 .
Revised: The results from both environments also support H2 .

Original: In the Bouncing Balls environment, DARD gives low distances for the well-ﬁt models (FEASIBILITY, REGRESS, REGRESS OOD) as a result of only evaluating them in their training distribution.
Revised: In the Bouncing Balls environment, DARD gives low distances for the well-ﬁt models (FEASIBILITY, REGRESS, REGRESS OOD) as a result of only evaluating them in their training distribution.

Original: In the Reacher environment, DARD exhibits a gap between REGRESS and REGRESS OOD, which indicates that the algorithm’s evaluation of the reward model on slightly out-of-distribution transitions does hurt performance.
Revised: In the Reacher environment, DARD exhibits a gap between REGRESS and REGRESS OOD, which indicates that the algorithm’s evaluation of the reward model on slightly out-of-distribution transitions does hurt performance.

Original: Nevertheless, on the reward models that can be learned in practice (REGRESS, PREF) DARD produces distances that are more predictive of episode return compared to baseline methods.
Revised: Nevertheless, on the reward models that can be learned in practice (REGRESS, PREF) DARD produces distances that are more predictive of episode return compared to baseline methods.

Original: The results also demonstrate that DARD is predictive of episode return across coverage distributions ( H3 ).
Revised: The results also demonstrate that DARD is predictive of episode return across coverage distributions ( H3 ).

Original: DARD is sensitive to the choice of coverage distribution because the transitions the metric samples are tightly grouped around this distribution.
Revised: DARD is sensitive to the choice of coverage distribution because the transitions the metric samples are tightly grouped around this distribution.

Original: This is reﬂected in the different distance scales between π uni and π ∗ .
Revised: This is reﬂected in the different distance scales between π uni and π ∗ .

Original: Nevertheless, the relative value of DARD within a sampling distribution is still predictive of episode return.
Revised: Nevertheless, the relative value of DARD within a sampling distribution is still predictive of episode return.

Original: For example, the metric maintains a consistent ordering of the models across π uni and π ∗ in both environments.
Revised: For example, the metric maintains a consistent ordering of the models across π uni and π ∗ in both environments.

Original: Finally, the results largely support the hypothesis ( H4 )
Revised: Finally, the results largely support the hypothesis ( H4 )

Original: that replacing ground truth transition models in DARD with high-ﬁdelity learned ones still results in distance values predictive of policy return.
Revised: that replacing ground truth transition models in DARD with high-ﬁdelity learned ones still results in distance values predictive of policy return.

Original: This conclusion is supported by DARD-L’s predictiveness of policy return across both environments and coverage distributions.
Revised: This conclusion is supported by DARD-L’s predictiveness of policy return across both environments and coverage distributions.

Original: In most cases, the distance values of DARD-L closely match those of
Revised: In most cases, the distance values of DARD-L closely match those of

Original: DARD.
Revised: DARD.

Original: Three reasons for this close correspondence are: (i) high-ﬁdelity transition models can be ﬁt in the relatively low-dimensional environments considered, (ii) DARD uses transition models to make single-step predictions, which avoids the compounding errors that may occur when sampling rollouts, and (iii) the environments and coverage distributions considered generally produce reward functions insensitive to small errors in the predicted states.
Revised: Three reasons for this close correspondence are: (i) high-ﬁdelity transition models can be ﬁt in the relatively low-dimensional environments considered, (ii) DARD uses transition models to make single-step predictions, which avoids the compounding errors that may occur when sampling rollouts, and (iii) the environments and coverage distributions considered generally produce reward functions insensitive to small errors in the predicted states.

Original: The lower distance values of DARD-L in the Reacher environment when using the expert coverage distribution are an exception to this trend.
Revised: The lower distance values of DARD-L in the Reacher environment when using the expert coverage distribution are an exception to this trend.

Original: By analyzing the errors made by the learned transition model, we concluded that the lower distances of DARD-L result from the reward functions being quite sensitive to small errors in predicted states in this case (see Appendix A.3.2 for detailed analysis).
Revised: By analyzing the errors made by the learned transition model, we concluded that the lower distances of DARD-L result from the reward functions being quite sensitive to small errors in predicted states in this case (see Appendix A.3.2 for detailed analysis).

Original: Although the distance values are still predictive of policy return, this result is an instance of the false positive equivalence that can result from errors in the transition model (as discussed in Appendix A.3.1 (case 2.B)), and illustrates the importance of applying DARD with well-ﬁt learned transitions models.
Revised: Although the distance values are still predictive of policy return, this result is an instance of the false positive equivalence that can result from errors in the transition model (as discussed in Appendix A.3.1 (case 2.B)), and illustrates the importance of applying DARD with well-ﬁt learned transitions models.

Original: In our experiments, we discovered that DARD is poorly suited for reward comparison in environments where nearby transitions exclusively have similar reward values.
Revised: In our experiments, we discovered that DARD is poorly suited for reward comparison in environments where nearby transitions exclusively have similar reward values.

Original: This is the case, for example, in robotic domains with small timesteps and in which a smooth reward function is used (such as negative distance to the goal).
Revised: This is the case, for example, in robotic domains with small timesteps and in which a smooth reward function is used (such as negative distance to the goal).

Original: We illustrate this behavior and explain why it occurs in Appendix A.2.
Revised: We illustrate this behavior and explain why it occurs in Appendix A.2.

Original: with experiments on the Point Maze environment used by Gleave et al.
Revised: with experiments on the Point Maze environment used by Gleave et al.

Original: In short, because
Revised: In short, because

Original: DARD performs reward comparisons between transitions relative to a transition model, if all the transitions it considers (across all possible actions) have nearly-identical reward values, it can be sensitive to noise in the reward model.
Revised: DARD performs reward comparisons between transitions relative to a transition model, if all the transitions it considers (across all possible actions) have nearly-identical reward values, it can be sensitive to noise in the reward model.

Original: For this reason, DARD should be used in environments for which, at least for some parts of the state-action space, nearby transitions yield well-differentiated reward values.
Revised: For this reason, DARD should be used in environments for which, at least for some parts of the state-action space, nearby transitions yield well-differentiated reward values.

Original: Overall, these experiments demonstrate that in situations where we have conﬁdence the transition model will not change signiﬁcantly, as is the case in many physical settings, DARD allows for comparing learned reward functions with other reward functions without policy optimization.
Revised: Overall, these experiments demonstrate that in situations where we have conﬁdence the transition model will not change signiﬁcantly, as is the case in many physical settings, DARD allows for comparing learned reward functions with other reward functions without policy optimization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The ability to reliably compare reward functions is a critical tool for ensuring that intelligent systems have objectives that are aligned with human intentions.
Revised: The ability to reliably compare reward functions is a critical tool for ensuring that intelligent systems have objectives that are aligned with human intentions.

Original: The typical approach to comparing reward functions, which involves learning policies on them, conﬂates errors in the reward with errors in policy optimization.
Revised: The typical approach to comparing reward functions, which involves learning policies on them, conﬂates errors in the reward with errors in policy optimization.

Original: We present Dynamics-Aware Reward Distance (DARD), a pseudometric that allows for comparing rewards directly while being invariant to reward shaping.
Revised: We present Dynamics-Aware Reward Distance (DARD), a pseudometric that allows for comparing rewards directly while being invariant to reward shaping.

Original: DARD uses knowledge of the transition dynamics to avoid evaluating learned reward functions far outside their training distribution.
Revised: DARD uses knowledge of the transition dynamics to avoid evaluating learned reward functions far outside their training distribution.

Original: Experiments suggest that for environments where we can assume the transition dynamics will not change signiﬁcantly between evaluation and deployment time that DARD reliably predicts policy performance associated with a reward function, even learned ones, without learning policies and without requiring online environment interaction.
Revised: Experiments suggest that for environments where we can assume the transition dynamics will not change signiﬁcantly between evaluation and deployment time that DARD reliably predicts policy performance associated with a reward function, even learned ones, without learning policies and without requiring online environment interaction.

Original: There are a number of exciting areas for future work.
Revised: There are a number of exciting areas for future work.

Original: First, DARD could be used to deﬁne benchmarks for evaluating reward learning methods (for example in autonomous driving).
Revised: First, DARD could be used to deﬁne benchmarks for evaluating reward learning methods (for example in autonomous driving).

Original: Second, DARD could be applied to high-dimensional (e.g., visual) domains where learned transition models are necessary and for which OOD evaluation issues are likely particularly challenging to overcome.
Revised: Second, DARD could be applied to high-dimensional (e.g., visual) domains where learned transition models are necessary and for which OOD evaluation issues are likely particularly challenging to overcome.

Original: Last, we are interested in investigating whether the rapid reward evaluation provided by DARD can be used to enable contrastive learning of reward functions.
Revised: Last, we are interested in investigating whether the rapid reward evaluation provided by DARD can be used to enable contrastive learning of reward functions.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We have attempted to make the experimental results easily reproducible by providing the following: ﬁrst, source code necessary to reproduce results is available in the supplementary material.
Revised: We have attempted to make the experimental results easily reproducible by providing the following: ﬁrst, source code necessary to reproduce results is available in the supplementary material.

Original: Second, the software for this research is designed to allow for reproducing the results in the main body of the text through a single program execution.
Revised: Second, the software for this research is designed to allow for reproducing the results in the main body of the text through a single program execution.

Original: Third, results are computed over ﬁve random seeds and standard error values are reported in appendix A.1.7 to provide reasonable bounds within which reproduced results should fall.
Revised: Third, results are computed over ﬁve random seeds and standard error values are reported in appendix A.1.7 to provide reasonable bounds within which reproduced results should fall.

Original: Fourth, we have attempted to provide extensive details regarding the experimental setup in appendix A.
Revised: Fourth, we have attempted to provide extensive details regarding the experimental setup in appendix A.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: The ability to reliably and efﬁciently evaluate reward functions without executing policies in the environment has the potential to ensure that AI systems will exhibit behavior that is aligned with the objective of the system designer before risking deploying these systems for making safety critical decisions.
Revised: The ability to reliably and efﬁciently evaluate reward functions without executing policies in the environment has the potential to ensure that AI systems will exhibit behavior that is aligned with the objective of the system designer before risking deploying these systems for making safety critical decisions.

Original: This has the potential to facilitate the application of reinforcement learning algorithms in settings in which this might otherwise be too risky, such as healthcare, ﬁnance, and robotics.
Revised: This has the potential to facilitate the application of reinforcement learning algorithms in settings in which this might otherwise be too risky, such as healthcare, ﬁnance, and robotics.

Original: A S UPPLEMENTARY M ATERIAL A. E XPERIMENTAL D ETAILS
Revised: A S UPPLEMENTARY M ATERIAL A. E XPERIMENTAL D ETAILS

Original: The following sub-sections provide details on the different components involved in the experiments
Revised: The following sub-sections provide details on the different components involved in the experiments

Original: . A.1. E NVIRONMENTS
Revised: . A.1. E NVIRONMENTS

Original: Bouncing Balls Environment This is a custom environment with traits of multi-agent navigation environments such as autonomous driving.
Revised: Bouncing Balls Environment This is a custom environment with traits of multi-agent navigation environments such as autonomous driving.

Original: The agent attempts to navigate to a random goal location while avoiding a set of balls (other agents) bouncing around the scene.
Revised: The agent attempts to navigate to a random goal location while avoiding a set of balls (other agents) bouncing around the scene.

Original: The state consists of the position and velocity of the balls (the agent included), as well as the goal location.
Revised: The state consists of the position and velocity of the balls (the agent included), as well as the goal location.

Original: The action space is a 2D continuous space of accelerations between [ − 5 , 5] to apply to the agent state.
Revised: The action space is a 2D continuous space of accelerations between [ − 5 , 5] to apply to the agent state.

Original: The transition function applies the agent accelerations to its state, while the other agents apply random accelerations sampled from a zero-mean, low variance Gaussian distribution.
Revised: The transition function applies the agent accelerations to its state, while the other agents apply random accelerations sampled from a zero-mean, low variance Gaussian distribution.

Original: The reward function gives a positive value once the agent is within some distance of the goal, which results in the agent and goal position being randomly moved.
Revised: The reward function gives a positive value once the agent is within some distance of the goal, which results in the agent and goal position being randomly moved.

Original: This environment executes for a ﬁxed horizon of 400 timesteps to approximate a continuing, inﬁnite horizon MDP.
Revised: This environment executes for a ﬁxed horizon of 400 timesteps to approximate a continuing, inﬁnite horizon MDP.

Original: We make the environment approximately continuing in order to ensure that S and S (cid:48) are distributed identically as is required by EPIC.
Revised: We make the environment approximately continuing in order to ensure that S and S (cid:48) are distributed identically as is required by EPIC.

Original: Sampling
Revised: Sampling

Original: S and S (cid:48) from their marginal distributions in the dataset results in transitions where the agents in s have completely different positions and velocities as in s (cid:48) , an issue that would be encountered in real-world multi-agent settings.
Revised: S and S (cid:48) from their marginal distributions in the dataset results in transitions where the agents in s have completely different positions and velocities as in s (cid:48) , an issue that would be encountered in real-world multi-agent settings.

Original: Reacher Environment This environment is adapted from the Reacher environment from OpenAI gym
Revised: Reacher Environment This environment is adapted from the Reacher environment from OpenAI gym

Original: (Brockman et al., 2016).
Revised: (Brockman et al., 2016).

Original: We adapt the environment in two ways.
Revised: We adapt the environment in two ways.

Original: First, we add a goal-based reward component.
Revised: First, we add a goal-based reward component.

Original: Second, we increase the timesteps between states to 5 from an original 2.
Revised: Second, we increase the timesteps between states to 5 from an original 2.

Original: We make both these changes so that the rewards of nearby transitions are sufﬁciently differentiated (see
Revised: We make both these changes so that the rewards of nearby transitions are sufﬁciently differentiated (see

Original: A.2.1 for detailed motivation).
Revised: A.2.1 for detailed motivation).

Original: A.1. P OLICY L EARNING
Revised: A.1. P OLICY L EARNING

Original: We used the RLlib (Liang et al., 2018) implementation of Proximal Policy Optimization (PPO) for learning control policies (Schulman et al., 2017).
Revised: We used the RLlib (Liang et al., 2018) implementation of Proximal Policy Optimization (PPO) for learning control policies (Schulman et al., 2017).

Original: Parameters are provided in table
Revised: Parameters are provided in table

Original: [Table]
Revised: [Table]

Original: We collect sets of datasets for use in reward learning and evaluation.
Revised: We collect sets of datasets for use in reward learning and evaluation.

Original: Each set contains three subdatasets for (i) reward model training, (ii) reward model validation
Revised: Each set contains three subdatasets for (i) reward model training, (ii) reward model validation

Original: [Table]
Revised: [Table]

Original: and (iii) reward evaluation.
Revised: and (iii) reward evaluation.

Original: We collect these dataset sets using two policies.
Revised: We collect these dataset sets using two policies.

Original: The ﬁrst policy ( π uni ) yields a uniform distribution on the action space.
Revised: The ﬁrst policy ( π uni ) yields a uniform distribution on the action space.

Original: The second policy is an expert ( π ∗ ) trained as described in A.1.2.
Revised: The second policy is an expert ( π ∗ ) trained as described in A.1.2.

Original: Table 3 gives the sizes of these datasets, which were set such that the validation and train losses were similar during reward learning.
Revised: Table 3 gives the sizes of these datasets, which were set such that the validation and train losses were similar during reward learning.

Original: [Table]
Revised: [Table]

Original: A.1. T RANSITION M ODEL L EARNING
Revised: A.1. T RANSITION M ODEL L EARNING

Original: This section provides details regarding the architecture and training of the learned transition models.
Revised: This section provides details regarding the architecture and training of the learned transition models.

Original: For the Bouncing Balls transition model, we split the state (originally containing the concatenated information for each ball in the scene and the goal position) into states for each ball and predict the motion of each independently.
Revised: For the Bouncing Balls transition model, we split the state (originally containing the concatenated information for each ball in the scene and the goal position) into states for each ball and predict the motion of each independently.

Original: We simplify the task in this manner in order to make learning the transition model easier, but a model that predicts the next state jointly could be learned instead.
Revised: We simplify the task in this manner in order to make learning the transition model easier, but a model that predicts the next state jointly could be learned instead.

Original: The absolute velocity of each ball is then passed to a fully-connected network, which outputs a position delta to be applied to the original state.
Revised: The absolute velocity of each ball is then passed to a fully-connected network, which outputs a position delta to be applied to the original state.

Original: In the case of the controlled ball, the action is also provided to the network.
Revised: In the case of the controlled ball, the action is also provided to the network.

Original: Given this formulation, the network is learning to translate a velocity into a change in position over some time period, optionally applying an acceleration value as well.
Revised: Given this formulation, the network is learning to translate a velocity into a change in position over some time period, optionally applying an acceleration value as well.

Original: For the Reacher environment, we pass the concatenated state and action to a fully-connected network, and again predict changes in the state.
Revised: For the Reacher environment, we pass the concatenated state and action to a fully-connected network, and again predict changes in the state.

Original: We normalize the scale of each state dimension by dividing by its standard deviation across the dataset.
Revised: We normalize the scale of each state dimension by dividing by its standard deviation across the dataset.

Original: In both cases
Revised: In both cases

Original: [Table]
Revised: [Table]

Original: we use a mean squared error loss computed between (i) the model output (i
Revised: we use a mean squared error loss computed between (i) the model output (i

Original: [Table]
Revised: [Table]

Original: e., state delta) applied to the original state and (ii) the ground truth next state.
Revised: e., state delta) applied to the original state and (ii) the ground truth next state.

Original: We train the transition model on data sampled from π uni , which is common in learning transition models (Ebert et al., 2018).
Revised: We train the transition model on data sampled from π uni , which is common in learning transition models (Ebert et al., 2018).

Original: Model and training hyperparameters are given in Table 4.
Revised: Model and training hyperparameters are given in Table 4.

Original: [Table]
Revised: [Table]

Original: on datasets used for training see section A.1.3.
Revised: on datasets used for training see section A.1.3.

Original: Superscripts indicate parameters speciﬁc to Bouncing Balls environment (BB) or
Revised: Superscripts indicate parameters speciﬁc to Bouncing Balls environment (BB) or

Original: Reacher environment (R).
Revised: Reacher environment (R).

Original: A.1. R EWARD L EARNING
Revised: A.1. R EWARD L EARNING

Original: Three methods of learning reward functions were considered in this research: (i) direct regression of per-timestep rewards (REGRESS), (ii) regression on a shufﬂed dataset matching the evaluation distribution of EPIC (REGRESS OOD), and (iii) preference-based learning of rewards.
Revised: Three methods of learning reward functions were considered in this research: (i) direct regression of per-timestep rewards (REGRESS), (ii) regression on a shufﬂed dataset matching the evaluation distribution of EPIC (REGRESS OOD), and (iii) preference-based learning of rewards.

Original: These methods each ﬁt a fully-connected neural network, and their model and training hyperparameters are provided in table 5.
Revised: These methods each ﬁt a fully-connected neural network, and their model and training hyperparameters are provided in table 5.

Original: We used early stopping based on validation loss for all models to determine the number of training epochs.
Revised: We used early stopping based on validation loss for all models to determine the number of training epochs.

Original: For the Bouncing Balls environment we input into the model a set of manually extracted features including distance of the agent to the goal in the current and next state, the difference in distances between states, an indicator of whether the goal has been reached, squared differences in position of other agents, and others.
Revised: For the Bouncing Balls environment we input into the model a set of manually extracted features including distance of the agent to the goal in the current and next state, the difference in distances between states, an indicator of whether the goal has been reached, squared differences in position of other agents, and others.

Original: Features involving squared distances are what make these model susceptible to OOD issues.
Revised: Features involving squared distances are what make these model susceptible to OOD issues.

Original: For the Reacher environment we concatenate the state, action, and next state, and pass that directly into the model.
Revised: For the Reacher environment we concatenate the state, action, and next state, and pass that directly into the model.

Original: In either case, these features are normalized by subtracting their mean and dividing by their standard deviation.
Revised: In either case, these features are normalized by subtracting their mean and dividing by their standard deviation.

Original: [Table]
Revised: [Table]

Original: A.1. R EWARD E VALUATION
Revised: A.1. R EWARD E VALUATION

Original: EPIC We compute the sample-based approximation of EPIC described in section A.1.1 of Gleave et al.
Revised: EPIC We compute the sample-based approximation of EPIC described in section A.1.1 of Gleave et al.

Original: Table 6 gives the parameters we use in computing EPIC.
Revised: Table 6 gives the parameters we use in computing EPIC.

Original: We use the same values of
Revised: We use the same values of

Original: N V and N M as Gleave et al.
Revised: N V and N M as Gleave et al.

Original: (2020), though use fewer seeds.
Revised: (2020), though use fewer seeds.

Original: In our case each seed represents a full execution of the pipeline (data collection, reward model training, reward evaluation), whereas in Gleave et al
Revised: In our case each seed represents a full execution of the pipeline (data collection, reward model training, reward evaluation), whereas in Gleave et al

Original: [Table]
Revised: [Table]

Original: (2020) it represents different sub-samplings of a single dataset.
Revised: (2020) it represents different sub-samplings of a single dataset.

Original: We take this approach in order to assess sensitivity to randomness in the data collection and reward learning processes
Revised: We take this approach in order to assess sensitivity to randomness in the data collection and reward learning processes

Original: [Table]
Revised: [Table]

Original: DARD Similarly to EPIC, we compute a sample-based approximation of DARD in the experiments.
Revised: DARD Similarly to EPIC, we compute a sample-based approximation of DARD in the experiments.

Original: We sample a batch B V of N V ( s, a, s (cid:48) ) transitions.
Revised: We sample a batch B V of N V ( s, a, s (cid:48) ) transitions.

Original: For computing the expectations E
Revised: For computing the expectations E

Original: [ R ( s, A, S (cid:48) )] and E [ R ( s (cid:48) , A, S (cid:48)(cid:48) )
Revised: [ R ( s, A, S (cid:48) )] and E [ R ( s (cid:48) , A, S (cid:48)(cid:48) )

Original: ] we sample N A actions u using a sampling policy (described in Appendix A.1.3).
Revised: ] we sample N A actions u using a sampling policy (described in Appendix A.1.3).

Original: We then sample N T samples of S (cid:48) or S (cid:48)(cid:48) , x (cid:48) and x (cid:48)(cid:48) , conditional on s or s (cid:48) and u .
Revised: We then sample N T samples of S (cid:48) or S (cid:48)(cid:48) , x (cid:48) and x (cid:48)(cid:48) , conditional on s or s (cid:48) and u .

Original: For the last expectation, E [ R ( S (cid:48) , A, S (cid:48)(cid:48) )] , we sample actions and next states similarly for S (cid:48) , but repeat the process for S (cid:48)(cid:48) , overall computing the mean over O (( N A N T ) 2 ) transitions.
Revised: For the last expectation, E [ R ( S (cid:48) , A, S (cid:48)(cid:48) )] , we sample actions and next states similarly for S (cid:48) , but repeat the process for S (cid:48)(cid:48) , overall computing the mean over O (( N A N T ) 2 ) transitions.

Original: We compute the dynamics-aware transformation (deﬁnition 3) for each transition by averaging over these sampled actions and next states:
Revised: We compute the dynamics-aware transformation (deﬁnition 3) for each transition by averaging over these sampled actions and next states:

Original: [Equation]
Revised: [Equation]

Original: Overall this yields an O ( N V N 2 A N 2 T ) algorithm.
Revised: Overall this yields an O ( N V N 2 A N 2 T ) algorithm.

Original: In our experiments the transition model is assumed to be deterministic, making N T = 1 .
Revised: In our experiments the transition model is assumed to be deterministic, making N T = 1 .

Original: The actions are sampled from the N-dimensional action space by taking the cross product of linearly-sampled values for each dimension.
Revised: The actions are sampled from the N-dimensional action space by taking the cross product of linearly-sampled values for each dimension.

Original: For example, in the Bouncing Balls environment, we sample 8 actions linearly between [ − 5 , 5] for each of the two dimensions, overall yielding N A = 64 .
Revised: For example, in the Bouncing Balls environment, we sample 8 actions linearly between [ − 5 , 5] for each of the two dimensions, overall yielding N A = 64 .

Original: The complexity of computing E [ R ( S (cid:48) , A, S (cid:48)(cid:48) )] could be reduced by instead sampling N A actions and N T transitions for each of x (cid:48) and x (cid:48)(cid:48) .
Revised: The complexity of computing E [ R ( S (cid:48) , A, S (cid:48)(cid:48) )] could be reduced by instead sampling N A actions and N T transitions for each of x (cid:48) and x (cid:48)(cid:48) .

Original: For the Bouncing Balls environment, we approximate the transition function with a constant velocity model for all agents, implemented in Pytorch and executed on the GPU (Paszke et al., 2019).
Revised: For the Bouncing Balls environment, we approximate the transition function with a constant velocity model for all agents, implemented in Pytorch and executed on the GPU (Paszke et al., 2019).

Original: For the Reacher environment, we use the ground truth simulator, which we execute in parallel acrosscores.
Revised: For the Reacher environment, we use the ground truth simulator, which we execute in parallel acrosscores.

Original: See table 7 for a summary of the DARD parameters used in the experiments.
Revised: See table 7 for a summary of the DARD parameters used in the experiments.

Original: [Table]
Revised: [Table]

Original: Policy Evaluation We compute average episode returns of policies trained on both manuallydeﬁned and learned reward models.
Revised: Policy Evaluation We compute average episode returns of policies trained on both manuallydeﬁned and learned reward models.

Original: Training parameters are the same as those used for learning the expert policy (see Table 2).
Revised: Training parameters are the same as those used for learning the expert policy (see Table 2).

Original: We evaluate policies for 1 million steps in the respective environment, and report the average, undiscounted episode return.
Revised: We evaluate policies for 1 million steps in the respective environment, and report the average, undiscounted episode return.

Original: Similarly to the other reward evaluation steps, we train one policy for each random seed (of which there are 5).
Revised: Similarly to the other reward evaluation steps, we train one policy for each random seed (of which there are 5).

Original: A.1. S TANDARD E RROR OF R ESULTS
Revised: A.1. S TANDARD E RROR OF R ESULTS

Original: Tables 8 and 9 provide the standard errors in the estimates of the mean distances and episode returns, computed over 5 random seeds.[Table]
Revised: Tables 8 and 9 provide the standard errors in the estimates of the mean distances and episode returns, computed over 5 random seeds.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A. A DDITIONAL E XPERIMENTS A.2. P OINT M AZE E XPERIMENTS
Revised: A. A DDITIONAL E XPERIMENTS A.2. P OINT M AZE E XPERIMENTS

Original: In this section we present results for the Point Maze environment considered in Gleave et al.
Revised: In this section we present results for the Point Maze environment considered in Gleave et al.

Original: These results illustrate a disadvantage of DARD.
Revised: These results illustrate a disadvantage of DARD.

Original: As discussed in section 5, because DARD performs reward comparisons between transitions relative to a transition model, if all the transitions it considers (across all possible actions) have near-identical reward values, it can be sensitive to noise in the reward model.
Revised: As discussed in section 5, because DARD performs reward comparisons between transitions relative to a transition model, if all the transitions it considers (across all possible actions) have near-identical reward values, it can be sensitive to noise in the reward model.

Original: The Point Maze environment ﬁts this description, and DARD performs poorly on it as a result (see table 10).
Revised: The Point Maze environment ﬁts this description, and DARD performs poorly on it as a result (see table 10).

Original: We provide evidence for the hypothesis that the lack of reward-differentiated transitions causes this poor performance by also providing results for an adapted Point Maze environment, which is identical to the original except that we increase the time difference between timesteps by a factor of 5 (such that nearby transitions yield different reward values).
Revised: We provide evidence for the hypothesis that the lack of reward-differentiated transitions causes this poor performance by also providing results for an adapted Point Maze environment, which is identical to the original except that we increase the time difference between timesteps by a factor of 5 (such that nearby transitions yield different reward values).

Original: In this adapted case, DARD and EPIC distances are signiﬁcantly closer than in the original environment.
Revised: In this adapted case, DARD and EPIC distances are signiﬁcantly closer than in the original environment.

Original: Two additional traits of the environment are relevant.
Revised: Two additional traits of the environment are relevant.

Original: First, the ground truth reward is only a function of the current state and action, but not the next state.
Revised: First, the ground truth reward is only a function of the current state and action, but not the next state.

Original: As a result, learned reward models tend to ignore the next state, which minimizes the effect of the OOD reward evaluations performed by
Revised: As a result, learned reward models tend to ignore the next state, which minimizes the effect of the OOD reward evaluations performed by

Original: EPIC.
Revised: EPIC.

Original: Second, in these experiments (unlike those in section 5) we train reward models to ﬁt the ground truth reward model (as opposed to a shaped version of it).
Revised: Second, in these experiments (unlike those in section 5) we train reward models to ﬁt the ground truth reward model (as opposed to a shaped version of it).

Original: We do this to be consistent with the experiments provided in Gleave et al.
Revised: We do this to be consistent with the experiments provided in Gleave et al.

Original: (2020); however, this results in the learned reward models exhibiting little shaping (as illustrated by the low distance values of D p ).
Revised: (2020); however, this results in the learned reward models exhibiting little shaping (as illustrated by the low distance values of D p ).

Original: This is a property of the reward learning algorithm and speciﬁc environment in which it is applied, but learned reward models may exhibit signiﬁcant shaping in general (Fu et al., 2017).
Revised: This is a property of the reward learning algorithm and speciﬁc environment in which it is applied, but learned reward models may exhibit signiﬁcant shaping in general (Fu et al., 2017).

Original: A.2. M ETRIC S AMPLE S IZE S ENSITIVITY
Revised: A.2. M ETRIC S AMPLE S IZE S ENSITIVITY

Original: How sensitive are reward comparison metrics to the size of the dataset on which they are computed?
Revised: How sensitive are reward comparison metrics to the size of the dataset on which they are computed?

Original: The answer depends on the speciﬁc environment, reward functions, and comparison metric under consideration.
Revised: The answer depends on the speciﬁc environment, reward functions, and comparison metric under consideration.

Original: As such, in this section we evaluate the sensitivity to sample size variations of DARD and EPIC in the Bouncing Balls and Reacher environments across a variety of manually deﬁned and learned reward functions.
Revised: As such, in this section we evaluate the sensitivity to sample size variations of DARD and EPIC in the Bouncing Balls and Reacher environments across a variety of manually deﬁned and learned reward functions.

Original: Evaluation Procedure Because we do not have “ground-truth” values that each metric should assume for reward functions in general, we focus instead on the variability of each reward comparison metric for a given sample size.
Revised: Evaluation Procedure Because we do not have “ground-truth” values that each metric should assume for reward functions in general, we focus instead on the variability of each reward comparison metric for a given sample size.

Original: We quantify this variability by computing, for each metric, a 95% conﬁdence interval (CI) of the mean reward distance across different sample sizes.
Revised: We quantify this variability by computing, for each metric, a 95% conﬁdence interval (CI) of the mean reward distance across different sample sizes.

Original: In this context, a smaller CI width indicates that a metric is less sensitive to the random variations in the sampling of the data.
Revised: In this context, a smaller CI width indicates that a metric is less sensitive to the random variations in the sampling of the data.

Original: We compute the CI by sampling K = 100 data subsets for each sample size N  from a much larger population of samples (we use the reward evaluation dataset described in Appendix A.1.3).
Revised: We compute the CI by sampling K = 100 datasets of size N for varying N from a much larger population of samples (we use the reward evaluation dataset described in Appendix A.1.3).

Original: For each subset, k, we compute each metric comparing each reward function to the ground-truth reward function.
Revised: For each dataset, we compute each metric comparing each reward function to the ground-truth reward function.

Original: We compute the CI based on the K resulting distances. For the evaluation datasets, we use π ∗ for the Bouncing Balls environment, and π uni for the Reacher environment.
Revised: We compute the CI based on the K resulting distances for each sample size N. For the evaluation datasets, we use π ∗ for the Bouncing Balls environment, and π uni for the Reacher environment.

Original: We use π ∗ for Bouncing Balls because the reward is sufﬁciently sparse that for small sample sizes, N, the metric values are frequently undeﬁned (due to only having samples with zero reward).
Revised: We use π ∗ for Bouncing Balls because the reward is sufﬁciently sparse that for small sample sizes, N, the metric values are frequently undeﬁned (due to only having samples with zero reward).

Original: Using π ∗ instead of π uni increases the fraction of transitions within the goal threshold addressing the problem.
Revised: Using π ∗ instead of π uni increases the fraction of transitions within the goal threshold addressing the problem.

Original: Reward Functions
Revised: Reward Functions

Original: We compare a set of manually deﬁned and learned reward functions.
Revised: We compare a set of manually deﬁned and learned reward functions.

Original: For the manually deﬁned reward functions, we desire a set of rewards that smoothly vary in similarity to the ground-truth reward.
Revised: For the manually deﬁned reward functions, we desire a set of rewards that smoothly vary in similarity to the ground-truth reward.

Original: To accomplish this, we use a noisy version of the ground-truth reward deﬁned as R noisy ( s, a, s (cid:48) )
Revised: To accomplish this, we use a noisy version of the ground-truth reward deﬁned as R noisy ( s, a, s (cid:48) )

Original: =
Revised: =

Original: R ( s, a, s (cid:48) )
Revised: R ( s, a, s (cid:48) )

Original: + (cid:15) for (cid:15) ∼ N (0 , σ ) .
Revised: + (cid:15) for (cid:15) ∼ N (0 , σ ) .

Original: By increasing σ we produce a sequence of reward functions that are increasingly different from the ground-truth reward.
Revised: By increasing σ we produce a sequence of reward functions that are increasingly different from the ground-truth reward.

Original: For the learned rewards, we use the REGRESS and REGRESS OOD reward models introduced in Section 5.
Revised: For the learned rewards, we use the REGRESS and REGRESS OOD reward models introduced in Section 5.

Original: Recall that REGRESS is trained on transitions sampled from a dataset of demonstrations collected from the environment, whereas REGRESS OOD is trained on the distribution of transitions resulting from sampling actions and next states independently from the starting state.
Revised: Recall that REGRESS is trained on transitions sampled from a dataset of demonstrations collected from the environment, whereas REGRESS OOD is trained on the distribution of transitions resulting from sampling actions and next states independently from the starting state.

Original: The resulting training distribution matches that used by EPIC in comparing rewards.
Revised: The resulting training distribution matches that used by EPIC in comparing rewards.

Original: Results Fig.
Revised: Results Fig.

Original: 2 shows the results of the experiment, with each row corresponding to a different reward function, and each column corresponding to a different environment (Bouncing Balls on the left and Reacher on the right).
Revised: 2 shows the results of the experiment, with each row corresponding to a different reward function, and each column corresponding to a different environment (Bouncing Balls on the left and Reacher on the right).

Original: We see that the 95% CI width for DARD and EPIC are quite similar for the noisy ground-truth reward functions (top three rows of the ﬁgure) across all sample sizes.
Revised: We see that the 95% CI width for DARD and EPIC are quite similar for the noisy ground-truth reward functions (top three rows of the ﬁgure) across all sample sizes.

Original: This indicates that, for these types of reward functions, the two metrics are similarly sensitive to the size of the dataset.
Revised: This indicates that, for these types of reward functions, the two metrics are similarly sensitive to the size of the dataset.

Original: Note that as the amount of noise increases so does the width of the CI, but the similarity of results across metrics holds.
Revised: Note that as the amount of noise increases so does the width of the CI, but the similarity of results across metrics holds.

Original: For the learned reward models we do observe different behavior between the metrics.
Revised: For the learned reward models we do observe different behavior between the metrics.

Original: In the case of the REGRESS model (4th row from the top), we see that DARD yields signiﬁcantly smaller CI widths than does EPIC.
Revised: In the case of the REGRESS model (4th row from the top), we see that DARD yields signiﬁcantly smaller CI widths than does EPIC.

Original: The reason for this is that EPIC evaluates the reward function at transitions outside its training distribution, which can yield arbitrary values.
Revised: The reason for this is that EPIC evaluates the reward function at transitions outside its training distribution, which can yield arbitrary values.

Original: The additional randomness (with respect to the random sampling of the data subset) incurred in these reward calculations results in more variation in the reward distance, which produces larger CIs.
Revised: The additional randomness (with respect to the random sampling of the data subset) incurred in these reward calculations results in more variation in the reward distance, which produces larger CIs.

Original: In contrast, DARD avoids these
Revised: In contrast, DARD avoids these

Original: OOD evaluations of the reward, instead staying close to the reward function training distribution where it is more consistent (across data subsets), producing smaller CIs.
Revised: OOD evaluations of the reward, instead staying close to the reward function training distribution where it is more consistent (across data subsets), producing smaller CIs.

Original: For the REGRESS OOD reward model (bottom row), we observe the inverse: EPIC produces smaller CI widths than does DARD.
Revised: For the REGRESS OOD reward model (bottom row), we observe the inverse: EPIC produces smaller CI widths than does DARD.

Original: This is because the reward model training distribution matches the one used by EPIC.
Revised: This is because the reward model training distribution matches the one used by EPIC.

Original: As a result, the “imagined” transitions of EPIC effectively increase the size of the evaluation dataset producing less variation for a given sample size.
Revised: As a result, the “imagined” transitions of EPIC effectively increase the size of the evaluation dataset producing less variation for a given sample size.

Original: DARD also uses “imagined” transitions, but only from the distribution of transitions encountered in practice in the environment.
Revised: DARD also uses “imagined” transitions, but only from the distribution of transitions encountered in practice in the environment.

Original: Conclusion We evaluated the sensitivity of DARD and EPIC to variations in sample size across different environments and reward models.
Revised: Conclusion We evaluated the sensitivity of DARD and EPIC to variations in sample size across different environments and reward models.

Original: We found that for noisy ground-truth reward functions, the two metrics performed similarly, and that for learned reward functions, the relative sensitivity of the metrics depends on the training distribution of the reward function under consideration.
Revised: We found that for noisy ground-truth reward functions, the two metrics performed similarly, and that for learned reward functions, the relative sensitivity of the metrics depends on the training distribution of the reward function under consideration.

Original: A. A DDITIONAL D ISCUSSION A.3. T HE I MPACT OF T RANSITION M ODEL E RRORS ON DARD
Revised: A.2. R ANDOMIZED R EWARD F UNCTIONS

Revised: The results in Section 5 focus on a special set of reward functions that are relatively similar to the ground-truth reward function.

Original: In this section, we discuss how errors made by the transition model can impact the performance of DARD.
Revised: In this section we instead attempt to address the question, how does DARD perform in the full range of reward similarity?

Original: We organize our discussion of this topic around different cases of reward equivalence and different types of errors that a transition model might make. The high-level summary of the following discussion is that errors in the transition model only impact the performance of DARD on reward functions that are not equivalent everywhere (case 2 below), and in this case DARD may produce both false negative and false positive equivalences as a result of transition model errors.
Revised: We investigate this question by (i) randomly generating a large number of reward functions in the Reacher environment, (ii) optimizing a policy for each reward function, (iii) evaluating that policy against the ground-truth reward function, and (iv) computing the DARD distance between the randomly generated and ground-truth rewards. Evaluation Procedure The rewards are generated by randomly sampling weights for the three reward components: distance from goal , action magnitude , goal-reached

Original: [Figure]
Revised: [Figure]

Original: C on ﬁ d e n ce
Revised: C on ﬁ d e n ce

Original: I n t e r v a l W i d
Revised: I n t e r v a l W i d

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Original: [Figure]
Revised: [Figure]

Original: C on ﬁ d e n ce
Revised: C on ﬁ d e n ce

Original: I n t e r v a l W i d t h
Revised: I n t e r v a l W i d t h

Original: Ground Truth + Noise
Revised: Ground Truth + Noise

Original: [Figure]
Revised: [Figure]

Original: σ = 0.1)
Revised: σ = 0.1)

Original: [Figure]
Revised: [Figure]

Original: C on ﬁ d e n ce
Revised: C on ﬁ d e n ce

Original: I n t e r v a l W i d t h
Revised: I n t e r v a l W i d t h

Original: [Figure]
Revised: [Figure]

Original: σ = 1)
Revised: σ = 1)

Original: [Figure]
Revised: [Figure]

Original: C on ﬁ d e n ce
Revised: C on ﬁ d e n ce

Original: I n t e r v a l W i d t h
Revised: I n t e r v a l W i d t h

Original: Regression
Revised: Regression

Original: [Figure]
Revised: [Figure]

Original: C on ﬁ d e n ce
Revised: C on ﬁ d e n ce

Original: I n t e r v a l W i d t h
Revised: I n t e r v a l W i d t h

Original: Regression Out-of
Revised: Regression Out-of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: indicator .

Revised: Weights for the ﬁrst two components are sampled uniformly between [0 , 1] .

Revised: The reason we disallow negative weights for these components is that they make policy optimization challenging or ill-posed: in the distance from goal case it creates a challenging exploration problem, and in the action magnitude case it encourages the policy to output increasingly large actions.

Revised: Weights for the goal-reached indicator component are sampled uniformly from [ − 1 , 1] .

Revised: The reward function is then computed as a linear combination of the reward components weighted by these sampled values.

Revised: We sample 128 reward functions in total.

Original: Case 1: Equivalent Reward
Revised: Policy optimization, reward evaluation, and policy evaluation are performed as described in Appendix A.1. Results Fig. 3 visualizes the results from the experiment, computing DARD on the random ( π uni ) and expert ( π ∗ ) coverage distributions. We see that in both cases DARD is predictive of policy return as indicated by the association between small DARD distances and large policy return.

Revised: The

Revised: DARD distance and policy return are both heavily dependent on the sign of the goal-reached indicator weight, which we visualize by coloring points based on the sign of this weight.

Revised: When the sign is negative, the policy learns to avoid getting too close to the goal, thereby dramatically reducing its expected return under the ground-truth reward.

Revised: The π uni plot shows policy return decaying more quickly as a function of DARD distance than does the plot of π ∗ .

Revised: The reason for this is that in the π uni case, the action magnitude contributes much more signiﬁcantly to the DARD distance than in the π ∗ case (because π ∗ generally consists of small actions near the goal state).

Revised: There are some outliers in the plot, which seem to result from extreme combinations of weight values that can cause policy optimization issues.

Revised: For example, cases of low DARD distance and low policy return in general tend to have small distance from goal weights and large, positive goal-reached indicator weights.

Revised: In these cases, the reward is quite sparse, causing problems for policy optimization (based on prior experiments, the policy does eventually achieve high expected return, but only with more training timesteps).

Revised: In the π uni case, instances of large DARD distance and high policy return tend to result from extremely small action magnitude weights and large, positive goal-reached indicator weights.

Revised: In these cases, the optimized policy takes large magnitude actions to quickly reach the goal state.

Revised: This incurs a large penalty over the ﬁrst few timesteps of the episode, after which the agent remains mostly stationary, accruing the goal-reached reward.

Revised: This trend does not hold for π ∗ , which demonstrates DARD’s sensitivity to the coverage distribution.

Revised: Conclusion In this section, we evaluated DARD on a large number of random reward functions in the Reacher environment, many of which differ signiﬁcantly from the ground-truth reward.

Revised: Results demonstrate that DARD maintains its predictiveness of policy return in this setting across both coverage distributions.

Revised: A. A DDITIONAL D ISCUSSION A.3. T HE I MPACT OF T RANSITION M ODEL E RRORS ON DARD

Revised: In this section, we discuss how errors made by the transition model can impact the performance of DARD.

Revised: We organize our discussion of this topic around different cases of reward equivalence and different types of errors that a transition model might make.

Revised: The high-level summary of the following discussion is that errors in the transition model only impact the performance of DARD on reward functions that are not equivalent everywhere (case 2 below), and in this case DARD may produce both false negative and false positive equivalences as a result of transition model errors.

Original:  Functions Consider two reward functions that are equivalent across the full space of transitions accounting for reward shaping (this includes the case of identical rewards).
Revised: Case 1: Equivalent Reward Functions Consider two reward functions that are equivalent across the full space of transitions accounting for reward shaping (this includes the case of identical rewards).

Original: The proof of Proposition 2 does not rely on any proprieties of the transition model.
Revised: The proof of Proposition 2 does not rely on any proprieties of the transition model.

Original: As a result, we can conclude that even arbitrary errors in the transition model will not result in two everywhere-equivalent reward functions being considered different.
Revised: As a result, we can conclude that even arbitrary errors in the transition model will not result in two everywhere-equivalent reward functions being considered different.

Original: This logic indicates that errors in the transition model may only impact the performance of DARD when the reward functions are not everywhere-equivalent as discussed next.
Revised: This logic indicates that errors in the transition model may only impact the performance of DARD when the reward functions are not everywhere-equivalent as discussed next.

Original: Case 2: Different Reward FunctionsS UBCASE A: R EWARDS ARE E QUIVALENT ON ALL Realizable T RANSITIONS
Revised: Case 2: Different Reward Functions

Revised: [Figure] S UBCASE A: R EWARDS ARE E QUIVALENT ON ALL Realizable T RANSITIONS

Original: Given the ground truth transition model, DARD would correctly identify the two rewards as equivalent (with respect to realizable transitions).
Revised: Given the ground truth transition model, DARD would correctly identify the two rewards as equivalent (with respect to realizable transitions).

Original: However, if the transition model erroneously samples unrealizable transitions on which the rewards differ, the transformation term of Eq.
Revised: However, if the transition model erroneously samples unrealizable transitions on which the rewards differ, the transformation term of Eq.

Original: (3) could assume arbitrary values, thereby resulting in the two rewards being considered different.
Revised: (3) could assume arbitrary values, thereby resulting in the two rewards being considered different.

Original: This might occur, for example, when dealing with learned reward functions for which sampled next states are OOD with respect to their training distribution.
Revised: This might occur, for example, when dealing with learned reward functions for which sampled next states are OOD with respect to their training distribution.

Original: S UBCASE B: R EWARDS D IFFER ON S OME R EALIZABLE T RANSITIONS
Revised: S UBCASE B: R EWARDS D IFFER ON S OME R EALIZABLE T RANSITIONS

Original: Given the ground truth transition model and sampling of the full action space, DARD would correctly identify the two rewards as different (assuming states producing transitions on which the rewards differ exist in the dataset).
Revised: Given the ground truth transition model and sampling of the full action space, DARD would correctly identify the two rewards as different (assuming states producing transitions on which the rewards differ exist in the dataset).

Original: However, if the transition model is incorrect in such a way that it never samples those transitions on which the reward models differ, DARD may produce false positive equivalences.
Revised: However, if the transition model is incorrect in such a way that it never samples those transitions on which the reward models differ, DARD may produce false positive equivalences.

Original: For example, suppose a transition model is learned for an autonomous driving environment.
Revised: For example, suppose a transition model is learned for an autonomous driving environment.

Original: Because collisions are rare in this environment, the transition model might never sample transitions involving collisions.
Revised: Because collisions are rare in this environment, the transition model might never sample transitions involving collisions.

Original: As a result, two reward models that differ only with respect to transitions involving collisions may be considered equivalent by DARD.
Revised: As a result, two reward models that differ only with respect to transitions involving collisions may be considered equivalent by DARD.

Original: In the preceding text, we say that DARD may produce false positive equivalences in this case because even with an erroneous transition model, if the dataset on which the rewards are compared itself contains transitions on which the rewards differ
Revised: In the preceding text, we say that DARD may produce false positive equivalences in this case because even with an erroneous transition model, if the dataset on which the rewards are compared itself contains transitions on which the rewards differ

Original: DARD will still recognize them as different (due to the R ( s, a, s (cid:48) ) term of Eq.
Revised: DARD will still recognize them as different (due to the R ( s, a, s (cid:48) ) term of Eq.

Original: Continuing the autonomous driving example: if the dataset itself contains collisions, then the rewards will be identiﬁed as different even if the transition models do not sample collisions.
Revised: Continuing the autonomous driving example: if the dataset itself contains collisions, then the rewards will be identiﬁed as different even if the transition models do not sample collisions.

Original: This case illustrates the importance of using a comprehensive coverage distribution in comparing reward functions.
Revised: This case illustrates the importance of using a comprehensive coverage distribution in comparing reward functions.

Original: A.3. A NALYSIS OF DARD-L IN R EACHER WITH π ∗ C OVERAGE D ISTRIBUTION
Revised: A.3. A NALYSIS OF DARD-L IN R EACHER WITH π ∗ C OVERAGE D ISTRIBUTION

Original: This section provides a detailed analysis of the results for DARD-L in the Reacher environment when using the coverage distribution of the expert policy.
Revised: This section provides a detailed analysis of the results for DARD-L in the Reacher environment when using the coverage distribution of the expert policy.

Original: Unlike the π uni coverage distribution, the states sampled from the expert distribution in the Reacher environment generally have small distances between the end-effector and goal location.
Revised: Unlike the π uni coverage distribution, the states sampled from the expert distribution in the Reacher environment generally have small distances between the end-effector and goal location.

Original: As a result, the reward is dominated by whether or not the end-effector is within the threshold distance at which the “goal reached” reward is given.
Revised: As a result, the reward is dominated by whether or not the end-effector is within the threshold distance at which the “goal reached” reward is given.

Original: This causes the reward functions to be sensitive to small errors in the predicted state because those errors can move the end-effector in or out of range of the goal. 
Revised: This causes the reward functions to be sensitive to small errors in the predicted state because those errors can move the end-effector in or out of range of the goal. [Figure]

Original:  On average, the learned transition model predicts larger next state end-effector displacements than the ground truth transition model.
Revised: [Figure] On average, the learned transition model predicts larger next state end-effector displacements than the ground truth transition model.

Original: This can be seen in Fig.
Revised: This can be seen in Fig.

Original: 3 where we plot the histogram of ground truth and predicted displacements over a set of transitions for which the transformed reward differs the most between the ground truth and learned transition models.
Revised: 4 where we plot the histogram of ground truth and predicted displacements over a set of transitions for which the transformed reward differs the most between the ground truth and learned transition models.

Original: These erroneous, larger predicted distances result from using a mean squared error loss, which causes displacements near zero to contribute a minuscule amount to the loss.
Revised: These erroneous, larger predicted distances result from using a mean squared error loss, which causes displacements near zero to contribute a minuscule amount to the loss.

Original: As a result, the next states predicted by the learned transition model are on average easier to assign rewards to than those of the ground-truth transition model: they are more clearly beyond the limit of the “goal reached” threshold, and (due to being farther away) reward shaping (which is easier to predict) constitutes a larger fraction of the reward.
Revised: As a result, the next states predicted by the learned transition model are on average easier to assign rewards to than those of the ground-truth transition model: they are more clearly beyond the limit of the “goal reached” threshold, and (due to being farther away) reward shaping (which is easier to predict) constitutes a larger fraction of the reward.

Original: Overall this results in the learned transition model yielding lower DARD distances than those of the ground truth transition model.
Revised: Overall this results in the learned transition model yielding lower DARD distances than those of the ground truth transition model.

Original: This is an instance of Case 2.B in Appendix A.3.1: states at which the reward models differ more are sampled relatively less frequently due to errors in the learned transition model.[Figure] [Figure]
Revised: This is an instance of Case 2.B in Appendix A.3.1: states at which the reward models differ more are sampled relatively less frequently due to errors in the learned transition model.

Original: A.3. A PPLICATIONS OF R EWARD C OMPARISON M ETRICS
Revised: A.3. A PPLICATIONS OF R EWARD C OMPARISON M ETRICS

Original: This section discusses two potential applications of reward comparison metrics.
Revised: This section discusses two potential applications of reward comparison metrics.

Original: Benchmarking Reward Learning Algorithms Suppose we have a large amount of (potentially suboptimal) demonstration data from which we would like to learn a reward function using one of a variety of potential IRL algorithms.
Revised: Benchmarking Reward Learning Algorithms Suppose we have a large amount of (potentially suboptimal) demonstration data from which we would like to learn a reward function using one of a variety of potential IRL algorithms.

Original: Additionally, suppose that we have (through an expensive process) manually labeled rewards in a relatively small set of demonstrations.
Revised: Additionally, suppose that we have (through an expensive process) manually labeled rewards in a relatively small set of demonstrations.

Original: Furthermore, assume that evaluating a policy online involves an expensive or dangerous process, and that learning a policy for this environment is itself challenging (e.g., sometimes fails or produces policies of highly-variable quality).
Revised: Furthermore, assume that evaluating a policy online involves an expensive or dangerous process, and that learn- ing a policy for this environment is itself challenging (e.g., sometimes fails or produces policies of highly-variable quality).

Original: Note that this setting might occur in practice in physical robotic domains such as autonomous driving or robotic manipulation.
Revised: Note that this setting might occur in practice in physical robotic domains such as autonomous driving or robotic manipulation.

Original: In summary, we would like to know which of the set of possible IRL algorithms produces the best reward function using the large set of demonstrations, but we would prefer to avoid policy learning and evaluation.
Revised: In summary, we would like to know which of the set of possible IRL algorithms produces the best reward function using the large set of demonstrations, but we would prefer to avoid policy learning and evaluation.

Original: One way to accomplish this goal would be to learn a “benchmark” reward model on the small set of manually labeled rewards (e.g., through supervised learning as in Cabi et al.
Revised: One way to accomplish this goal would be to learn a “benchmark” reward model on the small set of manually labeled rewards (e.g., through supervised learning as in Cabi et al.

Original: (2019)), and then to compare that with the rewards produced by the various IRL algorithms.
Revised: (2019)), and then to compare that with the rewards produced by the various IRL algorithms.

Original: If a reward function learned via IRL is similar to the benchmark reward function, this indicates that it is similar to the reward function manually deﬁned by experts, and should therefore be preferred.
Revised: If a reward function learned via IRL is similar to the benchmark reward function, this indicates that it is similar to the reward function manually deﬁned by experts, and should therefore be preferred.

Original: In this way, we can evaluate the different IRL methods we are considering, and choose the best for our problem.
Revised: In this way, we can evaluate the different IRL methods we are considering, and choose the best for our problem.

Original: Furthermore, this approach could be applied in creating an open benchmark enabling the comparison of reward learning methods.
Revised: Furthermore, this approach could be applied in creating an open benchmark enabling the comparison of reward learning methods.

Original: Validating Learned Reward Functions Suppose that we have deployed a system that uses a learned reward function.
Revised: Validating Learned Reward Functions Suppose that we have deployed a system that uses a learned reward function.

Original: With deployed learning systems, it is often necessary to continuously collect data and retrain models in order to account for changing data distributions arising from a variety of sources (Breck et al., 2017).
Revised: With deployed learning systems, it is often necessary to continuously collect data and retrain models in order to account for changing data distributions arising from a variety of sources (Breck et al., 2017).

Original: As such, we would like to frequently retrain the reward function on newly-collected data; however, we would also like to ensure that the reward function has not changed “too much” from previous versions that we know to be of high quality (through expensive deployment).
Revised: As such, we would like to frequently retrain the reward function on newly-collected data; however, we would also like to ensure that the reward function has not changed “too much” from previous versions that we know to be of high quality (through expensive deployment).

Original: Similar to the previous application, suppose that online policy evaluation is expensive or dangerous, and that policy optimization is challenging.
Revised: Similar to the previous application, suppose that online policy evaluation is expensive or dangerous, and that policy optimization is challenging.

Original: Given this problem setting, we could use reward comparisons as follows.
Revised: Given this problem setting, we could use reward comparisons as follows.

Original: First, learn a new reward function on all the data collected so far.
Revised: First, learn a new reward function on all the data collected so far.

Original: Second, compare that reward function with the old reward function on the validation dataset of the old reward.
Revised: Second, compare that reward function with the old reward function on the validation dataset of the old reward.

Original: If the comparison yields a small distance between rewards, this indicates that the reward functions are likely quite similar, thereby providing support for continuing the process of deploying the new reward function.
Revised: If the comparison yields a small distance between rewards, this indicates that the reward functions are likely quite similar, thereby providing support for continuing the process of deploying the new reward function.

Original: Alternatively, if the comparison yields a large distance between rewards, this indicates that the new reward is potentially quite different from the old one, and that we should carefully inspect the cases where the rewards differ before proceeding with deployment.
Revised: Alternatively, if the comparison yields a large distance between rewards, this indicates that the new reward is potentially quite different from the old one, and that we should carefully inspect the cases where the rewards differ before proceeding with deployment.

Original: In this way, we have deﬁned (part of) a validation procedure for continuously learning and deploying reward functions.
Revised: In this way, we have deﬁned (part of) a validation procedure for continuously learning and deploying reward functions.

Original: A. P ROOFS A.4. C OMPARING L EARNED R EWARD F UNCTIONS
Revised: A. P ROOFS A.4. C OMPARING L EARNED R EWARD F UNCTIONS

Original: Proposition 3 (The dynamics-aware transformed reward is invariant to shaping) .
Revised: Proposition 3 (The dynamics-aware transformed reward is invariant to shaping) .

Original: Let R : S ×
Revised: Let R : S ×

Original: A × S → R be a reward function, and Φ : S → R a potential function.
Revised: A × S → R be a reward function, and Φ : S → R a potential function.

Original: Let γ ∈ [0 , 1] be a discount rate, and D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions, and
Revised: Let γ ∈ [0 , 1] be a discount rate, and D S ∈ ∆( S ) and D A ∈ ∆( A ) be distributions over states and actions, and

Original: T : S ×
Revised: T : S ×

Original: A → ∆( S ) be a conditional distribution over next states.
Revised: A → ∆( S ) be a conditional distribution over next states.

Original: Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s 
Revised: Let R (cid:48) denote R shaped by Φ : R (cid:48) ( s, a, s (cid:48) )

Original: [Equation] ) =[Equation]
Revised: =

Original:  ( s, a, s (cid:48) ) + γ Φ( s [Equation]
Revised: R ( s, a, s (cid:48) ) + γ Φ( s (cid:48) )

Original: ) − Φ( s ) .
Revised: − Φ( s ) .

Original: Then the dynamics-aware transformations of R (cid:48) and R are equal: C T ( R (cid:48) )
Revised: Then the dynamics-aware transformations of R (cid:48) and R are equal: C T ( R (cid:48) )

Original: =
Revised: =

Original: C T ( R ) .
Revised: C T ( R ) .

Original: Proof.
Revised: Proof.

Original: Let s, a, s (cid:48)
Revised: Let s, a, s (cid:48)

Original: ∈
Revised: ∈

Original: S ×
Revised: S ×

Original: A × S .
Revised: A × S .

Original: Then by substituting in the deﬁnition of R (cid:48) and using linearity of expectation:
Revised: Then by substituting in the deﬁnition of R (cid:48) and using linearity of expectation:

Original: [Equation]
Revised: [Equation]

Original: Lemma A.1 (The Pearson distance D ρ is a pseudometric) .
Revised: Lemma A.1 (The Pearson distance D ρ is a pseudometric) .

Original: (Lemma 4.5 from Gleave et al.
Revised: (Lemma 4.5 from Gleave et al.

Original: Namely, for X a set of random variables, D ρ satisﬁes the following properties:
Revised: Namely, for X a set of random variables, D ρ satisﬁes the following properties:

Original: Identity: D ρ ( X, X ) = 

Revised: Identity: D ρ ( X, X ) =

Original: ∀ X ∈ X ,
Revised: ∀ X ∈ X ,

Original: Symmetry:
Revised: Symmetry:

Original: D ρ ( X, Y ) D ρ ( Y, X ) ∀ X, Y ∈ X , and
Revised: D ρ ( X, Y ) D ρ ( Y, X ) ∀ X, Y ∈ X , and

Original: Triangle Inequality: D ρ ( X, Z ) ≤ D ρ ( X, Y )
Revised: Triangle Inequality: D ρ ( X, Z ) ≤ D ρ ( X, Y )

Original: + D ρ ( Y, Z ) ∀ X, Y, Z ∈ X .
Revised: + D ρ ( Y, Z ) ∀ X, Y, Z ∈ X .

Original: Proof: See Gleave et al.
Revised: Proof: See Gleave et al.

Original: Proposition 4 (The DARD distance metric is a pseudometric) .
Revised: Proposition 4 (The DARD distance metric is a pseudometric) .

Original: Let S , A , S (cid:48) be random variables jointly following some coverage transition distribution, and let T be the transition model of the corresponding environment.
Revised: Let S , A , S (cid:48) be random variables jointly following some coverage transition distribution, and let T be the transition model of the corresponding environment.

Original: We deﬁne the DARD pseudometric between rewards R A and R B as follows:
Revised: We deﬁne the DARD pseudometric between rewards R A and R B as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This proof closely follows the proof of Theorem 4.7 from Gleave et al.
Revised: This proof closely follows the proof of Theorem 4.7 from Gleave et al.

Original: Namely, the result follows from direct application of Lemma A.1.
Revised: Namely, the result follows from direct application of Lemma A.1.

Original: Let R A , R B , and R C be reward functions mapping from transitions S ×
Revised: Let R A , R B , and R C be reward functions mapping from transitions S ×

Original: A × S to real numbers R .
Revised: A × S to real numbers R .

Original: Identity: It immediately follows that
Revised: Identity: It immediately follows that

Original: [Equation]
Revised: [Equation]

Original:  D [Equation] is a pseudometric
Revised: 0 since D ρ is a pseudometric,

Original: [Equation]{+D ρ ( X, X ) = 
+}
Revised: {+D ρ ( X, X ) = 
+}

Revised: ∀ X ∈ X .

Original: Symmetry:
Revised: Symmetry:

Original: This again immediately follows from the fact that D ρ is a pseudometric.
Revised: This again immediately follows from the fact that D ρ is a pseudometric.

Original: We have that:
Revised: We have that:

Original: [Equation]
Revised: [Equation]

Original: since D ρ ( X, Y ) =
Revised: since D ρ ( X, Y ) =

Original: D ρ ( Y, X )
Revised: D ρ ( Y, X )

Original: because D ρ is a pseudometric.
Revised: because D ρ is a pseudometric.

Original: Triangle Inequality: This again immediately follows from the fact that D ρ is a pseudometric.
Revised: Triangle Inequality: This again immediately follows from the fact that D ρ is a pseudometric.

Original: We have that:
Revised: We have that:

Original: [Equation]
Revised: [Equation]

Original: since D ρ ( X, Z ) ≤ D ρ ( X, Y )
Revised: since D ρ ( X, Z ) ≤ D ρ ( X, Y )

Original: + D ρ ( Y, Z ) because D ρ is a pseudometric.
Revised: + D ρ ( Y, Z ) because D ρ is a pseudometric.