Original: P

Original: LAN -B ASED

Original: R

Original: ELAXED R EWARD

Original: S

Original: HAPING

Original:  FOR G OAL -D IRECTED T ASKSAnonymous authors Paper under double-blind review
Revised: P LAN -B ASED R ELAXED R EWARD S HAPING FOR G OAL -D IRECTED T ASKS

Original: A BSTRACT
Revised: A BSTRACT

Original: In high-dimensional state spaces, the usefulness of Reinforcement Learning (RL) is limited by the problem of exploration.
Revised: In high-dimensional state spaces the usefulness of Reinforcement Learning (RL) is limited by the problem of exploration.

Original: This issue has been addressed using potential-based reward shaping (PB-RS) previously.
Revised: This issue has been addressed using potential-based reward shaping (PB-RS) previously.

Original: In the present work, we introduce Final-Volume-Preserving Reward Shaping (FV-RS).
Revised: In the present work, we introduce Final-Volume-Preserving Reward Shaping (FV-RS).

Original: FV-RS relaxes the strict optimality guarantees of PB-RS to a guarantee of preserved long-term behavior.
Revised: FV-RS relaxes the strict optimality guarantees of PB-RS to a guarantee of preserved long-term behavior.

Original: Being less restrictive, FV-RS allows for reward shaping functions that are even better suited for improving the sample efﬁciency of RL algorithms.
Revised: Being less restrictive, FV-RS allows for reward shaping functions that are even better suited for improving the sample efﬁciency of RL algorithms.

Original: In particular, we consider settings in which the agent has access to an approximate plan.
Revised: In particular, we consider settings in which the agent has access to an approximate plan.

Original: Here, we use examples of simulated robotic manipulation tasks to demonstrate that plan-based FV-RS can indeed signiﬁcantly improve the sample efﬁciency of
Revised: Here, we use examples of simulated robotic manipulation tasks to demonstrate that plan-based FV-RS can indeed signiﬁcantly improve the sample efﬁciency of

Original: RL over plan-based PB-RS.
Revised: RL over plan-based PB-RS.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement Learning (RL) provides a general framework for autonomous agents to learn complex behavior, adapt to changing environments, and generalize to unseen tasks and environments with little human interference or engineering effort.
Revised: Reinforcement Learning (RL) provides a general framework for autonomous agents to learn complex behavior, adapt to changing environments, and generalize to unseen tasks and environments with little human interference or engineering effort.

Original: However, RL in high-dimensional state spaces generally suffers from a difﬁcult exploration problem, making learning prohibitively slow and sample-inefﬁcient for many real-world tasks with sparse rewards.
Revised: However, RL in high-dimensional state spaces generally suffers from a difﬁcult exploration problem, making learning prohibitively slow and sample-inefﬁcient for many real-world tasks with sparse rewards.

Original: A possible strategy to increase the sample efﬁciency of RL algorithms is reward shaping (Mataric, 1994; Randløv & Alstrøm, 1998), in particular potential-based reward shaping (PB-RS)
Revised: A possible strategy to increase the sample efﬁciency of RL algorithms is reward shaping (Mataric, 1994; Randløv & Alstrøm, 1998), in particular potential-based reward shaping (PB-RS)

Original: (Ng et al., 1999).
Revised: (Ng et al., 1999).

Original: Reward shaping provides a dense reward signal to the RL agent, enabling it to converge faster to the optimal policy.
Revised: Reward shaping provides a dense reward signal to the RL agent, enabling it to converge faster to the optimal policy.

Original: In robotics tasks, approximate domain knowledge is often available and can be used by a planning algorithm to generate approximate plans.
Revised: In robotics tasks, approximate domain knowledge is often available and can be used by a planning algorithm to generate approximate plans.

Original: Here, the resulting plan can be provided to the RL agent using plan-based reward shaping (Grzes & Kudenko, 2008; Brys et al., 2015).
Revised: Here, the resulting plan can be provided to the RL agent using plan-based reward shaping (Grzes & Kudenko, 2008; Brys et al., 2015).

Original: Thus, plan-based reward shaping offers a natural way to combine the efﬁciency of planning with the ﬂexibility of RL.
Revised: Thus, plan-based reward shaping offers a natural way to combine the efﬁciency of planning with the ﬂexibility of RL.

Original: We analyze the use of plan-based reward shaping for RL.
Revised: We analyze the use of plan-based reward shaping for RL.

Original: The key novelty is that we theoretically introduce Final-Volume-Preserving Reward Shaping (FV-RS), a superset of
Revised: The key novelty is that we theoretically introduce Final-Volume-Preserving Reward Shaping (FV-RS), a superset of

Original: PB-RS.
Revised: PB-RS.

Original: Intuitively speaking, FV-RS allows for shaping rewards that convey the information encoded in the shaping reward in a more direct way than PB-RS, since the value of following a policy is not only determined by the shaping reward at the end of the trajectory, but can also depend on all intermediate states.
Revised: Intuitively speaking, FV-RS allows for shaping rewards that convey the information encoded in the shaping reward in a more direct way than PB-RS, since the value of following a policy is not only determined by the shaping reward at the end of the trajectory, but can also depend on all intermediate states.

Original: While FV-RS inevitably relaxes the optimality guarantees provided by PB-RS, we show in the experiments that FV-RS can signiﬁcantly improve sample efﬁciency beyond PB-RS, e.g. allowing RL agents to learn simulated 10-dimensional continuous robotic manipulation tasks after ca.
Revised: While FV-RS inevitably relaxes the optimality guarantees provided by PB-RS, we show in the experiments that FV-RS can signiﬁcantly improve sample efﬁciency beyond PB-RS, e.g. allowing RL agents to learn simulated 10-dimensional continuous robotic manipulation tasks after ca.

Original: 300 rollout episodes.
Revised: 300 rollout episodes.

Original: We argue that the strict notion of optimality in PB-RS is not necessary in many robotics applications, while on the other hand relaxing PB-RS to FV-RS facilitates speeding up the learning process.
Revised: We argue that the strict notion of optimality in PB-RS is not necessary in many robotics applications, while on the other hand relaxing PB-RS to FV-RS facilitates speeding up the learning process.

Original: Using FV-RS could be a better trade-off between optimality and sample efﬁciency in many domains.
Revised: Using FV-RS could be a better trade-off between optimality and sample efﬁciency in many domains.

Original: The contributions of this work are:
Revised: The contributions of this work are:

Original: • We introduce FV-RS as a new class of reward shaping for RL methods in general.
Revised: • We introduce FV-RS as a new class of reward shaping for RL methods in general.

Original: • We propose to speciﬁcally use FV-RS for plan-based reward shaping.
Revised: • We propose to speciﬁcally use FV-RS for plan-based reward shaping.

Original: • We show that compared to no RS and plan-based PB-RS, plan-based FV-RS signiﬁcantly increases the sample efﬁciency in several robotic manipulation tasks.
Revised: • We show that compared to no RS and plan-based PB-RS, plan-based FV-RS signiﬁcantly increases the sample efﬁciency in several robotic manipulation tasks.

Original:  R ELATED W ORK 2. S PARSE R EWARDS AND R EWARD S HAPING
Revised:  R ELATED W ORK 2. S PARSE R EWARDS AND R EWARD S HAPING

Original: In many real-world RL settings, the agent is only given sparse rewards, exacerbating the exploration problem.
Revised: In many real-world RL settings, the agent is only given sparse rewards, exacerbating the exploration problem.

Original: There exist several approaches in the literature to overcome this issue.
Revised: There exist several approaches in the literature to overcome this issue.

Original: These include mechanisms of intrinsic motivation and curiosity (Barto et al., 2004; Oudeyer et al., 2007; Schembri et al., 2007), which provide the agent with additional intrinsic rewards for events that are novel, salient, or particularly useful for the learning process.
Revised: These include mechanisms of intrinsic motivation and curiosity (Barto et al., 2004; Oudeyer et al., 2007; Schembri et al., 2007), which provide the agent with additional intrinsic rewards for events that are novel, salient, or particularly useful for the learning process.

Original: In reward optimization (Sorg et al., 2010;
Revised: In reward optimization (Sorg et al., 2010;

Original: Sequeira et al., 2011; 2014), the reward function itself is being optimized to allow for efﬁcient learning.
Revised: Sequeira et al., 2011; 2014), the reward function itself is being optimized to allow for efﬁcient learning.

Original: Similarly, reward shaping (Mataric, 1994; Randløv & Alstrøm, 1998) is a technique to give the agent additional rewards in order to guide it during training.
Revised: Similarly, reward shaping (Mataric, 1994; Randløv & Alstrøm, 1998) is a technique to give the agent additional rewards in order to guide it during training.

Original: In PB-RS (Ng et al., 1999; Wiewiora, 2003;
Revised: In PB-RS (Ng et al., 1999; Wiewiora, 2003;

Original: Wiewiora et al., 2003; Devlin & Kudenko, 2012), this is done in a way that ensures that the resulting optimal policy is the same with and without shaping.
Revised: Wiewiora et al., 2003; Devlin & Kudenko, 2012), this is done in a way that ensures that the resulting optimal policy is the same with and without shaping.

Original: Ng et al.
Revised: Ng et al.

Original: (1999) also showed that the reverse statement holds as well; PB-RS is the only type of modiﬁcation to the reward function that can guarantee such an invariance if no other assumptions about the Markov Decision Process (MDP) are made.
Revised: (1999) also showed that the reverse statement holds as well; PB-RS is the only type of modiﬁcation to the reward function that can guarantee such an invariance if no other assumptions about the Markov Decision Process (MDP) are made.

Original: In this work, we introduce Final-Volume-Preserving Reward Shaping (FV-RS), a subclass of reward shaping that is broader than PB-RS and not necessarily potential-based, and therefore is not guaranteed to leave the optimal policy invariant.
Revised: In this work, we introduce Final-Volume-Preserving Reward Shaping (FV-RS), a subclass of reward shaping that is broader than PB-RS and not necessarily potential-based, and therefore is not guaranteed to leave the optimal policy invariant.

Original: However, FV-RS still guarantees the invariance of the asymptotic state of the MDP under optimal control.
Revised: However, FV-RS still guarantees the invariance of the asymptotic state of the MDP under optimal control.

Original: In the experiments section, we show that this relaxed notion of reward shaping allows us to substantially improve the sample efﬁciency during training.
Revised: In the experiments section, we show that this relaxed notion of reward shaping allows us to substantially improve the sample efﬁciency during training.

Original: 2. D EMONSTRATION - AND P LAN -B ASED R EWARD S HAPING
Revised: 2. D EMONSTRATION - AND P LAN -B ASED R EWARD S HAPING

Original: Learning from Demonstration (LfD) aims at creating a behavioral policy from expert demonstrations.
Revised: Learning from Demonstration (LfD) aims at creating a behavioral policy from expert demonstrations.

Original: Existing approaches differ considerably in how the demonstration examples are collected and how the policy is derived from this (Argall et al., 2009; Ravichandar et al., 2020).
Revised: Existing approaches differ considerably in how the demonstration examples are collected and how the policy is derived from this (Argall et al., 2009; Ravichandar et al., 2020).

Original: The HAT algorithm (Taylor et al., 2011) introduces an intermediate policy summarization step, in which the demonstrated data is translated into an approximate policy that is then used to bias exploration in a ﬁnal RL stage.
Revised: The HAT algorithm (Taylor et al., 2011) introduces an intermediate policy summarization step, in which the demonstrated data is translated into an approximate policy that is then used to bias exploration in a ﬁnal RL stage.

Original: In Hester et al.
Revised: In Hester et al.

Original: (2017), the policy is simultaneously trained on expert data and collected data, using a combination of supervised and temporal difference losses.
Revised: (2017), the policy is simultaneously trained on expert data and collected data, using a combination of supervised and temporal difference losses.

Original: In Salimans &
Revised: In Salimans &

Original: Chen (2018), the RL agent is at the start of each episode reset to a state in the single demonstration.
Revised: Chen (2018), the RL agent is at the start of each episode reset to a state in the single demonstration.

Original: Other approaches (Thomaz et al., 2006; Knox & Stone, 2010) rely on interactive human feedback during the training process.
Revised: Other approaches (Thomaz et al., 2006; Knox & Stone, 2010) rely on interactive human feedback during the training process.

Original: At the intersection of RL and LfD, reward shaping offers a natural way to include expert demonstrations or plans of the correct behavior into an RL training process.
Revised: At the intersection of RL and LfD, reward shaping offers a natural way to include expert demonstrations or plans of the correct behavior into an RL training process.

Original: Prior work in this area includes using abstract plan knowledge represented in the form of STRIPS operators to create a potential function for PB-RS (Grzes & Kudenko, 2008; Efthymiadis et al., 2016; Devlin & Kudenko, 2016), which has been applied to the strategy game of Starcraft (Efthymiadis & Kudenko, 2013).
Revised: Prior work in this area includes using abstract plan knowledge represented in the form of STRIPS operators to create a potential function for PB-RS (Grzes & Kudenko, 2008; Efthymiadis et al., 2016; Devlin & Kudenko, 2016), which has been applied to the strategy game of Starcraft (Efthymiadis & Kudenko, 2013).

Original: Brys et al.
Revised: Brys et al.

Original: (2015) use expert demonstrations to directly construct a Gaussian potential function, and in Suay et al.
Revised: (2015) use expert demonstrations to directly construct a Gaussian potential function, and in Suay et al.

Original: (2016), this is extended to include multiple demonstrations that are translated into a potential function using Inverse Reinforcement Learning as an intermediate step.
Revised: (2016), this is extended to include multiple demonstrations that are translated into a potential function using Inverse Reinforcement Learning as an intermediate step.

Original: In this work, we use a planned sequence in state-space to construct a shaping function similar to
Revised: In this work, we use a planned sequence in state-space to construct a shaping function similar to

Original: Brys et al. (2015), but in contrast to the aforementioned work, we do not use this shaping function as a potential function for PB-RS.
Revised: Brys et al. (2015), but in contrast to the aforementioned work, we do not use this shaping function as a potential function for PB-RS.

Original: Instead, we use it directly as a reward function for FV-RS.
Revised: Instead, we use it directly as a reward function for FV-RS.

Original: We show that this signiﬁcantly improves the sample efﬁciency during training.
Revised: We show that this signiﬁcantly improves the sample efﬁciency during training.

Original:  B ACKGROUND 3. M ARKOV D ECISION P ROCESSES AND R EINFORCEMENT L EARNING
Revised:  B ACKGROUND 3. M ARKOV D ECISION P ROCESSES AND R EINFORCEMENT L EARNING

Original: We consider decision problems that can be described as a discrete-time MDPs (Bellman
Revised: We consider decision problems that can be described as a discrete-time MDPs (Bellman, 1957) (cid:104) S , A , T, γ, R (cid:105) .

Revised: Here, S is the set of all possible states, and A is the set of all possible actions.

Original: T : S ×
Revised: T : S ×

Original: A × S → [0 , 1] describes the dynamics of the system; T ( s (cid:48) | s, a ) is the probability (density) of the next state being s (cid:48) , provided that the current state is s and the action taken is a .
Revised: A × S → [0 , 1] describes the dynamics of the system; T ( s (cid:48) | s, a ) is the probability (density) of the next state being s (cid:48) , provided that the current state is s and the action taken is a .

Original: After the transition, the agent is given a reward R ( s, a, s (cid:48) ) .
Revised: After the transition, the agent is given a reward R ( s, a, s (cid:48) ) .

Original: The discount factor γ ∈ [0 , 1) trades off immediate and future reward.
Revised: The discount factor γ ∈ [0 , 1) trades off immediate and future reward.

Original: The goal of RL is to learn an optimal policy π ∗ : S , A → [0 , 1] that maximizes the expected discounted return, i.e.
Revised: The goal of RL is to learn an optimal policy π ∗ : S , A → [0 , 1] that maximizes the expected discounted return, i.e.

Original: [Equation]
Revised: [Equation]

Original: where p 0 is the initial-state distribution, from collected transition and reward data D
Revised: where p 0 is the initial-state distribution, from collected transition and reward data D

Original: = { ( s i , a i , s (cid:48) i ) } ni =0 .
Revised: = { ( s i , a i , s (cid:48) i ) } n i =0 .

Original: The Q -function of a given policy π is the expected return for choosing action a 0 in state s 0 , and following π thereafter, i.e.
Revised: The Q -function of a given policy π is the expected return for choosing action a 0 in state s 0 , and following π thereafter, i.e.

Original: [Equation]
Revised: [Equation]

Original: There exists a range of RL algorithms to solve equation 1.
Revised: There exists a range of RL algorithms to solve equation 1.

Original: Our reward-shaping approach only modiﬁes the reward and therefore can be combined with any RL algorithm.
Revised: Our reward-shaping approach only modiﬁes the reward and therefore can be combined with any RL algorithm.

Original: In this work, we are interested in applications in robotics, where both S and A are typically continuous.
Revised: In this work, we are interested in applications in robotics, where both S and A are typically continuous.

Original: A popular algorithm in this case is Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), which will be used for most of the robotic manipulation examples in this work.
Revised: A popular algorithm in this case is Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), which will be used for most of the robotic manipulation examples in this work.

Original: An exception to this are the examples in appendix A.4, which use Proximal Policy Optimization (PPO)
Revised: An exception to this are the examples in appendix A.4, which use Proximal Policy Optimization (PPO)

Original: (Schulman et al., 2017).
Revised: (Schulman et al., 2017).

Original: 3. P OTENTIAL -B ASED R EWARD S HAPING
Revised: 3. P OTENTIAL -B ASED R EWARD S HAPING

Original: In many RL problems, rewards are sparse, making it harder for the RL algorithm to converge to the optimal policy.
Revised: In many RL problems, rewards are sparse, making it harder for the RL algorithm to converge to the optimal policy.

Original: One possibility to alleviate this problem is to modify the reward R of the original
Revised: One possibility to alleviate this problem is to modify the reward R of the original

Original: MDP with a shaping reward F .
Revised: MDP with a shaping reward F .

Original: [Equation]
Revised: [Equation]

Original: In general, the optimal policy ˜ π ∗ of the MDP ˜ M with the shaped reward ˜ R is different from the optimal policy π ∗ of the MDP M with the original reward R .
Revised: In general, the optimal policy ˜ π ∗ of the MDP ˜ M with the shaped reward ˜ R is different from the optimal policy π ∗ of the MDP M with the original reward R .

Original: Ng et al.
Revised: Ng et al.

Original: (1999) showed however that ˜ π ∗ ≡ π ∗ if and only if F ( s, a, s (cid:48) ) is derived from a potential function Φ :
Revised: (1999) showed however that ˜ π ∗ ≡ π ∗ if and only if F ( s, a, s (cid:48) ) is derived from a potential function Φ :

Original: S → R :
Revised: S → R :

Original: [Equation]
Revised: [Equation]

Original: This proof was extended (Wiewiora et al., 2003) to potential functions of both actions and states, provided that the next action taken, a (cid:48) , is known already.
Revised: This proof was extended (Wiewiora et al., 2003) to potential functions of both actions and states, provided that the next action taken, a (cid:48) , is known already.

Original: A further generalization to time-dependent potential functions was introduced in Devlin & Kudenko (2012).
Revised: A further generalization to time-dependent potential functions was introduced in Devlin & Kudenko (2012).

Original:  F INAL -V OLUME -P RESERVING R EWARD S HAPING
Revised:  F INAL -V OLUME -P RESERVING R EWARD S HAPING

Original: In the following, we introduce the notion of the optimal ﬁnal volume, which will then allow us to introduce Final-Volume-Preserving Reward Shaping (FV-RS).
Revised: In the following, we introduce the notion of the optimal ﬁnal volume, which will then allow us to introduce Final-Volume-Preserving Reward Shaping (FV-RS).

Original: FV-RS does not guarantee the invariance of the optimal policy like PB-RS does, but instead provides a less restrictive guarantee of preserved long-term behavior.
Revised: FV-RS does not guarantee the invariance of the optimal policy like PB-RS does, but instead provides a less restrictive guarantee of preserved long-term behavior.

Original: 4. O PTIMAL F INAL V OLUME
Revised: 4. O PTIMAL F INAL V OLUME

Original: Deﬁnition 4.1.
Revised: Deﬁnition 4.1.

Original: Let M be an MDP with state-space S .
Revised: Let M be an MDP with state-space S .

Original: If the set G = ∩ G a ∈
Revised: If the set G = ∩ G a ∈

Original: A G a is non-empty, we call G the optimal ﬁnal volume of M .
Revised: A G a is non-empty, we call G the optimal ﬁnal volume of M .

Original: Here, A is the set of non-empty sets G a ⊆ S such that
Revised: Here, A is the set of non-empty sets G a ⊆ S such that

Original: [Equation]
Revised: [Equation]

Original: Thus, the optimal ﬁnal volume of an MDP M is the minimal subset of the state space S of M that the MDP can be found in under optimal control with probability 1 after ﬁnite time.
Revised: Thus, the optimal ﬁnal volume of an MDP M is the minimal subset of the state space S of M that the MDP can be found in under optimal control with probability 1 after ﬁnite time.

Original: Notice that in order to keep the notation more concise, we use the strict requirement P s t ∼ q π ∗ ( s t ∈ G )= 1 instead of e.g. P s
Revised: Notice that in order to keep the notation more concise, we use the strict requirement P s t ∼ q π ∗ ( s t ∈ G )

Original: [Equation] ∼ q [Equation] ∗ ( s t ∈ G ) ≥ 1 − (cid:15) in deﬁnition 4.1.
Revised: = 1 instead of e.g. P s t ∼ q π ∗ ( s t ∈ G ) ≥ 1 − (cid:15) in deﬁnition 4.1.

Original: For many stochastic MDPs, the stationary distribution under optimal policy has nonzero density in the entire state space S .
Revised: For many stochastic MDPs, the stationary distribution under optimal policy has nonzero density in the entire state space S .

Original: This means that strictly speaking, our deﬁnition would result in the trivial optimal ﬁnal volume G = S for these MDPs.
Revised: This means that strictly speaking, our deﬁnition would result in the trivial optimal ﬁnal volume G = S for these MDPs.

Original: In practice however, many interesting MDPs are such that we can ﬁnd a volume G for which it is so unlikely that the optimal agent will leave it after some ﬁnite time has passed that we can readily assume that this probability is 0 for all practical purposes.
Revised: In practice however, many interesting MDPs are such that we can ﬁnd a volume G for which it is so unlikely that the optimal agent will leave it after some ﬁnite time has passed that we can readily assume that this probability is 0 for all practical purposes.

Original: 4. C OMPATIBLE MDP S AND F INAL -V OLUME -P RESERVING R EWARD S HAPING
Revised: 4. C OMPATIBLE MDP S AND F INAL -V OLUME -P RESERVING R EWARD S HAPING

Original: Deﬁnition 4.2.
Revised: Deﬁnition 4.2.

Original: Let M = (cid:104) S , A , T, γ, R (cid:105) and
Revised: Let M = (cid:104) S , A , T, γ, R (cid:105) and

Original: ˜ M = (cid:104) S , A , T, ˜ γ, ˜ R (cid:105) be MDPs that are identical apart from different reward functions R and ˜ R , as well as different discount factors γ and ˜ γ .
Revised: ˜ M = (cid:104) S , A , T, ˜ γ, ˜ R (cid:105) be MDPs that are identical apart from different reward functions R and ˜ R , as well as different discount factors γ and ˜ γ .

Original: Let both M and ˜ M have optimal ﬁnal volumes G and ˜ G , respectively.
Revised: Let both M and ˜ M have optimal ﬁnal volumes G and ˜ G , respectively.

Original: We call ˜ M a compatible MDP to M iff ˜ G ⊆ G , and we write ˜ M ⊆ M .
Revised: We call ˜ M a compatible MDP to M iff ˜ G ⊆ G , and we write ˜ M ⊆ M .

Original: If this is the case, we also call ˜ R a ﬁnal-volume-preserving reward shaping function.
Revised: If this is the case, we also call ˜ R a ﬁnal-volume-preserving reward shaping function.

Original: If ˜ M ⊆ M , this means that although the rewards and discounts ( R, γ ) and ( ˜ R, ˜ γ ) are different and, in contrast to PB-RS, in general result in different optimal policies, they are similar in the sense that after a ﬁnite time, the state of ˜ M under optimal control will be inside the optimal ﬁnal volume
Revised: If ˜ M ⊆ M , this means that although the rewards and discounts ( R, γ ) and ( ˜ R, ˜ γ ) are different and, in contrast to PB-RS, in general result in different optimal policies, they are similar in the sense that after a ﬁnite time, the state of ˜ M under optimal control will be inside the optimal ﬁnal volume

Original: G of M .
Revised: G of M .

Original: In other words; in the long run, behaving optimally in ˜ M will have the same “result” as behaving optimally in M .
Revised: In other words; in the long run, behaving optimally in ˜ M will have the same “result” as behaving optimally in M .

Original: Since PB-RS leaves the optimal policy invariant, M and ˜ M are also compatible if ˜ M is the PB-RS counterpart to M .
Revised: Since PB-RS leaves the optimal policy invariant, M and ˜ M are also compatible if ˜ M is the PB-RS counterpart to M .

Original: Thus, the notion of compatibility is less restrictive than the notion of an invariant optimal policy; FV-RS is a superset of PB-RS.
Revised: Thus, the notion of compatibility is less restrictive than the notion of an invariant optimal policy; FV-RS is a superset of PB-RS.

Original: 4. FV-RS
Revised: 4. FV-RS

Original: We now proceed to describe speciﬁc recipes to ﬁnd compatible counterparts for MDPs with sparse reward.
Revised: We now proceed to describe speciﬁc recipes to ﬁnd compatible counterparts for MDPs with sparse reward.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let M be an MDP with state space S and with the sparse reward function
Revised: Let M be an MDP with state space S and with the sparse reward function

Original: [Equation]
Revised: [Equation]

Original: where B ⊆ S is non-empty.
Revised: where B ⊆ S is non-empty.

Original: Let M have the optimal ﬁnal volume G , and let G ⊆ B .
Revised: Let M have the optimal ﬁnal volume G , and let G ⊆ B .

Original: Let ˜ R
Revised: Let ˜ R

Original: ( s, a, s (cid:48) ) be a reward function that fulﬁlls
Revised: ( s, a, s (cid:48) ) be a reward function that fulﬁlls

Original: [Equation]
Revised: [Equation]

Original: Then, for every 0 < ∆ < 1 there exists a discount factor 0 < ˜ γ < 1 such that the MDP ˜ M corresponding to ˜ R and ˜ γ is compatible with M .
Revised: Then, for every 0 < ∆ < 1 there exists a discount factor 0 < ˜ γ < 1 such that the MDP ˜ M corresponding to ˜ R and ˜ γ is compatible with M .

Original: Proof.
Revised: Proof.

Original: See appendix A.5.
Revised: See appendix A.5.

Original: [Equation]
Revised: [Equation]

Original: This theorem gives us a worst-case bound on the reward function: Independently of how small the difference 1 − ∆ between the reward in B and elsewhere is, as long as the conditions in equationare fulﬁlled, we can select a sufﬁciently large ˜ γ < 1 that renders ˜ M a compatible MDP to M .
Revised: This theorem gives us a worst-case bound on the reward function: Independently of how small the difference 1 − ∆ between the reward in B and elsewhere is, as long as the conditions in equationare fulﬁlled, we can select a sufﬁciently large ˜ γ < 1 that renders ˜ M a compatible MDP to M .

Original: Corollary 4.1.1.
Revised: Corollary 4.1.1.

Original: This lower bound on ˜ γ is ˜ γ > ∆ 1 / ( t 0 − 1) .
Revised: This lower bound on ˜ γ is ˜ γ > ∆ 1 / ( t 0 − 1) .

Original: Proof.
Revised: Proof.

Original: Follows directly from the proof of theorem 4.1.
Revised: Follows directly from the proof of theorem 4.1.

Original: [Equation]
Revised: [Equation]

Original: Even if t 0 is unknown but ﬁnite, ˜ M ⊆ M if ˜ γ is chosen large enough.
Revised: Even if t 0 is unknown but ﬁnite, ˜ M ⊆ M if ˜ γ is chosen large enough.

Original: Note that so far, we have imposed no restrictions on ˜ R ( s, a, s (cid:48) ) for s (cid:48) / ∈ B , other than ˜ R ( s, a, s (cid:48) )
Revised: Note that so far, we have imposed no restrictions on ˜ R ( s, a, s (cid:48) ) for s (cid:48) / ∈ B , other than ˜ R ( s, a, s (cid:48) )

Original: ≤ ∆ .
Revised: ≤ ∆ .

Original: In that sense, the statements above represent an “upper bound on the lower bound of ˜ γ ” if we can not assume any additional structure for ˜ R .
Revised: In that sense, the statements above represent an “upper bound on the lower bound of ˜ γ ” if we can not assume any additional structure for ˜ R .

Original: However, if we put more restrictions on ˜ R , this worst-case bound can be relaxed.
Revised: However, if we put more restrictions on ˜ R , this worst-case bound can be relaxed.

Original: As an example, we discuss the case of a reward function that the agent can follow in a step-wise manner in appendix A.6.
Revised: As an example, we discuss the case of a reward function that the agent can follow in a step-wise manner in appendix A.6.

Original: There it is shown that, depending on the step width w , the lower bound ˜ γ > ∆ 1 / ( t 0 − 1) can be relaxed to ˜ γ > ∆ 1 / ( w − 1) .
Revised: There it is shown that, depending on the step width w , the lower bound ˜ γ > ∆ 1 / ( t 0 − 1) can be relaxed to ˜ γ > ∆ 1 / ( w − 1) .

Original: In the important case that the agent can follow the reward function monotonically (i.e. w = 1 ), this lower bound becomes 0 .
Revised: In the important case that the agent can follow the reward function monotonically (i.e. w = 1 ), this lower bound becomes 0 .

Original: For more details and a precise deﬁnition of the step width w , please refer to appendix A.6.
Revised: For more details and a precise deﬁnition of the step width w , please refer to appendix A.6.

Original: 4. P LAN -B ASED FV-RS
Revised: 4. P LAN -B ASED FV-RS

Original: We now describe how to construct a plan-based FV-RS shaping reward function from a plan, given as a sequence in state space.
Revised: We now describe how to construct a plan-based FV-RS shaping reward function from a plan, given as a sequence in state space.

Original: The goal is to use the plan to create a continuous reward function that gives dense feedback to the policy to guide it along the plan.
Revised: The goal is to use the plan to create a continuous reward function that gives dense feedback to the policy to guide it along the plan.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Let M be an MDP with metric state-space ( S , d ) with metric d : S × S
Revised: Let M be an MDP with metric state-space ( S , d ) with metric d : S × S

Original: → R and with the reward function
Revised: → R and with the reward function

Original: [Equation]
Revised: [Equation]

Original: where B ⊆ S is non-empty.
Revised: where B ⊆ S is non-empty.

Original: Let M have the optimal ﬁnal volume G , and let G ⊆ B .
Revised: Let M have the optimal ﬁnal volume G , and let G ⊆ B .

Original: From a planned sequence ( p 0 , p 1 , ..., p L − 1 ) in S with p L − 1 ∈ B , we can construct the reward function 
Revised: From a planned sequence ( p 0 , p 1 , ..., p L − 1 ) in S with p L − 1 ∈ B , we can construct the reward function ˜ R ( s, a, s (cid:48) ) = [Equation] if s (cid:48) ∈ B ;

Original: [Equation]
Revised: [Equation]

Original: where k ( s (cid:48) )
Revised: where k ( s (cid:48) )

Original: =
Revised: =

Original: argmin i ( d ( p i , s (cid:48) )) and g : R 0+ → (0 , 1] is strictly monotonically decreasing, where g (0)
Revised: argmin i ( d ( p i , s (cid:48) )) and g : R 0+ → (0 , 1] is strictly monotonically decreasing, where g (0)

Original: = 1 .
Revised: = 1 .

Original: Then, for every 0 < ∆ < 1 there exists 0 < ˜ γ < 1 such that the MDP ˜ M corresponding to ˜ R and ˜ γ is a compatible MDP to M .
Revised: Then, for every 0 < ∆ < 1 there exists 0 < ˜ γ < 1 such that the MDP ˜ M corresponding to ˜ R and ˜ γ is a compatible MDP to M .

Original: Proof.
Revised: Proof.

Original: Special case of theorem 4.1.
Revised: Special case of theorem 4.1.

Original: [Equation]
Revised: (cid:4)

Original: This translates the plan into a continuous reward function that leads towards the target area B .
Revised: This translates the plan into a continuous reward function that leads towards the target area B .

Original: The corresponding MDP ˜ M is guaranteed to result in the same optimal ﬁnal volume as M .
Revised: The corresponding MDP ˜ M is guaranteed to result in the same optimal ﬁnal volume as M .

Original: Thus, we established the machinery to translate any MDP M with the sparse reward function in equationinto its plan-based FV-RS counterpart MDP ˜ M .
Revised: Thus, we established the machinery to translate any MDP M with the sparse reward function in equationinto its plan-based FV-RS counterpart MDP ˜ M .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We demonstrate the efﬁciency of using FV-RS for plan-based reward shaping using several examples.
Revised: We demonstrate the efﬁciency of using FV-RS for plan-based reward shaping using several examples.

Original: We start with a simple discrete toy example to illustrate the difference between PB-RS and FVRS that is not potential-based.
Revised: We start with a simple discrete toy example to illustrate the difference between PB-RS and FVRS that is not potential-based.

Original: We then compare PB-RS and FV-RS using a realistic 10-dimensional simulated robotic pushing example.
Revised: We then compare PB-RS and FV-RS using a realistic 10-dimensional simulated robotic pushing example.

Original: This is one of several robotic pushing and pick-and-place examples we investigated.
Revised: This is one of several robotic pushing and pick-and-place examples we investigated.

Original: All results can be found in appendix A.2.
Revised: All results can be found in appendix A.2.

Original: Videos of all manipulation examples can be found in the supplementary material.
Revised: Videos of all manipulation examples can be found in the supplementary material.

Original: 5. D ISCRETE T OY E XAMPLE
Revised: 5. D ISCRETE T OY E XAMPLE

Original: We start off considering a simple discrete example as shown in ﬁgure 1a.
Revised: We start off considering a simple discrete example as shown in ﬁgure 1a.

Original: The agent starts in the middle and can choose between the actions A = { left , right , grab } , where grab does not change the state, but allows the agent to grab the ﬂag provided that it is already at the position of the ﬂag.
Revised: The agent starts in the middle and can choose between the actions A = { left , right , grab } , where grab does not change the state, but allows the agent to grab the ﬂag provided that it is already at the position of the ﬂag.

Original: The reward function is shown in ﬁgure 1b.
Revised: The reward function is shown in ﬁgure 1b.

Original: Without shaping, reward 1 is only given if the ﬂag is captured.
Revised: Without shaping, reward 1 is only given if the ﬂag is captured.

Original: We also deﬁne the potential-based reward function
Revised: We also deﬁne the potential-based reward function

Original: [Equation]
Revised: [Equation]

Original:  where the shaping potential Φ( s ) assigns value to being at the correct position, even if the ﬂag is not captured.
Revised: else [Equation] 10) where the shaping potential Φ( s ) assigns value to being at the correct position, even if the ﬂag is not captured.

Original: Similarly, we deﬁne the reward function
Revised: Similarly, we deﬁne the reward function

Original: [Equation]
Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: if

Revised: [Equation]

Revised: finish and a

Revised: [Equation]

Revised: [Equation]

Revised: finish and

Revised: [Equation]

Revised: grab else ,

Revised: [Equation]

Revised: [Figure]

Original: which is not potential-based but ﬁnal-volume-preserving for the discount factor γ = ˜ γ = 0 .
Revised: which is not potential-based but ﬁnal-volume-preserving for the discount factor γ = ˜ γ = 0 .

Original: 7 of the original MDP.
Revised: 7 of the original MDP.

Original: The agent collects data during 2 rollout episodes starting in the middle, during one of which it always chooses to move right until the end of the episode, and another episode during which the agent always moves left .
Revised: The agent collects data during 2 rollout episodes starting in the middle, during one of which it always chooses to move right until the end of the episode, and another episode during which the agent always moves left .

Original: Assume that this data is used in an actor-critic setup, where the actor policy is not converged yet and therefore always chooses the action right .
Revised: Assume that this data is used in an actor-critic setup, where the actor policy is not converged yet and therefore always chooses the action right .

Original: For[Figure] this given policy and given data, the resulting Q -functions are shown in ﬁgure 1c.
Revised: For this given policy and given data, the resulting Q -functions are shown in ﬁgure 1c.

Original: Without reward shaping, there is no reward signal collected during the rollouts, and therefore naturally, the Qfunction contains no information.
Revised: Without reward shaping, there is no reward signal collected during the rollouts, and therefore naturally, the Qfunction contains no information.

Original: With PB-RS, the reward for moving to the correct position is exactly canceled out by the discounted penalty for moving away from it again.
Revised: With PB-RS, the reward for moving to the correct position is exactly canceled out by the discounted penalty for moving away from it again.

Original: As a result, the values of the Q -function at the starting position contain no information about which action is preferable after these two training rollouts.
Revised: As a result, the values of the Q -function at the starting position contain no information about which action is preferable after these two training rollouts.

Original: With FV-RS, the shaping reward is not canceled out and propagated all the way through to the origin.
Revised: With FV-RS, the shaping reward is not canceled out and propagated all the way through to the origin.

Original: In this case, the Q -function provides the agent with the information that moving to the right is preferable if it ﬁnds itself at the starting position.
Revised: In this case, the Q -function provides the agent with the information that moving to the right is preferable if it ﬁnds itself at the starting position.

Original: With PB-RS, the shaped return that is assigned to following a certain policy only depends on the discounted difference of the rewards at the ﬁnal step and initial step of the recorded episode (Grzes, 2017).
Revised: With PB-RS, the shaped return that is assigned to following a certain policy only depends on the discounted difference of the rewards at the ﬁnal step and initial step of the recorded episode (Grzes, 2017).

Original: For non potential-based shaping reward functions like ˜ R FV however, the shaped return depends on all intermediate rewards on the way there as well.
Revised: For non potential-based shaping reward functions like ˜ R FV however, the shaped return depends on all intermediate rewards on the way there as well.

Original: In that sense, FV-RS allows for shaped reward functions that propagate the reward information of intermediate states faster than PB-RS reward functions.
Revised: In that sense, FV-RS allows for shaped reward functions that propagate the reward information of intermediate states faster than PB-RS reward functions.

Original: To use a physical analogy, PB-RS is analogous to a conservative force ﬁeld, in which the potential energy of a particle only depends on its current position.
Revised: To use a physical analogy, PB-RS is analogous to a conservative force ﬁeld, in which the potential energy of a particle only depends on its current position.

Original: FV-RS that is not potential-based is like a non-conservative (e.g. friction) force, for which the energy of a particle is not only a function of its position, but a function of the entire past trajectory.
Revised: FV-RS that is not potential-based is like a non-conservative (e.g. friction) force, for which the energy of a particle is not only a function of its position, but a function of the entire past trajectory.

Original: 5. R OBOTIC P USHING T ASK
Revised: 5. R OBOTIC P USHING T ASK

Original: In this example, we test the efﬁciency of FV-RS in a simulated robotic pushing task with realistic physics.
Revised: In this example, we test the efﬁciency of FV-RS in a simulated robotic pushing task with realistic physics.

Original: The task is very similar to the FetchPush-v1 environment in the OpenAI Gym (Brockman et al., 2016), but is implemented using open-source software.
Revised: The task is very similar to the FetchPush-v1 environment in the OpenAI Gym (Brockman et al., 2016), but is implemented using open-source software.

Original: Speciﬁcally, we use the NVIDIA
Revised: Speciﬁcally, we use the NVIDIA

Original: PhysX engine (phy, 2020) to simulate a box of size 0 .
Revised: PhysX engine (phy, 2020) to simulate a box of size 0 .

Original: 4 × 0 . 4 × 0 .
Revised: 4 × 0 . 4 × 0 .

Original: 2 lying on a table of size 3 × 3 , as shown in ﬁgure 2a. 
Revised: 2 lying on a table of size 3 × 3 , as shown in ﬁgure 2a. [Figure]

Revised: a c t i o n o f c o n s i s t

Revised: e n t l y s u

Revised: [Figure]

Revised: l

Original: The agent controls the 3 -dimensional velocity of a spherical end effector of radius 0 .
Revised: [Figure] s [Figure] Without reward shaping or with PB-RS, none of the agents were able to consistently reach the goal in this experiment.

Revised: Training progress of the top quartile of agents out of category.

Revised: The agent controls the 3 -dimensional velocity of a spherical end effector of radius 0 .

Original: The maximum velocity in any direction is 0 .
Revised: The maximum velocity in any direction is 0 .

Original: 1 per time step.
Revised: 1 per time step.

Original: The actual movement of the end effector is noisy; in 10% of the time steps, the end effector moves randomly.
Revised: The actual movement of the end effector is noisy; in 10% of the time steps, the end effector moves randomly.

Original: Using quaternion representation, the state space S has 7 + 3 = 10 degrees of freedom.
Revised: Using quaternion representation, the state space S has 7 + 3 = 10 degrees of freedom.

Original: The goal is to push the box within a distance 
Revised: The goal is to push the box within a distance of 0 .

Original: [Figure]

Original: F r a c t i o n o f c o n s i s t e n t l

Original: y s u cc

Original: [Figure]

Original: ss f u l

Original: [Figure]

Original: g e n t s [Figure] Without reward shaping or with PB-RS, none of the agents were able to consistently reach the goal in this experiment of 0 . 1 from  position  any orientation.
Revised: 1 from the goal position shown in green, with any orientation.

Original: Let B be the set of states that fulﬁll this requirement.
Revised: Let B be the set of states that fulﬁll this requirement.

Original: The unshaped reward R ( s, a, s (cid:48) ) is 1 if s (cid:48) ∈ B and is 0 elsewhere.
Revised: The unshaped reward R ( s, a, s (cid:48) ) is 1 if s (cid:48) ∈ B and is 0 elsewhere.

Original: The plan ( p 0 , p 1 , ..., p L − 1 ) is given as a sequence of states, as shown in ﬁgure 2b.
Revised: The plan ( p 0 , p 1 , ..., p L − 1 ) is given as a sequence of states, as shown in ﬁgure 2b.

Original: The planned sequence has been created manually using a noise-free model, and therefore can not be followed directly in the realistic simulation.
Revised: The planned sequence has been created manually using a noise-free model, and therefore can not be followed directly in the realistic simulation.

Original: Instead, it is used to create a plan-based shaped reward from it.
Revised: Instead, it is used to create a plan-based shaped reward from it.

Original: Following the suggestion of theorem 4.2, we speciﬁcally compare the reward functions
Revised: Following the suggestion of theorem 4.2, we speciﬁcally compare the reward functions

Original: [Equation]
Revised: [Equation]

Original: If Φ FV is chosen suitably following theorem 4.2, ˜ R FV is ﬁnal-volume-preserving with respect to reward function R of the original MDP.
Revised: If Φ FV is chosen suitably following theorem 4.2, ˜ R FV is ﬁnal-volume-preserving with respect to reward function R of the original MDP.

Original: The potential-based function ˜ R PB acts as the baseline.
Revised: The potential-based function ˜ R PB acts as the baseline.

Original: Notice that Φ FV and Φ PB are chosen independently in order to facilitate a fair comparison of FV-RS and
Revised: Notice that Φ FV and Φ PB are chosen independently in order to facilitate a fair comparison of FV-RS and

Original: PB-RS.
Revised: PB-RS.

Original: We discuss and compare different choices both for Φ FV and Φ PB in appendix A.1.
Revised: We discuss and compare different choices both for Φ FV and Φ PB in appendix A.1.

Original: There we compare different gaussian plan-based functions, similar to the ones used in Brys et al.
Revised: There we compare different gaussian plan-based functions, similar to the ones used in Brys et al.

Original: (2015), as well as a more simple function that only depends on the distance of the box to the target.
Revised: (2015), as well as a more simple function that only depends on the distance of the box to the target.

Original: Our analysis reveals that using the gaussian plan-based function
Revised: Our analysis reveals that using the gaussian plan-based function

Original: [Equation]
Revised: [Equation]

Original: with σ = 0 .
Revised: with σ = 0 .

Original: 2 results in a robust performance for FV-RS.
Revised: 2 results in a robust performance for FV-RS.

Original: Here, k ( s ) = argmin i ( d ( p i , s )) , and d ( · , · ) is the euclidean distance ignoring the coordinates corresponding to the orientation of the box.
Revised: Here, k ( s ) = argmin i ( d ( p i , s )) , and d ( · , · ) is the euclidean distance ignoring the coordinates corresponding to the orientation of the box.

Original: The value of this function increases if the agent moves further along the planned path or closer towards the planned path.
Revised: The value of this function increases if the agent moves further along the planned path or closer towards the planned path.

Original: Similarly, our analysis also shows that using Φ PB ( s ) =
Revised: Similarly, our analysis also shows that using Φ PB ( s ) =

Original: K Φ FV ( s ) with K =results in a robust performance for PB-RS.
Revised: K Φ FV ( s ) with K =results in a robust performance for PB-RS.

Original: We would like to emphasize that we did not impose that Φ PB ( s ) is a multiple of Φ FV ( s ) , rather our analysis revealed that this is the most suitable choice.
Revised: We would like to emphasize that we did not impose that Φ PB ( s ) is a multiple of Φ FV ( s ) , rather our analysis revealed that this is the most suitable choice.

Original: For more details, please refer to appendix A.1.
Revised: For more details, please refer to appendix A.1.

Original: We apply DDPG (Lillicrap et al., 2015) for the examples presented throughout this paper, with the exception of the example presented in appendix A.4 using PPO (Schulman et al., 2017).
Revised: We apply DDPG (Lillicrap et al., 2015) for the examples presented throughout this paper, with the exception of the example presented in appendix A.4 using PPO (Schulman et al., 2017).

Original: We use γ = ˜ γ = 0 . 9 as discount factor.
Revised: We use γ = ˜ γ = 0 . 9 as discount factor.

Original: The agent collects data in rollout episodes of random length sampled uniformly between 0 and 300 .
Revised: The agent collects data in rollout episodes of random length sampled uniformly between 0 and 300 .

Original: Each of these episodes starts at the same initial position indicated in ﬁgure 2a; we do not assume that the system can be reset to arbitrary states.
Revised: Each of these episodes starts at the same initial position indicated in ﬁgure 2a; we do not assume that the system can be reset to arbitrary states.

Original: The exploration policy acts (cid:15) -greedily with respect to the current actor, where (cid:15) = 0 .
Revised: The exploration policy acts (cid:15) -greedily with respect to the current actor, where (cid:15) = 0 .

Original: After every rollout, actor and critic are updated using the replay buffer.
Revised: After every rollout, actor and critic are updated using the replay buffer.

Original: Both actor and critic are implemented as neural networks in
Revised: Both actor and critic are implemented as neural networks in

Original: Tensorﬂow (Abadi et al., 2016).
Revised: Tensorﬂow (Abadi et al., 2016).

Original: Implementation details can be found in the supplementary code material.
Revised: Implementation details can be found in the supplementary code material.

Original: The data reported in ﬁgure 2 is obtained as follows:
Revised: The data reported in ﬁgure 2 is obtained as follows:

Original: For each of the shaping strategies s ∈ { No RS , PB-RS , FV-RS } , multiple agents a ∈ { 1 , ..., A } are trained independently from scratch.
Revised: For each of the shaping strategies s ∈ { No RS , PB-RS , FV-RS } , multiple agents a ∈ { 1 , ..., A } are trained independently from scratch.

Original: After N rollout episodes, multiple test episodes m ∈ { 1 , ..., M } are run for 300 time steps or until the goal is reached.
Revised: After N rollout episodes, multiple test episodes m ∈ { 1 , ..., M } are run for 300 time steps or until the goal is reached.

Original: d ( s ) am ( N ) is the distance of the box to the goal at the end of such a test episode.
Revised: d ( s ) am ( N ) is the distance of the box to the goal at the end of such a test episode.

Original: We classify an agent as being consistently successful once ¯ d ( s ) a = 1 /M (cid:80) m d ( s ) am ( N ) < 0 .
Revised: We classify an agent as being consistently successful once ¯ d ( s ) a = 1 /M (cid:80) m d ( s ) am ( N ) < 0 .

Original: We emphasize that this is a rather strict criterion.
Revised: We emphasize that this is a rather strict criterion.

Original: Over M test rollouts, a consistently successful agent must achieve an average asymptotic distance to the goal that is within the margin of 0 . 1 , meaning that essentially all test rollouts have to be within the margin.
Revised: Over M test rollouts, a consistently successful agent must achieve an average asymptotic distance to the goal that is within the margin of 0 . 1 , meaning that essentially all test rollouts have to be within the margin.

Original: We use A = 40 and M = 30 in all experiments reported.
Revised: We use A = 40 and M = 30 in all experiments reported.

Original: The training of an agent a is halted once it is consistently successful.
Revised: The training of an agent a is halted once it is consistently successful.

Original: We report data from all agents in ﬁgure 2c and focus on the best performing quartile of agents out of each category in ﬁgure 2d.
Revised: We report data from all agents in ﬁgure 2c and focus on the best performing quartile of agents out of each category in ﬁgure 2d.

Original: The performance of agents is ranked by how quickly they become consistently successful, or by how close to the goal they get in case they never become consistently successful.
Revised: The performance of agents is ranked by how quickly they become consistently successful, or by how close to the goal they get in case they never become consistently successful.

Original: FV-RS helps to learn the task with signiﬁcantly higher sample efﬁciency than PB-RS and without reward shaping.
Revised: FV-RS helps to learn the task with signiﬁcantly higher sample efﬁciency than PB-RS and without reward shaping.

Original: Some agents (see ﬁgure 2c) are consistently successful after ca.rollout episodes (corresponding to ca.
Revised: Some agents (see ﬁgure 2c) are consistently successful after ca.rollout episodes (corresponding to ca.

Original: 45 000 MDP transitions) using FV-RS.
Revised: 45 000 MDP transitions) using FV-RS.

Original: In contrast, none of the agents using PB-RS or no reward shaping were able to consistently reach the goal aftertraining rollouts in this example.
Revised: In contrast, none of the agents using PB-RS or no reward shaping were able to consistently reach the goal aftertraining rollouts in this example.

Original: Apart from this pushing example, we also investigated other robotic pushing or pick-and-place settings in appendix A.2.
Revised: Apart from this pushing example, we also investigated other robotic pushing or pick-and-place settings in appendix A.2.

Original: Furthermore, we compare FV-RS and PB-RS using PPO instead of DDPG in appendix A.4.
Revised: Furthermore, we compare FV-RS and PB-RS using PPO instead of DDPG in appendix A.4.

Original: While DDPG is off-policy and deterministic, PPO is on-policy and stochastic.
Revised: While DDPG is off-policy and deterministic, PPO is on-policy and stochastic.

Original: Consistently across all experiments, we ﬁnd a signiﬁcant advantage of using FV-RS over PB-RS, which qualitatively conﬁrms the ﬁndings discussed here.
Revised: Consistently across all experiments, we ﬁnd a signiﬁcant advantage of using FV-RS over PB-RS, which qualitatively conﬁrms the ﬁndings discussed here.

Original: Videos of this and other manipulation examples can also be found in the supplementary material.
Revised: Videos of this and other manipulation examples can also be found in the supplementary material.

Original:  D ISCUSSION AND C ONCLUSIONS
Revised:  D ISCUSSION AND C ONCLUSIONS

Original: We introduced FV-RS, a subclass of reward shaping that relaxes the invariance guarantee of PB-RS to a guarantee of preserved long-time behavior.
Revised: We introduced FV-RS, a subclass of reward shaping that relaxes the invariance guarantee of PB-RS to a guarantee of preserved long-time behavior.

Original: We illustrated that since FV-RS is not necessarily potential-based, it can convey the information provided by the shaping reward in a more direct way than PB-RS.
Revised: We illustrated that since FV-RS is not necessarily potential-based, it can convey the information provided by the shaping reward in a more direct way than PB-RS.

Original: Using FV-RS, the value of following a policy does not necessarily depend on the discounted ﬁnal reward only, but can also depend on all intermediate rewards.
Revised: Using FV-RS, the value of following a policy does not necessarily depend on the discounted ﬁnal reward only, but can also depend on all intermediate rewards.

Original: Thus, FV-RS is analogous to a non-conservative force (e.g. friction), and PB-RS to a conservative force (e.g. gravity).
Revised: Thus, FV-RS is analogous to a non-conservative force (e.g. friction), and PB-RS to a conservative force (e.g. gravity).

Original: We proposed to use FV-RS in order to increase the sample efﬁciency of plan-informed RL algorithms.
Revised: We proposed to use FV-RS in order to increase the sample efﬁciency of plan-informed RL algorithms.

Original: We demonstrated in the experiments that, compared to plan-based PB-RS, plan-based FV-RS can increase the sample efﬁciency of RL signiﬁcantly.
Revised: We demonstrated in the experiments that, compared to plan-based PB-RS, plan-based FV-RS can increase the sample efﬁciency of RL signiﬁcantly.

Original: This observation is consistent across multiple manipulation examples and RL algorithms.
Revised: This observation is consistent across multiple manipulation examples and RL algorithms.

Original: In all manipulation examples tested, there were agents that were consistently successful after ca.
Revised: In all manipulation examples tested, there were agents that were consistently successful after ca.

Original: 300 rollout episodes using FV-RS and DDPG.
Revised: 300 rollout episodes using FV-RS and DDPG.

Original: An important limitation of plan-based reward shaping in general arises if the plan used for the shaping is globally wrong.
Revised: An important limitation of plan-based reward shaping in general arises if the plan used for the shaping is globally wrong.

Original: In such a case, the shaped reward would be of no use for the exploration and potentially misleading.
Revised: In such a case, the shaped reward would be of no use for the exploration and potentially misleading.

Original: Here, combining FV-RS with other exploration strategies could possibly remedy this limitation.
Revised: Here, combining FV-RS with other exploration strategies could possibly remedy this limitation.

Original: Another limitation of the presented method is that it is currently formulated for a goal-driven sparse-reward setting.
Revised: Another limitation of the presented method is that it is currently formulated for a goal-driven sparse-reward setting.

Original: We restricted the present paper to the discussion of this class of MDPs since it is often encountered e.g. in robotic manipulation tasks.
Revised: We restricted the present paper to the discussion of this class of MDPs since it is often encountered e.g. in robotic manipulation tasks.

Original: The direct application of
Revised: The direct application of

Original: RL still poses a major obstacle here, making reward shaping an important tool.
Revised: RL still poses a major obstacle here, making reward shaping an important tool.

Original: Since reward shaping only modiﬁes the reward itself, our approach makes no other assumptions about the RL algorithm used.
Revised: Since reward shaping only modiﬁes the reward itself, our approach makes no other assumptions about the RL algorithm used.

Original: It is orthogonal to and can be combined with other techniques used in RL, including popular methods for sparse-reward settings like Hindsight Experience Replay (Andrychowicz et al., 2017).
Revised: It is orthogonal to and can be combined with other techniques used in RL, including popular methods for sparse-reward settings like Hindsight Experience Replay (Andrychowicz et al., 2017).

Original: The same holds true for methods in LfD that also exploit task information provided in the form of an approximate demonstration or plan.
Revised: The same holds true for methods in LfD that also exploit task information provided in the form of an approximate demonstration or plan.

Original: Plan-based FV-RS only relies on a relatively simple representation of the approximate plan in the form of a sequence in state space.
Revised: Plan-based FV-RS only relies on a relatively simple representation of the approximate plan in the form of a sequence in state space.

Original: In addition, our method does not require the system to be reset to arbitrary states, as it is the case e.g. in Salimans & Chen (2018).
Revised: In addition, our method does not require the system to be reset to arbitrary states, as it is the case e.g. in Salimans & Chen (2018).

Original: Both these aspects make it a practical choice for real-world robotic applications.
Revised: Both these aspects make it a practical choice for real-world robotic applications.

Original: In many robotic manipulation settings, an approximate planner for the task is available, but the resulting plan can not be directly executed on the real robot.
Revised: In many robotic manipulation settings, an approximate planner for the task is available, but the resulting plan can not be directly executed on the real robot.

Original: Plan-based FV-RS could facilitate this sim-2-real transfer in an efﬁcient and principled way using RL.
Revised: Plan-based FV-RS could facilitate this sim-2-real transfer in an efﬁcient and principled way using RL.

Original: Trading strict optimality guarantees for increased sample efﬁciency (while still guaranteeing preserved long-time behavior) could be beneﬁcial in these cases.
Revised: Trading strict optimality guarantees for increased sample efﬁciency (while still guaranteeing preserved long-time behavior) could be beneﬁcial in these cases.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Wewould like to thank the anonymous reviewers for their insightful suggestions and comments
Revised: We thank the anonymous reviewers for their insightful suggestions and comments. We thank the

Revised: MPI for Intelligent Systems for the Max Planck Fellowship funding

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Appendix A.1 compares different choices of shaping functions used for PB-RS and for FV-RS.
Revised: Appendix A.1 compares different choices of shaping functions used for PB-RS and for FV-RS.

Original: The particular choice that was used for the experiments in this paper is also discussed and justiﬁed there.
Revised: The particular choice that was used for the experiments in this paper is also discussed and justiﬁed there.

Original: Appendix A.2 contains additional robotic manipulation examples.
Revised: Appendix A.2 contains additional robotic manipulation examples.

Original: These are instances of both the simulated robotic pushing task presented in section 5.2 and a simulated robotic pick-and-place task presented in appendix A.3.
Revised: These are instances of both the simulated robotic pushing task presented in section 5.2 and a simulated robotic pick-and-place task presented in appendix A.3.

Original: Appendix A.4 contains robotic pushing examples using PPO as RL algorithm instead of DDPG.
Revised: Appendix A.4 contains robotic pushing examples using PPO as RL algorithm instead of DDPG.

Original: Appendix A.5 contains the proof of theorem 4.1.
Revised: Appendix A.5 contains the proof of theorem 4.1.

Original: In appendix A.6, a special case of theorem 4.1 is discussed, additionally assuming a step-wise reward function that the agent can follow monotonically
Revised: In appendix A.6, a special case of theorem 4.1 is discussed, additionally assuming a step-wise reward function that the agent can follow monotonically.

Original: . A. E XPERIMENTS ON THE CHOICE OF SHAPING FUNCTIONS USED FOR PB - RS AND FV- RS
Revised: A. E XPERIMENTS ON THE CHOICE OF SHAPING FUNCTIONS USED FOR PB - RS AND FV- RS

Original: We assume that the plan ( p 0 , p 1 , ..., p L − 1 ) is given as a sequence of states.
Revised: We assume that the plan ( p 0 , p 1 , ..., p L − 1 ) is given as a sequence of states.

Original: The functions ˜ R FV and ˜
Revised: The functions ˜ R FV and ˜

Original: R PB , and therefore Φ FV and Φ PB , should provide a reward that is higher the closer the state of the environment is to the plan, and the further advanced it is along the plan towards the goal.
Revised: R PB , and therefore Φ FV and Φ PB , should provide a reward that is higher the closer the state of the environment is to the plan, and the further advanced it is along the plan towards the goal.

Original: Furthermore, Φ FV ( s ) <
Revised: Furthermore, Φ FV ( s ) <

Original: 1 ∀ s ∈ S has to be fulﬁlled in order to ensure that ˜ R FV fulﬁlls the requirements of theorem 4.2.
Revised: 1 ∀ s ∈ S has to be fulﬁlled in order to ensure that ˜ R FV fulﬁlls the requirements of theorem 4.2.

Original: A canonical choice that satisﬁes these constraints are the Gaussian functions
Revised: A canonical choice that satisﬁes these constraints are the Gaussian functions

Original: [Equation]
Revised: [Equation]

Original:  where k ( s ) = argmin i( d ( p i , s )) , d ( · , · ) is the euclidean distance ignoring the coordinates corresponding to the orientation of the box, and K = 50 is a factor that is applied in order to render ˜ R FV and ˜ R PB comparable in scale.
Revised: 14) where k ( s ) = argmin i d ( p i , s )) , d ( · , · ) is the euclidean distance ignoring the coordinates corresponding to the orientation of the box, and K = 50 is a factor that is applied in order to render ˜ R FV and ˜ R PB comparable in scale.

Original: Φ GaussPB is similar to the shaping potential that was used for potentialbased reward shaping from demonstration in Brys et al.
Revised: Φ GaussPB is similar to the shaping potential that was used for potentialbased reward shaping from demonstration in Brys et al.

Original: As a simpler alternative, we also compare against the negative distance functions
Revised: As a simpler alternative, we also compare against the negative distance functions

Original: [Equation]
Revised: [Equation]

Original: where d Box ( · , p L − 1 ) denotes the distance of the box the goal located at p L − 1 .
Revised: where d Box ( · , p L − 1 ) denotes the distance of the box the goal located at p L − 1 .

Original: These functions do not take the planned sequence into account, but only the intended ﬁnal state, i.e. the goal.
Revised: These functions do not take the planned sequence into account, but only the intended ﬁnal state, i.e. the goal.

Original: It is easy to verify that using Φ DistFV results in a ﬁnal-volume-preserving reward function.
Revised: It is easy to verify that using Φ DistFV results in a ﬁnal-volume-preserving reward function.

Original: We compare the performance of FV-RS agents using Φ DistFV or Φ GaussFV ( · , σ ) for different σ against PBRS agents using Φ DistPB or Φ GaussPB ( · , σ ) for different σ .
Revised: We compare the performance of FV-RS agents using Φ DistFV or Φ GaussFV ( · , σ ) for different σ against PBRS agents using Φ DistPB or Φ GaussPB ( · , σ ) for different σ .

Original: We use two different robotic pushing examples for this comparison.
Revised: We use two different robotic pushing examples for this comparison.

Original: These are instances of the pushing example described in section 5.2, but with [Figure] cc e ss f u l a g e n t s
Revised: These are instances of the pushing example described in section 5.2, but with different initial positions and different goal positions.

Original: FV-RS No RS GaussFV ( ,0.1) GaussFV (
Revised: We measure the asymptotic distance over different agents and rollouts as detailed in section 5.2.

Original: [Figure] [Figure] [Figure]
Revised: The results are presented in ﬁgure 3.

Original: [Figure]
Revised: [Figure]

Original: DistFV (f) different initial positions and different goal positions. We measure the asymptotic distance over different agents and rollouts as detailed in section 5.2. The results are presented in ﬁgure
Revised: (f) With FV-RS, a similar result is obtained, although Φ GaussFV ( · , 0 . 2) performs signiﬁcantly better here.

Original: Interestingly, Φ DistPB and Φ DistFV perform best in the simple example.
Revised: Interestingly, Φ DistPB and Φ DistFV perform best in the simple example.

Original: However, using these functions for reward shaping does not scale well to the more difﬁcult example.
Revised: However, using these functions for reward shaping does not scale well to the more difﬁcult example.

Original: This can be ascribed to the missing path information when using purely distance-based information for reward shaping.
Revised: This can be ascribed to the missing path information when using purely distance-based information for reward shaping.

Original: On the other hand, using Φ GaussPB ( · , 0 . 2) for PB-RS seems to be the second-best choice for the simple example, and one of the best for the more difﬁcult one.
Revised: On the other hand, using Φ GaussPB ( · , 0 . 2) for PB-RS seems to be the second-best choice for the simple example, and one of the best for the more difﬁcult one.

Original: The same holds true for using Φ GaussFV ( · , 0 . 2) for FVRS.
Revised: The same holds true for using Φ GaussFV ( · , 0 . 2) for FVRS.

Original: Since the manipulation problems we are interested in in this paper resemble the more difﬁcult example more than the simpler one, these experiments suggest that using Φ GaussFV ( · , 0 . 2) for FV-RS is the best and most robust choice.
Revised: Since the manipulation problems we are interested in in this paper resemble the more difﬁcult example more than the simpler one, these experiments suggest that using Φ GaussFV ( · , 0 . 2) for FV-RS is the best and most robust choice.

Original: We refer to this function as Φ FV throughout the rest of the paper.
Revised: We refer to this function as Φ FV throughout the rest of the paper.

Original: Furthermore, using Φ GaussPB ( · , 0 . 2) for the baseline method PB-RS allows for a fair comparison, since this choice has shown the best performance on the more difﬁcult problem using FV-RS.
Revised: Furthermore, using Φ GaussPB ( · , 0 . 2) for the baseline method PB-RS allows for a fair comparison, since this choice has shown the best performance on the more difﬁcult problem using FV-RS.

Original: We refer to this function as Φ PB throughout the rest of the paper.
Revised: We refer to this function as Φ PB throughout the rest of the paper.

Revised: [Figure]

Original: On a different note, we observe that independently of the shaping function used, the performance of
Revised: On a different note, we observe that independently of the shaping function used, the performance of

Original: PB-RS decreases faster when moving to the more difﬁcult example than the performance of FV-RS.
Revised: PB-RS decreases faster when moving to the more difﬁcult example than the performance of FV-RS.

Original: For the example discussed in section 5.2, plots of the function Φ GaussFV ( · , 0 . 2) are shown in ﬁgure 4.
Revised: For the example discussed in section 5.2, plots of the function Φ GaussFV ( · , 0 . 2) are shown in ﬁgure 4.

Original: Φ GaussFV ( · , 0 . 2) is larger the closer s is to the approximate plan ( p 0 , p 1 , ..., p L − 1 ) given as a sequence of states, and the further advanced along the plan the state p
Revised: Φ GaussFV ( · , 0 . 2) is larger the closer s is to the approximate plan ( p 0 , p 1 , ..., p L − 1 ) given as a sequence of states, and the further advanced along the plan the state p

Original: i closest to s is (compare e.g. the scale of ﬁgure 4a to ﬁgure 4c).
Revised: i closest to s is (compare e.g. the scale of ﬁgure 4a to ﬁgure 4c).

Original: Using the approximate plan, the reward shaping function ˜ R FV that is[Figure] constructed using Φ GaussFV ( · , 0 . 2) provides a signal to the RL agent that helps 
Revised: Using the approximate plan, the reward shaping function ˜ R FV that is constructed using Φ GaussFV ( · , 0 . 2) provides a signal to the RL agent that helps guiding it towards the goal.

Original: A. A DDITIONAL R OBOTIC M ANIPULATION E XAMPLES
Revised: A. A DDITIONAL R OBOTIC M ANIPULATION E XAMPLES

Original: The robotic manipulation examples presented in the following are instances of the simulated robotic pushing task presented in section 5.2 and a simulated robotic pick-and-place task presented in appendix A.3.
Revised: The robotic manipulation examples presented in the following are instances of the simulated robotic pushing task presented in section 5.2 and a simulated robotic pick-and-place task presented in appendix A.3.

Original: The results are summarized in ﬁgure 5 and ﬁgure 6, respectively.
Revised: The results are summarized in ﬁgure 5 and ﬁgure 6, respectively.

Original: Qualitatively, the ﬁndings discussed in section 5.2 are conﬁrmed across all examples.
Revised: Qualitatively, the ﬁndings discussed in section 5.2 are conﬁrmed across all examples.

Original: In both the pushing and the pick-and-place setting, FV-RS helps to learn the manipulation task with signiﬁcantly higher sample efﬁciency than PB-RS.
Revised: In both the pushing and the pick-and-place setting, FV-RS helps to learn the manipulation task with signiﬁcantly higher sample efﬁciency than PB-RS.

Original: In all examples, there were agents that learned to consistently fulﬁll the task after ca.
Revised: In all examples, there were agents that learned to consistently fulﬁll the task after ca.

Original: 300 rollout episodes using FV-RS, which corresponds to 45 

Revised: 300 rollout episodes using FV-RS, which corresponds to 45

Original: MDP transitions.
Revised: MDP transitions.

Original: The ﬁrst pushing example of ﬁgure 5 can be seen as an easier version of the second example.
Revised: The ﬁrst pushing example of ﬁgure 5 can be seen as an easier version of the second example.

Original: Interestingly, no agent is able to consistently succeed at the harder task with PB-RS, while one agents out of A = 40 is able to consistently succeed at the easier one with PB-RS.
Revised: Interestingly, no agent is able to consistently succeed at the harder task with PB-RS, while one agents out of A = 40 is able to consistently succeed at the easier one with PB-RS.

Original: In contrast to that, the majority of agents learns to consistently fulﬁll both tasks after ca.
Revised: In contrast to that, the majority of agents learns to consistently fulﬁll both tasks after ca.

Original: 300 training rollouts in both examples using FV-RS.
Revised: 300 training rollouts in both examples using FV-RS.

Original: A video of all experiments is included in the supplementary material.
Revised: A video of all experiments is included in the supplementary material.

Original: A. R OBOTIC P ICK - AND -P LACE T ASK
Revised: A. R OBOTIC P ICK - AND -P LACE T ASK

Original: For the robotic pick-and-place task, we use the NVIDIA PhysX engine (phy, 2020) to simulate a disk of radius 0 .
Revised: For the robotic pick-and-place task, we use the NVIDIA PhysX engine (phy, 2020) to simulate a disk of radius 0 .

Original: 2 lying on a table of size 3 × 3 , as shown in ﬁgure 7a.
Revised: 2 lying on a table of size 3 × 3 , as shown in ﬁgure 7a.

Original: We assume that the grasp is automatically established if the end effector touches the disk in the red area.
Revised: We assume that the grasp is automatically established if the end effector touches the disk in the red area.

Original: Action and state space are the same as in section 5.2, apart from the information whether a grasp has been established or not, which is appended to the state of the system (binary signal).
Revised: Action and state space are the same as in section 5.2, apart from the information whether a grasp has been established or not, which is appended to the state of the system (binary signal).

Original: Just as in section 5.2, the movement of the end effector is noisy.
Revised: Just as in section 5.2, the movement of the end effector is noisy.

Original: The goal is to place the disk within a distance of 0 .
Revised: The goal is to place the disk within a distance of 0 .

Original: 1 from the goal position, shown in green in in ﬁgure 7a.
Revised: 1 from the goal position, shown in green in in ﬁgure 7a.

Original: Let B be the set of states that fulﬁll this requirement.
Revised: Let B be the set of states that fulﬁll this requirement.

Original: The unshaped reward R ( s, a, s (cid:48) ) is 1 if s (cid:48) ∈ B and is 0 elsewhere.
Revised: The unshaped reward R ( s, a, s (cid:48) ) is 1 if s (cid:48) ∈ B and is 0 elsewhere.

Original: We choose γ = ˜ γ = 0 .
Revised: We choose γ = ˜ γ = 0 .

Original:  Again, the plan (see ﬁgure 7b) is given as a sequence of states ( p 0 , p 1 , ..., p L − 1 ) , and we construct the potential-based shaped reward ˜ R PB and the ﬁnal-volume-preserving reward ˜ R FV analogously to equation 12 and equation 13.
Revised: [Figure] Again, the plan (see ﬁgure 7b) is given as a sequence of states ( p 0 , p 1 , ..., p L − 1 ) , and we construct the potential-based shaped reward ˜ R PB and the ﬁnal-volume-preserving reward ˜ R FV analogously to equation 12 and equation 13.

Original: This is motivated by the analysis done in appendix A.1, combined with the observation that the scale and complexity pick-and-place example is comparable to the scale and complexity of the pushing example.
Revised: This is motivated by the analysis done in appendix A.1, combined with the observation that the scale and complexity pick-and-place example is comparable to the scale and complexity of the pushing example.

Original: In contrast to equation 13 however, the euclidean distance d ( · , · ) also takes into account the binary information on whether a grasp has been established or[Figure] not.
Revised: In contrast to equation 13 however, the euclidean distance d ( · , · ) also takes into account the binary information on whether a grasp has been established or not.

Original: For PB-RS, we scale the potential function by K = 30 .
Revised: For PB-RS, we scale the potential function by K = 30 .

Original: This is done in order to ensure that the reward functions of PB-RS and FV-RS are of similar scale.
Revised: This is done in order to ensure that the reward functions of PB-RS and FV-RS are of similar scale.

Original: The results for 2 different pick-and-place examples are discussed in appendix A.2, speciﬁcally in ﬁgure 6.
Revised: The results for 2 different pick-and-place examples are discussed in appendix A.2, speciﬁcally in ﬁgure 6.

Original: Here, we measure the asymptotic distance over different agents and rollouts as detailed in section 5.2. 
Revised: Here, we measure the asymptotic distance over different agents and rollouts as detailed in section 5.2. [Figure]

Revised: reward shaping, none of the agents were able to consistently reach the goal in these experiments.

Revised: The training progress of the top quartile of agents is signiﬁcantly faster with FV-RS than with PB-RS or without reward shaping.

Revised: [Figure]

Original: A. R OBOTIC P USHING E XAMPLES USING PPO
Revised: A. R OBOTIC P USHING E XAMPLES USING PPO

Original: We repeat the experiments presented in appendix A.1 using PPO instead of DDPG as RL algorithm.
Revised: We repeat the experiments presented in appendix A.1 using PPO instead of DDPG as RL algorithm.

Original: In DDPG, a deterministic policy is learned from off-policy data.
Revised: In DDPG, a deterministic policy is learned from off-policy data.

Original: In contrast to this, PPO is an on-policy algorithm for learning a stochastic policy.
Revised: In contrast to this, PPO is an on-policy algorithm for learning a stochastic policy.

Original: We use an open-source implementation from Barhate (2020).
Revised: We use an open-source implementation from Barhate (2020).

Original: The actor network outputs mean values between − 0 .
Revised: The actor network outputs mean values between − 0 .

Original: 1 and 0 .
Revised: 1 and 0 .

Original: 1 for the Gaussian policy, while the standard deviation of the policy [Figure]
Revised: 1 for the Gaussian policy, while the standard deviation of the policy output is ﬁxed to 0 .

Original: without reward shaping, none of the agents were able to consistently reach the goal in these experiments.

Original: The training progress of the top quartile of agents is signiﬁcantly faster with FV-RS than with PB-RS or without reward shaping.

Original: [Figure]

Original: output is ﬁxed to 0 .

Original: We use the clipping parameter (cid:15) = 0 .
Revised: We use the clipping parameter (cid:15) = 0 .

Original: 1 (see Schulman et al.
Revised: 1 (see Schulman et al.

Original: For more implementation details, please also refer to the supplementary material.
Revised: For more implementation details, please also refer to the supplementary material

Original:  Since the environment is unchanged, we use the shaping functions deﬁned in equation 12 and equation 13.
Revised: [Figure] Since the environment is unchanged [Figure] we use the shaping functions deﬁned in equation 12 and equation 13.

Original: The results are presented in ﬁgure 8.
Revised: The results are presented in ﬁgure 8.

Original: The relative advantage of using FV-RS over PB-RS
Revised: The relative advantage of using FV-RS over PB-RS

Original: [Figure]
Revised: [Figure]

Original: -RS.

Original: (d, e, f)

Original: This difference increases however on the more difﬁcult example.

Original: This relative advantage of using FV-RS over PB-RS using

Original: PPO is very similar to the one reported in ﬁgure 3 for DDPG.

Original: with PPO is very similar to the one reported in ﬁgure 3, where results for the same examples using
Revised: with PPO is very similar to the one reported in ﬁgure 3, where results for the same examples using

Original: DDPG are reported.
Revised: DDPG are reported.

Original: This relative advantage is relatively small on the simpler example, but increases drastically for the more difﬁcult example.
Revised: This relative advantage is relatively small on the simpler example, but increases drastically for the more difﬁcult example.

Original: As a side note, the absolute number of samples needed by an agent to consistently learn the task is typically much higher using the on-policy PPO algorithm.
Revised: As a side note, the absolute number of samples needed by an agent to consistently learn the task is typically much higher using the on-policy PPO algorithm.

Original: This is especially true for the more difﬁcult example.
Revised: This is especially true for the more difﬁcult example.

Original: Another interesting observation is that without any reward shaping (No RS), the stochastic PPO policy performs better than the deterministic DDPG policy in ﬁgure 3.
Revised: Another interesting observation is that without any reward shaping (No RS), the stochastic PPO policy performs better than the deterministic DDPG policy in ﬁgure 3.

Original: This is however not the case in the more difﬁcult example.
Revised: This is however not the case in the more difﬁcult example.

Original: A. P ROOF OF T HEOREM 4. 
Revised: A. P ROOF OF T HEOREM 4.

Original: We adopt notation from section 4.
Revised: We adopt notation from section 4.

Original: The largest reward 1 can only be collected if s (cid:48) is within B .
Revised: The largest reward 1 can only be collected if s (cid:48) is within B .

Original: Since
Revised: Since

Original: M has the optimal ﬁnal volume G ⊆ B and M and ˜ M have the same dynamics T , there exists a policy (the optimal policy of M ) to drive the state of ˜ M to G ⊆ B in ﬁnite time t 0 .
Revised: M has the optimal ﬁnal volume G ⊆ B and M and ˜ M have the same dynamics T , there exists a policy (the optimal policy of M ) to drive the state of ˜ M to G ⊆ B in ﬁnite time t 0 .

Original: The only question that remains is whether it is suboptimal for a policy to drive the state of ˜ M to G ⊆ B if this takes many steps, and whether it might be optimal to stay at a local maximum of ˜ R ( s, a, s (cid:48) ) .
Revised: The only question that remains is whether it is suboptimal for a policy to drive the state of ˜ M to G ⊆ B if this takes many steps, and whether it might be optimal to stay at a local maximum of ˜ R ( s, a, s (cid:48) ) .

Original: The maximum return that can be collected outside of B is ∆ / (1 − ˜ γ ) .
Revised: The maximum return that can be collected outside of B is ∆ / (1 − ˜ γ ) .

Original: On the other hand, directly navigating to G ⊆ B returns at least ˜ γ t 0 − 1 / (1 − ˜ γ ) .
Revised: On the other hand, directly navigating to G ⊆ B returns at least ˜ γ t 0 − 1 / (1 − ˜ γ ) .

Original: For every 0 < ∆ < 1 , we can therefore ﬁnd a 0 < ˜ γ < 1 such that directly navigating to G ⊆ B is favorable under reward ˜ R ( s, a, s (cid:48) ) and discount ˜ γ as well.
Revised: For every 0 < ∆ < 1 , we can therefore ﬁnd a 0 < ˜ γ < 1 such that directly navigating to G ⊆ B is favorable under reward ˜ R ( s, a, s (cid:48) ) and discount ˜ γ as well.

Original: Once the state is within G , the optimal policy of M can be applied to keep the state within
Revised: Once the state is within G , the optimal policy of M can be applied to keep the state within

Original: G .
Revised: G .

Original: Again, this is feasible since M has the optimal ﬁnal volume G .
Revised: Again, this is feasible since M has the optimal ﬁnal volume G .

Original: Thus, ˜ M ⊆ M .
Revised: Thus, ˜ M ⊆ M .

Original: A. S TEP - WISE R EWARD F UNCTION
Revised: A. S TEP - WISE R EWARD F UNCTION

Original: We adopt notation from section 4.
Revised: We adopt notation from section 4.

Original: We discuss a possible relaxation of the lower bound on ˜ γ given in corollary 4.1.1 in case more assumptions are made about ˜ R .
Revised: We discuss a possible relaxation of the lower bound on ˜ γ given in corollary 4.1.1 in case more assumptions are made about ˜ R .

Original: Speciﬁcally, we consider the case of a step-wise reward function that the agent can follow monotonically.
Revised: Speciﬁcally, we consider the case of a step-wise reward function that the agent can follow monotonically.

Original: Deﬁnition A.1.
Revised: Deﬁnition A.1.

Original: Let a reward function ˜ R fulﬁll
Revised: Let a reward function ˜ R fulﬁll

Original: [Equation]
Revised: [Equation]

Original: where B ⊆ S is non-empty.
Revised: where B ⊆ S is non-empty.

Original: For w ∈ N \{ 0 } , we call an MDP M monotonically ( w, ∆) controllable with respect to ˜ R , iff there exists a policy π and 0 < ∆ < 1 such that
Revised: For w ∈ N \{ 0 } , we call an MDP M monotonically ( w, ∆) controllable with respect to ˜ R , iff there exists a policy π and 0 < ∆ < 1 such that

Original: [Equation]
Revised: [Equation]

Original: where actions and states follow the distribution
Revised: where actions and states follow the distribution

Original: [Equation]
Revised: [Equation]

Original: We call w M ’s step width and we call ∆ M ’s step height .
Revised: We call w M ’s step width and we call ∆ M ’s step height .

Original: In words, we additionally require that given the dynamics of M , the reward function must be deﬁned in a way that guarantees that the agent can advance to the next ”step” within w time steps.
Revised: In words, we additionally require that given the dynamics of M , the reward function must be deﬁned in a way that guarantees that the agent can advance to the next ”step” within w time steps.

Original: We acknowledge that constructing such a reward function might not always be feasible, but it allows us to illustrate the relationship between the lower bound on ˜ γ and the step width w .
Revised: We acknowledge that constructing such a reward function might not always be feasible, but it allows us to illustrate the relationship between the lower bound on ˜ γ and the step width w .

Original: The following theorem states this relationship.
Revised: The following theorem states this relationship.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: Let M be an MDP with metric state-space ( S , d ) and with the sparse reward function
Revised: Let M be an MDP with metric state-space ( S , d ) and with the sparse reward function

Original: [Equation]
Revised: [Equation]

Original: where B ⊆ S is non-empty.
Revised: where B ⊆ S is non-empty.

Original: Let M have the optimal ﬁnal volume G ⊆ B .
Revised: Let M have the optimal ﬁnal volume G ⊆ B .

Original: Let ˜ R
Revised: Let ˜ R

Original: ( s, a, s (cid:48) ) be a reward function that fulﬁlls
Revised: ( s, a, s (cid:48) ) be a reward function that fulﬁlls

Original: [Equation]
Revised: [Equation]

Original: where 0 < ∆ < 1 .
Revised: where 0 < ∆ < 1 .

Original: Additionally, let M be monotonically ( w, ∆) controllable with respect to ˜ R .
Revised: Additionally, let M be monotonically ( w, ∆) controllable with respect to ˜ R .

Original: Then, for every 0 < ∆ < 1 there exists a discount factor 0 < ˜ γ < 1 such that the MDP ˜ M corresponding to ˜ R and ˜ γ is compatible to M .
Revised: Then, for every 0 < ∆ < 1 there exists a discount factor 0 < ˜ γ < 1 such that the MDP ˜ M corresponding to ˜ R and ˜ γ is compatible to M .

Original: Proof.
Revised: Proof.

Original: This is a special case of theorem 4.1.
Revised: This is a special case of theorem 4.1.

Original: [Equation]
Revised: [Equation]

Original: Corollary A.1.1.
Revised: Corollary A.1.1.

Original: In this setting, ˜ M is compatible to M , if ˜ γ > ∆ 1 / ( w − 1) .
Revised: In this setting, ˜ M is compatible to M , if ˜ γ > ∆ 1 / ( w − 1) .

Original: Proof.
Revised: Proof.

Original: Assume that the agent is in an area where it can collect the reward ∆ k .
Revised: Assume that the agent is in an area where it can collect the reward ∆ k .

Original: The optimal agent has to decide if it advances to the next ”step” (where it can sustainably collect the reward ∆ k − 1 ) or if it stays in lower-reward areas.
Revised: The optimal agent has to decide if it advances to the next ”step” (where it can sustainably collect the reward ∆ k − 1 ) or if it stays in lower-reward areas.

Original: The return of the former is at least ∆ k − 1 ˜ γ w − 1 / (1 − ˜ γ ) , the return of the latter is at most ∆ k / (1 − ˜ γ ) .
Revised: The return of the former is at least ∆ k − 1 ˜ γ w − 1 / (1 − ˜ γ ) , the return of the latter is at most ∆ k / (1 − ˜ γ ) .

Original: Thus, the optimal agent will decide to steer towards the next step as long as ˜ γ > ∆ 1 / ( w − 1) .
Revised: Thus, the optimal agent will decide to steer towards the next step as long as ˜ γ > ∆ 1 / ( w − 1) .

Original: [Equation]
Revised: [Equation]

Original: Comparing this to corollary 4.1.1, we see that if w < t 0 , this relaxes the lower bound on ˜ γ .
Revised: Comparing this to corollary 4.1.1, we see that if w < t 0 , this relaxes the lower bound on ˜ γ .

Original: A notable special case is w = 1 , in which there exists a policy so that the reward is increased at every step.
Revised: A notable special case is w = 1 , in which there exists a policy so that the reward is increased at every step.

Original: In this case, any ˜ γ > 0 is sufﬁcient so ensure compatibility.
Revised: In this case, any ˜ γ > 0 is sufﬁcient so ensure compatibility.

Original: For reward functions that do not strictly adhere to deﬁnition A.1 but can be followed in a step-wise manner, the relation stated in theorem A.1 still qualitatively illustrates the following: The smaller the number of steps after which the agent can advance to a region of higher reward (i.e. the step width w ), the more relaxed the lower bound on ˜ γ becomes.
Revised: For reward functions that do not strictly adhere to deﬁnition A.1 but can be followed in a step-wise manner, the relation stated in theorem A.1 still qualitatively illustrates the following: The smaller the number of steps after which the agent can advance to a region of higher reward (i.e. the step width w ), the more relaxed the lower bound on ˜ γ becomes.

Original: In the important case that this step width becomes 1 , this lower bound is 0 .
Revised: In the important case that this step width becomes 1 , this lower bound is 0 .