Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards
Abstract
We study the problem of reward shaping to accelerate the training process of a reinforcement learning agent.
Existing works have considered a number of different reward shaping formulations; however, they either require external domain knowledge or fail in environments with extremely sparse rewards.
In this paper, we propose a novel framework, Exploration-Guided Reward Shaping (E XPLO RS), that operates in a fully self-supervised manner and can accelerate an agent’s learning even in sparse-reward environments.
The key idea of E XPLO RS is to learn an intrinsic reward function in combination with exploration-based bonuses to maximize the agent’s utility w.r.t.
extrinsic rewards.
We theoretically showcase the usefulness of our reward shaping framework in a special family of MDPs.
Experimental results on several environments with sparse/noisy reward signals demonstrate the effectiveness of E XPLO RS.
Introduction
Training reinforcement learning (RL) agents in environments with extremely sparse or distracting rewards is challenging.
Existing works have studied several approaches to design informative rewards that speed up the agent’s convergence [1–7].
One well-studied line of work is potential-based reward shaping, where a potential function is speciﬁed by an expert or obtained via transfer learning techniques (see [3, 8–17]).
Another popular approach is to learn rewards via Inverse-RL using expert demonstrations [18].
Alternatively, one could also consider a manual speciﬁcation of rewards, e g., using distance-based metrics [19].
However, these reward design techniques typically rely on high-quality domain knowledge and may fail in practice.
In fact, the RL agents can easily exploit poorly designed rewards and get stuck in local optima.
This naturally leads to the fundamental question of how to do online reward shaping without relying on expert domain knowledge.
More concretely, can we design informative rewards that will accelerate the agent’s training process by leveraging experience gained online during the agent’s training lifetime itself?
To tackle this question, recent works [24–26] have explored fully self-supervised learning of parametric intrinsic rewards that can improve the performance of RL agents.
In particular, these methods alternate between intrinsic reward parameter learning and the agent’s policy optimization w.r.t.
the learned reward.
For instance, Learning Intrinsic Rewards for Policy Gradient (LIRPG) technique [25] updates the intrinsic reward parameters to maximize the extrinsic rewards received by the policy from the environment.
Self-supervised Online Reward Shaping (SORS) technique [26] infers an intrinsic reward using a classiﬁcation-based reward inference algorithm, TREX [27].
However, these fully self-supervised reward shaping techniques might fail to produce meaningful agent behavior in environments with extremely sparse rewards (called hard-exploration domains) as they lack an 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
explicit explorative component.
Intuitively, these techniques will not be able to make updates to parameters of their intrinsic reward functions, without receiving a non-zero extrinsic reward signal.
In a parallel line of work, several techniques have been proposed to speciﬁcally tackle the challenges of extreme sparsity and exploration.
One such line of work is to add more stochasticity in the agent’s behavior (e.g., [28–30]); however such techniques typically succeed in tasks with already well-shaped rewards.
Another important line of work, relevant to our proposed framework, is bonusdriven exploration techniques for tackling hard-exploration domains – these techniques augment the extrinsic rewards with additional intrinsic bonus signals to encourage extra exploration [31].
A popular category of intrinsic bonuses is count-based bonuses that encourage RL agents to experience infrequently visited states [32–34].
Another category of intrinsic bonuses is providing rewards for improving the agent’s knowledge about the environment [35–40].
However, simply relying on these bonus-driven signals can mislead the agent towards sub-optimal or bad behaviors — for instance, in noisy-distractive domains such as the “noisy TV” problem [41], unpredictable random or noisy outputs would attract the agent’s attention forever.
An important research question that we seek to address is: How can we design an online intrinsic reward function, without any domain knowledge, that can speed up the agent’s learning process even in environments with extremely sparse rewards and noisy distractions?
To this end, we propose a novel framework, Exploration-Guided Reward Shaping (E XPLO RS), that learns an intrinsic reward function in combination with exploration-based bonuses to maximize the agent’s utility.
E XPLO RS operates in a fully self-supervised manner, and alternates between reward learning and policy optimization.
Our main results and contributions are:
I. We propose a novel reward shaping framework, E XPLO RS, that operates in a fully selfsupervised manner and can accelerate an agent’s learning even in sparse-reward environments.
(Section 3.1).
II.
We derive intuitive meta-gradients for updating the intrinsic reward component of E XPLO RS that enables our framework to be broadly applicable to any RL agent and not only policygradient based agents (Sections 3.2 and 3.3).
III.
We theoretically showcase the usefulness of our reward shaping framework in accelerating an agent’s learning in a special family of chain environments (Section 3.4).
IV.
We empirically demonstrate the effectiveness of E XPLO RS on several environments with sparse and noisy reward signals (Section 4).
Problem Setup
In Section 2.1, we present a general framework of online reward shaping technique for RL agents.
In
Section 2.2, we discuss the limitations of existing reward shaping techniques.
2.1 General Framework of Online Reward Shaping
Preliminaries.
An environment is deﬁned as a Markov Decision Process (MDP) M :
= ( S , A , T, P 0 , γ, R ) , where the state and action spaces are denoted by S and A respectively.
T : S × S ×
A → [0 , 1] captures the state transition dynamics, i.e., T ( s (cid:48) | s, a ) denotes the probability of landing in state s (cid:48) by taking action a from state s .
γ is the discounting factor, and Pis the initial state distribution.
The reward function is given by R : S ×
A → [ − R max , R max ] , for some R max > 0 .
We denote the true underlying extrinsic reward function by R and the designed reward function by (cid:98) R .
We denote a stochastic policy π : S → ∆ ( A ) as a mapping from a state to a probability distribution over actions, and a deterministic policy π : S
→ A as a mapping from a state to an action.
For any trajectory ξ = { ( s t , a t ) } t =0 , 1 ,...,H , we deﬁne its cumulative return w.r.t.
reward function R as J ( ξ, R ) :
= (cid:80) H t =
γ t · R ( s t , a t ) .
Then, the expected cumulative return (value) of a policy π w.r.t.
R is deﬁned as J ( π, R ) :
=
E [ J ( ξ, R ) | P 0 , T, π ] , where s 0 ∼ P 0 ( · ) , a t ∼ π ( ·| s t ) , and s t +1 ∼ T ( ·| s t , a t ) .
The learner seeks to ﬁnd a policy that has maximum value w.r.t.
the extrinsic reward function R , i.e., max π J ( π, R ) .
Online reward shaping.
A general framework of online reward shaping for RL agents is given in
Algorithm 1.
A natural objective here is to design informative rewards (cid:98) R k at each round k so that the resulting ﬁnal policy π K performs better (i.e., has high value w.r.t.
R )
compared to the corresponding policy obtained via the standard training with (cid:98) R k = R .
Note that we consider a single lifetime training setting for an RL agent on a single task, i.e., there is no resetting of the policy between rounds.
2.2 Existing Techniques and Issues
A popular technique for reward shaping is potential-based reward shaping (PBRS) which guarantees that any optimal policy induced by the designed reward function is also optimal under the extrinsic reward function [3].
However, for PBRS to be effective in accelerating the training process of an RL agent, we need to have access to good potential functions based on expert domain knowledge [42].
The focus of our work is on designing fully self-supervised reward shaping techniques.
Below, we provide a discussion of existing techniques that do not require any expert guidance or domain knowledge, and also discuss their limitations.
Reward shaping based on exploration bonuses.
In the bonus-driven exploration framework [32– 34], a count-based intrinsic bonus B k ( s ) is given to the agent to encourage exploration.
The bonus
B k ( s ) measures the “novelty” of a state s given the history of all transitions up to round k .
The authors in [34] extend the classic exploration methods with count-based intrinsic bonuses [43–46] to high-dimensional, continuous state spaces.
However, these “exploration-only” reward shaping techniques do not appropriately combine the successful extrinsic reward signals received from the environment.
When there are distractive zones in the state space, these methods will keep on exploring the state space even after obtaining extrinsic reward signals.
Fully self-supervised reward shaping: LIRPG [25].
Learning Intrinsic Rewards for Policy Gradient (LIRPG) technique [25] considers a parametric reward function of the form (cid:98) R LIRPG ( s, a )
=
R ( s, a ) + R φ ( s, a ) , and learns the parameter φ of the intrinsic reward function R φ in a fully self-supervised manner.
LIRPG alternates between learning the intrinsic reward parameter φ and the agent’s policy optimization w.r.t.
the learned reward (cid:98) R LIRPG .
At round k , for ﬁxed π k ,
LIRPG updates the parameter φ k − 1 to φ k by considering the effect such a change would have on the expected cumulative return (w.r.t.
R ) of the learner through the change in the policy π k , i.e., update φ using the gradient (cid:2) ∇ φ J ( L ( π k , (cid:98) R LIRPG ) , R ) (cid:3) φ k − 1 .
In order to develop an update rule for φ , LIRPG considers policy gradient style learning algorithm L with parametric policies (cid:8) π θ : θ ∈ R d θ (cid:9) .
More concretely, for a parameter θ k at round k s.t.
π k :=
π θ k , the learner’s policy update depends on φ as L ( π k , (cid:98) R LIRPG ) :
= π θ ( φ ) , where θ ( φ )
=
θ k
+ α · (cid:104) ∇ θ J
( π θ , (cid:98) R LIRPG ) (cid:105) θ k .
Based on this learner update, the LIRPG update for the intrinsic reward parameters, at round k , is based on the following meta-gradients:
[Equation]
1 , where η is the learning rate.
We note that the LIRPG technique could fail in environments with extremely sparse rewards as the agent may not receive a non-zero extrinsic reward signal needed to update the parameter φ .
Moreover, the LIRPG technique is applicable only to policy-gradient based RL agents.
Fully self-supervised reward shaping: SORS [26].
Self-supervised Online Reward Shaping (SORS) technique [26] considers a reward function of the form (cid:98) R SORS ( s, a ) =
R φ ( s, a ) , and infers the parameter φ using a classiﬁcation-based reward inference algorithm, T-REX [27].
However, unlike T-REX that requires rankings over the trajectories as input, SORS uses the extrinsic reward R as a self-supervised learning signal to rank the trajectories generated by the agent during training.
By de- sign, SORS only enforces the relative pairwise ordering over the trajectories w.r.t.
R when training R φ and ignores the scale of the returns associated with trajectories w.r.t.
R .
This makes training a policy challenging when the environment has noisy or distractive reward signals.
Further, similar to LIRPG, the SORS technique could fail in environments with extremely sparse rewards as the agent may not obtain any trajectories with non-zero extrinsic reward signal needed to update the parameter φ .
In this paper, we seek to develop an online reward shaping technique that can accelerate the agent’s training process in environments with extremely sparse and distractive rewards, without any expert domain knowledge.
As discussed above, techniques that rely only on intrinsic bonuses [32–34] could mislead the agent towards sub-optimal behaviors in noisy-distractive domains.
Similarly, the fully self-supervised reward shaping techniques (LIRPG and SORS) might be ineffective in environments with extremely sparse rewards.
We overcome these limitations by designing a novel reward shaping framework that appropriately balances exploration (via an intrinsic bonus component) and exploitation (via an intrinsic reward component) of extrinsic reward signals.
Exploration-Guided Reward Shaping
In Sections 3.1, 3.2, and 3.3, we propose an exploration-guided reward shaping framework, E XPLO RS, to accelerate an RL agent’s training process.
In Section 3.4, we theoretically showcase the usefulness of our framework in a chain environment.
3.1 Our Reward Formulation
We consider the following parametric reward function for E XPLO RS (see Algorithm 1):
[Equation]
where φ ∈ R d φ and w ∈ R d w .
Here, R S ELF RS φ corresponds to the intrinsic rewards in self-supervised reward shaping techniques, and B E XPLO B w corresponds to the intrinsic bonuses in exploration-only reward shaping techniques.
At round k of Algorithm 1, (cid:98) R E XPLO RS k −( s, a ) is designed with parameters ( φ k − 1 , w k − 1 ) .
Then, given updated policy π k , we update the parameters ( φ k − 1 , w k − 1 ) to ( φ k , w k ) .
Notation.
For the remainder of this section, we drop the superscripts (E XPLO RS, S ELF RS, and
E XPLO B) when referring to the reward functions in Eq.
In the subscript of the expectations E , let π ( a | s ) mean a ∼ π ( ·| s ) , µ π ( s, a ) mean s ∼ d π , a ∼ π ( ·| s ) , and µ π ( s ) mean s ∼ d π .
Further, we use shorthand notation µ ks,a and µ ks to refer µ π θk ( s, a ) and µ π θk ( s ) , respectively.
Intrinsic reward R φ .
We model the intrinsic reward R φ using any parameterized function.
At round k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k by considering the effect such a change would have on the the expected cumulative return w.r.t.
R through the change in the policy π k [24, 25].
In particular, we update φ using the gradient (cid:2) ∇ φ J ( L ( π k , (cid:98) R ) , R ) (cid:3) φ k − 1 , where (cid:98) R ( s, a )
=
R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
However, when considering L with neural policies, it is challenging to directly analyze the impact of φ in the policy π k .
Since our goal is to design a reward shaping technique that is applicable to any RL agent, we consider a simple surrogate learning algorithm (cid:101) L for our analysis.
In particular, we consider (cid:101) L with parametric policies (cid:8)
π θ : θ ∈ R d θ (cid:9) that does single-step vanilla policy gradient update with Q -values computed using h -depth planning.
We map the policy π k to a parameter θ k ∈ R d θ and deﬁne:
[Equation]
where α is the learning rate and Q π θk (cid:98) R,
h
[Equation]
s, a ) =
E (cid:104)(cid:80) ht
=
γ t (cid:98) R ( s t , a t ) (cid:12)(cid:12) s
= s, a 0 = a, T, π θ k (cid:105) is the h -depth Q -value w.r.t.
(cid:98) R .
Then, we update φ using the following bi-level optimization:
[Equation]
where (cid:98) R ( s, a ) :
= R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
In the above bi-level formulation, (cid:101) L with h -depth planning for small values of h essentially requires designing more informative intrinsic rewards to beneﬁt the agent’s training process [24].
Intrinsic bonus B w .
Given a state abstraction ψ : S → X ψ (with |X ψ | = d w ), we maintain the visitation count of the abstracted states in w , i.e., w [ x ] corresponds to the visitation counts of the states { s ∈ S : ψ ( s ) = x } .
This allows us to implicitly maintain pseudo-counts N w ( s ) of visiting states s ∈ S .
In particular, we set N w ( s ) =
(cid:0) λB max (cid:1) 2 + w [ ψ ( s )] for some B max , λ > 0 .
Then, we deﬁne the intrinsic bonus as follows:
B w ( s ) =
λ √ N w ( s ) .
We update w based on the rollouts in round k [32–34].
3.2 Derivation of Gradient Updates for R φ
In this subsection, we ﬁrst obtain high-level meta-gradient updates for R φ similar to LIRPG [25].
Then, we derive intuitive meta-gradient updates that would allow E XPLO RS to be compatible with any RL agent.
High-level gradient updates for R φ .
We solve the bi-level optimization problem (P1.U)-(P1.L) of the intrinsic reward component in an iterative manner using the gradient updates that we derive below.
At round k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k as follows:
[Equation]
where η is the learning rate, the equality in ( a ) is due to chain rule, and the approximation in ( b ) is made by assuming a smoothness condition of (cid:13)(cid:13)(cid:13)(cid:2) ∇ θ J ( π θ , R ) (cid:3) θ ( φ k − 1 ) − (cid:2) ∇ θ J ( π θ , R ) (cid:3) θ k (cid:13)(cid:13)(cid:13) 2 ≤ c · (cid:107) θ ( φ k − 1 ) − θ k (cid:107) 2 for some c > 0 .
By using the meta-gradient derivations in [47–49], we write the term 1 (cid:13) as follows: [ ∇ φ
[Equation]
φ )]
[Equation]
k (cid:105) , where (cid:98) R ( s, a ) :=
R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
By using the policy gradient theorem [50], we write the term 2 (cid:13) as follows:
[Equation]
The above gradient update of φ k , involving the terms 1 (cid:13) and 2 (cid:13) , resembles the LIRPG [25] update.
However, both the terms 1 (cid:13) and 2 (cid:13) require computing the gradient of the policy, i.e., ∇ θ log π θ ( a | s ) .
This requirement makes the above update applicable only for policy-gradient based agents.
Below, we derive intuitive simpliﬁcations of the above two terms, 1 (cid:13) and 2 (cid:13) , that would enable our technique to be applicable to any RL agent, and not only policy-gradient based agents.
Intuitive gradient updates for R φ .
In order to obtain intuitive forms of the terms 1 (cid:13) and 2 (cid:13) , we consider further simpliﬁcations to the surrogate learning algorithm (cid:101) L introduced in Section 3.1.
In particular, for our analysis and derivation, we let (cid:101)
L use tabular representation θ ∈ R |S|·|A| and softmax policy given by π θ ( a | s ) :
=
[Equation]
, ∀ s ∈ S , a ∈ A .
We deﬁne A
[Equation]
s
[Equation]
:=
Q π θk (cid:98) R,h ( s, a ) − V π θk (cid:98) R,h ( s ) and A π θk R ( s, a ) :
=
Q π θk R ( s, a ) − V π θk R ( s ) .
Based on this, the following proposition provides intuitive gradient updates for R φ .
Proposition 1.
For the simpliﬁed surrogate learning algorithm (cid:101) L with h -depth planning, the gradient term [ ∇ φ θ ( φ )] φ k − 1 · (cid:2) ∇ θ J ( π θ , R ) (cid:3) θ k in Eq.
(2) takes the following form:
[Equation]
For the special case of h =
[Equation]
the gradient term further simpliﬁes to the following form:
[Equation]
Compared to Eq.
(2), the intuitive gradient update term in the above proposition does not require computing the policy gradient ∇ θ log π θ ( a | s ) .
This allows us to develop an update rule for intrinsic reward parameter φ that is applicable to any RL agent.
In particular, given the current policy π k (possibly without any differentiable parameterization), we simplify Eq.
(2) and propose the following gradient update rule for parameter φ :
[Equation]
where η (cid:48)
=
η · α .
Note that the above gradient update only requires black-box access to the policy π k in the form of trajectory rollouts as in the SORS technique [26].
3.3 Empirical Updates and Practical Aspects
In this subsection, we present a concrete pseudocode for training an RL agent with E XPLO RS reward shaping technique.
Algorithm 2 provides a sketch of the overall training process, interleaving the agent’s training with E XPLO RS.
The sketch presented in Algorithm 2 is adapted from the training process proposed for the SORS technique [26].
Further, we consider rollouts where each round corresponds to a single rollout, instead of environment steps, as in SORS.
Below, we discuss the empirical updates for intrinsic reward and bonus components of Eq.
Empirical updates for intrinsic reward R φ .
We translate the ﬁnal expectation-based update of φ k in Eq.
(3) to its empirical counterpart using the rollout data D collected by executing the current policy π k (or recent policies) in the MDP M .
At any round k , let D contain a collection of trajectories (cid:8) ξ i (cid:9) n i =1 , where ξ i =
( s i 0 , a i 0 , s i 1 , a i 1 , . . .
, s iH ) .
For a given trajectory ξ i and time index t , we denote a partial trajectory as ξ it =
( s it , a it , . . . , s iH ) .
Based on this notation, we empirically update the parameter φ as follows:
[Equation]
where we absorb the normalization factors into η φk , ignore the term µ π k ( s it ) , and set (cid:104) ∇
[Equation]
Similar to LIRPG [25]
[Equation]
we also maintain a critic V R, (cid:101) φ k − 1 ( · ) to approximate V π k R ( · ) in Eq.
We update the parameters of the critic, (cid:101) φ k − 1 to (cid:101) φ k , using the same rollout data D and learning rate η (cid:101) φk .
In Algorithm 2, hyperparameters N r and N π control the frequency of updates for the intrinsic reward R φ and policy π , respectively.
For stability reasons, we update the policy more frequently compared to the intrinsic reward, i.e., N π < N r .
We provide full implementation details in Section 4 and appendices.
Empirical updates for intrinsic bonus B w .
We update B w based on the history of all the states visited up to round k .
Similar to #Exploration [34], we use the count-based intrinsic bonuses with a state abstraction ψ : S → X ψ .
We maintain the visitation count of the abstracted states in w .
For each rollout ξ k , we update the parameter w of the intrinsic bonus as follows:
[Equation]
Similar to the existing count-based exploration techniques [32–34], we use a lookahead step when incorporating the bonus term (see line 5 in Algorithm 2).
In our implementation, we update the intrinsic bonus at a more ﬁne-grained level, i.e., we update B w at each environment step t within each round k directly, instead of waiting for the rollout to ﬁnish.
However, for clear presentation in
Algorithm 2, we write the B w update at the level of round k , not at the level of environment step t .
We provide full implementation details in Section 4 and appendices.
3.4 Theoretical Analysis
In this subsection, we theoretically showcase the usefulness of our exploration-guided reward shaping framework in accelerating an agent’s learning in a chain environment with extremely-sparse rewards and distractive zones in the state space.
Our analysis considers a stylized learning setting with simpliﬁed versions of different reward shaping techniques.
do update 4: if k % π
Deﬁne reward (cid:98) R k − 1 ( s, a, s (cid:48) ) :
= R ( s, a ) + R φ k − 1 ( s, a ) + B w k − 1 ( s (cid:48) ) 6:
Obtain updated policy π k ← L ( π k − 1 , (cid:98) R k − using the latest rollouts in D 7: else 8:
Keep previous policy π k ← π k −// data collection 9:
Rollout the policy π k in the MDP M to obtain a trajectory ξ k = (cid:0) s k 0 , a k 0 , s k 1 , a k 1 , . . .
, s kH (cid:1) 10:
Store ξ k in the buffer D . add ( ξ the
D is full, remove the oldest trajectory // intrinsic reward update 11: if k % N r =
then 12:
Obtain updated reward parameter φ k from φ k − 1 as in Eq.
(4) using D and learning rate η φk 13:
Obtain updated critic parameter (cid:101) φ k from (cid:101) φ k − 1 using D and learning rate η (cid:101) φk 14: else 15:
Keep previous parameters
φ k ← φ k − 1 and (cid:101) φ k ← (cid:101) φ k −// intrinsic bonus update 16:
Update k 17:
Deﬁne bonus B w k ( s ) =
λ √ N wk ( s ) , where N w k ( s ) = (cid:0) λB max (cid:1)
+ w k
[ ψ ( s )] 18: Output: Policy π K
Chain environment.
We consider a chain environment M = (cid:0) S , A , T, P 0 , γ, R (cid:1) of length n 1 + n 2 + 1 .
Let the state space be S
= { x − n 2 , . .
, x − 1 , x 0 , x 1 , . . .
, x n 1 } , and the action space be A = {← , →} .
We always start in the state x 0 , i.e., the initial state distribution is
P 0 ( x 0 )
= 1 .
The transition dynamics is deterministic and given as follows: T ( x i +1 | x i , → )
=for − n 2 ≤ i ≤ n 1 − 1 , T ( x i − 1 | x i , ← )
= 1 for − ( n 2 − 1) ≤ i ≤ n 1 , T ( terminal | x n 1 , → ) = 1 , and T ( terminal | x − n 2 , ← )
= 1 .
The reward function is deﬁned as follows:
R ( x i , → ) = 0 for − n 2 ≤ i ≤ n 1 − 1 , R ( x n 1 , → ) = 1 , and R ( x i , ← ) = 0 for − n 2 ≤ i ≤ n 1 .
We consider an inﬁnite horizon setting with discounted returns, i.e., H → ∞ and γ < 1 .
Learning algorithm and reward shaping techniques.
For our theoretical analysis, we consider a stylized learning setting with a TD-style RL algorithm L and simpliﬁed versions of different reward shaping techniques; details are provided in appendices.
We analyze the total number time steps required for L to learn an optimal policy in the chain environment under four different settings: (i)
Case L ( S ELF RS = 0 , E XPLO B = 0) is a default setting without any shaping; (ii) Case L ( S ELF RS = 0 , E XPLO B = 1) uses only the intrinsic bonuses; (iii) Case L ( S ELF RS = 1 , E XPLO B = 0) uses only the intrinsic rewards; (iv) Case L ( S ELF RS = 1 , E XPLO B = 1) combines intrinsic bonuses with intrinsic rewards.
The following theorem compares these four settings and showcases the usefulness of our framework, i.e., Case L ( S ELF RS = 1 , E XPLO B = 1) – proof is provided in appendices.
Theorem 1.
Consider the chain environment M and the algorithm L deﬁned above.
Let cost ( L ( S ELF RS , E XPLO B )) denote the total number time steps required for L ( S ELF RS , E XPLO B ) to learn an optimal policy in M .
Then, we have the following (expected) costs for the four settings:
(i) E [ cost ( L ( S ELF RS = 0 , E XPLO B = 0))] ≥ 2 n 1 − 1 ; (ii) cost ( L ( S ELF RS = 0 , E XPLO B = 1))
=
n 1 · ( n 1 + n 2 + 2) ; (iii) E [ cost ( L ( S ELF RS = 1 ,
E XPLO B = 0))] ≥ 2 n 1 − 1 ; (iv) cost ( L ( S ELF RS = 1 , E XPLO B = 1)) ≤ n 1 + n 2 +
The proof and additional details about the learning setting are provided in appendices.
Experimental Evaluation
In this section, we evaluate our reward shaping framework on three environments: C HAIN (Section 4.1), R OOM (Section 4.2), and L INE K (Section 4.3).
C HAIN corresponds to a navigation task in a chain, adapted from the environment used for theoretical analysis in Section 3.4; this is a canonical environment used for studying extremely sparse-reward settings [7].
R OOM corresponds to a navigation task in a grid-world where the agent has to learn a policy to quickly reach the goal location in one of four rooms, starting from an initial location.
Even though this environment has a small state/action space, it provides a very rich and intuitive problem setting to validate different reward shaping techniques.
In fact, variants of R OOM have been used extensively in the literature [10, 11, 14, 17, 51–54]—the environment used in our experiments is adapted from [54].
L INE K corresponds to a navigation task in a one-dimensional space where the agent has to ﬁrst pick the correct key and then reach the goal.
The agent’s location is represented as a point on a line segment.
This environment is inspired by variants of navigation tasks in the literature where an agent needs to perform subtasks [3, 54, 55]—the environment used in our experiments is adapted from [54].
We give an overview of main results here, and provide a more detailed description of the setup and additional implementation details in appendices.
4.1 Evaluation on C HAIN
C HAIN (Figure 1).
We represent the chain environment of length n 1 + n 2 + 1 as an MDP with state-space S consisting of an initial location x 0 (shown as “blue-circle”), n 1 nodes to the right of x 0 , and n 2 nodes to the left of x 0 .
The rightmost node of the chain is the “goal” state (shown as “green-star”).
In the left part of the chain, there can be a “distractor” state (shown as “green-plus”).
The agent can take two actions given by
A := { “left” , “right” } –
Evaluation setup.
We conduct our experiments with two different types of RL agents for C HAIN : tabular REINFORCE agent [7] and tabular Q-learning agent [7].
Algorithm 2 provides a sketch of the overall training process, and shows how agent’s training interleaves with reward shaping techniques.
We compare the performance of the following reward shaping techniques: (i) (cid:98) R O RIG :=
R is a default baseline without any shaping; (ii) (cid:98) R SORS’ :
= R + R SORS φ is based on the SORS technique [26] (see
Section 2.2); 2 (iii) (cid:98) R LIRPG’ is obtained via adapting the LIRPG technique [25] to our training pipeline (see Algorithm 2, Sections 2.2 and 3.2)—note that (cid:98) R LIRPG’ is not applicable to Q-learning agent; 3 (iv) (cid:98) R E XPLO B := R + B E XPLO B w uses only the intrinsic bonuses; (v) (cid:98) R S ELF RS :
= R + R S ELF RS φ uses only the intrinsic rewards; (vi) (cid:98) R E XPLO RS :
= R + R S ELF RS φ + B E XPLO B w combines intrinsic bonuses with intrinsic rewards.
We provide full details about the implementation and hyperparameters in appendices.
Results.
During training, the agent receives rewards based on (cid:98) R and is evaluated based on R .
Figureshows results for both the variants of C HAIN environment; the reported results are averaged over 20 runs and convergence plots show the mean performance with standard error bars.
These results demonstrate the effectiveness of our exploration-guided reward shaping framework ( (cid:98) R E XPLO RS ), in comparison to baselines ( (cid:98) R O RIG , (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R E XPLO B , (cid:98) R S ELF RS ).
Next, we summarize some of our key ﬁndings.
First, our results show that (cid:98) R E XPLO RS outperforms the baselines in both C HAINand C HAIN + environments, irrespective of the RL agent (REINFORCE and Q-learning).
Second, the
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(a) C HAIN 0 , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(b) C HAIN + , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG (c) C HAIN 0 , Q-learning 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
[Figure]
learning performance of (cid:98) R E XPLO RS is better than variants which only use either intrinsic bonuses or intrinsic rewards, i.e., (cid:98) R E XPLO B or (cid:98) R S ELF RS – this demonstrates the utility of combining these two signals.
Third, results in Figures 2b and 2d show that three reward shaping techniques ( (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS ) could fail or lead to sub-optimal policies because of the presence of distractor states.
4.2 Evaluation on R OOM
R OOM (Figure 3).
This environment is based on the work of [54]; however, we adapted it to have a “distractor” state (shown as “greenplus”) that provides a small reward.
Similar to the two variants of
C HAIN , we have two variants of this environment: (i) R OOM 0 has
R dis =
at the distractor state shown as “green-plus” (equivalently, there is no distractor state); (ii) R OOM + has R dis = 0 .
01 at the distractor state.
The environment-speciﬁc parameters (including p rand , R max , γ ) are kept same as in Section 4.1.
We defer full details to appendices.
Evaluation setup and results.
Our evaluation setup for this environment is exactly same as that used for C HAIN environment (described in Section 4.1); here, we consider only the tabular REINFORCE agent.
In particular, all the hyperparameters (related to the REINFORCE agent, reward shaping techniques, and training process) are the same as in Section 4.1.
Figures 5a and 5b show the agent’s performance for environments R OOM 0 and R OOM + (averaged over 20 runs).
These results, along with results obtained in Figure 2, further demonstrate the effectiveness and robustness of (cid:98) R E XPLO RS across different environments in comparison to baselines.
4.3 Evaluation on L INE K
L INE K (Figure 4).
This environment corresponds to a navigation task in a one-dimensional space where the agent has to ﬁrst pick the correct key and then reach the goal.
The environment used in our experiments is based on the work of [54]; however, we adapted it to have multiple keys (only one being correct) and “distractor” states that provide a small reward at goal locations even without the correct key.
The environment comprises of the following main elements: (a) an agent whose current location (shown as “blue-circle”) is a point x
Similar to Sections 4.1 and 4.
we use the environment
(i) L
K 0 with ( k ii
+
We defer full details to appendices.
Experimental setup.
We conduct our experiments with a neural REINFORCE agent using a twolayered neural network architecture (i.e., one fully connected hidden layer with 256 nodes and RELU activation) [7].
Similar to Section 4.1, we compare the performance of six techniques.
As a crucial
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(a) R OOM 0 , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(b)
R OOM + , REINFORCE     5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(c) L INE K 0 , REINFORCE     5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(d)
L INE K + , REINFORCE difference, here we use neural-network based reward functions for (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS , and (cid:98) R E XPLO RS (see Footnotes 2 and 3).
Based on [25, 26], we use the same neural-network architecture for intrinsic reward functions as used for the agent’s policy by applying appropriate transformations at the output layer (e.g., instead of using soft-max, use tanh -clipping to get output reward values for actions).
We provide full details about the implementation and hyperparameters in appendices.
Results.
During training, the agent receives rewards based on (cid:98) R and is evaluated based on R .
Figures 5c and 5d show results for both the variants of L INE K environment; the reported results are averaged over 30 runs and convergence plots show the mean performance with standard error bars.
These plots showcase the performance of different techniques as we vary R dis ∈ { 0 . 00 , 0 . 01 } – this in turn decides whether there are any distractor states that can serve as local minima for the agent.
The convergence behavior in Figures 5c and 5d demonstrates the effectiveness of our explorationguided reward shaping framework ( (cid:98) R E XPLO RS ), in comparison to baselines ( (cid:98) R O RIG , (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R E XPLO B , (cid:98) R S ELF RS ).
Next, we summarize some of our key ﬁndings.
First, our results show that (cid:98) R E XPLO RS outperforms all the baselines in both L INE K 0 and L INE K + environments.
Second, results in Figure 5d show that three reward shaping techniques ( (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS ) performed worse than (cid:98) R O RIG – this is because of the presence of distractor states which create local minima for the agent and these shaped functions could further encourage learning a sub-optimal policy.
In contrast, (cid:98) R E XPLO RS combines the beneﬁts of intrinsic rewards ( (cid:98) R S ELF RS ) and intrinsic bonuses ( (cid:98) R E XPLO B ) to speed up agent’s learning in a robust and efﬁcient manner.
Overall, these results demonstrate that our shaping technique (cid:98) R E XPLO RS results in efﬁcient learning even when dealing with complex state representations and when learning neural-network based intrinsic reward functions.
Concluding Discussions
We proposed a novel reward shaping framework, E XPLO RS, that operates in a fully self-supervised manner and could accelerate an agent’s learning even in sparse-reward environments.
Next, we discuss a few limitations of our work and outline a future plan to address them.
First, the experimental evaluation is conducted on simpler environments to study the performance of techniques w.r.t.
the three characteristics of (a) hard exploration, (b) local minima, and (c) “noisy TV” problem.
It would be interesting to evaluate different reward design techniques in more complex environments (e.g., with continuous state/action spaces); this would also require designing benchmark environments that systematically capture the above three characteristics.
Second, E XPLO RS combines the intrinsic rewards and intrinsic bonuses that allows it to overcome the limitations of state-of-the-art techniques.
It would be interesting to develop more principled ways to combine these two signals.
Third, it would be useful to provide rigorous analysis of E XPLO RS in terms of convergence speed and stability of an agent.
Acknowledgments
This work was supported in part by the European Research Council (ERC) under the Horizon Europe programme (ERC StG, grant agreement No. 101039090).
Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute.
A Table of Contents
In this section, we give a brief description of the content provided in the appendices of the paper.
• Appendix B provides derivations for the intuitive gradient updates for R φ .
(Section 3.2) • Appendix C provides proof for the theoretical analysis.
(Section 3.4) • Appendix D provides additional details for C HAIN .
(Section 4.1) • Appendix E provides additional details for R OOM .
(Section 4.2) • Appendix F provides additional details for L INE K. (Section 4.3)
B Derivation of Gradient Updates for R φ : Proof (Section 3.2)
Proof of Proposition 1.
For any s ∈ S
[Equation]
a ∈ A
[Equation]
let 1 s
[Equation]
a ∈ R |S|·|A| denote a vector with 1 in the ( s, a ) -th entry and 0 elsewhere.
First
[Equation]
we simplify the term 1 (cid:13) as follows:
[Equation]
[Table]
[Equation]
Then
[Equation]
we simplify the term 2 (cid:13) as follows:
[Equation]
[Table]
[Equation]
Finally
[Equation]
we consider the following:
[Equation]
[Table]
C Theoretical Analysis: Proof (Section 3.4)
Proof of Theorem 1.
We prove Theorem 1 via case-by-case analysis.
Case L ( S ELF RS = 0 , E XPLO B = 0) .
This case corresponds to learning without any reward shaping, i.e., learning with the extrinsic reward only: R ( s, a ) .
Then, we note the following:
I. Initially, we have a random policy except at state x n 1 , where we take the optimal action → (line 7).
We maintain zero value function V t for all the states (line 9) until we obtain the ﬁrst success complete rollout, i.e., s t +1 is terminal and R ( s t , a t )
= 1 . II.
With an initial random policy and starting from x 0 , probability of obtaining a success complete rollout is (cid:0) 12 (cid:1) n 1 + (cid:0) 12 (cid:1) n 1 +2 + (cid:0) 12 (cid:1) n 1 +4 + . . . , which is upper bounded by p max
= (cid:80) ∞
i =0 (cid:0) 12 (cid:1) n 1 + i = (cid:0) 12 (cid:1) n 1 − 1 .
III.
Let E [ T 1 ] be the expected number of steps required for the ﬁrst occurrence of the above successful rollout.
Then, we have: E [ T 1 ] ≥ 1 p max = 2 n 1 − 1 .
IV.
After the ﬁrst successful rollout, we will have V t ( x n 1 ) = 1 and zero elsewhere (line 9).
Then, we will have a random policy except at x n 1 and x ( n 1 − 1) , where we take the optimal action (line 7).
This effectively repeats the same steps above for the chain without x n 1 .
V. Let E [ T 2 ] be the expected number of steps required for the second occurrence of the above successful rollout.
Then, we have: E [ T 2 ] ≥
1: Input: Binary ﬂags S ELF RS and E XPLO B 2: Initialize: V 0 ( s ) = 0 ; R ( s, a ) = 0 , B ( s )
= 1 , ∀ s ∈ S , a ∈ A ; λ ∈ (0 , 1) 3: s 1 = x 0 ; B ( s 1 ) =
λ 4: for each t = 1 , 2 , . . .
do 5: if
E XPLO B = 0 then 6:
B ( s ) = 0 , ∀ s ∈ S // bonus component used for action selection 7: a t = arg max a (cid:48)
R ( s t , a (cid:48) )
+ R ( s t , a (cid:48) )
+ B ( T ( s t , a (cid:48) ))
+ γ · V t − 1 ( T ( s t , a (cid:48) )) 8: s t
+
=
T ( s t , a t ) //
we do not consider the bonus component when updating the value function 9:
V t ( s t ) =
R ( s t , a t ) + R ( s t , a t ) + γ · V t − 1 ( s t +1 ) 10: if s t +1 = terminal then 11: if R ( s t , a t ) = 1 and S ELF RS = 1 then // update the intrinsic reward component 12: φ ( s ) = 0 , ∀ s ∈ S 13:
Update φ ( s ) for all the states in the current rollout as the discounted return 14:
R ( s, a ) = γ · φ ( T ( s, a )) − φ ( s ) , ∀ s ∈ S , a ∈ A // reset the value function to account for change in R 15:
V t ( s ) = 0 , ∀ s ∈ S 16: reset s t +1 =
x// update the bonus component 17:
B ( s t +1 )
=
λ · B ( s t +1 ) 18: Output: policy π t
VI.
After the second successful rollout, we will have V t ( x n 1 )
= 1 , V t ( x ( n 1 − 1) )
=
γ , and zero elsewhere (line 9).
Then, we will have a random policy except at x n 1 , x ( n 1 − 1) , and x ( n 1 − 2) , where we take the optimal action (line 7).
This effectively repeats the same steps above for the chain without x n 1 and x ( n 1 − 1) .
VII.
After following the above procedure for n 1 success rollouts, we will have the optimal value/policy learnt for the chain (solving the MDP).
Thus, the expected sample complexity is lower bounded by E [ cost ( L ( S ELF RS = 0 , E XPLO B = 0))]
= (cid:80) n
i
=
1 E [ T i ] ≥ (cid:80) n 1 i =1 2 n 1 − i .
Case L ( S ELF RS = 0 , E XPLO B = 1)
This case corresponds to learning with the extrinsic reward and intrinsic bonus: R ( s, a ) + B ( T ( s, a )) .
Then, we note the following (here, we need λ ≤ γ ):
I. We have zero value function (line 9) until we get the ﬁrst success complete rollout, i.e., s t +1 is terminal and R ( s t , a t )
= 1 .
II.
W.l.o.g.
we take → action at time t = 1 at x 0 .
Then, we continue to take → action (for n 1 +steps) until we reach rightmost terminal state, since λ < 1 (lines 7 and 17).
III.
After the ﬁrst successful rollout, we will have V t ( x n 1 ) = 1 and zero elsewhere (line 9).
Note that V t ( terminal ) = 0 , ∀ t .
IV.
Once we reset to x 0 , we take ← since λ < 1 (line 7).
Then, we continue to take ← action (for n 2 + 1 steps) until we reach leftmost terminal state, since λ < 1 (lines 7 and 17).
V. This alternating one-sided navigation process will continue until V t values are updated for all the nodes right to x 0 (one node at a time per one full cycle).
The condition λ ≤ γ ensures that after all the nodes right to x 0 get updated with right V t values, there will be no further exploration on the left-side of x 0 .
Thus, the sample complexity is given by cost ( L ( S ELF RS = 0 , E XPLO B = 1))
=
n 1 · ( n 1 + n 2 + 2) .
Case L ( S ELF RS = 1 , E XPLO B = 0)
This case corresponds to learning with the extrinsic reward and intrinsic reward : R ( s, a ) + R ( s, a ) .
Then, we note the following:
I. From the analysis for the case L (1 , 1) , we have: E [ T 1 ] ≥ 1 p max = 2 n 1 − 1 .
II.
However, after the ﬁrst successful rollout, we obtain the optimal policy (line 7) immediately since the shaping reward (line 14) contains myopic-optimality information.
Thus, the expected sample complexity is lower bounded by E [ cost ( L ( S ELF RS = 1 , E XPLO B = 0))]
=
E [ T 1 ] ≥ 2 n 1 − 1 .
Case L ( S ELF RS = 1 , E XPLO B = 1)
This case corresponds to learning with the extrinsic reward and intrinsic reward and bonus: R ( s, a ) + R ( s, a ) + B ( T ( s, a )) .
Then, we note the following (here, we need λ 2 ≤ γ n 1 ):
I. From the analysis for the case L (1 , 0) , we obtain ﬁrst successful trajectory after n 1 + n 2 +steps (utmost).
Then, as in the case of L (0 , 1) , shaping reward (line 14) will propagate myopicoptimality information immediately.
The condition λ 2 ≤ γ n 1 ensures that after all the nodes right to x 0 get updated with right V t values, there will be no further exploration on the left-side of x 0 .
Thus, the sample complexity is upper bounded by cost ( L ( S ELF RS = 1 , E XPLO B = 1)) ≤ n 1 + n 2 + 2 .
D Evaluation on C HAIN : Additional Details (Section 4.1)
C HAIN (Figure 1).
We expand on the details of the C HAIN environment, introduced in Section 4.1.
We represent the chain environment of length n 1 + n 2 + 1 as an MDP with state-space S consisting of an initial location x 0 (shown as “blue-circle”), n 1 nodes to the right of x 0 , and n 2 nodes to the left of x 0 .
The rightmost node of the chain is the “goal” state (shown as “green-star”).
In the left part of the chain, there can be a “distractor” state (shown as “green-plus”)
Evaluation setup: agents.
As mentioned in Section 4. we conduct our experiments with two different types of RL agents for C HAIN : tabular REINFORCE agent [7] and tabular Q-learning agent [7].
First, we consider tabular REINFORCE agent that maintain scores θ [ s, a ] for each stateaction pair and applies soft-max operation over the scores to obtain the policy π .
When computing the agent’s performance during evaluation, we also use the agent’s soft-max policy (instead of choosing actions greedily).
Second, we consider tabular Q-learning agent with exploration factor (cid:15) = 0 .
When computing the agent’s performance during evaluation, we also use the agent’s (cid:15) -greedy policy (instead of choosing actions greedily).
Algorithm 2 provides a sketch of the overall training process, and shows how agent’s training interleaves with reward shaping techniques – the agent’s policy is updated in lines 4 – 8 of the algorithm.
For the agent’s training process, we use a ﬁxed set of hyperparameters irrespective of the type of agent or the reward shaping technique.
More concretely, we have the following: (a) the agent’s learning rate is set to 0 .
1 ; (b) frequency of updates N π is set to be 2 , i.e., update after every 2 rollouts in the environment; (c) a rollout buffer (ﬁrst-in-ﬁrst-out) D of size 10 is maintained and we update the agent’s policy using the last 5 rollouts in D .
In the tabular setting with C HAIN , we ﬁnd that the overall quantitative results are robust to these hyperparameters – we use the exact same set of hyperparameters for evaluation on R OOM , described in Section 4.2.
Evaluation setup: shaping techniques.
Next, we describe different reward shaping techniques used during the agent’s training phase.
Speciﬁcally, during training, the agent receives rewards based on the shaped reward function (cid:98) R ; the performance (as reported in the plots) is always evaluated w.r.t.
the extrinsic reward function R .
More concretely, we have the following shaping techniques:
• (cid:98) R O RIG :=
R .
This serves as a default baseline where extrinsic reward function is used during training without any shaping.
• (cid:98) R SORS’ :
= R + R SORS φ .
This is based on the SORS technique[26]; see additional details in Section 2.2 (also see Footnote 2 about (cid:98) R SORS’ ).
For C HAIN environment, we use tabular representation for R SORS φ and perform gradient updates as described in the work of [26].
Algorithmprovides a sketch of the overall training process – the R SORS φ updates would be applied in lines 11 – 15 in the algorithm.
In fact, the training process presented in Algorithm 2 is adapted from the training process proposed for the SORS technique [26].
We update the intrinsic reward function using the following hyperparameters: (a) the learning rate is set to 0 .
01 ; (b) frequency of updates N r is set to be 5 , i
e., update after every 5 rollouts in the environment; (c) we have a rollout buffer D of size 10 and sample a set of 10 pairs of rollouts for the gradient updates (in our implementation, we prioritized sampling of pairs that have non-zero gap between returns).
• (cid:98) R LIRPG’ :
= R + R LIRPG’ φ .
This is obtained via adapting the LIRPG technique of [ to our training pipeline; see Algorithm 2, Sections 2.2 and 3.2 (also see Footnote 3 about (cid:98) R LIRPG’ ).
More speciﬁcally, when considering tabular REINFORCE agent, we implemented (cid:98) R LIRPG’ as an adaptation of (cid:98) R S ELF RS where we
set h → ∞ instead of see Section 3.2) – the rest of the
implementation
is
same as
described below for (cid:98) R S ELF RS .
Note that the LIRPG technique is not applicable to Q-learning agent.
• (cid:98) R E XPLO B := R + B E XPLO B w .
This corresponds to a part of our reward shaping technique which uses only the intrinsic bonuses B E XPLO B w .
As discussed in Sections 3.1 and 3.3, we use a count-based bonus B E XPLO B w .
For C HAIN environment, we use a tabular representation for B E XPLO B w where w [ s ] captures the state-visitation counts for a state s .
Algorithm 2 provides a sketch of the overall training process – the B E XPLO B w updates are applied in lines 16 – 17 in the algorithm.
We set the hyperparameters B max and λ to be same as R max ( = 1 . 0 for C HAIN ).•
(cid:98) R S ELF RS :
= R + R S ELF RS φ .
This corresponds to a part of our reward shaping technique which uses only the intrinsic rewards R S ELF RS φ .
For C HAIN environment, we use a tabular representation for R S ELF RS φ
where φ [ s, a ] reward values are learned for each state-action pair and R S ELF RS φ ( s, a ) :
=
φ [ s, a ] ∀ ( s, a ) .
Along with R S ELF RS φ , a tabular value-function V R, (cid:101) φ is maintained w.r.t.
R , serving as critic to compute values V π k R ( s ) as needed for the empirical updates (see Section 3.3).
For updating V R, (cid:101) φ , we use Monte Carlo updates based on the trajectory returns as target and using a (cid:96) 2 -norm loss function [7].
Algorithm 2 provides a sketch of the overall training process – the
R S ELF RS φ updates are applied in lines 11 – 15 in the algorithm.
We set the following values for hyperparameters: (a) learning rate for updating φ parameters is set to 0 .
01 ; (b) learning rate for updating (cid:101) φ parameters is set to
c) frequency of updates N r is set to be 5 ,
i
e update after every 5 rollouts in the environment; (d) we have a rollout buffer D of size 10 .
Furthermore, in all our experiments with Q-learning agent, we clipped the values of φ in the range [ − 0 .
(see Section 4.3 and Appendix F for another variant of clipping used with neural agents).
• (cid:98) R E XPLO RS := R + R S ELF RS φ + B E XPLO B w .
This is our exploration-guided reward shaping technique that combines intrinsic bonuses with intrinsic rewards.
Algorithm 2 provides a sketch of the overall training process; we update R S ELF RS φ and B E XPLO B w in the same way as described in the previous two points above.
Note that, for stability, we update the policy more frequently than the intrinsic reward ( N π = 2 vs.
N r = 5 ) and at a higher learning rate ( 0 . 1 vs. 0 . 01 ), as considered in the work of [25, 26].
In the tabular setting with C HAIN , we ﬁnd that the overall quantitative results are robust to hyperparameters mentioned above – we use the exact same set of hyperparameters for evaluation on R OOM in Section 4.2.
Evaluation setup: compute resources.
We ran the experiments on a cluster comprising of machines with 3 . 30 GHz Intel Xeon CPU E5-2667 v2 processor and 256 GB RAM.
E Evaluation on R OOM : Additional Details (Section 4.2)
R OOM (Figure 3).
The environment used in our experiments is based on the work of [54]; however, we adapted it to have a “distractor” state (shown as “green-plus”) that could provide a small positive reward.
Next, we present additional details about the environment.
We represent the environment as an MDP with S states, each corresponding to cells in the grid-world indicating the agent’s current location (shown as “blue-circle”).
The goal (shown as “green-star”) is located at the top-right corner cell; in the bottom-left room, there can be a “distractor” state (shown as “green-plus”) that could provide a small positive reward.
The agent can take four actions given by A :
= { “up” , “left” , “down” , “right” } .
An action takes the agent to the neighbouring cell represented by the direction of the action; however, if there is a wall (shown as “brown-segment”), the agent stays at the current location.
There are also a few terminal walls (shown as “thick-red-segment”) that terminate the episode, located at the bottom-left corner cell, where “left” and “down” actions terminate the episode.
Furthermore, when an agent takes an action a ∈ A , there is p rand probability that an action a (cid:48) ∈
A \ { a } will be executed instead of a .
The agent gets a reward of R max after it has navigated to the goal and then takes a “right” action (i.e., the reward can be accumulated in this state); similarly, the “up” action in the distractor state gives a reward of R dis .
The reward is 0 for all other state-action pairs.
There is a discount factor γ and an episode terminates after H = 30 steps.
The environment-speciﬁc parameters (including p rand , R max , R dis , γ ) are kept same as in Section 4.1, i.e., p rand = 0 .
05 , R max = 1 , R dis
= 0 or 0 .
01 , and γ = 0 . 99 .
Similar to the two variants of C HAIN environment, we have two variants of this environment: (a) R OOM 0 has R dis
= 0 at the distractor state shown as “green-plus” (equivalently, there is no distractor state); (b) R OOM + has R dis = 0 .
01 at the distractor state.
F Evaluation on L INE K: Additional Details (Section 4.3)
L INE K (Figure 4).
We expand on the details of the L INE K environment, introduced in Section 4.3.
As discussed in Section 4.3, this environment corresponds to a navigation task in a one-dimensional space where the agent has to ﬁrst pick the correct key and then reach the goal.
The environment used in our experiments is based on the work of [54]; however, we adapted it to have multiple keys (only one being correct) and “distractor” states that provide a small reward at goal locations even without the correct key.
The environment comprises of the following main elements: (a) an agent whose current location (shown as “blue-circle”) is a point x in [0 , 1] ; (b) goal (shown as “green-star”) is available in locations on the segment [0 .
9 , 1] ; (c) a set of k keys that are available in locations on the segment [0 . 0 , 0 . 1] ; (d) among k keys, only 1 key is correct and the remaining k − 1 keys are wrong (i.e., irrelevant at the goal).
The agent’s initial location is sampled from [0 . 3 , 0 . 4] .
The agent can take four actions given by A :
= { “left” , “right” , “pickCorrect” , “pickWrong” } .
“pickCorrect” action does not change the agent’s location, however, when executed in locations where keys are available, the agent acquires the correct key required at the goal; if the agent already possesses any key, the action has no effect.
“pickWrong” action does not change the agent’s location, however, when executed in locations where keys are available, the agent acquires one of the k − 1 wrong keys (chosen at random); if agent possesses a key, the action has no effect.
A move action of type “left” or “right” takes the agent from the current location in the direction of the move with the dynamics of the ﬁnal location captured by two hyperparameters (∆ a, 1 , ∆ a, 2 ) ; for instance, with current location x and action “left”, the new location x (cid:48) is sampled uniformly among locations from ( x − ∆ a, 1 − ∆ a, 2 ) to ( x − ∆ a, 1 + ∆ a, 2 ) .
The agent’s move action is not applied if the new location crosses the wall, and there is p rand probability of a random action.
The agent receives rewards as follows: (a) R max once it has navigated to the goal location after acquiring the correct key and then takes a “right” action (the action doesn’t terminate the episode and reward can be accumulated); (b) R dis after it has navigated to the goal location without acquiring the correct key and then takes a “right” action (the action doesn’t terminate the episode and reward can be accumulated); (c) the reward is 0 elsewhere.
We have a discount factor γ and the environment resets after a horizon of H .
We set p rand = 0 .
05 , R max = 1 , R dis
= 0 or 0 .
01 , H = 60 , γ = 0 .
99 , ∆ a, 1 = 0 .
075 , and ∆ a, 2 = 0 .
We obtain different variants of the environment by changing the values of R dis and number of keys k .
Similar to Sections 4.1 and 4.2, we use two adaptations of the environment: (i) L INE K 0 with ( k = 10 , R dis = 0)
(i.e., without any distractor state); (ii) L INE K + with ( k = 10 , R dis = 0 . 01) (i.e., with distractor states).
In our experiments, we represent the environment as an MDP with S states comprising of the following: (a) the agent’s current location (a point x in [0 , 1] ); (b) one bit indicating if the agent is on a segment with keys; (c) one bit indicating if the agent is on a segment with the goal; (d) k bits, corresponding to each of the k keys, indicating whether agent has that key or not (at most one of these bits can be one, as the agent can acquire only one key at any point in time, according to the transition dynamics speciﬁed above).
This state representation is the input observation space for neural networks used by our policy and intrinsic reward functions.
Evaluation setup: agents.
We conduct our experiments with a neural REINFORCE agent using a two-layered neural network architecture (i.e., one fully connected hidden layer with 256 nodes and RELU activation) [7].
In all the experiments that used neural-network based policies for agents, we also kept an exploration factor of (cid:15) = 0 .
05 , i.e., the agent uses soft-max neural policy with probability (1 − (cid:15) ) and chooses a random action with (cid:15) .
Algorithm 2 provides a sketch of the overall training process, and shows how agent’s training interleaves with reward shaping techniques – the agent’s policy is updated in lines 4 – 8 of the algorithm.
For the agent’s training process, we use a ﬁxed set of hyperparameters irrespective of the type of reward shaping technique or speciﬁc variant of the environment.
More concretely, we have the following: (a) the agent’s learning rate is set to 10 − 5 ; (b) frequency of updates N π is set to be 2 , i.e., update after every 2 rollouts in the environment; (c) a rollout buffer (ﬁrst-in-ﬁrst-out) D of size 10 is maintained and we update the agent’s policy using the last 5 rollouts in D .
Most of these hyperparameters are close to what we used for the tabular REINFORCE agent in the C HAIN environment, described in Appendix D.
Evaluation setup: shaping techniques.
Next, we describe different reward shaping techniques used during the agent’s training phase.
Speciﬁcally, during training, the agent receives rewards based on the shaped reward function (cid:98) R ; the performance (as reported in the plots) is always evaluated w.r.t.
the extrinsic reward function R .
Similar to Section 4.1, we compare the performance of six techniques.
As a crucial difference, here we use neural-network based reward functions for (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS , and (cid:98) R E XPLO RS .
We provide details of the different reward shaping techniques below:
• (cid:98) R O RIG :=
R .
This serves as a default baseline where extrinsic reward function is used during training without any shaping.
• (cid:98) R SORS’ :
= R + R SORS φ This is
based on the SORS technique [ ]; see additional details in
Section 2.2 (also see Footnote 2 about (cid:98) R SORS’ ).
Following the neural architectures used for reward functions in [25, 26], we use the same neural-network architecture as used for the agent’s policy – instead of using soft-max at the output layer to compute probability distribution over actions, here we use tanh -clipping (with a scaling factor of 0 . 10 ) to get output reward values for actions.
Algorithm 2 provides a sketch of the overall training process – the R SORS φ updates would be applied in lines 11 – 15 in the algorithm.
We update the intrinsic reward function using the following hyperparameters: (a) the learning rate is set to 10 − 3 ; (b) frequency of updates N r is set to be 20 , i. e., update after every 20 rollouts in the environment; (c) we have a rollout buffer D of size 10 and sample a set of 10 pairs of rollouts for the gradient updates (in our implementation, we prioritized sampling of pairs that have non-zero gap between returns).
• (cid:98) R LIRPG’ :
= R + R LIRPG’ φ .
This is obtained via adapting the LIRPG technique of [25] to our training pipeline; see Algorithm 2, Sections 2.2 and 3.2 (also see Footnote 3 about (cid:98) R LIRPG’ ).
More speciﬁcally, in our experiments, we implemented (cid:98) R LIRPG’ as an adaptation of (cid:98) R S ELF RS where we set h → ∞ instead of 1 in A π θk (cid:98) R,h ( s, a ) (see Section 3.2) – the rest of the implementation is same as described below for (cid:98) R S ELF RS .
When computing A π θk (cid:98) R,h ( s, a ) for h > 1 , we need an additional rollout to be able to compute this quantity.
In our experiments with L INE K, we set h → ∞ only for the starting state of the episode and kept h = 1 for the rest of the trajectory – this helped in reducing the computation time and variance.
• (cid:98) R E XPLO B := R + B E XPLO B w .
This corresponds to a part of our reward shaping technique which uses only the intrinsic bonuses B E XPLO B w .
As discussed in Sections 3.1 and 3.3, we use a count-based bonus B E XPLO B w .
For this environment, we use an abstraction that discretizes the continuous location part of the state to 0 .
1 -length segments, i.e., creating 10 segments in total; the bits used to represent different indicator ﬂags are then used along with these segments to represent an abstracted state.
Given this abstraction, the rest of the process and hyperparameters for updating B E XPLO B w are the same as discussed in Appendix D. • (cid:98) R S ELF RS :
= R + R S ELF RS φ .
This corresponds to a part of our reward shaping technique which uses only the intrinsic rewards R S ELF RS φ .
By following the neural architectures used for reward functions in [25, 26], we use the same neural-network architecture as used for the agent’s policy.
In particular, we use two networks for (cid:98) R S ELF RS : (a) one network is used for the reward function R S ELF RS φ that applies tanh -clipping (with a scaling factor of 0 . 10 ) instead of soft-max to get output reward values for actions; (b) the second network is used for learning value-function V R, (cid:101) φ that applies a linear layer instead of a soft-max layer to obtain state-values.
For updating V R, (cid:101) φ , we use Monte Carlo updates based on the trajectory returns as target and using a (cid:96) 2 -norm loss function [7].
Algorithm 2 provides a sketch of the overall training process – the R S ELF RS φ updates are applied in lines 11 – 15 in the algorithm.
We set the following values for hyperparameters: (a ) learning rate for updating φ parameters is set to 10 − 3 ; (b) learning rate for updating (cid:101) φ parameters is set to 5 · 10 − 3 ; (c) frequency of updates N r is set to be 20 , i.e., update after every 20 rollouts in the environment; (d) we have a rollout buffer D of size 10 .
• (cid:98) R E XPLO RS := R + R S ELF RS φ + B E XPLO B w .
This is our exploration-guided reward shaping technique that combines intrinsic bonuses with intrinsic rewards.
Algorithm 2 provides a sketch of the overall training process; we update R S ELF RS φ and B E XPLO B w in the same way as described in the previous two points above.
We update the policy more frequently than the intrinsic reward ( N π = 2 vs. N r = 20 ), as considered in the work of [25, 26].
Moreover, for the ﬁrst 5000 episodes of training, we do not supply intrinsic reward signals from neural network components of (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS , or (cid:98) R E XPLO RS (even though we keep updating their neural network components as usual) – this helps in preventing spuriourous reward signals associated with initialization of neural networks.