Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards
Abstract
We study the problem of reward shaping to accelerate the training process of a reinforcement learning agent.
Existing works have considered a number of different reward shaping formulations; however, they either require external domain knowledge or fail in environments with extremely sparse rewards.
In this paper, we propose a novel framework, Exploration-Guided Reward Shaping (E XPLO RS), that operates in a fully self-supervised manner and can accelerate an agent’s learning even in sparse-reward environments.
The key idea of E XPLO RS is to learn an intrinsic reward function in combination with exploration-based bonuses to maximize the agent’s utility w.r.t.
extrinsic rewards.
We theoretically showcase the usefulness of our reward shaping framework in a special family of MDPs.
Experimental results on several environments with sparse/noisy reward signals demonstrate the effectiveness of E XPLO RS.
Introduction
Training reinforcement learning (RL) agents in environments with extremely sparse or distracting rewards is challenging.
Existing works have studied several approaches to design informative rewards that speed up the agent’s convergence [1–7].
One well-studied line of work is potential-based reward shaping, where a potential function is speciﬁed by an expert or obtained via transfer learning techniques (see [3, 8–17]).
Another popular approach is to learn rewards via Inverse-RL using expert demonstrations [18].
Alternatively, one could also consider a manual speciﬁcation of rewards, e g., using distance-based metrics [19].
However, these reward design techniques typically rely on high-quality domain knowledge and may fail in practice.
In fact, the RL agents can easily exploit poorly designed rewards and get stuck in local optima.
This naturally leads to the fundamental question of how to do online reward shaping without relying on expert domain knowledge.
More concretely, can we design informative rewards that will accelerate the agent’s training process by leveraging experience gained online during the agent’s training lifetime itself?
To tackle this question, recent works [24–26] have explored fully self-supervised learning of parametric intrinsic rewards that can improve the performance of RL agents.
In particular, these methods alternate between intrinsic reward parameter learning and the agent’s policy optimization w.r.t.
the learned reward.
For instance, Learning Intrinsic Rewards for Policy Gradient (LIRPG) technique [25] updates the intrinsic reward parameters to maximize the extrinsic rewards received by the policy from the environment.
Self-supervised Online Reward Shaping (SORS) technique [26] infers an intrinsic reward using a classiﬁcation-based reward inference algorithm, TREX [27].
However, these fully self-supervised reward shaping techniques might fail to produce meaningful agent behavior in environments with extremely sparse rewards (called hard-exploration domains) as they lack an 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
explicit explorative component.
Intuitively, these techniques will not be able to make updates to parameters of their intrinsic reward functions, without receiving a non-zero extrinsic reward signal.
In a parallel line of work, several techniques have been proposed to speciﬁcally tackle the challenges of extreme sparsity and exploration.
One such line of work is to add more stochasticity in the agent’s behavior (e.g., [28–30]); however such techniques typically succeed in tasks with already well-shaped rewards.
Another important line of work, relevant to our proposed framework, is bonusdriven exploration techniques for tackling hard-exploration domains – these techniques augment the extrinsic rewards with additional intrinsic bonus signals to encourage extra exploration [31].
A popular category of intrinsic bonuses is count-based bonuses that encourage RL agents to experience infrequently visited states [32–34].
Another category of intrinsic bonuses is providing rewards for improving the agent’s knowledge about the environment [35–40].
However, simply relying on these bonus-driven signals can mislead the agent towards sub-optimal or bad behaviors — for instance, in noisy-distractive domains such as the “noisy TV” problem [41], unpredictable random or noisy outputs would attract the agent’s attention forever.
An important research question that we seek to address is: How can we design an online intrinsic reward function, without any domain knowledge, that can speed up the agent’s learning process even in environments with extremely sparse rewards and noisy distractions?
To this end, we propose a novel framework, Exploration-Guided Reward Shaping (E XPLO RS), that learns an intrinsic reward function in combination with exploration-based bonuses to maximize the agent’s utility.
E XPLO RS operates in a fully self-supervised manner, and alternates between reward learning and policy optimization.
Our main results and contributions are:
I. We propose a novel reward shaping framework, E XPLO RS, that operates in a fully selfsupervised manner and can accelerate an agent’s learning even in sparse-reward environments.
(Section 3.1).
II.
We derive intuitive meta-gradients for updating the intrinsic reward component of E XPLO RS that enables our framework to be broadly applicable to any RL agent and not only policygradient based agents (Sections 3.2 and 3.3).
III.
We theoretically showcase the usefulness of our reward shaping framework in accelerating an agent’s learning in a special family of chain environments (Section 3.4).
IV.
We empirically demonstrate the effectiveness of E XPLO RS on several environments with sparse and noisy reward signals (Section 4).
Problem Setup
In Section 2.1, we present a general framework of online reward shaping technique for RL agents.
In
Section 2.2, we discuss the limitations of existing reward shaping techniques.
2.1 General Framework of Online Reward Shaping
Preliminaries.
An environment is deﬁned as a Markov Decision Process (MDP) M :
= ( S , A , T, P 0 , γ, R ) , where the state and action spaces are denoted by S and A respectively.
T : S × S ×
A → [0 , 1] captures the state transition dynamics, i.e., T ( s (cid:48) | s, a ) denotes the probability of landing in state s (cid:48) by taking action a from state s .
γ is the discounting factor, and Pis the initial state distribution.
The reward function is given by R : S ×
A → [ − R max , R max ] , for some R max > 0 .
We denote the true underlying extrinsic reward function by R and the designed reward function by (cid:98) R .
We denote a stochastic policy π : S → ∆ ( A ) as a mapping from a state to a probability distribution over actions, and a deterministic policy π : S
→ A as a mapping from a state to an action.
For any trajectory ξ = { ( s t , a t ) } t =0 , 1 ,...,H , we deﬁne its cumulative return w.r.t.
reward function R as J ( ξ, R ) :
= (cid:80) H t =
γ t · R ( s t , a t ) .
Then, the expected cumulative return (value) of a policy π w.r.t.
R is deﬁned as J ( π, R ) :
=
E [ J ( ξ, R ) | P 0 , T, π ] , where s 0 ∼ P 0 ( · ) , a t ∼ π ( ·| s t ) , and s t +1 ∼ T ( ·| s t , a t ) .
The learner seeks to ﬁnd a policy that has maximum value w.r.t.
the extrinsic reward function R , i.e., max π J ( π, R ) .
Online reward shaping.
A general framework of online reward shaping for RL agents is given in
Algorithm 1.
A natural objective here is to design informative rewards (cid:98) R k at each round k so that the resulting ﬁnal policy π K performs better (i.e., has high value w.r.t.
R )
compared to the corresponding policy obtained via the standard training with (cid:98) R k = R .
Note that we consider a single lifetime training setting for an RL agent on a single task, i.e., there is no resetting of the policy between rounds.
2.2 Existing Techniques and Issues
A popular technique for reward shaping is potential-based reward shaping (PBRS) which guarantees that any optimal policy induced by the designed reward function is also optimal under the extrinsic reward function [3].
However, for PBRS to be effective in accelerating the training process of an RL agent, we need to have access to good potential functions based on expert domain knowledge [42].
The focus of our work is on designing fully self-supervised reward shaping techniques.
Below, we provide a discussion of existing techniques that do not require any expert guidance or domain knowledge, and also discuss their limitations.
Reward shaping based on exploration bonuses.
In the bonus-driven exploration framework [32– 34], a count-based intrinsic bonus B k ( s ) is given to the agent to encourage exploration.
The bonus
B k ( s ) measures the “novelty” of a state s given the history of all transitions up to round k .
The authors in [34] extend the classic exploration methods with count-based intrinsic bonuses [43–46] to high-dimensional, continuous state spaces.
However, these “exploration-only” reward shaping techniques do not appropriately combine the successful extrinsic reward signals received from the environment.
When there are distractive zones in the state space, these methods will keep on exploring the state space even after obtaining extrinsic reward signals.
Fully self-supervised reward shaping: LIRPG [25].
Learning Intrinsic Rewards for Policy Gradient (LIRPG) technique [25] considers a parametric reward function of the form (cid:98) R LIRPG ( s, a )
=
R ( s, a ) + R φ ( s, a ) , and learns the parameter φ of the intrinsic reward function R φ in a fully self-supervised manner.
LIRPG alternates between learning the intrinsic reward parameter φ and the agent’s policy optimization w.r.t.
the learned reward (cid:98) R LIRPG .
At round k , for ﬁxed π k ,
LIRPG updates the parameter φ k − 1 to φ k by considering the effect such a change would have on the expected cumulative return (w.r.t.
R ) of the learner through the change in the policy π k , i.e., update φ using the gradient (cid:2) ∇ φ J ( L ( π k , (cid:98) R LIRPG ) , R ) (cid:3) φ k − 1 .
In order to develop an update rule for φ , LIRPG considers policy gradient style learning algorithm L with parametric policies (cid:8) π θ : θ ∈ R d θ (cid:9) .
More concretely, for a parameter θ k at round k s.t.
π k :=
π θ k , the learner’s policy update depends on φ as L ( π k , (cid:98) R LIRPG ) :
= π θ ( φ ) , where θ ( φ )
=
θ k
+ α · (cid:104) ∇ θ J
( π θ , (cid:98) R LIRPG ) (cid:105) θ k .
Based on this learner update, the LIRPG update for the intrinsic reward parameters, at round k , is based on the following meta-gradients:
[Equation]
1 , where η is the learning rate.
We note that the LIRPG technique could fail in environments with extremely sparse rewards as the agent may not receive a non-zero extrinsic reward signal needed to update the parameter φ .
Moreover, the LIRPG technique is applicable only to policy-gradient based RL agents.
Fully self-supervised reward shaping: SORS [26].
Self-supervised Online Reward Shaping (SORS) technique [26] considers a reward function of the form (cid:98) R SORS ( s, a ) =
R φ ( s, a ) , and infers the parameter φ using a classiﬁcation-based reward inference algorithm, T-REX [27].
However, unlike T-REX that requires rankings over the trajectories as input, SORS uses the extrinsic reward R as a self-supervised learning signal to rank the trajectories generated by the agent during training.
By de- sign, SORS only enforces the relative pairwise ordering over the trajectories w.r.t.
R when training R φ and ignores the scale of the returns associated with trajectories w.r.t.
R .
This makes training a policy challenging when the environment has noisy or distractive reward signals.
Further, similar to LIRPG, the SORS technique could fail in environments with extremely sparse rewards as the agent may not obtain any trajectories with non-zero extrinsic reward signal needed to update the parameter φ .
In this paper, we seek to develop an online reward shaping technique that can accelerate the agent’s training process in environments with extremely sparse and distractive rewards, without any expert domain knowledge.
As discussed above, techniques that rely only on intrinsic bonuses [32–34] could mislead the agent towards sub-optimal behaviors in noisy-distractive domains.
Similarly, the fully self-supervised reward shaping techniques (LIRPG and SORS) might be ineffective in environments with extremely sparse rewards.
We overcome these limitations by designing a novel reward shaping framework that appropriately balances exploration (via an intrinsic bonus component) and exploitation (via an intrinsic reward component) of extrinsic reward signals.
Exploration-Guided Reward Shaping
In Sections 3.1, 3.2, and 3.3, we propose an exploration-guided reward shaping framework, E XPLO RS, to accelerate an RL agent’s training process.
In Section 3.4, we theoretically showcase the usefulness of our framework in a chain environment.
3.1 Our Reward Formulation
We consider the following parametric reward function for E XPLO RS (see Algorithm 1):
[Equation]
where φ ∈ R d φ and w ∈ R d w .
Here, R S ELF RS φ corresponds to the intrinsic rewards in self-supervised reward shaping techniques, and B E XPLO B w corresponds to the intrinsic bonuses in exploration-only reward shaping techniques.
At round k of Algorithm 1, (cid:98) R E XPLO RS k −( s, a ) is designed with parameters ( φ k − 1 , w k − 1 ) .
Then, given updated policy π k , we update the parameters ( φ k − 1 , w k − 1 ) to ( φ k , w k ) .
Notation.
For the remainder of this section, we drop the superscripts (E XPLO RS, S ELF RS, and
E XPLO B) when referring to the reward functions in Eq.
In the subscript of the expectations E , let π ( a | s ) mean a ∼ π ( ·| s ) , µ π ( s, a ) mean s ∼ d π , a ∼ π ( ·| s ) , and µ π ( s ) mean s ∼ d π .
Further, we use shorthand notation µ ks,a and µ ks to refer µ π θk ( s, a ) and µ π θk ( s ) , respectively.
Intrinsic reward R φ .
We model the intrinsic reward R φ using any parameterized function.
At round k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k by considering the effect such a change would have on the the expected cumulative return w.r.t.
R through the change in the policy π k [24, 25].
In particular, we update φ using the gradient (cid:2) ∇ φ J ( L ( π k , (cid:98) R ) , R ) (cid:3) φ k − 1 , where (cid:98) R ( s, a )
=
R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
However, when considering L with neural policies, it is challenging to directly analyze the impact of φ in the policy π k .
Since our goal is to design a reward shaping technique that is applicable to any RL agent, we consider a simple surrogate learning algorithm (cid:101) L for our analysis.
In particular, we consider (cid:101) L with parametric policies (cid:8)
π θ : θ ∈ R d θ (cid:9) that does single-step vanilla policy gradient update with Q -values computed using h -depth planning.
We map the policy π k to a parameter θ k ∈ R d θ and deﬁne:
[Equation]
where α is the learning rate and Q π θk (cid:98) R,
h
[Equation]
s, a ) =
E (cid:104)(cid:80) ht
=
γ t (cid:98) R ( s t , a t ) (cid:12)(cid:12) s
= s, a 0 = a, T, π θ k (cid:105) is the h -depth Q -value w.r.t.
(cid:98) R .
Then, we update φ using the following bi-level optimization:
[Equation]
where (cid:98) R ( s, a ) :
= R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
In the above bi-level formulation, (cid:101) L with h -depth planning for small values of h essentially requires designing more informative intrinsic rewards to beneﬁt the agent’s training process [24].
Intrinsic bonus B w .
Given a state abstraction ψ : S → X ψ (with |X ψ | = d w ), we maintain the visitation count of the abstracted states in w , i.e., w [ x ] corresponds to the visitation counts of the states { s ∈ S : ψ ( s ) = x } .
This allows us to implicitly maintain pseudo-counts N w ( s ) of visiting states s ∈ S .
In particular, we set N w ( s ) =
(cid:0) λB max (cid:1) 2 + w [ ψ ( s )] for some B max , λ > 0 .
Then, we deﬁne the intrinsic bonus as follows:
B w ( s ) =
λ √ N w ( s ) .
We update w based on the rollouts in round k [32–34].
3.2 Derivation of Gradient Updates for R φ
In this subsection, we ﬁrst obtain high-level meta-gradient updates for R φ similar to LIRPG [25].
Then, we derive intuitive meta-gradient updates that would allow E XPLO RS to be compatible with any RL agent.
High-level gradient updates for R φ .
We solve the bi-level optimization problem (P1.U)-(P1.L) of the intrinsic reward component in an iterative manner using the gradient updates that we derive below.
At round k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k as follows:
[Equation]
where η is the learning rate, the equality in ( a ) is due to chain rule, and the approximation in ( b ) is made by assuming a smoothness condition of (cid:13)(cid:13)(cid:13)(cid:2) ∇ θ J ( π θ , R ) (cid:3) θ ( φ k − 1 ) − (cid:2) ∇ θ J ( π θ , R ) (cid:3) θ k (cid:13)(cid:13)(cid:13) 2 ≤ c · (cid:107) θ ( φ k − 1 ) − θ k (cid:107) 2 for some c > 0 .
By using the meta-gradient derivations in [47–49], we write the term 1 (cid:13) as follows: [ ∇ φ
[Equation]
φ )]
[Equation]
k (cid:105) , where (cid:98) R ( s, a ) :=
R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
By using the policy gradient theorem [50], we write the term 2 (cid:13) as follows:
[Equation]
The above gradient update of φ k , involving the terms 1 (cid:13) and 2 (cid:13) , resembles the LIRPG [25] update.
However, both the terms 1 (cid:13) and 2 (cid:13) require computing the gradient of the policy, i.e., ∇ θ log π θ ( a | s ) .
This requirement makes the above update applicable only for policy-gradient based agents.
Below, we derive intuitive simpliﬁcations of the above two terms, 1 (cid:13) and 2 (cid:13) , that would enable our technique to be applicable to any RL agent, and not only policy-gradient based agents.
Intuitive gradient updates for R φ .
In order to obtain intuitive forms of the terms 1 (cid:13) and 2 (cid:13) , we consider further simpliﬁcations to the surrogate learning algorithm (cid:101) L introduced in Section 3.1.
In particular, for our analysis and derivation, we let (cid:101)
L use tabular representation θ ∈ R |S|·|A| and softmax policy given by π θ ( a | s ) :
=
[Equation]
, ∀ s ∈ S , a ∈ A .
We deﬁne A
[Equation]
s
[Equation]
:=
Q π θk (cid:98) R,h ( s, a ) − V π θk (cid:98) R,h ( s ) and A π θk R ( s, a ) :
=
Q π θk R ( s, a ) − V π θk R ( s ) .
Based on this, the following proposition provides intuitive gradient updates for R φ .
Proposition 1.
For the simpliﬁed surrogate learning algorithm (cid:101) L with h -depth planning, the gradient term [ ∇ φ θ ( φ )] φ k − 1 · (cid:2) ∇ θ J ( π θ , R ) (cid:3) θ k in Eq.
(2) takes the following form:
[Equation]
For the special case of h =
[Equation]
the gradient term further simpliﬁes to the following form:
[Equation]
Compared to Eq.
(2), the intuitive gradient update term in the above proposition does not require computing the policy gradient ∇ θ log π θ ( a | s ) .
This allows us to develop an update rule for intrinsic reward parameter φ that is applicable to any RL agent.
In particular, given the current policy π k (possibly without any differentiable parameterization), we simplify Eq.
(2) and propose the following gradient update rule for parameter φ :
[Equation]
where η (cid:48)
=
η · α .
Note that the above gradient update only requires black-box access to the policy π k in the form of trajectory rollouts as in the SORS technique [26].
3.3 Empirical Updates and Practical Aspects
In this subsection, we present a concrete pseudocode for training an RL agent with E XPLO RS reward shaping technique.
Algorithm 2 provides a sketch of the overall training process, interleaving the agent’s training with E XPLO RS.
The sketch presented in Algorithm 2 is adapted from the training process proposed for the SORS technique [26].
Further, we consider rollouts where each round corresponds to a single rollout, instead of environment steps, as in SORS.
Below, we discuss the empirical updates for intrinsic reward and bonus components of Eq.
Empirical updates for intrinsic reward R φ .
We translate the ﬁnal expectation-based update of φ k in Eq.
(3) to its empirical counterpart using the rollout data D collected by executing the current policy π k (or recent policies) in the MDP M .
At any round k , let D contain a collection of trajectories (cid:8) ξ i (cid:9) n i =1 , where ξ i =
( s i 0 , a i 0 , s i 1 , a i 1 , . . .
, s iH ) .
For a given trajectory ξ i and time index t , we denote a partial trajectory as ξ it =
( s it , a it , . . . , s iH ) .
Based on this notation, we empirically update the parameter φ as follows:
[Equation]
where we absorb the normalization factors into η φk , ignore the term µ π k ( s it ) , and set (cid:104) ∇
[Equation]
Similar to LIRPG [25]
[Equation]
we also maintain a critic V R, (cid:101) φ k − 1 ( · ) to approximate V π k R ( · ) in Eq.
We update the parameters of the critic, (cid:101) φ k − 1 to (cid:101) φ k , using the same rollout data D and learning rate η (cid:101) φk .
In Algorithm 2, hyperparameters N r and N π control the frequency of updates for the intrinsic reward R φ and policy π , respectively.
For stability reasons, we update the policy more frequently compared to the intrinsic reward, i.e., N π < N r .
We provide full implementation details in Section 4 and appendices.
Empirical updates for intrinsic bonus B w .
We update B w based on the history of all the states visited up to round k .
Similar to #Exploration [34], we use the count-based intrinsic bonuses with a state abstraction ψ : S → X ψ .
We maintain the visitation count of the abstracted states in w .
For each rollout ξ k , we update the parameter w of the intrinsic bonus as follows:
[Equation]
Similar to the existing count-based exploration techniques [32–34], we use a lookahead step when incorporating the bonus term (see line 5 in Algorithm 2).
In our implementation, we update the intrinsic bonus at a more ﬁne-grained level, i.e., we update B w at each environment step t within each round k directly, instead of waiting for the rollout to ﬁnish.
However, for clear presentation in
Algorithm 2, we write the B w update at the level of round k , not at the level of environment step t .
We provide full implementation details in Section 4 and appendices.
3.4 Theoretical Analysis
In this subsection, we theoretically showcase the usefulness of our exploration-guided reward shaping framework in accelerating an agent’s learning in a chain environment with extremely-sparse rewards and distractive zones in the state space.
Our analysis considers a stylized learning setting with simpliﬁed versions of different reward shaping techniques.
do update 4: if k % π
Deﬁne reward (cid:98) R k − 1 ( s, a, s (cid:48) ) :
= R ( s, a ) + R φ k − 1 ( s, a ) + B w k − 1 ( s (cid:48) ) 6:
Obtain updated policy π k ← L ( π k − 1 , (cid:98) R k − using the latest rollouts in D 7: else 8:
Keep previous policy π k ← π k −// data collection 9:
Rollout the policy π k in the MDP M to obtain a trajectory ξ k = (cid:0) s k 0 , a k 0 , s k 1 , a k 1 , . . .
, s kH (cid:1) 10:
Store ξ k in the buffer D . add ( ξ the
D is full, remove the oldest trajectory // intrinsic reward update 11: if k % N r =
then 12:
Obtain updated reward parameter φ k from φ k − 1 as in Eq.
(4) using D and learning rate η φk 13:
Obtain updated critic parameter (cid:101) φ k from (cid:101) φ k − 1 using D and learning rate η (cid:101) φk 14: else 15:
Keep previous parameters
φ k ← φ k − 1 and (cid:101) φ k ← (cid:101) φ k −// intrinsic bonus update 16:
Update k 17:
Deﬁne bonus B w k ( s ) =
λ √ N wk ( s ) , where N w k ( s ) = (cid:0) λB max (cid:1)
+ w k
[ ψ ( s )] 18: Output: Policy π K
Chain environment.
We consider a chain environment M = (cid:0) S , A , T, P 0 , γ, R (cid:1) of length n 1 + n 2 + 1 .
Let the state space be S
= { x − n 2 , . .
, x − 1 , x 0 , x 1 , . . .
, x n 1 } , and the action space be A = {← , →} .
We always start in the state x 0 , i.e., the initial state distribution is
P 0 ( x 0 )
= 1 .
The transition dynamics is deterministic and given as follows: T ( x i +1 | x i , → )
=for − n 2 ≤ i ≤ n 1 − 1 , T ( x i − 1 | x i , ← )
= 1 for − ( n 2 − 1) ≤ i ≤ n 1 , T ( terminal | x n 1 , → ) = 1 , and T ( terminal | x − n 2 , ← )
= 1 .
The reward function is deﬁned as follows:
R ( x i , → ) = 0 for − n 2 ≤ i ≤ n 1 − 1 , R ( x n 1 , → ) = 1 , and R ( x i , ← ) = 0 for − n 2 ≤ i ≤ n 1 .
We consider an inﬁnite horizon setting with discounted returns, i.e., H → ∞ and γ < 1 .
Learning algorithm and reward shaping techniques.
For our theoretical analysis, we consider a stylized learning setting with a TD-style RL algorithm L and simpliﬁed versions of different reward shaping techniques; details are provided in appendices.
We analyze the total number time steps required for L to learn an optimal policy in the chain environment under four different settings: (i)
Case L ( S ELF RS = 0 , E XPLO B = 0) is a default setting without any shaping; (ii) Case L ( S ELF RS = 0 , E XPLO B = 1) uses only the intrinsic bonuses; (iii) Case L ( S ELF RS = 1 , E XPLO B = 0) uses only the intrinsic rewards; (iv) Case L ( S ELF RS = 1 , E XPLO B = 1) combines intrinsic bonuses with intrinsic rewards.
The following theorem compares these four settings and showcases the usefulness of our framework, i.e., Case L ( S ELF RS = 1 , E XPLO B = 1) – proof is provided in appendices.
Theorem 1.
Consider the chain environment M and the algorithm L deﬁned above.
Let cost ( L ( S ELF RS , E XPLO B )) denote the total number time steps required for L ( S ELF RS , E XPLO B ) to learn an optimal policy in M .
Then, we have the following (expected) costs for the four settings:
(i) E [ cost ( L ( S ELF RS = 0 , E XPLO B = 0))] ≥ 2 n 1 − 1 ; (ii) cost ( L ( S ELF RS = 0 , E XPLO B = 1))
=
n 1 · ( n 1 + n 2 + 2) ; (iii) E [ cost ( L ( S ELF RS = 1 ,
E XPLO B = 0))] ≥ 2 n 1 − 1 ; (iv) cost ( L ( S ELF RS = 1 , E XPLO B = 1)) ≤ n 1 + n 2 +
The proof and additional details about the learning setting are provided in appendices.
Experimental Evaluation
In this section, we evaluate our reward shaping framework on three environments: C HAIN (Section 4.1), R OOM (Section 4.2), and L INE K (Section 4.3).
C HAIN corresponds to a navigation task in a chain, adapted from the environment used for theoretical analysis in Section 3.4; this is a canonical environment used for studying extremely sparse-reward settings [7].
R OOM corresponds to a navigation task in a grid-world where the agent has to learn a policy to quickly reach the goal location in one of four rooms, starting from an initial location.
Even though this environment has a small state/action space, it provides a very rich and intuitive problem setting to validate different reward shaping techniques.
In fact, variants of R OOM have been used extensively in the literature [10, 11, 14, 17, 51–54]—the environment used in our experiments is adapted from [54].
L INE K corresponds to a navigation task in a one-dimensional space where the agent has to ﬁrst pick the correct key and then reach the goal.
The agent’s location is represented as a point on a line segment.
This environment is inspired by variants of navigation tasks in the literature where an agent needs to perform subtasks [3, 54, 55]—the environment used in our experiments is adapted from [54].
We give an overview of main results here, and provide a more detailed description of the setup and additional implementation details in appendices.
4.1 Evaluation on C HAIN
C HAIN (Figure 1).
We represent the chain environment of length n 1 + n 2 + 1 as an MDP with state-space S consisting of an initial location x 0 (shown as “blue-circle”), n 1 nodes to the right of x 0 , and n 2 nodes to the left of x 0 .
The rightmost node of the chain is the “goal” state (shown as “green-star”).
In the left part of the chain, there can be a “distractor” state (shown as “green-plus”).
The agent can take two actions given by
A := { “left” , “right” } – node receives We We
Evaluation setup.
We conduct our experiments with two different types of RL agents for C HAIN : tabular REINFORCE agent [7] and tabular Q-learning agent [7].
Algorithm 2 provides a sketch of the overall training process, and shows how agent’s training interleaves with reward shaping techniques.
We compare the performance of the following reward shaping techniques: (i) (cid:98) R O RIG :=
R is a default baseline without any shaping; (ii) (cid:98) R SORS’ :
= R + R SORS φ is based on the SORS technique [26] (see
Section 2.2); 2 (iii) (cid:98) R LIRPG’ is obtained via adapting the LIRPG technique [25] to our training pipeline (see Algorithm 2, Sections 2.2 and 3.2)—note that (cid:98) R LIRPG’ is not applicable to Q-learning agent; 3 (iv) (cid:98) R E XPLO B := R + B E XPLO B w uses only the intrinsic bonuses; (v) (cid:98) R S ELF RS :
= R + R S ELF RS φ uses only the intrinsic rewards; (vi) (cid:98) R E XPLO RS :
= R + R S ELF RS φ + B E XPLO B w combines intrinsic bonuses with intrinsic rewards.
We provide full details about the implementation and hyperparameters in appendices.
Results.
During training, the agent receives rewards based on (cid:98) R and is evaluated based on R .
Figureshows results for both the variants of C HAIN environment; the reported results are averaged over 20 runs and convergence plots show the mean performance with standard error bars.
These results demonstrate the effectiveness of our exploration-guided reward shaping framework ( (cid:98) R E XPLO RS ), in comparison to baselines ( (cid:98) R O RIG , (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R E XPLO B , (cid:98) R S ELF RS ).
Next, we summarize some of our key ﬁndings.
First, our results show that (cid:98) R E XPLO RS outperforms the baselines in both C HAINand C HAIN + environments, irrespective of the RL agent (REINFORCE and Q-learning).
Second, the performance of (cid:98) R E XPLO RS is better than variants which only use either intrinsic bonuses or intrinsic SORS reward
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(a) C HAIN 0 , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(b) C HAIN + , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG (c) C HAIN 0 , Q-learning 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG (
[Figure]
, Q-learning rewards, i.e., (cid:98) R E XPLO B or (cid:98) R S ELF RS – this demonstrates the utility of combining these two signals.
Third, results in Figures 2b and 2d show that three reward shaping techniques ( (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS ) could fail or lead to sub-optimal policies because of the presence of distractor states.
4.2 Evaluation on R OOM
3: R OOM 0 / R OOM +
R OOM (Figure 3).
This environment is based on the work of [54]; however, we adapted it to have a “distractor” state (shown as “greenplus”) that provides a small reward.
Similar to the two variants of
C HAIN , we have two variants of this environment: (i) R OOM 0 has
R dis =
at the distractor state shown as “green-plus” (equivalently, there is no distractor state); (ii) R OOM + has R dis = 0 .
01 at the distractor state.
The environment-speciﬁc parameters (including p rand , R max , γ ) are kept same as in Section 4.1.
We defer full details to appendices.
Evaluation setup and results.
Our evaluation setup for this environment is exactly same as that used for C HAIN environment (described in Section 4.1); here, we consider only the tabular REINFORCE agent.
In particular, all the hyperparameters (related to the REINFORCE agent, reward shaping techniques, and training process) are the same as in Section 4.1.
Figures 5a and 5b show the agent’s performance for environments R OOM 0 and R OOM + (averaged over 20 runs).
These results, along with results obtained in Figure 2, further demonstrate the effectiveness and robustness of (cid:98) R E XPLO RS across different environments in comparison to baselines.
4.3 Evaluation on L INE K
L INE K (Figure 4).
This environment corresponds to a navigation task in a one-dimensional space where the agent has to ﬁrst pick the correct key and then reach the goal.
The environment used in our experiments is based on the work of [54]; however, we adapted it to have multiple keys (only one being correct) and “distractor” states that provide a small reward at goal locations even without the correct key.
The environment comprises of the following main elements: (a) an agent whose current location (shown as “blue-circle”) is a point x in [0 , 1] ; (b) goal (shown as “green-star”) is available in locations on the segment [0 .
9 , 1] ; (c) a set of k keys that are available in locations on the segment [0 . 0 , 0 . 1] , (d) among k keys, only 1 key is correct and the remaining k − 1 keys are wrong (i.e., irrelevant at the goal). Moreover, we consider the agent with two different actions related to picking a key: (a) “pickCorrect” makes the agent collect the correct key required at the goal; (b) “pickWrong” makes the agent collect one of the k − 1 wrong keys, chosen at random.
Similar to Sections 4.1 and 4.2, we use two adaptations of the environment: (i) L INE K 0 with ( k = 10 , R dis = 0) ; (ii) L INE K + with ( k = 10 , R dis = 0 . 01) .
We defer full details to appendices.
Experimental setup.
We conduct our experiments with a neural REINFORCE agent using a twolayered neural network architecture (i.e., one fully connected hidden layer with 256 nodes and RELU activation) [7].
Similar to Section 4.1, we compare the performance of six techniques.
As a crucial difference, here we use neural-network based reward functions for (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS , and
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(a) R OOM 0 , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(b)
R OOM + , REINFORCE     5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(c) L INE K 0 , REINFORCE     5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XPLO RS S ELF RS
E XPLO B
SORS
O RIG
LIRPG(d)
L INE K + , REINFORCE (cid:98) R E XPLO RS (see Footnotes 2 and 3).
Based on [25, 26], we use the same neural-network architecture for intrinsic reward functions as used for the agent’s policy by applying appropriate transformations at the output layer (e.g., instead of using soft-max, use tanh -clipping to get output reward values for actions).
We provide full details about the implementation and hyperparameters in appendices.
Results.
During training, the agent receives rewards based on (cid:98) R and is evaluated based on R .
Figures 5c and 5d show results for both the variants of L INE K environment; the reported results are averaged over 30 runs and convergence plots show the mean performance with standard error bars.
These plots showcase the performance of different techniques as we vary R dis ∈ { 0 . 00 , 0 . 01 } – this in turn decides whether there are any distractor states that can serve as local minima for the agent.
The convergence behavior in Figures 5c and 5d demonstrates the effectiveness of our explorationguided reward shaping framework ( (cid:98) R E XPLO RS ), in comparison to baselines ( (cid:98) R O RIG , (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R E XPLO B , (cid:98) R S ELF RS ).
Next, we summarize some of our key ﬁndings.
First, our results show that (cid:98) R E XPLO RS outperforms all the baselines in both L INE K 0 and L INE K + environments.
Second, results in Figure 5d show that three reward shaping techniques ( (cid:98) R SORS’ , (cid:98) R LIRPG’ , (cid:98) R S ELF RS ) performed worse than (cid:98) R O RIG – this is because of the presence of distractor states which create local minima for the agent and these shaped functions could further encourage learning a sub-optimal policy.
In contrast, (cid:98) R E XPLO RS combines the beneﬁts of intrinsic rewards ( (cid:98) R S ELF RS ) and intrinsic bonuses ( (cid:98) R E XPLO B ) to speed up agent’s learning in a robust and efﬁcient manner.
Overall, these results demonstrate that our shaping technique (cid:98) R E XPLO RS results in efﬁcient learning even when dealing with complex state representations and when learning neural-network based intrinsic reward functions.
Concluding Discussions
We proposed a novel reward shaping framework, E XPLO RS, that operates in a fully self-supervised manner and could accelerate an agent’s learning even in sparse-reward environments.
Next, we discuss a few limitations of our work and outline a future plan to address them.
First, the experimental evaluation is conducted on simpler environments to study the performance of techniques w.r.t.
the three characteristics of (a) hard exploration, (b) local minima, and (c) “noisy TV” problem.
It would be interesting to evaluate different reward design techniques in more complex environments (e.g., with continuous state/action spaces); this would also require designing benchmark environments that systematically capture the above three characteristics.
Second, E XPLO RS combines the intrinsic rewards and intrinsic bonuses that allows it to overcome the limitations of state-of-the-art techniques.
It would be interesting to develop more principled ways to combine these two signals.
Third, it would be useful to provide rigorous analysis of E XPLO RS in terms of convergence speed and stability of an agent.
Acknowledgments
Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute.
Funded/Cofunded by the European Union (ERC, TOPS, 101039090).
[-+
-]
[-= (cid:80) n 
-]
Views and opinions expressed are however those of the author(s) only and do not necessarily reﬂect those of the European Union or the European Research Council.
Neither the European Union nor the granting authority can be held responsible for them.