Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards
Abstract
We study the problem of reward shaping to accelerate the training process ofa reinforcement learning agent.
Existing works have considered a number ofdifferent reward shaping formulations; however, they either require external domainknowledge or fail in environments with extremely sparse rewards.
In this paper, wepropose a novel framework, Exploration-Guided Reward Shaping (E XP RS), thatoperates in a fully self-supervised manner and can accelerate an agent’s learningeven in sparse-reward environments. The key idea of E XP RS is to learn an intrinsicreward function in combination with exploration-based bonuses to maximize theagent’s utility w.r.t. extrinsic rewards.
We theoretically showcase the utility of E XP RS in a special family of MDPs.
Experimental results on several environmentswith sparse/noisy reward signals demonstrate the effectiveness of our framework
Introduction
Training reinforcement learning (RL) agents in environments with extremely sparse or distractingrewards is challenging.
Existing works have studied several approaches to design informative rewardsthat speed up the agent’s convergence [1–7].
One well-studied line of work is potential-based rewardshaping, where the potential function is speciﬁed by an expert or obtained via transfer learningtechniques (see [3, 8–17]).
Another popular approach is to learn rewards via Inverse-RL usingexpert demonstrations [18].
Alternatively, one could also consider a manual speciﬁcation of rewards,e.g., using distance-based metrics [19].
However, these reward design techniques typically rely onhigh-quality domain knowledge and may fail in practice.
In fact, the RL agents can easily exploitpoorly designed rewards and get stuck in local optima.
This naturally leads to the fundamentalquestion of how to do online reward shaping without relying on expert domain knowledge.
Moreconcretely, can we design informative rewards that will accelerate the agent’s training process byleveraging experience gained online during the agent’s training lifetime itself?
To tackle this question, recent works [24–26] have explored fully self-supervised learning of parametric intrinsic rewards that can improve the performance of RL agents.
In particular, these methodsalternate between intrinsic reward parameter learning and the agent’s policy optimization w.r.t.
thelearned reward.
For instance, the Learning Intrinsic Rewards for Policy Gradient (LIRPG) technique[25] updates the intrinsic reward parameters to maximize the extrinsic rewards received by the policyfrom the environment.
Self-supervised Online Reward Shaping (SORS) technique [26] infers anintrinsic reward using a classiﬁcation-based reward inference algorithm, TREX [27].
However, thesefully self-supervised reward shaping techniques might fail to produce meaningful agent behaviorin environments with extremely sparse rewards (called hard-exploration domains) as they lack anexplicit explorative component.
Intuitively, these techniques will not be able to make updates toparameters of their intrinsic reward functions, without receiving a non-zero extrinsic reward signal.
In a parallel line of work, several techniques have been proposed to speciﬁcally tackle the challengesof extreme sparsity and exploration.
One such line of work is to add more stochasticity in the agent’s behavior (e.g., [28–30]); however such techniques typically succeed in tasks with already well-shapedrewards.
Another important line of work, relevant to our proposed framework, is bonus-drivenexploration techniques for tackling hard-exploration domains – these techniques typically augmentthe extrinsic rewards with additional intrinsic bonus signals to encourage extra exploration [31].
Theidea behind count-based bonuses is that it encourages RL agents to experience infrequently visitedstates [32, 32–34].
For instance, [34] proposed a simple generalization of the classic explorationmethod with count-based intrinsic bonuses [35–38]; in particular, they discretize the state space witha hash function.
Another category of intrinsic bonuses is providing rewards for improving the agent’sknowledge about the environment [39–44].
However, simply relying on these bonus-driven signalscan mislead the agent towards sub-optimal or bad behaviors — for instance, in noisy-distractivedomains such as the “noisy TV problem” [45], unpredictable random noise outputs would attract theagent’s attention forever.
An important research question that we seek to address is: How can we design an online intrinsicreward function, without any domain knowledge or external supervision, that can speed up theagent’s learning process even in environments with extreme sparsity and noisy distractions?
Tothis end, we propose a novel framework, Exploration-Guided Reward Shaping, that combines bothexploration-focused and exploitation-focused intrinsic reward components.
Our framework of onlinereward shaping alternates between reward learning and policy optimization.
Similar to SORS [26],our method is compatible with any existing RL algorithm, and not only policy-gradient style learnersas considered in [25].
Our main contributions are:
I. We propose a novel framework, Exploration-Guided Reward Shaping (E XP RS), that appropri-
ately balances exploration (via an intrinsic bonus component) and exploitation (via an intrinsic
reward component) of extrinsic signals (Sections 2 and 3.1).
II.
We derive intuitive meta-gradients for updating the intrinsic reward component that enables
our method to be broadly applicable to any RL agent.
We theoretically
showcase the utility of
E XP RS in a chain environment (Sections 3.2 and 3.4).
III.
We empirically demonstrate the effectiveness of E XP RS on several environments with sparseand noisy reward signals (Section 4).
Problem Setup2.1 Online Reward Shaping
Preliminaries. An environment is deﬁned as a Markov Decision Process (MDP) M :=( S , A , T, P 0 , γ, R ) , where the state and action spaces are denoted by S and A respectively.
T : S × S ×
A → [0 , 1] captures the state transition dynamics, i.e., T ( s (cid:48) | s, a ) denotes theprobability of landing in state s (cid:48) by taking action a from state s .
γ is the discounting factor, and P 0is the initial state distribution.
The reward function is given by R : S ×
A → [ − R max , R max ] , forsome R max > 0 .
We denote the true underlying extrinsic reward function by R and the designedreward function by (cid:98) R .
We denote a stochastic policy π : S → ∆ ( A ) as a mapping from a state toa probability distribution over actions, and a deterministic policy π : S
→ A as a mapping from astate to an action.
For any trajectory ξ = { ( s t , a t ) } t =0 , 1 ,...,H , we deﬁne its cumulative return w.r.t.reward function R as J ( ξ, R ) :
= (cid:80)
Ht
=
γ t · R ( s t , a t ) .
Then, the expected cumulative return (value)of a policy π w.r.t.
R is deﬁned as J ( π, R ) :
=
E [ J ( ξ, R ) | P 0 , T, π ] , where s 0 ∼ P 0 ( · ) , a t ∼ π ( ·| s t ) ,and s t +1 ∼ T ( ·| s t , a t ) .
The learner seeks to ﬁnd a policy that has maximum value w.r.t.
the extrinsicreward function R , i.e., max π J ( π, R ) .
A general framework of online reward shaping for RL agents is given in Algorithm 1.
A naturalobjective here is to design informative rewards (cid:98) R k
at each round k
so that the resulting ﬁnal policyπ K performs better (i.e., has high value w.r.t.
R )
compared to the corresponding policy obtained viathe standard training with (cid:98) R k = R .
Note that, we consider a single lifetime training setting for an RLagent, on a single task, i.e., between rounds we do not reset the policy.
2.2 Existing Techniques and Issues
Reward shaping based on expert domain knowledge.
A popular technique for reward shapingis potential-based reward shaping (PBRS) which guarantees that any optimal policy induced by thedesigned function is also optimal under the extrinsic reward function [3].
Given a potential functionΦ : S → R that quantiﬁes “goodness” of the states, the reward function (cid:98) R PBRS produced by the PBRS framework is given by: (cid:98) R PBRS ( s, a ) :
=
R ( s, a ) + γ (cid:80) s (cid:48) ∈S T ( s (cid:48) | s, a ) · Φ ( s (cid:48) )
− Φ ( s ) .
For the above reward function to be helpful for training the RL agent, we need to have access togood potential functions based on expert domain knowledge.
It has already been noted that usingthe wrong potential functions could drastically slow down the learning process [46].
The authorsin [12, 47] have extended the PBRS framework to incorporate dynamic potential functions that arelearned online, alongside agent’s training.
As noted in [26], the effects of dynamic PBRS can be“learned away” over time, as they are equivalent to value function initialization [48].
Reward shaping based on exploration bonuses.
In the bonus-driven exploration framework [32–34], a count-based intrinsic bonus B k ( s ) is given to the agent to encourage exploration.
The bonus
B k ( s ) measures the “novelty” of a state s given the history of all transitions up to round k .
Theauthors in [34] extend the classic exploration methods with count-based intrinsic bonuses [35–38]to high-dimensional, continuous state spaces.
The states are mapped to hash codes, which allowsto count their occurrences with a hash table.
However, these “exploration-only” reward shapingmethods do not appropriately combine the successful extrinsic reward signals received from theenvironment.
When there are distractive zones in the state space, these methods will keep onexploring the state space even after obtaining extrinsic reward signals.
Reward shaping in fully self-supervised way: LIRPG [25] and PGRD [24].
The Learning Intrinsic Rewards for Policy Gradient (LIRPG) technique [25] considers a parametric reward function of theform (cid:98) R LIRPG ( s, a ) = R ( s, a )+ R φ ( s, a ) , and learns the parameter φ of the intrinsic reward function
R φ in a fully self-supervised manner. LIRPG method alternates between learning intrinsic rewardparameter φ and the agent’s policy optimization w.r.t.
the learned reward (cid:98) R LIRPG ( s, a ) = R ( s, a ) +
R φ ( s, a ) . At round k , for ﬁxed π k , LIRPG updates the parameter φ k − 1 to φ k by considering the effectsuch a change would have on the expected cumulative return (w.r.t. R ) of the learner through thechange in the policy π k , i.e., update φ using the gradient (cid:2) ∇ φ J ( L ( π k , (cid:98) R LIRPG ) , R ) (cid:3) φ k − 1 . In order todevelop a concrete update rule for φ , LIRPG considers policy gradient style learning algorithm L withparametric policies (cid:8) π θ : θ ∈ R d θ (cid:9) .
For any parameter θ k s.t. π k
= π θ k , and for any φ , we updateθ ( φ ) = θ k + α · (cid:104) ∇ θ J ( π θ , (cid:98) R LIRPG ) (cid:105) θ k .
Further, we set L ( π k , (cid:98) R LIRPG ) = π θ ( φ ) . The LIRPG update,at round k , is given as follows: φ k = φ k − 1 + η · [ ∇ φ θ ( φ )] φ k − 1 · (cid:2) ∇ θ ( φ ) J ( π θ ( φ ) , R ) (cid:3) φ k − 1 , where ηis the learning rate.
This form allows LIRPG to be applicable only to policy-based or policy-gradientstyle RL methods.
There is a predecessor work to LIRPG, called Policy Gradient for Reward Design(PGRD [24]), which aims to design optimal rewards for RL agents with computational limitationsresulting from small bounds on the depth of planning.
To this end, they speciﬁcally consider aplanning agent with h -step/depth Q -values in the tabular setting.
In [24], the authors have empiricallyfound that when the planning depth h is small, the role of the intrinsic reward component R φ becomesmore signiﬁcant in (cid:98) R PGRD ( s, a ) , where (cid:98) R PGRD ( s, a ) =
R ( s, a ) + R φ ( s, a ) .
Reward shaping in fully self-supervised way: SORS [26].
The Self-supervised Online
Reward Shaping (SORS) technique [26] considers a parametric reward function of the form(cid:98)
R SORS ( s, a ) =
R φ ( s, a ) , and infers the intrinsic reward parameter φ using a classiﬁcation-basedreward inference algorithm, T-REX [27].
However, unlike T-REX that requires manual rankingsover the trajectories, SORS uses the extrinsic reward R as a self-supervised learning signal to rankthe trajectories generated by the agent during training.
SORS can work with any RL method incontrast to LIRPG and PGRD techniques, which work only for speciﬁc types (policy-gradient basedor planning-based) of RL agents.
By design, SORS only enforces the
[Table]
over the trajectories w.r.t.
R when training R φ . It ignores the weighting or importance of differenttrajectories w.r.t. the scaling of R at different parts of the state space. This makes training a policychallenging, when R has noisy/distractive signals.
In this paper, we are interested in developing an online reward shaping technique that can succeedin environments with extremely sparse and distractive rewards, without any expert guidance.
Thefully self-supervised reward shaping techniques (LIRPG, PGRD, and SORS) might fail to producemeaningful agent behavior in environments with extremely sparse rewards as they lack an explicitexploration component. Intuitively, these techniques will not be able to make updates to parametersof their intrinsic reward functions, without receiving a non-zero extrinsic reward signal. We overcomethe limitations of these techniques by incorporating an explicit explorative component in our rewardformulation. In contrast, the techniques that rely only on the intrinsic bonus [32–34] can misleadthe agent towards sub-optimal or bad behaviors in noisy-distractive domains.
We overcome thelimitations of these techniques by appropriately exploiting the extrinsic reward signals. Further, wederive intuitive meta-gradients for updating the intrinsic reward component that enables our methodto be broadly applicable, not just for speciﬁc RL agents as in LIRPG [25] and PGRD [24]. Similar to
SORS [26], our technique is compatible with any RL agent as long we have access to their rollouts.  Exploration-Guided Reward Shaping
3.1 Our Reward Formulation
We consider the following parametric reward function for our Exploration-Guided Reward Shaping(E XP RS) technique (see Algorithm 1):
[Equation]
where φ ∈ R d φ and w ∈ R d w .
Here, R S ELF RS φ corresponds to the intrinsic rewards in self-supervisedreward shaping techniques, and B E XP w corresponds to the intrinsic bonuses in exploration-onlyreward shaping techniques.
At round k of Algorithm 1, (cid:98) R E XP RS k − 1 ( s, a ) is designed with parameters( φ k − 1 , w k − 1 ) .
Then, given updated policy π k , we update the parameters ( φ k − 1 , w k − 1 ) to ( φ k , w k ) .
For the remainder of this section, we drop the superscripts in the reward/bonus terms of Eq.
Intrinsic reward R φ .
We model the intrinsic reward R φ using any general parameterized function.
At round k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k by considering the effectsuch a change would have on the the expected cumulative return (w.r.t. R ) of the learner throughthe change in the policy π k (as in LIRPG [25]).
In particular, we update φ using the gradient 162 (cid:2) ∇ φ J ( L ( π k , (cid:98) R ) , R ) (cid:3) φ k − 1 , where (cid:98) R ( s, a )
=
R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
However, we may nothave access to the learning algorithm L directly or it might be too complex to analyze the impact ofthe change in φ considering L .
Since our end goal is to speed up the learning process of any RL agent,it is actually useful to consider a simple learning algorithm with parametric policies (cid:8) π θ : θ ∈ R d θ (cid:9) .
To this end, we consider a learning algorithm (cid:101)
L that does 1-step vanilla policy gradient update with hdepth planning for Q -values.
In particular, we map the policy π k to a parameter θ k ∈ R d θ and deﬁne: [Equation] [Equation]
[Equation]
, where Q π θk (cid:98) R,h ( s, a ) =
E (cid:104)(cid:80) ht
=
γ t · (cid:98) R ( s t , a t ) | s 0 =
s, a 0 = a, T, π θ k (cid:105) and α is the learning rate.
Then, we update the parameter φ using the following bi-level optimization:arg
[Equation]
(P
[Equation]
subject to θ ( φ ) ← (cid:101) L ( θ k , (cid:98) R ) (P1.L) where (cid:98) R ( s, a ) :
= R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
Note that the above bi-level formulation explicitly accounts for the requirement of accelerating thetraining process of an RL agent, with h -depth planning (inspired by PGRD [24]).
Intrinsic bonus B w .
Given a state abstraction ψ : S → X ψ (with |X ψ | = d w ), we maintain thevisitation count of the abstracted states in w , i.e., w [ x ] corresponds to the visitation counts of the states{ s ∈ S : ψ ( s ) = x } .
This allows us to implicitly maintain pseudo-counts N w ( s ) of visiting states s ∈ S .
In particular, we set N w ( s ) =
(cid:16) λB max (cid:17) 2 + w [ ψ ( s )] for some B max , λ > 0 .
Then, we deﬁne theintrinsic bonus as follows:
B w ( s ) =
λ √ N w ( s ) .
We update w based on the rollouts in round k [32–34].
3.2 Parameter Updates for R φ
In this section, we ﬁrst obtain high-level meta-gradient updates for R φ similar to LIRPG [25]. Then,we derive intuitive meta-gradient updates that would allow our Exploration-Guided Reward Shaping(E XP RS) technique to be compatible with any RL algorithm. Notation. In the subscript of the expectations, let π ( a | s ) mean a ∼ π ( ·| s ) , µ π ( s, a ) mean s ∼d π , a ∼ π ( ·| s ) , and µ π ( s ) mean s ∼ d π .
Further, we write µ s,a = µ π θk ( s, a ) , and µ s = µ π θk ( s ) .
High-level gradient updates for R φ . We solve the bi-level optimization problem (P1.U)-(P1.L) ofthe intrinsic reward component in an iterative manner using the gradient updates that we derive below.
At round k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k as follows:
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) 2 (cid:13) , where η is the learning rate, ( a ) is due to chain rule, and ( b ) is due to the following smoothness condition: (cid:13)(cid:13)(cid:13)(cid:2) ∇ θ J ( π θ , R ) (cid:3)
θ ( φ k − 1 ) − (cid:2) ∇ θ J ( π θ , R ) (cid:3) θ k (cid:13)(cid:13)(cid:13) 2 ≤ (cid:107) θ ( φ k − 1 ) − θ k (cid:107) 2 .
By using themeta-gradient derivations in [49–51], we write the term 1 (cid:13) as follows: [ ∇ φ θ ( φ )] φ k − 1 = α ·
E µ s,a (cid:104)(cid:2) ∇ φ Q π θk (cid:98) R,h ( s, a ) (cid:3) φ k − 1 · (cid:2) ∇ θ log π θ ( a | s ) (cid:3) (cid:62) θ k (cid:105) , where (cid:98) R ( s, a ) :
= R ( s, a )+ R φ ( s, a )+ B w k − 1 ( s ) .
By using the policy gradient theorem [52], we write the term 2 (cid:13) as follows: (cid:2) ∇ θ J ( π θ , R ) (cid:3) θ k = E µ s,a (cid:104)(cid:2) ∇ θ log π θ ( a | s ) (cid:3) θ k · Q π θk R ( s, a ) (cid:105) .
The above gradient update of φ k , involving the terms 1 (cid:13)and 2 (cid:13) , resembles the LIRPG [25] update.
However, both the terms 1 (cid:13) and 2 (cid:13) require computingthe gradient of the policy network, i.e., ∇ θ log π θ ( a | s ) .
This requirement makes the above (LIRPG)update applicable only for policy-gradient based agents. Below, we aim to derive intuitive simpliﬁcations of the above two terms, 1 (cid:13) and 2 (cid:13) , that would enable our method to be applicable to any RLagent, and not just policy-gradient based agents.
Intuitive gradient updates for R φ .
In order to obtain intuitive forms of the terms 1 (cid:13) and 2 (cid:13) , weconsider a simpliﬁed learner model.
However, we note that our ﬁnal algorithm does not rely on thissimpliﬁed model, i.e., our ﬁnal algorithm is applicable to any RL agent.
In particular, we consider alearner model using tabular representation θ ∈ R |S|·|A| , and deﬁne π θ ( a | s ) := [Equation] (cid:0)
[Equation]
(cid:1) (cid:80)
[Equation]
s
[Equation]
) (cid:1) , ∀ s ∈
S , a ∈ A . Note that, reward function R φ is still represented by a parametric function.
For anys ∈ S , a ∈ A , let 1 s,a ∈ R |S|·|A| denote a vector with 1 in the ( s, a ) -th entry and 0 else where.
Wedeﬁne A π θk (cid:98) R,h ( s, a ) :=
Q π θk (cid:98) R,h ( s, a ) − V π θk (cid:98) R,h ( s ) and A π θk R ( s, a ) :=
Q π θk R ( s, a ) − V π θk R ( s ) . Proposition 1.
For the learner model described above, we have the following:
[Equation]
Finally, we get the expectation-based update of φ as follows (with η (cid:48)
=
η · α ):
[Equation]
Remark. Compared to the high-level gradient update of φ as in LIRPG [25], the above intuitiveupdate with a simpliﬁed learner model does not require computing the gradient of the policy network. This allows the above update to be applicable to any RL agent. Given the current policy π k (possiblywithout any differentiable parameterization), we can write the above gradient update of φ as follows:
[Equation]
which only requires black-box access to (or sample rollouts from) policy π k .
In particular, weuse h = 1 depth planning for Q -values in (cid:101) L .
Then, in the above gradient update of φ , we have: 212 (cid:104)
[Equation]
3.3 Our Algorithm with Empirical Updates
In this section, we present a concrete pseudocode of our Exploration-Guided Reward Shaping(E XP RS) technique.
We closely follow the algorithmic structure of SORS [26].
Further, we focus onrollouts (each round corresponds to a single rollout), instead of environment steps, as in SORS.
Below,we discuss the empirical updates for intrinsic reward and bonus components of our reward (cid:98) R E XP RS . Empirical updates for intrinsic reward R φ .
We translate the ﬁnal expectation-based update of φ kin Section 3.2 to its empirical counterpart using the rollout data D collected by executing the currentpolicy π k (or the recent policies) in the MDP M .
In particular, we update the parameter φ as follows:
[Equation]
where we absorb 1 |D| into η φk , ignore the term µ π k ( s it ) , denote ξ it = ( s it , a it , . . . , s iH ) , and set
[Equation]
= (cid:104) ∇ φ (cid:16) R φ ( s it , a it ) − E π k ( b | s it ) (cid:2) R φ ( s it , b ) (cid:3) (cid:17)(cid:105) φ k .
Similar to LIRPG [25], wealso maintain a critic V R, (cid:101) φ k ( · ) to approximate V π k R ( · ) . We update the parameters of the critic, (cid:101) φ kto (cid:101) φ k +1 , using the same rollout data D and learning rate η (cid:101) φk .
Empirical updates for intrinsic bonus B w .
In comparison to the intrinsic reward, we update theintrinsic bonus at a more ﬁne-grained level, i.e., we update B w at each environment/rollout step 227 t within each round k .
However, for clear presentation, we write the B w update at the level of roundk , not at the level of environment step t .
We update B w based on the history of all the states visitedup to round k . Similar to #Exploration [34], we use the count-based intrinsic bonuses with a stateabstraction ψ : S → X ψ .
We maintain the visitation count of the abstracted states in w . For eachrollout ξ k , we update the parameter w of the intrinsic bonus as follows (component-wise):
[Equation]
Based on the above empirical updates for intrinsic reward and bonus components, we derive our ﬁnalalgorithm given in Algorithm 2.
For stability reasons, we update the policy more frequently comparedto the intrinsic reward, i.e., N π < N r .
Similar to the existing count-based exploration techniques [32–34], we use a lookahead step when incorporating the bonus term (see line 14 in Algorithm 2).
3.4 Theoretical Analysis
In this section, we theoretically showcase the utility of our exploration-guided reward shaping.
Inparticular, we show that our shaping technique helps to learn an optimal policy in a sample efﬁcientmanner in an environment with both extremely-sparse and distractive zones in the state space.
Environment.
We consider a chain environment M = (cid:0) S , A , T, P 0 , γ, R (cid:1) of length n 1 + n 2 + 1 .
Let the state space be S = (cid:8) x − n 2 , x − ( n 2 − 1) , . . .
, x − 1 , x 0 , x 1 , . . .
, x ( n 1 − 1) , x n 1 (cid:9) , and the actionspace be A = {← , →} .
We always start (or reset) in the state x 0 , i.e., the initial state distributionis P 0 ( x 0 )
= 1 .
The transition dynamics is deterministic and given by: T ( x i +1 | x i , → ) = 1 for− n 2 ≤ i ≤ n 1 − 1 , T ( x i − 1 | x i , ← )
= 1 for − ( n 2 − 1) ≤ i ≤ n 1 , T ( terminal | x n 1 , → ) = 1 ,and T ( terminal | x − n 2 , ← )
= 1 .
The reward function is deﬁned as follows:
R ( x i , → ) = 0 for− n 2 ≤ i ≤ n 1 − 1 , R ( x n 1 , → ) = 1 , and R ( x i , ← ) = 0 for − n 2 ≤ i ≤ n 1 .
We consider an inﬁnitehorizon setting with discounted returns ( H → ∞ , and γ < 1 ).
Learning algorithm and reward shaping techniques. For our theoretical analysis of differentreward shaping techniques, we consider a simpliﬁed TD-style RL algorithm L given in Appendix C.
The algorithm L takes two binary ﬂags, no-self and no-exp , as input that correspond to theintrinsic reward and bonus components, R and B , respectively.
For example, L (1 , 1) correspondsto the setting of learning with extrinsic reward R only.
The following theorem compares the samplecomplexity of algorithm L under different reward shaping techniques to learn an optimal policy. 1: Inputs and hyperparameters: L ﬁrst ﬁrst ; abstraction ψ ; rates
2: Initialization initialize
3: for K do // data collection 4: Rollout the policy π k in the MDP M to obtain a trajectory ξ k = (cid:0) s k 0 , a k 0 , s k 1 , a k 1 , . . . , s kH (cid:1) 5:
Store ξ k in the buffer D . add ( ξ k ) ; if the buffer D is full, remove the oldest trajectory // intrinsic bonus update 6:
Update w k according to Eq.
(4) using the states visited in the trajectory ξ k 7:
{+Deﬁne bonus B w k +
+}
( s )
=
λ √ N wk +1 ( s ) , where N w k +1 ( s )
{+= (cid:16) λB max (cid:17) 2 + w k +
+}
[ ψ ( s )] // intrinsic reward update 8: if k % N r = 0 then 9:
Update φ k according to Eq.
(3) using full data in D 10:
Update critic V R, (cid:101) φ k to V R, (cid:101) φ k +1 using full data in D and the learning rate η (cid:101) φk 11: else 12:
Maintain the previous values φ k +1 ← φ k and (cid:101) φ k +1 ← (cid:101) φ k // policy update 13: if k % N π = 0 then 14:
{+Deﬁne reward (cid:98) R k +
 ( s, a, s (cid:48) ) : =+}
R ( s, a ) + R φ k +1 ( {+s, a ) + B w k +
+}
( s (cid:48) )
Update policy π k +1 ← L ( π k , (cid:98) R k +1 ) using the latest rollouts in D 16: else 17:
Maintain the previous value π k +1 ← π k 18: Output: Policy π K
Theorem 1.
Consider the chain environment M deﬁned above, and the learning algorithm Lgiven in Appendix C. Let cost ( L ( no-self , no-exp )) denote the number time steps requiredfor L ( no-self , [-a (cid:48) )) 8: s t +
-]no-exp ) to learn an optimal policy in M .
Then, with appropriate choice ofhyperparameters, we have the following: (i) without any reward shaping, E [ cost ( L (1 , 1))] ≥ 2 n 1 − 1 ;(ii) with exploration-only reward shaping, cost ( L (1 , 0))
= n 1 · ( n 1 + n 2 + 2)
; (iii) with fullyself-supervised reward shaping, E [ cost ( L (0 , 1))] ≥ 2 n 1 − 1 ; (iv) with exploration-guided rewardshaping, cost ( L (0 , 0)) ≤ n 1 + n 2 + 2 .
Experimental Evaluation
In this section, we evaluate our reward shaping framework on three environments: C HAIN (Section 4.1), R OOM (Section 4.2), and L INE K (Section 4.3).
C HAIN corresponds to a navigation task ina chain, adapted from the environment used for theoretical analysis in Section 3.4; this is a canonicalenvironment used for studying extremely sparse-reward settings [7].
R OOM corresponds to a navigation task in a grid-world where the agent has to learn a policy to quickly reach the goal location in oneof four rooms, starting from an initial location.
Even though this environment has a small state/actionspace, it provides a very rich and intuitive problem setting to validate different reward shaping techniques.
In fact, variants of R OOM have been used extensively in the literature [10, 11, 14, 17, 53–56]—the environment used in our experiments is adapted from [56].
L INE K corresponds to a navigationtask in a one-dimensional space where the agent has to ﬁrst pick the correct key and then reach the goal.
The agent’s location is represented as a point on a line segment.
This environment is inspired by variants of navigation tasks in the literature where an agent needs to perform subtasks [3, 56, 57]—the environment used in our experiments is adapted from [56].
We give an overview of main results here, andprovide a more detailed description of the setup and additional results in the supplementary material.
4.1 Evaluation on C HAIN
C HAIN (Figure 1).
We represent the chain environment oflength n 1 + n 2 + 1 as an MDP with state-space S consistingof an initial location x 0 (shown as “blue-circle”), n 1 nodes tothe right of x 0 , and n 2 nodes to the left of x 0 . The rightmost 3 Episode (x 10 4 ) E xp ec t e d r e w a r d
E XP RS S ELF RS E XP SORS O RIG LIRPG (a) C HAIN 0 , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XP RS S ELF RS
E XP
SORS O RIG
LIRPG (
[Figure]
C HAIN + , REINFORCE 1.
3 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XP RS S ELF RS
E XP
SORS O RIG
[Figure]
HAIN 0 , Q
[Figure] learning 1. 3 Episode (x 10 4 ) E xp ec t e d r e w a r d
E XP RS S ELF RS E XP
SORS O RIG [Figure] node of the chain is the “goal” state (shown as “green-star”). In the left part of the chain, therecan be a “distractor” state (shown as “green-plus”).
The agent can take two actions given by A := { “left” , “right” } – an action takes the agent to the intended neighboring node with probabilityof (1 − p rand ) for p rand = 0 .
The agent receives a reward of R max = 1 for the “right” action at thegoal state, R dis for the “left” action at the distractor state, and 0 for all other state-action pairs.
Thereis a discount factor γ = 0 . 99 and the environment resets after a horizon of H = 40 .
We considertwo different variants of the chain environment: (i) C HAIN 0 with ( n 1 = 20 , n 2 = 40 , R dis = 0) ; (ii)
C HAIN + with ( n 1 = 20 , n 2 = 40 , R dis = 0 . 01) .
We defer the full environment details to appendices.
Evaluation setup and results.
We conduct our experiments with two different types of RL agentsfor C HAIN : tabular REINFORCE agent [7] and tabular Q-learning agent [7]. Algorithm 2 provides asketch of the overall training process, and shows how agent’s training interleaves with reward shapingtechniques.
We compare the performance of the following reward shaping techniques: (i) (cid:98) R O RIG :=
Ris a default baseline without any shaping; (ii) (cid:98) R SORS is obtained via the SORS technique [26] (see
Section 2.2); (iii) (cid:98) R LIRPG is obtained via adapting the LIRPG technique [25] to our training pipeline(see Algorithm 2, Sections 2.2 and 3.2)—note that (cid:98) R LIRPG is not applicable to Q-learning agent;(iv) (cid:98) R E XP :
= R + B E XP w uses only the intrinsic bonuses; (v) (cid:98) R S ELF RS :
= R + R S ELF RS φ uses only theintrinsic rewards; (vi) (cid:98) R E XP RS :
= R + R S ELF RS φ + B E XP w combines intrinsic bonuses with intrinsicrewards. We provide full details of hyperparameters in appendices.
During training, the agentreceives rewards based on (cid:98) R and is evaluated based on R .
Figure 2 illustrates results for both variantsof C HAIN environment; the reported results are averaged over 30 runs and convergence plots showthe mean performance with standard error bars.
These results demonstrate the effectiveness of ourexploration-guided reward shaping framework ( (cid:98) R E XP RS ), in comparison to baselines ( (cid:98) R O RIG , (cid:98) R SORS ,(cid:98) R LIRPG , (cid:98) R E XP , (cid:98) R S ELF RS ).
We provide a more detailed analysis of these results in appendices.
4.2 Evaluation on R OOM
R OOM (Figure 3).
This environment is based on the work of [56]; however, we adapted it to have a “distractor” state (shown as “green-plus”)that provides a small reward.
Similar to the two variants of C HAIN ,we have two variants of this environment: (i) R OOM 0 has R dis = [-
-]0at
the distractor state shown as “green-plus” (equivalently, there is nodistractor state); (ii) R OOM + has R dis = 0 .
01 at the distractor state.
The environment-speciﬁc parameters (including p rand , R max , H , γ ) arekept same as in Section 4.1.
We defer full details to appendices.
Evaluation setup and results.
Our evaluation setup for this environment is exactly same as that usedfor C HAIN environment (described in Section 4.1); here, we consider only the tabular REINFORCEagent.
In particular, all the hyperparameters (related to the REINFORCE agent, reward shapingtechniques, and training process) are the same as in Section 4.1.
Figures 4a and 4b show the agent’sperformance for environments R OOM 0 and R OOM + (averaged over 20 runs).
These results, alongwith results obtained in Figure 2, further demonstrate the effectiveness and robustness of (cid:98) R E XP RSacross different environments in comparison to baselines.
5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XP RS S ELF RS E XP
SORS O RIG
LIRPG (a) R OOM 0 , REINFORCE     5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XP RS S ELF RS E XP
SORS O RIG
LIRPG (b) R OOM + , REINFORCE     5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XP RS S ELF RS E XP
SORS O RIG
LIRPG (c) L INE K 0 , REINFORCE     5 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XP RS S ELF RS E XP
SORS O RIG
LIRPG (d) L INE K + , REINFORCE
4.3 Evaluation on L INE K
L INE K (Figure 5).
This environment corresponds to a navigationtask in a one-dimensional space where the agent has to ﬁrst pickthe correct key and then reach the goal.
The environment used inour experiments is based on the work of [56]; however, we adaptedit to have multiple keys (only one being correct) and “distractor”states that provide a small reward at goal locations even withoutthe correct key.
The environment comprises of the following main elements: (a) an agent whosecurrent location (shown as “blue-circle”) is a point x in [0 , 1] ; (b) goal (shown as “green-star”) isavailable in locations on the segment [0 . 9 , 1] ; (c) a set of k keys that are available in locations onthe segment [0 . 0 , 0 . 1] , (d) among k keys, only 1 key is correct and the remaining k − 1 keys arewrong (i.e., irrelevant at the goal). Moreover, we consider the agent with two different actionsrelated to picking a key: (a) “pickCorrect” makes the agent collect the correct key required at thegoal; (b) “pickWrong” makes the agent collect one of the k − 1 wrong keys, chosen at random.
Similar to Sections 4.1 and 4.2, we use two adaptations of the environment: (i) L INE K 0 with( k = 10 , R dis = 0) ; (ii) L INE K + with ( k = 10 , R dis = 0 . 01) .
We defer full details to appendices.
Experimental setup and results.
We conduct our experiments with a neural REINFORCE agentusing a two-layered neural network architecture (i.e., one fully connected hidden layer with 256nodes and RELU activation) [7].
Similar to Section 4.1, we compare the performance of sixtechniques.
As a crucial difference, here we use neural-network based reward functions for (cid:98) R SORS ,(cid:98) R LIRPG , (cid:98) R S ELF RS , and (cid:98) R E XP RS .
Based on [25, 26], we use the same neural-network architecture forintrinsic reward functions as used for the agent’s policy by applying appropriate transformationsat the output layer (e.g., instead of using soft-max, use tanh -clipping to get output reward valuesfor actions).
Figures 4c and 4d show the agent’s performance for environments L INE K 0 and L INE K +(averaged over 20 runs).
In particular, Figure 4d shows results for L INE K + – the performance of thethree reward shaping techniques ( (cid:98) R SORS , (cid:98) R LIRPG , (cid:98) R S ELF RS ) is even worse compared to that of (cid:98) R O RIG .
These results showcase the effectiveness of (cid:98) R E XP RS in more complex learning settings that requireneural representations for reward functions.
We provide a more in-depth analysis in appendices.
Concluding Discussions
We proposed a novel reward shaping framework, E XP RS, that operates in a fully self-supervisedmanner and could accelerate an agent’s learning even in sparse-reward environments.
Next, wediscuss a few limitations of our work and outline a future plan to address them. First, the experimentalevaluation is conducted on simpler environments to study the performance of techniques w.r.t.
thethree characteristics of (a) hard exploration, (b) local minima, and (c) noisy TV problem.
It wouldbe interesting to evaluate different reward design techniques in more complex environments (e.g.,with continuous state/action spaces); this would also require designing benchmark environments thatsystematically capture the above three characteristics.
Second, E XP RS combines the intrinsic rewardsand intrinsic bonuses that allows it to overcome the limitations of state-of-the-art techniques. It wouldbe interesting to develop more principled ways to combine these two signals. Third, it would beuseful to provide rigorous analysis of E XP RS in terms of convergence speed and stability of an agent.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] The paper is organized according to the contributions
listed at the end of the introduction section.
(b) Did you describe the limitations of your work?
[Yes] In Section 5, we discuss the limitations of our work and outline a future plan to address these limitations.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This
work presents a new online reward shaping technique for reinforcement learning agents.
Given the algorithmic and empirical nature of our work applied to learning agents, we
do not foresee any direct negative societal impacts of our work in the present form.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We conﬁrm that our paper conforms with the ethics review guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions
are stated either as part of the theorem statements or the proofs.
(b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs of
all theoretical results are included in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code and
instructions are included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Details are provided in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Error bars are included in all the result graphs
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] Details are provided in thesupplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]