Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards
Abstract
We study the problem of reward shaping to accelerate the training process of areinforcement learning agent.
Existing works have considered a number of different reward shaping formulations; however, they either require external domainknowledge or fail in environments with extremely sparse rewards.
In this paper,our goal is to develop a reward shaping framework that can operate in a fully selfsupervised manner and can accelerate an agent’s learning even in sparse-rewardenvironments. We propose a novel framework, Exploration-Guided Reward Shaping, that combines exploration-based intrinsic bonuses along with extrinsic rewardsignals.
We theoretically showcase the utility of our reward shaping framework incontrast to existing approaches.
Experimental results on several environments withsparse/noisy reward signals demonstrate the effectiveness of our framework
Introduction
Training reinforcement learning (RL) agents in environments with extremely sparse or distractingrewards is challenging.
Existing works have studied several approaches to design informative rewardsthat speed up the agent’s convergence [1–7].
One well-studied line of work is potential-based rewardshaping, where the potential function is speciﬁed by an expert or obtained via transfer learningtechniques (see [3, 8–17]).
Another popular approach is to learn rewards via Inverse-RL usingexpert demonstrations [18].
Alternatively, one could also consider a manual speciﬁcation of rewards,e.g., using distance-based metrics [19].
However, these reward design techniques typically rely onhigh-quality domain knowledge and may fail in practice.
In fact, the RL agents can easily exploitpoorly designed rewards and get stuck in local optima.
This naturally leads to the fundamentalquestion of how to do online reward shaping without relying on expert domain knowledge.
Moreconcretely, can we design informative rewards that will accelerate the agent’s training process byleveraging experience gained online during the agent’s training lifetime itself?
To tackle this question, recent works [24–26] have explored fully self-supervised learning of parametric intrinsic rewards that can improve the performance of RL agents.
In particular, these methodsalternate between intrinsic reward parameter learning and the agent’s policy optimization w.r.t.
thelearned reward.
For instance, the Learning Intrinsic Rewards for Policy Gradient (LIPRG) technique[25] updates the intrinsic reward parameters to maximize the extrinsic rewards received by the policyfrom the environment.
Self-supervised Online Reward Shaping (SORS) technique [26] infers anintrinsic reward using a classiﬁcation-based reward inference algorithm, TREX [27].
However, thesefully self-supervised reward shaping techniques might fail to produce meaningful agent behaviorin environments with extremely sparse rewards (called hard-exploration domains) as they lack anexplicit explorative component.
Intuitively, these techniques will not be able to make updates toparameters of their intrinsic reward functions, without receiving a non-zero extrinsic reward signal.
In a parallel line of work, several techniques have been proposed to speciﬁcally tackle the challengesof extreme sparsity and exploration.
One such line of work is to add more stochasticity in the agent’s behavior (e.g., [28–30]); however such techniques typically succeed in tasks with already well-shapedrewards.
Another important line of work, relevant to our proposed framework, is bonus-drivenexploration techniques for tackling hard-exploration domains – these techniques typically augmentthe extrinsic rewards with additional intrinsic bonus signals to encourage extra exploration [31].
Theidea behind count-based bonuses is that it encourages RL agents to experience infrequently visitedstates [32, 32–34].
For instance, [34] proposed a simple generalization of the classic explorationmethod with count-based intrinsic bonuses [35–38]; in particular, they discretize the state space witha hash function.
Another category of intrinsic bonuses is providing rewards for improving the agent’sknowledge about the environment [39–44].
However, simply relying on these bonus-driven signalscan mislead the agent towards sub-optimal or bad behaviors — for instance, in noisy-distractivedomains such as the “noisy TV problem” [45], unpredictable random noise outputs would attract theagent’s attention forever.
An important research question that we seek to address is: How can we design an online intrinsicreward function, without any domain knowledge or external supervision, that can speed up theagent’s learning process even in environments with extreme sparsity and noisy distractions?
Tothis end, we propose a novel framework, Exploration-Guided Reward Shaping, that combines bothexploration-focused and exploitation-focused intrinsic reward components.
Our framework of onlinereward shaping alternates between reward learning and policy optimization.
Similar to SORS [26],our method is compatible with any existing RL algorithm, and not only policy-gradient style learnersas considered in [25].
Our main contributions are:
I. We propose a novel framework, Exploration-Guided Reward Shaping (E XP RS), that appropri-
ately balances exploration (via an intrinsic bonus component) and exploitation (via an intrinsic
reward component) of extrinsic signals (Sections 2 and 3.1).
II.
We derive intuitive meta-gradients for updating the intrinsic reward component that enables
our method to be broadly applicable to any RL agent.
We theoretically
showcase the utility of
E XP RS in a chain environment (Sections 3.2 and 3.3).
III.
We empirically demonstrate the effectiveness of E XP RS on several environments with sparseand noisy reward signals (Section 4).
Problem Setup2.1 Preliminaries
An environment is deﬁned as a Markov Decision Process (MDP) M : = ( S , A , T, P 0 , γ, R ) , wherethe state and action spaces are denoted by S and A respectively.
T : S × S ×
A → [0 , 1] capturesthe state transition dynamics, i.e., T ( s (cid:48) | s, a ) denotes the probability of landing in state s (cid:48) by takingaction a from state s .
γ is the discounting factor, and P 0 is the initial state distribution.
The rewardfunction is given by R : S ×
A → [ − R max , R max ] , for some R max > 0 .
We denote the trueunderlying extrinsic reward function by R and the designed reward function by (cid:98) R .
We denote a stochastic policy π : S → ∆ ( A ) as a mapping from a state to a probability distributionover actions, and a deterministic policy π : S
→ A as a mapping from a state to an action.
Forany trajectory ξ = { ( s t , a t ) } t =0 , 1 ,...,H , we deﬁne its cumulative return w.r.t. reward function R as J ( ξ, R ) :
= (cid:80)
Ht
=
γ t · R ( s t , a t ) .
Then, the expected cumulative return (value) of a policy π w.r.t.
R isdeﬁned as J ( π, R ) :
=
E [ J ( ξ, R ) | P 0 , T, π ] , where s 0 ∼ P 0 ( · ) , a t ∼ π ( ·| s t ) , and s t +1 ∼ T ( ·| s t , a t ) .
The learner seeks to ﬁnd a policy that has maximum value w.r.t.
the extrinsic reward function R ,i.e., max π J ( π, R ) .
2.2 Online Reward Shaping
A general framework of online reward shaping for RL agents is given in Algorithm 1.
A naturalobjective here is to design informative rewards (cid:98) R k
at each round k
so that the resulting ﬁnal policyπ K performs better (i.e., has high value w.r.t.
R )
compared to the corresponding policy obtained viathe standard training with (cid:98) R k = R .
Note that, we consider a single lifetime training setting for an RLagent, on a single task, i.e., between rounds we do not reset the policy.
2.3 Existing Techniques and Issues
Reward shaping based on expert domain knowledge.
A popular technique for reward shapingis potential-based reward shaping (PBRS) which guarantees that any optimal policy induced by thedesigned function is also optimal under the extrinsic reward function [3].
Given a potential functionΦ : S → R that quantiﬁes “goodness” of the states, the reward function (cid:98) R k produced by the PBRSframework is given by:
[Equation]
For the above reward function to be helpful for training the RL agent, we need to have access togood potential functions based on expert domain knowledge.
It has already been noted that usingthe wrong potential functions could drastically slow down the learning process [46].
The authorsin [12, 47] have extended the PBRS framework to incorporate dynamic potential functions that arelearned online, alongside agent’s training.
As noted in [26], the effects of dynamic PBRS can be“learned away” over time, as they are equivalent to value function initialization [48].
Reward shaping based on exploration bonuses.
In the bonus-driven exploration framework [32–34], a count-based intrinsic bonus B k ( s ) is given to the agent to encourage exploration.
The bonus
B k ( s ) measures the “novelty” of a state s given the history of all transitions up to round k .
Theauthors in [34] extend the classic exploration methods with count-based intrinsic bonuses [35–38]to high-dimensional, continuous state spaces.
The states are mapped to hash codes, which allowsto count their occurrences with a hash table.
These counts are then used to compute an intrinsicbonus according to the classic bonus-driven exploration theory.
At round k , the agent receives thereward (cid:98) R k
( s, a ) = R ( s, a ) + B k ( s ) . These “exploration-only” reward shaping methods do notappropriately combine the successful extrinsic reward signals received from the environment.
Whenthere are distractive zones in the state space, these methods will keep on exploring the space evenafter obtaining extrinsic reward signals.
Reward shaping in fully self-supervised way.
The self-supervised reward shaping methods [24–26]learn the parameters φ of an intrinsic reward function R φ by exploiting the successful extrinsic rewardsignals received from the environment.
These parameters are learned online to speed up the agent’slearning during its lifetime.
At round k , the LIPRG [25] technique provides the reward (cid:98) R k ( s, a ) =
R ( s, a ) + R φ k ( s, a ) and the SORS [26] technique provides the reward (cid:98) R k ( s, a ) =
R φ k ( s, a ) to theagent.
These self-supervised reward shaping methods lack an explicit explorative component.
When
R is extremely sparse, the learner would not get any extrinsic signal to update the parameters of R φ ,until a ﬁrst successful rollout is realized from π k .
In this paper, we are interested in developing an online reward shaping technique that can succeed inenvironments with extremely sparse and distractive rewards, without any expert guidance.
Exploration-Guided Reward Shaping
In Sections 3.1 and 3.2, we propose an exploration-guided reward shaping technique, E XP RS, toaccelerate an RL agent’s training process.
In Section 3.3, we theoretically compare our rewardshaping technique against baseline techniques in a chain environment.
3.1 Our Reward Formulation
We design the following parametric reward function for Algorithm 1: [Equation] RS [Equation]
φ
[Equation]
where φ ∈ R d φ and w ∈ R d w .
Here, R S ELF RS φ corresponds to the intrinsic rewards in self-supervisedreward shaping techniques, and B E XP w corresponds to the intrinsic bonuses in exploration-only rewardshaping techniques.
For the remainder of the section, we drop the superscripts in the reward/bonusterms.
At round k of Algorithm 1, (cid:98) R k − 1 ( s, a ) is designed with parameters ( φ k − 1 , w k − 1 ) .
Then,given updated policy π k , we update the parameters ( φ k − 1 , w k − 1 ) to new values ( φ k , w k ) .
Intrinsic reward R φ .
We model the intrinsic reward R φ using any general parameterized function.
At round k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k by considering the effectsuch a change would have on the the expected cumulative return (w.r.t. R ) of the learner throughthe change in the policy π k (as in LIPRG [25]).
In particular, we update φ using the gradient 134 (cid:2) ∇ φ J ( L ( π k , (cid:98) R ) , R ) (cid:3) φ k − 1 , where (cid:98) R ( s, a )
=
R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
However, we may nothave access to the learning algorithm L directly or it might be too complex to analyze the impact ofthe change in φ considering L .
Since our end goal is to speed up the learning process of any RL agent,it is actually useful to consider a simple learning algorithm with parametric policies (cid:8) π θ : θ ∈ R d θ (cid:9) .
To this end, we consider a learning algorithm (cid:101)
L that does 1-step vanilla policy gradient update with hdepth planning for Q -values.
In particular, we map the policy π k to a parameter θ k ∈ R d θ and deﬁne:(cid:101) L ( θ k , (cid:98) R ) : = θ k + α · (cid:104)
[Equation]
where Q π θk (cid:98) R,h ( s, a ) =
E (cid:104)(cid:80) ht
=
γ t · (cid:98) R ( s t , a t ) | s 0 =
s, a 0 = a, T, π θ k (cid:105) and α is the learning rate.
Then, we update the parameter φ using the following bi-level optimization:
[Equation]
subject to θ ( φ ) ← (cid:101)
[Equation]
[Equation] [Equation] where (cid:98) R ( s, a ) :
= R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
Note that the above bi-level formulation explicitly accounts for the requirement of acceleratingthe training process of an RL agent, with h -depth planning.
Further, we note that the LIPRG [25]technique is speciﬁcally designed for policy-gradient style RL agents, and their technique isincompatible with value-based RL agents.
Intrinsic bonus B w .
Given a state abstraction ψ : S → X ψ (with |X ψ | = d w ), we maintain thevisitation count of the abstracted states in w , i.e., w [ x ] corresponds to the visitation counts of thestates { s ∈ S : ψ ( s ) = x } .
This allows us to implicitly maintain pseudo-counts N w ( s ) of visitingstates s ∈ S .
In particular, we set N w ( s ) =
(cid:16) λR max (cid:17) 2 + w [ ψ ( s )] for some λ > 0 .
Then, we deﬁne theintrinsic bonus as follows:
B w ( s ) =
λ √ N w ( s ) .
We update w based on the rollouts in round k [32–34].
3.2 Proposed Algorithm
Parameter updates for R φ .
We solve the bi-level optimization problem (P1.U)-(P1.L) of theintrinsic reward component in iterative manner using the gradient updates that we derive below.
Atround k , for ﬁxed π k and w k − 1 , we update the parameter φ k − 1 to φ k as follows:
[Equation]
where η is the learning rate, ( a ) is due to chain rule, and ( b ) is due to the following smoothnesscondition: (cid:13)(cid:13) (cid:13) (cid:2) ∇ θ J ( π θ , R ) (cid:3) [Equation] [Equation] k
[Equation]
[Equation]
(cid:2)
[Equation]
J (
[Equation]
, R ) (cid:3) [Equation] k (cid:13)(cid:13) (cid:13) 2 ≤ (cid:107) θ ( φ k − 1 ) − θ k (cid:107) 2 .
Next, we analyze thetwo gradient terms, [ ∇ φ θ ( φ )] φ k − 1 ∈ R d
φ × d θ and (cid:2) ∇ θ J ( π θ , R ) (cid:3) θ k ∈ R d θ , separately. Below, weuse the following notation: µ π ( s, a ) means s ∼ d π , a ∼ π ( ·| s ) ; similarly, µ π ( s ) means s ∼ d π .
Byusing the policy gradient theorem, we have: [Equation] We obtain the meta-gradient [49–51] as follows:
[Equation] where (cid:98) R ( s, a ) :
= R ( s, a ) + R φ ( s, a ) + B w k − 1 ( s ) .
Here, we aim to derive intuitive simpliﬁcations of the above two gradient terms that enable ourmethod to be applicable to any RL agent, not just policy-gradient style agents (in comparison to
LIPRG [25]).
For the derivation purposes, we further simplify the learner model using tabularrepresentation.
However, the algorithm and updates do not require this.
In particular, we considerθ ∈ R S · A (where S = |S| and A = |A| ), and deﬁne π θ ( a | s ) :=
[Equation]
Note that the reward function R φ is still represented by a parametric function.
For any s ∈ S , a ∈ A ,let 1 s,a ∈ R S · A denote a vector that has one in the ( s, a ) -th entry and zero else where. We deﬁne
A π θk (cid:98) R,h ( s, a ) :
= Q π θk (cid:98) R,h ( s, a ) − V π θk (cid:98) R,h ( s ) and A π θk R ( s, a ) :=
Q π θk R ( s, a ) − V π θk R ( s ) .
Then, we write theabove two gradient terms as follows (with µ = µ π θk ( s, [Equation]
[Equation]
Finally, we get the full update as follows (with η (cid:48)
=
η · α , µ = µ π θk ( s, a ) and µ (cid:48) = µ π θk ( s
[Equation]
, a
[Equation]
[Equation]
Our algorithm with empirical updates.
First, we obtain empirical update rules for intrinsic rewardand bonus components.
To this end, we rely on the rollout data D k collected by executing the learner’scurrent policy π k in the MDP M . Then, based on Eq. [Equation]
we update the parameter φ of the intrinsicreward
[Equation]
for ﬁxed w k − 1 , as follows:
[Equation]
where we absorbed 1 |D k | into η (cid:48)(cid:48) , and ignored the term µ π k ( s i ) . Note that we can compute: (i)π k ( a i | s i ) using the learner’s policy π k , (ii) A π k R ( s i , a i ) using J ( ξ i , R ) or a critic corresponding to R , and (iii)
A π k (cid:98) R,h ( s i , a i ) using J ( ξ i , (cid:98) R ) or a critic corresponding to (cid:98) R .
When we use1 -depth planning for Q -values in (cid:101) L , we have: (cid:104) [Equation] [Equation]
∈A
[Equation]
k
[Equation] b | s i ) · ∇ φ R φ ( s i , b ) (cid:3) φ k − 1 .
Now, we update the parameter w of the intrinsic bonus, forﬁxed φ k − 1 , as follows (component-wise):
[Equation]
Then, we have B w k = λ √ N wk ( s ) , where N w k ( s ) = (cid:16) λR max (cid:17) 2 + w k [ ψ ( s )] . Finally, we obtain ourdesigned reward function at round k as follows: (cid:98) R k ( s, a ) =
R ( s, a ) + R φ k ( s, a ) + B w k ( s ) . Thecomplete procedure of our exploration-guided reward shaping is given in Algorithm 2. 1: Input: learning rates { η (cid:48)(cid:48) } , parameter λ , and reward update frequency f 2: Initialization: intrinsic parameters ( φ 0 , w 0 ) , and policy π3: collect rollouts using π 0 in M and store them in D4: for k = 1 , 2 , . .
, K do 5: deﬁne reward (cid:98) R k − 1 ( s, a ) :
= R ( s, a ) + R φ k − 1 ( s, a ) + B w k − 1 ( s ) 6: update policy π k ← L ( π k − 1 , (cid:98) R k − 1 ) using D k −7: collect rollouts using π k in M and store them in D k 8: if k % f = 0 then 9: update φ k according to Eq. (2) using D k 10: update w k according to Eq. (3) using D k 11: else 12: φ k ← φ k −13: w k ← w k −14: Output: policy π K
3.3 Theoretical Analysis
In this section, we theoretically showcase the utility of our exploration-guided reward shaping overfully self-supervised and exploration-only shaping techniques.
In particular, we show that our shapingtechnique helps to learn an optimal policy in a sample efﬁcient manner in an environment with bothextremely-sparse and distractive rewards.
Environment.
We consider a chain environment M = (cid:0) S , A , T, P 0 , γ, R (cid:1) of length n 1 + n 2 + 1 .
Let the state space be S = (cid:8) x − n 2 , x − ( n 2 − 1) , . . .
, x − 1 , x 0 , x 1 , . . .
, x ( n 1 − 1) , x n 1 (cid:9) , and the actionspace be A = {← , →} .
We always start (or reset) in the state x 0 , i.e., the initial state distributionis P 0 ( x 0 )
= 1 .
The transition dynamics is deterministic and given by: T ( x i +1 | x i , → ) = 1 for− n 2 ≤ i ≤ n 1 − 1 , T ( x i − 1 | x i , ← )
= 1 for − ( n 2 − 1) ≤ i ≤ n 1 , T ( terminal | x n 1 , → ) = 1 ,and T ( terminal | x − n 2 , ← )
= 1 .
The reward function is deﬁned as follows:
R ( x i , → ) = 0 for− n 2 ≤ i ≤ n 1 − 1 , R ( x n 1 , → ) = 1 , and R ( x i , ← ) = 0 for − n 2 ≤ i ≤ n 1 .
We consider an inﬁnitehorizon setting with discounted returns ( H → ∞ , and γ < 1 ).
Learning algorithm. For our theoretical analysis of different reward shaping techniques, we considera simpliﬁed Q -learning style RL algorithm L given in Algorithm 3.
The algorithm L takes two Boolean ﬂags, no-self and no-exp , as input that correspond to the intrinsic reward and bonuscomponents, R and B , respectively.
For example, L ( true , true ) corresponds to the setting oflearning with extrinsic reward R only.
The following theorem compares the sample complexity of different reward shaping techniques tolearn an optimal policy in the chain environment.
Theorem 1. Consider the chain environment M deﬁned above, and the learning algorithm Lgiven in Algorithm 3. Let cost ( L ( no-self , no-exp )) denote the number time steps required for
L ( no-self , no-exp ) to learn an optimal policy in M . Then, we have the following: without any reward shaping, E [ cost ( L ( true , true ))] ≥ (cid:80) n 1 i =1 2 n 1 − i with exploration-only reward shaping, cost ( L ( true , false ))
=
n 1 · ( n 1 + n 2 + 2)3.
with fully self-supervised reward shaping, E [ cost ( L ( false , true ))] ≥ 2 n 1 − 14.
with exploration-guided reward shaping, cost ( L ( false , false )) ≤ n 1 + n 2 +
1: Input: Boolean ﬂags no-self and no-exp 2:
Initialize: V 0 ( s ) = 0 ; R ( s, a ) = 0 , B [-w k +
-]( s )
= 1 , ∀ s ∈ S , a ∈ A ; λ ∈ (0 , 1) 3: s 1 = x 0 ; B ( s 1 ) =[-(cid:16) λB max (cid:17) 2 + w k +
-]
λ 4: for each t = 1 , 2 , . . . do 5: if no-exp = true then 6:
B ( s ) = 0 , ∀ s ∈ S 7: a t = arg max a (cid:48) R ( s t , a (cid:48) )
+ R ( s t , a (cid:48) ) + B ( T ( s t , a (cid:48) )) + γ · V t − 1 ( T ( s t , a (cid:48) )) 8: s t
+
=
[-R ( s, a ) + R φ k +1 ( s, a ) + B w k +
-]T ( s t , a t ) 9:
V t ( s t ) = R ( s t , a t ) + R ( s t , a t ) + γ · V t − 1 ( s t +1 ) 10: if s t +1 = terminal then 11: if R ( s t , a t ) = 1 and no-self = false then 12: φ ( s )
= 0 , ∀ s ∈ S 13: for all the states in the current rollout update φ ( · ) as the discounted return 14: R ( s, a ) = γ · φ ( T ( s, a )) − φ ( s ) , ∀ s ∈ S , a ∈ A 15:
V t ( s ) = 0 , ∀ s ∈ S 16: reset s t +1 = x17:
B ( s t +1 ) =
λ · B ( s t +1 ) 18: Output: policy π t
Experimental Evaluation
In this section, we evaluate our reward shaping framework on three environments: C HAIN (Section 4.1), R OOM (Section 4.2), and L INE K (Section 4.3).
C HAIN corresponds to a navigation task ina chain, adapted from the environment used for theoretical analysis in Section 3.3; this is a canonicalenvironment used for studying extremely sparse-reward settings [7].
R OOM corresponds to a navigation task in a grid-world where the agent has to learn a policy to quickly reach the goal location in oneof four rooms, starting from an initial location.
Even though this environment has a small state/actionspace, it provides a very rich and intuitive problem setting to validate different reward shaping techniques.
In fact, variants of R OOM have been used extensively in the literature [10, 11, 14, 17, 52–55]—the environment used in our experiments is adapted from [55].
L INE K corresponds to a navigationtask in a one-dimensional space where the agent has to ﬁrst pick the key and then reach the goal.
Theagent’s location is represented as a point on a line segment.
This environment is inspired by variantsof navigation tasks in the literature where an agent needs to perform subtasks [3, 55, 56]—the environment used in our experiments is adapted from [55].
We give an overview of main results here, andprovide a more detailed description of the setup and additional results in the supplementary material.
4.1 Evaluation on C HAIN
C HAIN (Figure 1).
We represent the chain environment oflength n 1 + n 2 + 1 as an MDP with state-space S consistingof an initial location x 0 (shown as “blue-circle”), n 1 nodesto the right of x 0 , and n 2 nodes to the left of x 0 .
Therightmost node of the chain is the “goal” state (shown as“green-star”).
In the left part of the chain, there can be a “distractor” state (shown as “green-plus”).
The agent can take two actions given by A :
= { “left” , “right” } .
An action takes the agent to theneighboring node represented by the direction of the action.
However, taking “left” action at theleftmost node (shown as “thick-red-circle”) leads to termination, and “right” action at the rightmostnode (goal) keeps the agent at the current location.
Furthermore, when an agent takes an actiona ∈ A , there is p rand probability that an action a (cid:48) ∈ A \ { a } will be executed instead of a .
The agentreceives a reward of: R max for the “right” action at the goal state, R dis for the “left” action at thedistractor state, and 0 for all other state-action pairs.
There is a discount factor γ and the environmentresets after a horizon of H .
In our evaluation, we set p rand = 0 .
05 , R max = 1 , R dis = 0 or 0 . H = 30 , and γ = 0 .
We obtain different variants of the chain environment by changing thevalues of ( n 1 , n 2 , R dis ) . In particular, C HAIN 0 denotes a set of two environments with the followingconﬁgurations: { (15 , 20 , 0) , (20 , 20 , 0) } . Similarly, C HAIN + denotes a set of two environments withthe following conﬁgurations: { (15 , 20 , 0 . 01) , (20 , 20 , 0 . 01) } . [Figure]
a [Figure] Evaluation setup.
We conduct our experiments with two different types of RL agents [7]: First, weconsider tabular REINFORCE agent that maintain scores θ [ s, a ] for each state-action pair and appliessoft-max operation over the scores to obtain the policy; for the agent’s updates, we set learning rateα = 0 . Second, we consider Q-learning agent with learning rate α = 0 . 1 , and exploration factor(cid:15) = 0 .
We compare the performance of the following reward shaping techniques: (i) (cid:98) R O RIG :=
R ,which represents the extrinsic reward function; (ii) (cid:98) R S ELF RS := [Equation] RS φ , where R S ELF RS φ islearned with a tabular parameterization φ along with a critic w.r.t.
R (see Section 3.1); (iii) (cid:98) R E XP RS := R + R S ELF RS φ + B E XP w , where B E XP w is a count-based bonus (see Section 3.1); (iv) (cid:98) R SORS is obtained viathe SORS technique [26]. Next, we discuss details of the full training process.
We maintain a rolloutbuffer (ﬁrst-in-ﬁrst-out) D of size 10 . For every 5 rounds (corresponding to 5 rollouts in the MDP),we update the intrinsic reward using all rollouts in the buffer D .
For every 2 rounds, we update theagent’s policy using the latest 5 rollouts in D . We update the intrinsic bonuses at every environmentstep during a rollout. Note that, for stability, we update the policy more frequently than the intrinsicreward and at a higher learning rate, as considered in the work of [25, 26].
In the supplementarymaterial, we have provided full details of technique-speciﬁc hyperparameters and learning rates. Results.
During training, the agent receives rewards based on (cid:98) R and is evaluated based on R .
Figure 2 illustrates results for both variants of C HAIN environment; the reported results are averagedover 30 runs and convergence plots show the mean performance with standard error bars.
Theconvergence behavior in Figure 2 demonstrates the effectiveness of our exploration-guided rewardshaping framework. In particular, the RL agents (both REINFORCE and Q-learning) trained with(cid:98)
R E XP RS outperforms all other techniques in both C HAIN 0 and C HAIN + environments. Speciﬁcally, weattribute the following reasons for the poor performance of SORS technique: (i) it is not speciﬁcallydesigned for environments with extremely-sparse rewards; (ii) it is agnostic to the scale of the extrinsicreward function as it only uses the information about pairwise labels.
4.2 Evaluation on R OOM
OOM +
R OOM (Figure 3).
The environment used in our experiments is basedon the work of [55]; however, we adapted it to have a “distractor” state(shown as “green-plus”) that provides a small reward.
Similar to thetwo variants of C HAIN , we have two variants of this environment: (a) R OOM 0 has R dis = {+
+}
at the distractor state shown as “green-plus”(equivalently, there is no distractor state); (b) R OOM + has R dis = 0 .
01at the distractor state.
The environment-speciﬁc parameters (includingp rand , R max , H , γ ) are kept same as in Section 4.1.
We defer the fulldetails of the environment to the supplementary material.
Evaluation setup and results.
Our evaluation setup for this environment is the same as the C HAINenvironment (described in Section 4.1); here, we consider only the tabular REINFORCE agent.
Inparticular, all the hyperparameters (related to the REINFORCE agent, reward shaping techniques,and training process) are the same as in Section 4.1.
Figures 4a and 4b show the agent’s performancefor environments R OOM 0 and R OOM + (averaged over 20 runs).
These results, along with results
10 Episode ( [Figure] [Figure]
E xp ec t e d r e w a r d
E XP RS S ELF RS
SORS O RIG
[Figure] REINFORCE [Figure] [Figure]
E xp ec t e d r e w a r d
E XP RS S ELF RS
SORS O RIG
[Figure] REINFORCE  10 Episode (x 10 4 )
E xp ec t e d r e w a r d
E XP RS S ELF RS
SORS O RIG
[Figure] 10 Episode (x 10 4 )
E xp ec t e d r e
[Figure]
a r d E XP RS S ELF RS
SORS O RIG
[Figure] obtained in further and , compared to
4.3 Evaluation on L INE K
L INE K (Figure 5).
This environment corresponds to a navigationtask in a one-dimensional space where the agent has to ﬁrst pickthe key and then reach the goal.
The environment used in ourexperiments is adapted from the work of [55]. In particular,
to
Based on the setting considered in [55], use an state space in our experiments obtained via discretization of the segment which is then passed in bitmap vector representation to neural networks.
We defer full details of the environment to the supplementary material.
Experimental setup and results.
We conduct our experiments with a neural REINFORCE agentusing a two-layered neural network architecture.
As before, we compare the performance of fourreward shaping techniques.
As a crucial difference to experiments in Sections 4.1 and 4.2, herewe use neural-network based reward functions for (cid:98) R S ELF RS , (cid:98) R E XP RS , and (cid:98) R SORS .
These networksuse a single non-linear layer that applies tanh -clipping for the output reward values (as consideredby [25, 26].)
Figures 4c and 4d show the agent’s performance for environments R OOM 0 and R OOM +(averaged over 5 runs).
These results showcase the effectiveness of (cid:98) R E XP RS in more complex learningsettings, requiring neural representations for reward functions.
We provide additional results in thesupplementary material.
Concluding Discussions
We proposed a novel reward shaping framework that operates in a fully self-supervised manner andcould accelerate an agent’s learning even in sparse-reward environments.
Our framework is basedon combining exploration-based intrinsic bonuses along with extrinsic reward signals to acceleratethe agent’s training process.
Experimental evaluation in different environments demonstrated theeffectiveness of our reward shaping framework. Next, we discuss a few limitations of our work andoutline a future plan on how our work can be extended to address them.
First, it would be interesting toextend the experimental evaluation to more complex environments (e.g., with continuous state/actionspaces).
Second, it would be useful to provide rigorous analysis of our reward shaping frameworkin terms of convergence speed and stability of an agent. Third, it would be useful to systematicallystudy our reward design framework across agents trained using different methods.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] The paper is organized according to the contributions
listed at the end of the introduction section.
(b) Did you describe the limitations of your work?
[Yes] In Section 5, we discuss thelimitations of our work and outline a future plan to address these limitations.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This
work presents a new online reward shaping technique for reinforcement learning agents.
Given the algorithmic and empirical nature of our work applied to learning agents, we
do not foresee any direct negative societal impacts of our work in the present form.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We conﬁrm that our paper conforms with the ethics review guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions
are stated either as part of the theorem statements or the proofs.
(b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs of
all theoretical results are included in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code and
instructions are included in the supplementary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Details are provided in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Error bars are included in all the result graphs
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Details are provided in the supplementary material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]