Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity
Anonymous Author(s)
AffiliationAddress email
Abstract
The success of reinforcement learning in a variety of challenging sequentialdecision-making problems has been much discussed, but often ignored in thisdiscussion is the consideration of how the choice of reward function affects the behavior of these algorithms.
Most practical RL algorithms require copious amountsof reward engineering in order to successfully solve challenging tasks.
The idea ofthis type of “reward-shaping” has been often discussed in the literature and is usedin practical instantiations, but there is relatively little formal characterization ofhow the choice of reward shaping can yield benefits in sample complexity for RLproblems.
In this work, we build on the framework of novelty-based exploration toprovide a simple scheme for incorporating shaped rewards into RL along with ananalysis tool to show that particular choices of reward shaping provably improvesample efficiency.
We characterize the class of problems where these gains areexpected to be significant and show how this can be connected to practical algorithms in the literature.
We show that these results hold in practice in experimentalevaluations as well, providing an insight into the mechanisms through which rewardshaping can significantly improve the complexity of reinforcement learning whileretaining asymptotic performance.
Introduction
Reinforcement learning (RL) in its most general form presents a very difficult optimization problem:when there are no constraints on the reward function or dynamics, a learning algorithm may need toexhaustively explore the entire state space to discover high-reward regions.
Naïve algorithms thatrely entirely on random exploration are known to be exponentially expensive [19, 2] and much of thetheoretical work on efficient RL algorithms has focused smarter exploration strategies that aim to moreefficiently cover state space, typically in time polynomial in the state space cardinality [5, 7, 17, 46,20, 25, 43].
Much of this work is based on upper confidence bound (UCB) principles and prescribessome kind of exploration bonus to prioritize exploration of rarely visited regions.
Analogous strategieshave also been employed in a number of practical RL algorithms [32, 31, 9, 11, 15, 27, 34, 33, 28].
However, surprisingly, much of the empirical work on reinforcement learning does not make use ofexplicit exploration bonuses or other dedicated exploration strategies, despite numerous theoreticalresults showing them to be essential to attain polynomial sample complexity.
Instead, practitioners often incorporate prior knowledge of each task into designing or shaping the reward function [24,29, 30, 35, 4, 10, 41, 45], preferring this heuristic approach over the more principled explorationstrategies.
At this point, one may wonder why is reward shaping often practically preferable to
A likely answer to this question lies in the fact that even the best general-purpose explorationstrategies still require visiting every state in the MDP at least once in the worst case.
This is ofcourse unavoidable without further assumptions.
However, in practice, sample complexity that ispolynomial in the size of the entire state space might still not be practical, and hence prior knowledgein the form of reward shaping is required to render such tasks tractable for RL.
Surprisingly, despitethe widespread popularity of reward shaping in RL applications, the analysis of reward shaping hasremained limited to proving policy invariance [24] or largely empirical observations.
In this work,we take a step towards studying the effect of reward shaping on the efficiency of RL algorithms, byasking the following question:
Can we theoretically justify the sample complexity benefits that reward shaping from prior domainknowledge can provide reinforcement learning?
In this work, we aim to provide a set of tools that formally analyze how reward shaping can improvethe complexity of tabula-rasa RL and better direct. exploration.
To perform this analysis, we firstpropose a simple RL algorithm “UCBVI-Shaped” that incorporates shaped rewards into optimismbased exploration.
We then provide a regret analysis framework that studies how this introductionof shaped rewards can (in certain cases) provide much more directed optimism than uninformedexploration, while maintaining asymptotic performance.
In our analysis, we specifically consider problems where shaping is provided through a term (cid:101) V , a(potentially suboptimal) approximation of the optimal value function V ⋆ .
In particular, we assumethat the shaping (cid:101) V is a multiplicatively bounded approximation of the optimal value function V ⋆ , i.e(cid:101) V ( s ) ≤
V ⋆ ( s ) ≤ β (cid:101) V ( s ) ∀ s for a finite multiplicative factor β .
This type of shaped reward function (cid:101) Vcan be incorporated into a standard RL algorithm like UCBVI [7] through two channels - (1) bonusscaling: simply reweighting a standard, decaying count-based bonus 1 √ N h ( s , a ) by the per-state rewardshaping and (2) value projection: adaptively projecting learned value functions into ranges of valuefunctions derived from the reward shaping.
We show that this relatively simple algorithmic instantiation provides significant complexity benefitswith shaping.
Intuitively, the key pieces in our complexity analysis of UCBVI-Shaped are two-fold:1)
The multiplicative sandwich condition (via β ) between (cid:101) V and V ⋆ allows for the bonus scaling todepend on β (cid:101) V instead of a coarse approximation of V ⋆ such as H .
This allows for a reduction ofcomplexity from a horizon H dependent term to one scaling with β (cid:101) V ≤ β V ⋆ by allowing for a fasterdecay of the exploration bonuses while still providing enough optimism; 2)
The projection of thevalue function prevents “over optimism” by hastening the convergence of the empirical (cid:98) Q functionsduring value iteration thus allowing for faster detection of sub-optimal actions.
This results in theability to prune out large parts of the state space, as we also validate empirically.
Related Work
Regret Analysis in Finite Horizon Episodic Tabular MDPs Recent research on regret analysis hasstudied both model-based and model-free RL methods.
Model-based methods [16, 3, 7, 14, 13, 26]first learn a model from previous experience and use the learned model for future decision making.
In contrast, model-free methods [17, 8, 46, 37, 23, 20] aim to learning the value function without themodel estimation stage and use the learned value function for decision making.
Our analysis lies inthe model-based framework and is similar to the setting of [7] but with additional assumptions onknowing (cid:101) V as a multiplicatively bounded approximation of V ⋆ .
The main difference between ourresults and the aforementioned works is that, the truncated interval assumption can reduce the statedependency S to the some smaller “effective state space” and reduce horizon dependency.
Regret Analysis with Linear Function Approximation A recent line of literature has investigatedthe linear function approximation setting by assuming the transition kernels or the value functionscan be represented by d -dimensional linear features [18, 44, 6, 47, 12, 39, 38, 36] or even generalfunction classes [40].
With the aforementioned assumptions, the regret analysis will only dependon the ambient dimension d (or other intrinsic complexity measure), instead of S , A in the tabularsetting [36, 39, 12, 47, 6, 43, 18], which could greatly decrease the complexity of learning.
Instead of posing structural assumptions on the function class representing the MDP’s values or dynamics, weask the question of whether having approximate knowledge of V ⋆ can improve the speed of learning.
Practical Reward Design and Reward Shaping Ng et al.
[24] proposed a potential-based shapingfunction F that ensures policy invariance under transformation.
However, unbiased potential basedreward shaping is rarely used in practice.
In most applications, heuristic reward design is carefullyperformed with potentially biased reward functions.
In some large-scale practical RL tasks [10,41, 42], the reward functions are heavily handcrafted based on prior knowledge.
Besides rewardengineering, a distinct line of work applies uninformed exploration algorithms like count-based rewardshaping or intrinsic rewards to encourage greater state visitation [32, 31, 9, 11, 15, 27, 34, 33, 28, 21].
Importantly, these methods optimize for the worst case, as they try to cover all states since the exactlocation of the reward is unknown.
In contrast, we look at the problem of incorporating domainknowledge via reward shaping into the exploration process.
Overview
We consider an episodic Markov Decision process M =
( S , A , P ⋆ , r , H ) where S corresponds tothe state space,
A is the action space, P ⋆ is the transition operator, r : S ×A → [ 0 , 1 ] is the rewardfunction and H is the problem horizon.
We use |S| , |A| to denote the number of state and actions,and we use P ⋆ ( ·| s , a ) to denote the transition probability of state action pair ( s , a ) .
The value function
V π ( s 0 ) of a policy π starting at an initial state s 0 is defined as V π ( s 0 ) :
= E π (cid:2)
∑ Hh = 0 r ( s h , a h ) (cid:3) , where 102 E π denote the transition dynamic of M under π .
Similarly, V ⋆ ( s 0 ) is the value function of the optimalpolicy π ⋆ .
We consider a sequential interaction between a learner and the MDP M occurring inrounds indexed by t ∈ N .
At the start of round t the learner selects a policy π t that is used to gather asample trajectory from M .
As is standard in the literature, we measure the learner’s performance upto round T by Regret ( T ) :
= ∑ Tt = 1 V ⋆ ( s 0 ) − V π t ( s 0 ) .
Our objective is to show that knowing a reward shaping term (cid:101) V allowsfor the design of a simple algorithm,that with high probability having asublinear regret upper bound with aleading term that depends on an “effective state space” of M which isdetermined by the quality of a rewardshaping term (cid:101) V and the nature of theparticular MDP being solved.
Thispruned effective state space can bemuch smaller than |S| .
Additionally,we will show an improved horizon dependence as the shaping term (cid:101) V allows for the bonus terms tobe smaller and therefore decay faster thus replacing horizon factors of H with β (cid:101) V ones.
To more intuitively understand how reward shaping can allow for the consideration of a reducedeffective state space, consider the maze environment in Fig.
1 with agent starting in the middle.
Without knowing where the goal is, an uninformed exploration algorithm needs to explore nearly theentire maze.
However, an effectively incorporated reward shaping term (cid:101) V would allow an algorithmto prune out the entire left half of the grid, effectively halving the effective state space size.
This canhappen in two ways —firstly if the shaping can directly indicate that certain actions are sub-optimalthen entire regions of the space can be eliminated from search.
Secondly, even if states are noteliminated, if their corresponding bonuses are scaled according to their shaped value (cid:101) V from theshaping rather than uniformly with H , it limits unnecessary exploration towards suboptimal states.
Based on this intuition, we introduce UCBVI-Shaped, a modification of the UCBVI algorithm
[7]that uses an additional reward shaping term (cid:101) V in two ways: (1) Bonus scaling: (cid:101) V is used to provide anexploration bonus that combines inverse state visitation counts with reward shaping.
This allows forthe reduction of overoptimism and the dropping of the horizon H dependence (2) Value projection:the shaping term is used to clip the empirical value function (cid:98) V h ( s ) , which enables pruning unnecessary elements of the state space and allows for a bound that depends only on the effective state space size.
Informally, our main result for UCBVI-Shaped can be summarized as follows:
Theorem 3.1 (Main Informal) .
With probability at least 1 − δ , the regret of UCBVI - Shaped satisfies
[Equation]
Here, S remain corresponds to the set of states in M where the information contained in (cid:101) V was notenough to rule them out, and Regret − UCBVI ( S remain A , T ) is a UCBVI regret upper bound functionevaluated on S remain and A .
B ( (cid:101) V , M ) is a time-agnostic complexity measure that depends solely onthe M and the quality of the shaping (cid:101) V.
We will define S remain and B ( (cid:101) V , M ) and describe their relationship to (cid:101) V precisely in Section 5.
If
S remain ≪ S , the regret of UCBVI - Shaped can be substantially smaller than Regret − UCBVI ( S , A ) ,particularly since the first term of the regret grows logarithmically while the second scales with √ T .
We state our assumptions next
Assumption 1.
The quality of the shaping term (cid:101) V is described by a parameter β .
We assume accessto shaping “value function” estimators (cid:101)
V h : S
×A → R such that, (cid:101) V h ( s ) ≤ V ⋆ h ( s )
≤ β (cid:101) V h ( s ) , for alls ∈ S [-,h 
-], h ∈
[ H ] and for some β ≥ 1 .
Assumption 2.
We assume the reward functions satisfy r ( s , a ) ∈ [ 0 , 1 ] , ∀ ( s , a ) ∈ S ×A .
Assumption 3.
The states S are h − indexed i.e. the states reachable at time
h [-
-]∈
[ H ] are disjoint fromthe states reachable at time h [-0 
-]′ ∈
[ H ] when h ̸ = h ′ .
Contributions.
Our analysis introduces the notions of pseudosuboptimal and pathpseudosuboptimal states to quantify the “effective” size of the state space as a function ofthe quality of the shaping term (cid:101) V , and we use these notions to show how UCBVI-Shaped can attainsignificantly improved regret rates.
We further show that incorporating shaping into the explorationbonus term improves the horizon-dependence in the bound.
The UCBVI-Shaped Algorithm
To perform analysis of reward shaping, we build on the framework of the UCBVI algorithm [7].
This forms a convenient base algorithm for incorporating shaped rewards in a way that admits fasterlearning while maintaining asymptotic performance (as we will discuss in Section 5). Our algorithm,
UCBVI-Shaped, is a combination of two changes to the upper confidence bound algorithm - first wemodify the bonus scaling to depend on (cid:101) V and second, we introduce a projection subroutine into valueiteration, that is implemented as a standard clipping operation.
▷ Bonus scaling
As in standard UCBVI, we define N th ( s , a ) to be the visitation count for the state-action pair ( s , a )at iteration t − 1 for horizon h : N th ( s , a ) :
= ∑ t − 1 i = 0 1 (cid:8) ( s ih , a ih ) =
( s , a ) (cid:9) .
Similarly to N th ( s , a ) , we use
N t h ( s , a , s ′ ) :
= ∑ t − 1 i = 0 1 (cid:8) ( s i h , a i h , s i h + 1 ) =
( s , a , s ′ ) (cid:9) as the visitation count of state-action pair ( s , a ) andthe subsequent state s ′ .
We then use (cid:98) P t ( s ′ | s , a ) :
= N th ( s , a , s ′ ) N th ( s , a ) , ∀ h , s , a , s ′ to denote the empirical transition kernels at iteration t .
UCBVI uses value iteration with the empirical transition function (cid:98) P t and areward function augmented with an exploration bonus, given by r h + b th .
This is defined as a dynamicprogramming procedure that starts at H and then proceeds backward in time to h = 0, updatingaccording to Algorithm 1 and 2.
The key algorithmic changes between UCBVI and UCBVI-Shapedare highlighted in blue - (1) scaling bonus b th by the shaping term (cid:101) V and (2) projecting the empiricalvalue function (cid:98)
V th ( s ) by an upper bound based on the sandwiched shaping, β (cid:101) V ( s , a ) .
[Table]
▷
[Table]
Analyzing UCBVI-Shaped
In this section, we will derive our main result on the sample complexity of the UCBVI-Shapedalgorithm, and along the way introduce pseudosuboptimal and path-pseudosuboptimal states as a toolfor deriving bounds that depend only on the “effective” state space as determined by the providedreward shaping.
We will first introduce some notation to use in our analysis.
As described in theprevious section, UCBVI-Shaped proceeds in rounds.
At the beginning of round t , the learner hasaccess to an empirical model (cid:98) P t built from the data collected up to iteration t − 1.
The bonus termswe consider are built by taking the empirical second moment of (cid:101) V .
This is related to the definition ofbonus_2 in [7].
Importantly, the empirical value functions (cid:98) V th are clipped above by β (cid:101)
V h .
[Equation]
Where (cid:101) V max h = max s ′ (cid:101)
V h ( s ′ ) and (cid:101) V max = max s ′ , h ′ (cid:101) V h ′ ( s ′ ) .
Despite clipping and a modified bonus term, the (cid:98) Q and (cid:98) V values of UCBVI-Shaped are optimistic:
Lemma 5.1 (Optimism) .
With probability at least 1 − δ we have
[Equation]
and
[Equation]
for all t ∈ N , where (cid:98) V t h is computed via Algorithm 2.
The proof of Lemma 5.1 can be found in Appendix A.3.
Optimism (Lemma 5.1 ) and the simulation
Lemma [1] implies that:
[Equation]
We now turn our attention to characterize the information contained in (cid:101) V .
5.1 Pruning of the State Space
Consider the following “surrogate” Q functions, (cid:101) Q lh , (cid:101) Q uh : S ×A → R induced by (cid:101) V via:
[Equation]
By Assumption 1, we can box Q ⋆ as Q ⋆ h ( s , a ) ∈ (cid:104) (cid:101) Q lh ( s , a ) , (cid:101) Q uh ( s , a ) (cid:105) .
The basis of our results is thefollowing observation.
If an action a satisfies (cid:101) Q uh ( s , a ) <
V ⋆ h ( s ) for state s , then a is a suboptimal action for state s .
Given a parameter ∆ > 0, we say that an action a is ∆ − pseudosuboptimal 2 for states if V ⋆ h ( s ) ≥ ∆ + (cid:101) Q uh ( s , a ) .
We denote the set of ∆ − pseudosuboptimal state action pairs as:
[Equation]
The intuition we want to capture is that all states ˜ s that can be accessed only through traversing astate action pair in PseudoSub ∆ , can be safely ignored, because we can determine their suboptimality,as soon as the state action pairs (or set of state action pairs) in PseudoSub ∆ that lead to ˜ s are visitedenough.
Now we define the set of ∆ -path-pseudosuboptimal states as the states that can be reached only bytraversing a ∆ − suboptimal state action pair:
PathPseudoSub ∆ = { s ∈ S s.t.
all feasible paths from initial states to s intersect PseudoSub ∆ } .
We will show that states in PathPseudoSub ∆ will not be explored by UCBVI-Shaped after a fewiterations.
To provide intuition of why this is the case,note the support of the optimal policydistribution does not contain any state in PathPseudoSub ∆ for all ∆ > 0, i.e., Support ( d π ⋆ ( s , a ))
∩
PathPseudoSub ∆ = /0 , ∀ ∆ > 0.
For any state action pair ( s , a ) define its set of neighboring states Neighbor ( s , a ) as the set ofstates with nonzero probability in P ⋆ ( ·| s , a ) .
By definition of PathPseudoSub ∆ , for all ( s , a ) [-
-]∈
S\ (( PathPseudoSub ∆ ×A ) ∪ PseudoSub ∆ ) , we have:
[Equation]
In other words, the neighborhood of any state action pair ( s , a ) whose state is not in PathPseudoSub ∆and such that ( s , a ) is not ∆ − pseudo-suboptimal, cannot be in PathPseudoSub ∆ .
We also introducethe notion of Boundary pseudosuboptimal state action pairs to capture the set of state action pairsthat are ∆ − suboptimal but whose states are not in PathPseudoSub ∆ .
BoundaryPseudoSub ∆ =
{ ( s , a ) ∈
PseudoSub ∆ and s ̸∈ PathPseudoSub ∆ } . One of these states has to be traversed by any trajectory that contains any state in PathPseudoSub ∆ .
Although we only use (cid:101) Q u in the definition of PseudoSub ∆ ,
PathPseudoSub ∆ , and
BoundaryPseudoSub ∆ , the size of these sets is modulated by the scale of β and the width ofthe intervals (cid:104) (cid:101) Q l ( s , a ) , (cid:101) Q u ( s , a ) (cid:105) .
We will show that (in the notation of Theorem 3.1) S prunned ≈
S\ PathPseudoSub ∆ .
With this notation, the formal version of our main results is stated as follows.
Theorem 5.2.
With probability at least 1 − 3 δ , the regret of UCBVI-Shaped is upper bounded by
T
[Equation]
for all T ∈ N .
Theorem 5.2 instantiates the desiderata of Theorem 3.1.
Although this regret upper bound cannotbe decomposed into a sum of two term as in Theorem 3.1.
For any fixed ∆ , the regret upper bounddoes have two components, one where S remain can be identified with |S\ PathPseudoSub ∆ | and asecond one that scales logarithmically in T .
In the next section we flesh out the steps in the proofof this result.
The full argument can be found in Appendix A.
The bonus scaling also allows us toameliorate the horizon dependence of the upper bound.
Instead of obtaining a H 2 dependence asthe bound of theorem 7.1 in [1], the first term depends on β H (cid:101) V max ≤ β H max s V ⋆ ( s ) .
Notice thatthe state dependence in the second term may be only be of order log ( |S| ) if there is a ∆ such that| BoundaryPseudoSub ∆ | ≪ |S| (albeit at the cost of a quadratic dependence on 1 / ∆ 2 ).
Setting ∆ to avalue smaller than the minimum gap recovers the UCBVI regret upper bounds Theorem 7.1 in [1]showing Theorem 5.2 is a strict generalization.
5.2 Proof Intuitions and Sketch for Theorem 5.
Improved Horizon Dependence.
An empirical Bernstein bound shows that adding a bonus scaling(up to low order terms) with O (cid:16)(cid:113) (cid:99)
Var s ′ ∼ (cid:98) P t ( ·| s , a ) ( V ⋆ h + 1 ( s ′ ) | s , a ) ln 2 | S ||
A | t δ / N th
( s , a ) (cid:17) is sufficient toensure optimism.
Since V ⋆ is not known, this variance term can be substituted by the empiricalsecond moment of β (cid:101)
V h
+ 1 .
Finally the scaling of these terms can be upper bounded by β (cid:101) V max .
Withoutknowledge of (cid:101) V , achieving this scaling would prove more challenging, since the only proxy for V ⋆ h + 1available is (cid:98) V th + 1 or H both of which may vastly overestimate it.
State Pruning.
The value function clipping mechanism ensures that E s [-
 ⇠ b [Equation] th-]′ ∼ (cid:98) P h ( ·| s , a ) (cid:104)(cid:98)
V th + 1 ( s ′ )
(cid:105)
≤β E s ′ ∼ (cid:98) P h ( ·| s , a ) (cid:104)(cid:101) V h
+ 1 ( s ′ ) (cid:105) and therefore the empirical gap between (cid:98) Q th ( s , a ) and V ⋆ ( s ) is decreasingat a rate of at most O (cid:16) β (cid:101) V max / (cid:112)
N th ( s , a ) (cid:17) .
Since optimism ensures that (cid:98) Q th ( s ,
π ⋆ ( a )) ≥ V ⋆ ( s ) ,once (cid:98) Q ( s , a ) <
V ⋆ ( s ) , action a will not be pulled anymore.
Thus for any ∆ , the number oftimes any state action pair in PseudoSub ∆ may be visited by UCBVI-Shaped is upper boundedby β 2 (cid:16) (cid:101) V max (cid:17) 2 / ∆ 2 .
Since any visit to a state action pair in PathPseudoSub ∆ × A ∪ PseudoSub ∆requires a visit to a state in BoundaryPseudoSub ∆ , this allows us to bound the total number ofvisits to a state (or a trajectory containing such a state) in PathPseudoSub ∆ ×
A ∪ PseudoSub ∆ by
H | BoundaryPseudoSub ∆ | β 2 (cid:16)(cid:101) V max (cid:17) 2 / ∆ 2 .
Finally, we can split the regret decomposition of Eqn.
3into two sums, one over state action pairs in U = PathPseudoSub
∆ ×A∪ PseudoSub ∆ (or trajectoriesintersecting U ) and a second one over state action pairs in U c (or trajectories without intersectionwith U ).
We can then apply the upper bound on the visitation of state action pairs (or trajectories ) in
PathPseudoSub ∆ ×A∪ PseudoSub ∆ to derive a regret upper bound over these states scaling with 1 / ∆ .
Using the trajectory decomposition yields the term scaling with 1 / ∆ 2 (but having only logarithmicstate dependence) in Theorem 5.2.
We handle the upper bound for the term summing over intersections with U c by noting that Eqn. 4 implies the transition operators over state action pairs in U c havesupport only over S\ PathPseudoSub ∆ .
This implies the error terms (cid:16)(cid:98) P t ( ·| s h , a h ) − P ⋆ ( ·| s h , a h ) (cid:17) · (cid:98) V π t h + 1can be bounded by a factor depending on |S\ PathPseudoSub ∆ | and not |S| .
Detailed proofs can befound in Appendix A.
Numerical Simulations
To show the practical relevance of our analysis on reward shaping we perform some numericalsimulations on a family of maze environments with tabular state-action representations, as shown in
Fig.
These environments have deterministic dynamics and reward.
The reward is 0 at all statesexcept a goal sink state, which has reward 1.
These simulations are aimed at studying the followingquestions: (1) Does reward shaping improves sample complexity in these types of maze environmentsover uninformed UCBVI?
What is the relative importance of the bonus reweighting and the (cid:101) Vprojection?
(3) How does the “suboptimality” of (cid:101) V impact the resulting sample complexity?
Does introducing decayed shaping actually allow for policy convergence?
In these experiments,(cid:101)
V is constructed by scaling the optimal value function V ⋆ by per-state scaling factors sampledindependently within the range β . .
[Figure]
6.1 Does reward shaping help direct exploration over optimism under uncertainty?
We conducted numerical simulations on tabular environments to understand how reward shaping via
UCBVI-Shaped provides benefits over standard UCBVI.
Fig.
3 shows cumulative regret accumulatedwith different variants of UCBVI-Shaped (with both projection and bonus scaling), UCBVI-Shaped-P(with only projection), UCBVI-Shaped-BS (with only bonus scaling), UCBVI (standard UCBVIwithout shaping, as described in [7]).
This is benchmarked across the three environments describedin Fig.
2, with various levels of imperfect shaping applied by varying β = { 1 . 5 , 1 . 9 } .
As seen from
Fig.
3, across all environments UCBVI-shaped with projection and bonus scaling performs mostfavorably, followed typically by UCBVI-Shaped-P, followed by UCBVI-Shaped-BS and UCBVI,suggesting that reward shaping can significantly help with learning efficiency.
[Figure]
6.2 How does the effectiveness of reward shaping vary across environments?
We next conducted some numerical simulations across environments to understand how the nature ofthe environment itself affects the sample complexity of learning with shaping via UCBVI-Shaped.
As shown in Fig.
4, we see that UCBVI shaped can be very effective in environments with manyirrelevant sub-optimal paths like the double corridor environment in Fig. 2, but is relatively lesseffective in environments where all exploration is directed the same way such as the single corridor.
Even incorrect but optimistic shaping will provide guidance towards the goal, making UCBVIrelatively less dominant in the single corridor environment as compared to double corridor.
Thissuggests that in environments where ruling out an entire part of the exploration space is easy from theshaping, we can expect to see larger benefits, as in double corridor.
6.3 How does the suboptimality of reward shaping affect learning?
[Figure]
We next compared how different levels of suboptimality of the β sandwich termin the reward shaping affectcumulative regret across environments.
As shown in
Fig.
5, we see that for environments with open paths,the shaping degradation hasminimal negative effect until it gets very suboptimal.
On the other hand, for corridor and double corridor, where there are only a few paths to the goal, suboptimal reward shaping [Figure] along those paths can significantly hamper progress.
This suggests that suboptimal reward shapingcan hurt learning, but mainly when β is quite large and in environments with bottlenecks. 6.4 Does decaying suboptimal reward shaping help over standard shaped rewards? [Figure] We ran simulations to understand whether the adaptive decay of (cid:101) V by 1 √ N ( s , a ) actually providestangible benefits, specifically when the shaping is suboptimal.
We compared UCBVI-Shaped with avariant where (cid:101) V is added to the reward linearly, as done in many practical RL implementations.
Asseen in Fig.
6, with suboptimal shaping, UCBVI-shaped dampens the effect of the suboptimal shapingand succeed, whereas simple addition of shaping leads to the agent becoming trapped.
This suggeststhat not only can reward shaping improve sample efficiency, but incorporating reward shaping via
UCBVI-Shaped can mitigate the potential bias from suboptimal shaping.
In this work, we take a step towards formally analyzing the benefits of reward shaping, proving thateffective reward shaping can lead to more efficient learning than uninformed exploration strategies.
In our analysis, we study an algorithm that incorporates reward shaping into a modified versionof UCBVI, using it to modify bonuses and value function projection.
Our analysis shows thatincorporating shaped rewards allows for pruning significant parts of the state space and sharpening ofoptimism in a task directed way.
This reduces the dependence of the regret bound on the state spacesize and on the horizon, depending on the quality of the shaping term and parameters of the MDP.
This shows how reward shaping can direct exploration and provide significant sample complexitybenefits while retaining asymptotic performance. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes
(b
) Did you describe the limitations of your work?
{+Yes] See Section 
+}
(c)
Did
you discuss any potential negative societal impacts of your work
[Yes]
See
Section 7(
d
) Have you read the ethics review guidelines
and
ensured that your paper conforms tothem?
[Yes] This work does not actually use human subjects, and is largely done insimulation.
But we have included a discussion in Section 72.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Math is used
as a theory/formalism, but we don’t make any provable claims about it.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix
A for link to URL and run instructions in the README in the github repo.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] All plots were created with 3 random seeds with std error
bars.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Envs we used are
cited in section
(b) Did you mention the license of the assets?
[Yes] This is in Appendix B
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We published the code and included all environments and assets as a part of this
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We used three open source domains and collected our own data
on these domains.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[No] We did not include full text since we didn’t use an exact script, but
we summarized the instructions and included images of the environments used.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Only human involvement was data
collection with our system. (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Human testers were volunteers