Unpacking Reward Shaping: Understanding the Beneﬁts of Reward Engineering on Sample Complexity
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The success of reinforcement learning in a variety of challenging sequentialdecision-making problems has been much discussed, but often ignored in thisdiscussion is the consideration of how the choice of reward function affects the behavior of these algorithms.
Most practical RL algorithms require copious amountsof reward engineering in order to successfully solve challenging tasks.
The idea ofthis type of “reward-shaping” has been often discussed in the literature and is usedin practical instantiations, but there is relatively little formal characterization ofhow the choice of reward shaping can yield beneﬁts in sample complexity for RLproblems.
In this work, we build on the framework of novelty-based exploration toprovide a simple scheme for incorporating shaped rewards into RL along with ananalysis tool to show that particular choices of reward shaping provably improvesample efﬁciency.
We characterize the class of problems where these gains areexpected to be signiﬁcant and show how this can be connected to practical algorithms in the literature.
We show that these results hold in practice in experimentalevaluations as well, providing an insight into the mechanisms through which rewardshaping can signiﬁcantly improve the complexity of reinforcement learning whileretaining asymptotic performance.
Introduction
Reinforcement learning (RL) in its most general form presents a very difﬁcult optimization problem:when there are no constraints on the reward function or dynamics, a learning algorithm may need toexhaustively explore the entire state space to discover high-reward regions.
Naïve algorithms thatrely entirely on random exploration are known to be exponentially expensive [22, 2] and much of thetheoretical work on efﬁcient RL algorithms has focused on smarter exploration strategies that aimto more efﬁciently cover state space, typically in time polynomial in the state space cardinality [5,7, 20, 50, 23, 28, 47].
Much of this work is based on upper conﬁdence bound (UCB) principlesand prescribes some kind of exploration bonus to prioritize exploration of rarely visited regions.
Analogous strategies have also been employed in a number of practical RL algorithms [36, 35, 9, 11,18, 31, 38, 37, 32].
However, surprisingly, much of the empirical work on reinforcement learningdoes not make use of explicit exploration bonuses or other dedicated exploration strategies, despitenumerous theoretical results showing them to be essential to attain polynomial sample complexity.
Instead, practitioners often incorporate prior knowledge of each task into designing or shaping thereward function [27, 33, 34, 39, 4, 10, 45, 49], preferring this heuristic approach over the moreprincipled exploration strategies.
At this point, one may wonder why is reward shaping oftenpractically preferable to dedicated exploration?
A likely answer to this question lies in the fact that even the best general-purpose explorationstrategies still require visiting every state in the MDP at least once in the worst case.
This is ofcourse unavoidable without further assumptions.
However, in practice, sample complexity that ispolynomial in the size of the entire state space might still not be practical, and hence prior knowledgein the form of reward shaping is required to render such tasks tractable for RL.
Surprisingly, despitethe widespread popularity of reward shaping in RL applications, the analysis of reward shaping hasremained limited to proving policy invariance [27] or largely empirical observations, often relegatingreward shaping to folk knowledge.
In this work, we take a step towards studying the effect of rewardshaping on the efﬁciency of RL algorithms, by asking the following question:
Can we theoretically justify the sample complexity beneﬁts that reward shaping from prior domain knowledge can provide for reinforcement learning
In this work, we aim to provide a set of tools that formally analyze how reward shaping can improvethe complexity of tabula-rasa RL and better direct exploration.
To perform this analysis, we ﬁrstpropose a simple modiﬁcation to standard RL algorithms —“UCBVI-Shaped” that incorporatesshaped rewards into optimism based exploration.
We use this algorithm instantiation to then providea regret analysis framework that studies how this introduction of shaped rewards can (in certain cases)provide much more directed optimism than uninformed exploration, while maintaining asymptoticperformance.
To approach our analysis, we speciﬁcally consider problems where the reward shaping is providedthrough a term e V , a (potentially suboptimal) approximation of the optimal value function V ? . Inparticular, we assume that the shaping e
V is a multiplicatively bounded approximation of the optimalvalue function V ? , i.e., e V [Equation] s )  V ?
[Equation] s )  b e V ( s ) , 8 s , for a ﬁnite multiplicative factor b .
This typeof shaped reward function e V can be incorporated into a standard RL algorithm like UCBVI [7]through two channels - (1) bonus scaling: simply reweighting a standard, decaying count-based bonus1 p N h ( s , a ) by the per-state reward shaping and (2) value projection: adaptively projecting learned valuefunctions into ranges of value functions derived from the reward shaping.
We show that this relatively simple algorithmic instantiation lends itself to a simple analysis whichshows signiﬁcant complexity beneﬁts with shaping.
Intuitively, the key pieces in our complexityanalysis of UCBVI-Shaped are two-fold: 1)
The multiplicative sandwich condition (via b ) betweene V and V ? allows for the bonus scaling to depend on b e V instead of a coarse approximation of V ? such as H .
This allows for a reduction of complexity from a horizon H dependent term to onescaling with b e V  b V ? by allowing for a faster decay of the exploration bonuses while still providingenough optimism; 2)
The projection of the value function prevents “over optimism” by hastening theconvergence of the empirical b Q functions during value iteration thus allowing for faster detection ofsub-optimal actions.
This results in the ability to prune out large parts of the state space, as we alsovalidate empirically.
To concretize, the key contribution of this work is mainly to characterize how reward shaping canprovably improve sample efﬁciency by providing gains in both |S| and H dependent terms.
We doso by analyzing the gains from reward shaping in two different ways —bonus scaling and valueprojection.
We show that the “quality” (determined by b ) of the provided shaping can signiﬁcantlyaffect the sample efﬁciency of the resulting reinforcement learning algorithm.
Related Work
Regret Analysis in Finite Horizon Episodic Tabular MDPs. Recent research on regret analysishas studied both model-based and model-free RL methods.
Model-based methods [19, 3, 7, 17, 14, 30]ﬁrst learn a model from previous experience and use the learned model for future decision making.
In contrast, model-free methods [20, 8, 50, 41, 26, 23] aim to learn the value function without themodel estimation stage and use the learned value function for decision making.
Our analysis lies inthe model-based framework and is similar to the setting of [7] but with additional assumptions onknowing e V as a multiplicatively bounded approximation of V ? .
The main difference between our results and the aforementioned works is that, the truncated interval assumption can reduce the statedependency |S| to the some smaller “effective state space” and reduce horizon dependency.
Regret Analysis with Linear Function Approximation. A recent line of literature has investigatedthe linear function approximation setting by assuming the transition kernels or the value functionscan be represented by d -dimensional linear features [21, 48, 6, 51, 12, 43, 42, 40] or even generalfunction classes [44].
With the aforementioned assumptions, the regret analysis will only dependon the ambient dimension d (or other intrinsic complexity measure), instead of S , A in the tabularsetting [40, 43, 12, 51, 6, 47, 21], which could greatly decrease the complexity of learning.
Instead ofposing structural assumptions on the function class representing the MDP’s values or dynamics, weask the question of whether having approximate knowledge of V ? can improve the speed of learning.
Practical Reward Design and Reward Shaping. Ng et al.
[27] proposed a potential-based shapingfunction F that ensures policy invariance under transformation.
However, unbiased potential basedreward shaping is rarely used in practice.
In most applications, heuristic reward design is carefullyperformed with potentially biased reward functions.
In some large-scale practical RL tasks [10,45, 46], the reward functions are heavily handcrafted based on prior knowledge.
Besides rewardengineering, a distinct line of work applies uninformed exploration algorithms like count-based rewardshaping or intrinsic rewards to encourage greater state visitation [36, 35, 9, 11, 18, 31, 38, 37, 32, 24].
Importantly, these methods optimize for the worst case, as they try to cover all states since the exactlocation of the reward is unknown.
In contrast, we look at the problem of incorporating domainknowledge via reward shaping into the exploration process.
Closely related to our work is that of
Cheng et al.
[13], which studies how to incorporate shaping (heuristics) into the RL process viareducing the effective horizon.
This work provides gains by reducing the horizon factor while ourwork provides gains by reducing the size of the effective state space that needs to be searched through.
Overview
We consider an episodic Markov Decision process M =
( S , A , P ? , r , H ) where S corresponds tothe state space,
A is the action space, P ? is the transition operator, r : S ⇥ A ! [ 0 , 1 ] is the rewardfunction and H is the problem horizon.
We use |S| , |A| to denote the number of state and actions,and we use P ? ( ·| s , a ) to denote the transition probability of state action pair ( s , a ) .
The value function
V p ( s 0 ) of a policy p starting at an initial state s 0 is deﬁned as V p ( s 0 ) :
= E p ⇥ Â
Hh = 0 r ( s h , a h ) ⇤ , where E p denote the transition dynamic of M under p .
Similarly, V ? ( s 0 ) is the value function of the optimalpolicy p ? .
We consider a sequential interaction between a learner and the MDP M occurring inrounds indexed by t 2 N .
At the start of round t the learner selects a policy p t that is used to gather asample trajectory from M .
As is standard in the literature, we measure the learner’s performance upto round T by Regret ( T ) :
= Â Tt = 1 V ? ( s 0 )  V p t ( s 0 ) .
Our objective is to show that knowing a reward shaping term e V allowsfor signiﬁcantly more sample efﬁcientlearning, which with high probability has a sublinear regret upper bound. This bound has a leading term that depends on an “effective state space” of M , determined by the quality of a reward shaping term e
V and the natureof the particular MDP being solved.
This pruned effective state space canbe much smaller than |S| .
Additionally, we will show an improved horizon dependence as the shaping terme V allows for the bonus terms to besmaller and therefore decay faster thus replacing horizon factors of H with b e V ones.
Intuitively, reward shaping allows for the consideration of a reduced effective state space, considerthe maze environment in Fig.
1 with agent starting in the middle.
Without knowing where the goal is,an uninformed exploration algorithm needs to explore nearly the entire maze.
However, an effectivelyincorporated reward shaping term e V would allow an algorithm to prune out the entire left half of thegrid, effectively halving the effective state space size.
This can happen in two ways —ﬁrstly if theshaping can directly indicate that certain actions are sub-optimal then entire regions of the space canbe eliminated from search.
Secondly, even if states are not eliminated, if their corresponding bonusesare scaled according to their shaped value e V from the shaping rather than uniformly with H , it limitsunnecessary exploration towards suboptimal states.
Based on this intuition, we propose slight modiﬁcations to a standard reinforcement algorithm thatallows us to perform this analysis effectively.
To aid our analysis, we introduce UCBVI-Shaped, amodiﬁcation of the UCBVI algorithm
[7] that uses an additional reward shaping term e V in two ways:(1) Bonus scaling: e V is used to provide an exploration bonus that combines inverse state visitationcounts with reward shaping.
This allows for the reduction of overoptimism and the dropping of thehorizon H dependence (2) Value projection: the shaping term is used to clip the empirical valuefunction b V h ( s ) , which enables pruning unnecessary elements of the state space and allows for a boundthat depends only on the effective state space size.
Informally, our main result for UCBVI-Shapedcan be summarized as follows:
Theorem 3.1 (Main Informal) .
With probability at least 1  d , the regret of UCBVI - Shaped satisﬁes
[Equation]
Here, S remain corresponds to the set of states in M where the information contained in e V was notenough to rule them out, and Regret  UCBVI ( S remain A , T ) is a UCBVI regret upper bound functionevaluated on S remain and A .
B ( e V , M ) is a time-agnostic complexity measure that depends solely onthe M and the quality of the shaping e V.
We will deﬁne S remain and B ( e V , M ) and describe their relationship to e V precisely in Section 5.
If
S remain ⌧ S , the regret of UCBVI - Shaped can be substantially smaller than Regret  UCBVI ( S , A ) ,particularly since the ﬁrst term of the regret grows logarithmically while the second scales with p T .
We state our assumptions next:
Assumption 1.
The quality of the shaping term e V is described by a parameter b .
We assume accessto shaping “value function” estimators e
V h : S !
R such that, e V h ( s )  V ?
h ( s )  b e V h ( s ) , for all s 2 S {+,h 
+}
[ H ] and for some b  1 .
Assumption 2.
We assume the reward functions satisfy r ( s , a ) 2 [ 0 , 1 ] , 8 ( s , a ) 2 S ⇥ A .
Assumption 3.
The states S are h  indexed, i.e., the states reachable at time
h {+
+}
[ H ] are disjointfrom the states reachable at time h {+0 
+}
[ H ] when h 6 = h 0 .
Contributions.
Our analysis introduces the notions of pseudosuboptimal and pathpseudosuboptimal states to quantify the “effective” size of the state space as a function ofthe quality of the shaping term e V , and we use these notions to show how UCBVI-Shaped can attainsigniﬁcantly improved regret rates.
We further show that incorporating shaping into the explorationbonus term improves the horizon-dependence in the bound when the shaping is good enough,allowing us to replace a leading H term with max s b e V ( s ) .
The UCBVI-Shaped Algorithm
To perform analysis of reward shaping, we build on the framework of the UCBVI algorithm [7].
UCBVI is an exploration algorithm based on the upper conﬁdence bound, described in detail in
Appendix D. This forms a convenient base algorithm for incorporating shaped rewards in a way thatadmits faster learning while maintaining asymptotic performance (as we will discuss in Section 5).
Our algorithm, UCBVI-Shaped, is a combination of two changes to the upper conﬁdence boundalgorithm - ﬁrst we modify the bonus scaling to depend on e V and second, we introduce a projectionsubroutine into value iteration, that is implemented as a standard clipping operation. b P t ( s 0 | s a ) :
= N th ( s , a , s 0 ) N th ( s , a ) , 8 h , s a s 0 .
. Bonus scaling
As in standard UCBVI, we deﬁne N th ( s , a ) to be the visitation count for the state-action pair ( s , a )at iteration t  1 for horizon h : N th
( s , a ) : = Â t  1 i = 0 ( s ih , a ih ) =
( s , a ) .
Similarly to N th ( s , a ) , we use
N th ( s , a , s 0 ) :
= Â t  1 i = 0 ( s ih , a ih , s ih + 1 ) =
( s , a , s 0 ) as the visitation count of state-action pair ( s , a ) andthe subsequent state s 0 .
We then use b P t ( s 0 | s , a ) :
= N th ( s , a , s 0 ) N th ( s , a ) , 8 h , s , a , s 0 to denote the empirical transitionkernels at iteration t .
UCBVI uses value iteration with the empirical transition function b P t and areward function augmented with an exploration bonus, given by r h + b th .
This is deﬁned as a dynamicprogramming procedure that starts at H and then proceeds backward in time to h b
V s ) by bound based , b e V ( s , a ) . 1: Input
[Equation]
[Equation]
3: While not converged [Equation] [Equation] projection 6: p
[Equation]
Analyzing UCBVI-Shaped
In this section, we will derive our main result on the sample complexity of the UCBVI-Shapedalgorithm, and along the way introduce pseudosuboptimal and path-pseudosuboptimal states as a toolfor deriving bounds that depend only on the “effective” state space as determined by the providedreward shaping.
We will ﬁrst introduce some notation to use in our analysis.
As described in theprevious section, UCBVI-Shaped proceeds in rounds.
At the beginning of round t , the learner hasaccess to an empirical model b P t built from the data collected up to iteration t  1.
The bonus termswe consider are built by taking the empirical second moment of e V .
This is related to the deﬁnition ofbonus 2 in Azar et al.
Importantly, the empirical value functions b V th are clipped above by b e
V h .
[Equation]
Where e V max h = max se
V h ( s 0 ) and e V max = max s 0 , he V h 0 ( s 0 ) .
Despite clipping and a modiﬁed bonus term, the b Q and b V values of UCBVI-Shaped are optimistic:
Lemma 5.1 (Optimism) .
With probability at least 1  d we have
[Equation]
and
[Equation]
for all t 2 N , where b V th is computed via Algorithm 2.
The proof of Lemma 5.1 can be found in Appendix A.1.
Optimism (Lemma 5.1) and the simulation
Lemma [1] implies that:
[Equation]
We now turn our attention to characterize the information contained in e V .
5.1 Pruning of the State Space
Consider the following “surrogate” Q functions, e Q [Equation] uh
[Equation]
⇥
A
[Equation]
R induced by e
V via:
[Equation] By Assumption 1, we can bound Q ? via Q ? h ( s , a ) [Equation]
e
Q uh
[Equation] i . The basis of our results isthe following observation.
If an action a satisﬁes e Q uh ( s , a ) <
V ? h ( s ) for state s , then a is a suboptimalaction for state s .
Given a parameter D > 0, we say that an action a is D  pseudosuboptimal 2 for states if V ? h ( s )  D + e Q uh ( s , a ) .
We denote the set of D  pseudosuboptimal state action pairs as:
[Equation]
The intuition we want to capture is that all states ˜ s that can be accessed only through traversing astate action pair in PseudoSub D , can be safely ignored because we can determine their suboptimalityas soon as the state action pairs in PseudoSub D that lead to ˜ s are visited enough.
Now we deﬁne the set of D -path-pseudosuboptimal states as the states that can be reached only bytraversing a D  pseudosuboptimal state action pair:
PathPseudoSub D = { s 2 S s.t.
all feasible paths from initial states to s intersect PseudoSub D } .
We will show that states in PathPseudoSub D will not be explored by UCBVI-Shaped after a fewiterations.
Intuitively, this happens because the support of the state visitation distribution of the optimalpolicy does not contain any state in PathPseudoSub D , 8 D > 0 (or equivalently, Support ( d p ?
( s , a )) \
PathPseudoSub D = /0 , 8 D > 0).
Hence, once the UCBVI-shaped identiﬁes some sub-optimal stateaction pairs, the algorithm will not visit these state-action pairs again.
For any state action pair ( s , a ) deﬁne its set of neighboring states Neighbor ( s , a ) as the set ofstates with nonzero probability in P ? ( ·| s , a ) .
By deﬁnition of PathPseudoSub D , for all ( s , a ) {+
+}
S\ (( PathPseudoSub D ⇥ A ) [ PseudoSub D ) , we have: Neighbor
[Equation]
s ,
[Equation]
D . In other words, the neighborhood of any state action pair ( s , a ) whose state is not in PathPseudoSub Dand such that ( s , a ) is not D  pseudo-suboptimal, are not in PathPseudoSub D .
We also introduce thenotion of “boundary pseudosuboptimal” state action pairs to capture the set of state action pairs thatare D  suboptimal but whose states are not in PathPseudoSub D .
BoundaryPseudoSub
[Equation]
Naturally, one of these states has to be traversed by any trajectory that contains any state in PathPseudoSub D .
Although we only use e Q u in the deﬁnition of PseudoSub D ,
PathPseudoSub D , and
BoundaryPseudoSub D , the size of these sets is modulated by the scale of b and the width ofthe intervals [Equation] i .
We will show that (in the notation of Theorem 3.1) S prunned ⇡
S\ PathPseudoSub D .
With this notation, the formal version of our main results is stated as follows
Theorem 5.2.
With probability at least 1  3 d , the regret of UCBVI-Shaped is upper bounded by
T Â t = 1 V ? ( s 0 )  V p t ( s 0 ) = O min D
H b e
V max | S \ PathPseudoSub D | s | A | T ln e
V max | S ||
A | T d + b⇣e
V max ⌘ 2 ln e V max | S || A | T d H 1 / 2 min | S | 3 / 2 | A | D , H 3 / 2 | BoundaryPseudoSub D |
D!!
, for all T 2 N .
Theorem 5.2 instantiates the desiderata of Theorem 3.1.
Although this regret upper bound cannotbe decomposed into a sum of two term as in Theorem 3.1.
For any ﬁxed D , the regret upper bounddoes have two components, one where S remain can be identiﬁed with |S\ PathPseudoSub D | and asecond one that scales logarithmically in T .
In the next section we ﬂesh out the steps in the proofof this result.
The full argument can be found in Appendix A.
The bonus scaling also allows us toameliorate the horizon dependence of the upper bound.
Instead of obtaining a H 2 dependence asthe bound of theorem 7.1 in [1], the ﬁrst term depends on b H max s V ? ( s )  b H e V max .
Notice thatthe state dependence in the second term may be only be of order log ( |S| ) if there is a D such that| BoundaryPseudoSub D | ⌧ |S| (albeit at the cost of a quadratic dependence on 1 / D 2 ).
Setting D to avalue smaller than the minimum gap recovers the UCBVI regret upper bounds Theorem 7.1 in [1]showing Theorem 5.2 is a strict generalization.
for Theorem 5.
Improved Horizon Dependence.
An empirical Bernstein bound shows that adding a bonus scaling(up to low order terms) with O
[Equation]
/ N th [Equation]
s , a ) ◆ is sufﬁcient toensure optimism.
Since V ? is not known, this variance term can be substituted by the empirical secondmoment of b e
V h
+ 1 .
Finally the scaling of these terms can be upper bounded by b e V max .
Withoutknowledge of e V , achieving this scaling would prove more challenging, since the only proxy for V ? h + 1available is b V th + 1 or H both of which may vastly overestimate it.
State Pruning.
The value function clipping mechanism ensures that E s {+
 ⇠ b [Equation] th+}
[Equation] hb [Equation] th
[Equation] [Equation] he
V [Equation] + 1 ( s [Equation]
) i and therefore the empirical gap between b Q th ( s , a ) and V ? ( s ) is decreasingat a rate of at most O ⇣ [Equation]
p N th ( s , a ) ⌘ .
Since optimism ensures that b Q th ( s , p ?
( a ))  V ? ( s ) ,once b Q ( s , a ) < V ?
( s ) , action a will not be pulled anymore.
Thus for any D , the number oftimes any state action pair in PseudoSub D may be visited by UCBVI-Shaped is upper boundedby b⇣e V max ⌘ 2 / D 2 .
Since any visit to a state action pair in PathPseudoSub D ⇥ A [ PseudoSub Drequires a visit to a state in BoundaryPseudoSub D , which allows us to bound the total number ofvisits to a state (or a trajectory containing such a state) in PathPseudoSub D ⇥
A [ PseudoSub D by
H | BoundaryPseudoSub D | b⇣e V max ⌘ 2 / D 2 .
Finally, we can split the regret decomposition of Eqn.
3 into two sums, one over state action pairs in U = PathPseudoSub D ⇥
A [ PseudoSub D (or trajectories intersecting U ) and a second one over stateaction pairs in U c (or trajectories without intersection with U ).
We can then apply the upper bound onthe visitation of state action pairs (or trajectories ) in PathPseudoSub D ⇥
A [ PseudoSub D to derive aregret upper bound over these states scaling with 1 / D .
Using the trajectory decomposition yields theterm scaling with 1 / D 2 (but having only logarithmic state dependence) in Theorem 5.2.
We handlethe upper bound for the term summing over intersections with U c by noting that Eqn. 4 implies thetransition operators over state action pairs in U c have support only over S\ PathPseudoSub D .
Thisimplies the error terms [Equation] [Equation] V p t h + 1 can be bounded by a factor depending on|S\ PathPseudoSub D | and not |S| .
Detailed proofs can be found in Appendix A.
Numerical Simulations
To show the practical relevance of our analysis on reward shaping we perform some numericalsimulations on a family of maze environments with tabular state-action representations, as shown in
Fig.
These environments have deterministic dynamics and reward.
The reward is 0 at all statesexcept a goal sink state, which has reward 1.
These simulations are aimed at studying the followingquestions: (1) Does reward shaping improves sample complexity in these types of maze environmentsover uninformed UCBVI?
What is the relative importance of the bonus reweighting and the e Vprojection?
(3) How does the “suboptimality” of e V impact the resulting sample complexity?
Does introducing decayed shaping actually allow for policy convergence?
In these experiments,e
V is constructed by scaling the optimal value function V ? by per-state scaling factors sampledindependently within the range b .
[Figure]
6.1 Does reward shaping help direct exploration over optimism under uncertainty?
We conducted numerical simulations on tabular environments to understand how reward shaping via
UCBVI-Shaped provides beneﬁts over standard UCBVI.
Fig.
3 shows cumulative regret accumulatedwith different variants of UCBVI-Shaped (with both projection and bonus scaling), UCBVI-Shaped-P(with only projection), UCBVI-Shaped-BS (with only bonus scaling), UCBVI (standard UCBVIwithout shaping, as described in [7]).
This is benchmarked across the three environments describedin Fig.
2, with various levels of imperfect shaping applied by varying b = { 1 . 5 , 1 . 9 } .
As seen from
Fig.
3, across all environments UCBVI-shaped with projection and bonus scaling performs mostfavorably, followed typically by UCBVI-Shaped-P, followed by UCBVI-Shaped-BS and UCBVI,suggesting that reward shaping can signiﬁcantly help with learning efﬁciency.
[Figure]
6.2 How does the effectiveness of reward shaping vary across environments?
We next conducted some numerical simulations across environments to understand how the nature ofthe environment itself affects the sample complexity of learning with shaping via UCBVI-Shaped.
As shown in Fig.
4, we see that UCBVI shaped can be very effective in environments with manyirrelevant sub-optimal paths like the double corridor environment in Fig. 2, but is relatively lesseffective in environments where all exploration is directed the same way such as the single corridor.
Even incorrect but optimistic shaping will provide guidance towards the goal, making UCBVIrelatively less dominant in the single corridor environment as compared to double corridor. This
[Figure]
suggests that in environments where ruling out an entire part of the exploration space is easy from theshaping, we can expect to see larger beneﬁts, as in double corridor.
6.3 How does the suboptimality of reward shaping affect learning?
[Figure]
We next compared how different levels of suboptimality of the b sandwich termin the reward shaping affectcumulative regret across environments.
As shown in
Fig.
3 (a) and (d), wesee that for environmentswith open paths (like theopen gridworld), the shaping degradation has minimal negative effect until itgets very suboptimal.
On the other hand, for corridor and double corridor (Fig. 5 (b)), where thereare only a few paths to the goal, suboptimal reward shaping along those paths signiﬁcantly hamperprogress.
Discussion
In this work, we take a step towards formally analyzing the beneﬁts of reward shaping, proving thateffective reward shaping can lead to more efﬁcient learning than uninformed exploration strategies.
In our analysis, we study an algorithm that incorporates reward shaping into a modiﬁed versionof UCBVI, using it to modify bonuses and value function projection.
Our analysis shows thatincorporating shaped rewards allows for pruning signiﬁcant parts of the state space and sharpening ofoptimism in a task directed way.
This reduces the dependence of the regret bound on the state spacesize and on the horizon, depending on the quality of the shaping term and parameters of the MDP.
This shows how reward shaping can direct exploration and provide signiﬁcant sample complexitybeneﬁts while retaining asymptotic performance.
[-Yes] See Section 
-]
We hope that this work is a step towards movingsample complexity analysis away from being reward agnostic to actually considering reward shapingmore formally in analysis.
We hope that a much broader class of reward shaping and more genericways to incorporate reward shaping are explored in the future.