Unpacking Reward Shaping: Understanding the Beneﬁts of Reward Engineering on Sample Complexity
Abstract
Reinforcement learning provides an automated framework for learning behaviors from high-level reward speciﬁcations, but in practice the choice of reward function can be crucial for good results – while in principle the reward only needs to specify what the task is, in reality practitioners often need to design more detailed rewards that provide the agent with some hints about how the task should be completed.
The idea of this type of “reward-shaping” has been often discussed in the literature, and is often a critical part of practical applications, but there is relatively little formal characterization of how the choice of reward shaping can yield beneﬁts in sample complexity.
In this work, we build on the framework of novelty-based exploration to provide a simple scheme for incorporating shaped rewards into RL along with an analysis tool to show that particular choices of reward shaping provably improve sample efﬁciency.
We characterize the class of problems where these gains are expected to be signiﬁcant and show how this can be connected to practical algorithms in the literature.
We conﬁrm that these results hold in practice in an experimental evaluation, providing an insight into the mechanisms through which reward shaping can signiﬁcantly improve the complexity of reinforcement learning while retaining asymptotic performance.
Introduction
Reinforcement learning (RL) in its most general form presents a very difﬁcult optimization problem: when there are no constraints on the reward function or dynamics, a learning algorithm may need to exhaustively explore the entire state space to discover high-reward regions.
Naïve algorithms that rely entirely on random exploration are known to be exponentially expensive [26, 3], and much of the theoretical work on efﬁcient RL algorithms has focused on smarter exploration strategies that aim to more efﬁciently cover state space, typically in time polynomial in the state space cardinality [6, 8, 24, 55, 27, 32, 52].
Much of this work is based on upper conﬁdence bound (UCB) principles and prescribes some kind of exploration bonus to prioritize exploration of rarely visited regions.
Analogous strategies have also been employed in a number of practical RL algorithms [41, 40, 10,
However, perhaps surprisingly, much of the empirical work on reinforcement learning does not make use of explicit exploration bonuses or other dedicated exploration strategies, despite numerous theoretical results showing them to be essential to attain tractable sample complexity.
Instead, practitioners often incorporate prior knowledge of each task into designing or shaping the reward function [31, 38, 39, 44, 5, 11, 50, 54], preferring this heuristic approach over the more principled exploration strategies.
At this point, one may wonder why is reward shaping often practically preferable to dedicated exploration?
A likely answer to this question lies in the fact that even the best general-purpose exploration strategies still require visiting every state in the MDP at least once in the worst case.
This is of course unavoidable without further assumptions.
However, in practice, sample complexity that is polynomial in the size of the entire state space might still not be practical, and hence prior knowledge in the form of reward shaping is required to render such tasks tractable.
Surprisingly, despite the widespread popularity of reward shaping in RL applications, the analysis of reward shaping has remained limited to proving policy invariance [31] or largely empirical observations, often relegating reward shaping to folk knowledge.
In this work, we take a step towards studying the effect of reward shaping on the efﬁciency of RL algorithms, by asking the following question:
Can we theoretically justify the sample complexity beneﬁts that reward shaping from prior domain knowledge can provide for reinforcement learning?
We aim to provide a set of tools that formally analyze how reward shaping can improve the complexity of tabula-rasa RL and better direct exploration.
To perform this analysis, we ﬁrst propose a simple modiﬁcation to standard RL algorithms —“UCBVI-Shaped” that incorporates shaped rewards into optimism based exploration.
We use this algorithm instantiation to then provide a regret analysis framework that studies how this introduction of shaped rewards can (in certain cases) provide much more directed optimism than uninformed exploration, while maintaining asymptotic performance.
To approach our analysis, we speciﬁcally consider problems where the reward shaping is provided through a term e
V , a (potentially suboptimal) approximation of the optimal value function V ? .
In particular, we assume that the shaping e
V is a multiplicatively bounded approximation of the optimal value function V ? , i.e., e
V ( s )  V ?
( s )  b e
V ( s ) , 8 s , for a ﬁnite multiplicative factor b .
This type of shaped reward function e
V can be incorporated into a standard RL algorithm like UCBVI [8] through two channels: (1) bonus scaling – simply reweighting a standard, decaying count-based bonus 1 p N h ( s , a ) by the per-state reward shaping and (2) value projection – adaptively projecting learned value functions into ranges of value functions derived from the reward shaping.
We show that this relatively simple algorithmic instantiation lends itself to an analysis that shows signiﬁcant sample complexity beneﬁts with shaping.
Intuitively, the key pieces in our complexity analysis of UCBVI-Shaped are: (1) A multiplicative sandwich condition (via b ) between e
V and V ? allows for the bonus scaling to depend on b e
V instead of a coarse approximation of V ?
such as H .
This allows for a reduction of complexity from a horizon H dependent term to one scaling with b
[Equation]
by allowing for a faster decay of the exploration bonuses while still providing enough optimism.
A projection of the value function prevents “over optimism” by hastening the convergence of the empirical b
Q functions during value iteration, thus allowing for faster detection of sub-optimal actions.
This results in the ability to prune out large parts of the state space, as we also validate empirically.
To summarize, the key contribution of this work is to characterize how reward shaping can provably improve sample efﬁciency by providing gains in both |S| and H dependent terms.
We do so by analyzing the gains from reward shaping in two different ways: bonus scaling and value projection.
We show that the “quality” (determined by b ) of the provided shaping can signiﬁcantly improve the sample efﬁciency of the resulting reinforcement learning algorithm, provide a set of analysis techniques to understand improvements in sample complexity from shaping, and conﬁrm our ﬁndings with numerical experiments.
Related Work
Regret Analysis in Finite Horizon Episodic Tabular MDPs.
Recent research on regret analysis has studied both model-based and model-free RL methods.
Model-based methods [23, 4, 8, 20, 17, 35] ﬁrst learn a model from previous experience and use the learned model for future decision making.
In contrast, model-free methods [24, 9, 55, 46, 30, 27] aim to learn the value function without the model estimation stage and use the learned value function for decision making.
Our analysis lies in the model-based framework and is similar to the setting of [8] but with additional assumptions on knowing e
V as a multiplicatively bounded approximation of V ? .
The main difference between our results and the aforementioned works is that we consider the reward shaping setting with a truncated interval assumption. As a result, we reduce the state dependency |S| to the some smaller “effective state space”. Our work is also closely related to [21]. We discuss this in Section 3.
Regret Analysis with Linear Function Approximation.
A recent line of literature has investigated the linear function approximation setting by assuming the transition kernels or the value functions can be represented by d -dimensional linear features [25, 53, 7, 56, 13, 48, 47, 45] or even general function classes [49].
With the aforementioned assumptions, the regret analysis will only depend on the ambient dimension d (or other intrinsic complexity measure), instead of S , A in the tabular setting [45, 48, 13, 56, 7, 52, 25], which could greatly decrease the complexity of learning.
Instead of posing structural assumptions on the function class representing the MDP’s values or dynamics, we ask the question of whether having approximate knowledge of V ?
can improve the speed of learning.
Practical Reward Design and Reward Shaping.
Ng et al.
[31] proposed a potential-based shaping function F that ensures policy invariance under transformation.
However, unbiased potential based reward shaping is rarely used in practice.
In most applications, heuristic reward design is carefully performed with potentially biased reward functions.
In some large-scale practical RL tasks [11, 50, 51], the reward functions are heavily handcrafted based on prior knowledge.
Besides reward engineering, a distinct line of work applies uninformed exploration algorithms like count-based reward shaping or intrinsic rewards to encourage greater state visitation [41, 40, 10, 12, 22, 36, 43, 42, 37, 28].
Importantly, these methods optimize for the worst case, as they try to cover all states since the exact location of the reward is unknown.
In contrast, we look at the problem of incorporating domain knowledge via reward shaping into the exploration process.
Closely related to our work is that of
Cheng et al.
[14], which studies how to incorporate shaping (heuristics) into the RL process via reducing the effective horizon.
This work provides gains by reducing the horizon factor while our work provides gains by reducing the size of the effective state space that needs to be searched through.
Overview
We consider an episodic Markov Decision process M =
( S , A , P ? , r , H ) where S corresponds to the state space,
A is the action space, P ? is the transition operator, r : S ⇥ A !
[ 0 , 1 ] is the reward function and H is the problem horizon.
We use |S| , |A| to denote the number of state and actions, and we use P ?
( ·| s , a ) to denote the transition probability of state action pair ( s , a ) .
The value function
V p ( s 0 ) of a policy p starting at an initial state s 0 is deﬁned as V p ( s 0 ) :
=
E p ⇥
Â
Hh = 0 r ( s h , a h ) ⇤ , where
E p denote the transition dynamic of M under p .
Similarly, V ?
( s 0 ) is the value function of the optimal policy p ? .
We consider a sequential interaction between a learner and the MDP M occurring in rounds indexed by t 2 N .
At the start of round t the learner selects a policy p t that is used to gather a sample trajectory from M .
As is standard in the literature, we measure the learner’s performance up to round T by Regret ( T ) :
= Â Tt = 1 V ?
( s 0 )
V p t ( s 0 ) .
Our goal is to show that knowing a reward shaping term [Equation]
allows for signiﬁcantly more sample efﬁcient learning, which with high probability has a sublinear regret upper bound.
This bound has a leading term that depends on an “effective state space” of M , determined by the quality of a reward shaping term e
V and the nature of the particular MDP being solved.
This pruned effective state space can be much smaller than |S| .
Additionally, we will show an improved horizon dependence as the shaping term e
V allows for the bonus terms to be smaller and therefore decay faster thus replacing horizon factors of H with b e
V ones.
Intuitively, reward shaping allows for the consideration of a reduced effective state space. Consider the maze environment in Fig.
1 with agent starting in the middle.
Without knowing where the goal is, an uninformed exploration algorithm needs to explore nearly the entire maze.
However, an effectively incorporated reward shaping term e
V would allow the algorithm to prune out the entire left half of the maze, effectively halving the effective state space size.
This can happen in two ways: ﬁrstly, if the shaping can directly indicate that certain actions are sub-optimal, then entire regions of the space can be eliminated from search.
Secondly, even if states are not eliminated, if their corresponding bonuses are scaled according to their shaped value e
V from the shaping rather than uniformly with H , it limits unnecessary exploration of suboptimal states.
Based on this intuition, we propose modiﬁcations to a standard reinforcement algorithm that allows us to perform this analysis effectively.
To aid our analysis, we introduce UCBVI-Shaped, a modiﬁcation of the UCBVI algorithm [8] that uses an additional reward shaping term e
V in two ways: (1) Bonus scaling: e
V is used to provide an exploration bonus that combines inverse state visitation counts with reward shaping.
This allows for the reduction of overoptimism and the dropping of the horizon H dependence (2) Value projection: the shaping term is used to clip the empirical value function b
V h ( s ) , which enables pruning unnecessary elements of the state space and allows for a bound that depends only on the effective state space size.
Informally, our main result for UCBVI-Shaped can be summarized as follows:
Theorem 3.1 (Main Informal) .
With probability at least 1  d , the regret of UCBVI - Shaped satisﬁes[-
-]
[Equation]
⌘ . Here, S remain corresponds to the set of states in M where the information contained in e
V was not enough to rule them out, and Regret  UCBVI ( S remain A , T ) is a UCBVI regret upper bound function evaluated on S remain and A .
B ( e
V , M ) is a time-agnostic complexity measure that depends solely on the M and the quality of the shaping e
V.
We will deﬁne S remain and B ( e
V , M ) and describe their relationship to e
V precisely in Section 5.
If
S remain ⌧ S , the regret of UCBVI - Shaped can be substantially smaller than Regret  UCBVI ( S , A ) , particularly since the ﬁrst term of the regret grows logarithmically while the second scales with p T .
We state our assumptions next:
Assumption 1.
The quality of the shaping term e
V is described by a parameter b .
We assume access to a shaping “value function” estimators e
V h : S !
R such that V ?
h ( s )  b e
V h ( s ) , for all s 2 S , h
[ H ] and for some b  1 .
Instead of absorbing b into the deﬁnition of e
V h we allow e
V h ( s ) < V ? h ( s ) for some states s 2 S . We’ll show that learning a value of b that turns this assumption true can be performed online.
This assumption is intimately related to the optimistic e
Q assumptions of [21].
A thorough comparison with this work can be found in Appendix B.2.
Assumption 2. We assume the reward functions satisfy r ( s , a ) 2 [ 0 , 1 ] , 8 ( s , a ) 2 S ⇥ A .
Assumption 3.
The states S are h  indexed, i.e., the states reachable at time
h
[ H ] are disjoint from the states reachable at time h 0
[ H ] when h 6 = h 0 .
Contributions.
Our main conceptual innovation is to introduce the notions of pseudosuboptimal and path-pseudosuboptimal states to quantify the “effective” size of the state space as a function of the quality of the shaping term e
V and use these notions to show how UCBVI-Shaped can attain signiﬁcantly improved regret rates.
In contrast with for example [21], where the regret rates will always scale at least with the number of states, our regret guarantees depend on an effective state size that may be orders of magnitude smaller.
We believe this captures the real complexity improvement that reward shaping may yield, namely, avoid exploration of unnecessary areas of the state space.
Other approaches such as [21] may (in general) at most yield an improvement in the dependence on the effective size of the action space.
We further show that incorporating shaping into the exploration bonus term improves the horizon-dependence in the bound when the shaping is good enough, allowing us to replace a leading H term with max s b e
V ( s ) .
The
To perform analysis of reward shaping, we build on the framework of the UCBVI algorithm [8].
UCBVI is an exploration algorithm based on the upper conﬁdence bound, described in detail in
Appendix D. This forms a convenient base algorithm for incorporating shaped rewards in a way that admits faster learning while maintaining asymptotic performance (as we will discuss in Section 5).
Our algorithm, UCBVI-Shaped, is a combination of two changes to the upper conﬁdence bound algorithm. First, we modify the bonus scaling to depend on e
V . Second, we introduce a projection subroutine into value iteration, implemented as a standard clipping operation. 1: Input reward function r assumed to be known), conﬁdence parameters 2: for t
,..., T 3: Compute b
P t using all previous empirical transition data as b [Equation] Compute reward bonus b th s e Run Value-Iteration
Projection Algorithm
Set p t as the returned policy of VI. 7: End for As in standard UCBVI, we deﬁne N th ( s , a ) to be the visitation count for the state-action pair ( s , a ) at iteration t  1 for horizon h :
N th
( s , a ) :
= Â t  1 i = 0
( s ih , a ih )
=
( s , a ) .
Similarly to N th ( s , a ) , we use
N th ( s , a , s 0 ) :
= Â t  1 i = 0
( s ih , a ih , s ih + 1 ) =
( s , a , s 0 ) as the visitation count of state-action pair ( s , a ) and the subsequent state s 0 .
We then use
[Equation]
: = N th ( s , a , s 0 )
N th ( s , a ) , 3 8 h , s , a , s 0 to denote the empirical transition kernels at iteration t .
UCBVI uses value iteration with the empirical transition function b
P t and a reward function augmented with an exploration bonus, given by r h + b th .
This is deﬁned as a dynamic programming procedure that starts at H and then proceeds backward in time to h = 0, updating according to Algorithm 1 and 2.
The key algorithmic changes between UCBVI and UCBVIShaped are highlighted in blue: (1) scaling bonus b th by the shaping term e V and (2) projecting the empirical value function b V th ( s ) by an upper bound based on the sandwiched shaping, b e
V ( s , a ) .
As we discuss in Section 6, we also show how the sandwich factor b does need to be provided beforehand but instead can be inferred through a straightforward technique for online model selection. While the approximate order of the bouns term is e V p N ( s , a ) , a more detailed description can be found in Section 5.
Algorithm nb P t , r + b th o H h = 0 .
b V tH ( s ) = 0 , 8 s , b Q th ( s , a ) b Q th ( s , a ) =
min n r h ( s , a )+ b th ( s , a )+ b P t ( ·| s , a ) · b V th + 1 , H o b
V th ( s ) =
min ⇣ max a b Q th ( s , a ) , b e V ( s ) ⌘ .
Value projection 6: p th ( s ) =
argmax a b
Q th ( s , a ) , 8 h , s , a .
Analyzing UCBVI-Shaped
In this section, we will derive our main result on the sample complexity of the UCBVI-Shaped algorithm, and along the way introduce pseudosuboptimal and path-pseudosuboptimal states as a tool for deriving bounds that depend only on the “effective” state space as determined by the provided reward shaping.
We will ﬁrst introduce some notation to use in our analysis.
As described in the previous section, UCBVI-Shaped proceeds in rounds.
At the beginning of round t , the learner has access to an empirical model b
P t built from the data collected up to iteration t  1.
The bonus terms we consider are built by taking the empirical second moment of e
V .
This is related to the deﬁnition of bonus 2 in Azar et al.
Importantly, the empirical value functions b
V th are clipped above by b e
V h :
[Equation]
where e
[Equation]
and e
[Equation]
Despite clipping and a modiﬁed bonus term, the b
Q and b
V values of UCBVI-Shaped are optimistic:
Lemma 5.1.
With probability at least 1  d we have
[Equation]
and
[Equation]
for all t , h 2 N ⇥ [ H ] , where b
V th is computed via Algorithm 2.
The proof of Lemma 5.1 can be found in Appendix A.1.
Optimism (Lemma 5.1) and the simulation
Lemma [2] imply that:
[Equation]
We now turn our attention to characterize the information contained in e
V .
5.1 Pruning of the State Space
Consider the following “surrogate” Q functions,
[Equation]
R induced by e
V via:
[Equation]
By Assumption 1, we can bound Q ? via Q ?
h ( s , a )  e
Q uh ( s , a ) .
The basis of our main results is the following observation.
If an action a satisﬁes e
Q uh ( s , a ) <
V ?
h ( s ) for state s , then a is a suboptimal action for state s .
The projection (Step 5 of Algorithm 2) ensures the ‘empirical’ Q  functions b
Q th ( s , a ) of Algorithm 1 will quickly converge to values upper bounded by e
Q uh ( s , a ) .
Since optimism guarantees that b
Q th ( s , p ⇤ ( s ))  V ? ( s ) , and the policy executed by UCBVI-Shaped is greedy w.r.t b Q th , actions belonging to state-action pairs such that e Q u h ( s , a ) < V ?
h ( s ) will quickly stopped being played by the algorithm.
Moreover, all states that are only accessible through state-action pairs of this kind will also stop being visited by the algorithm after only a few iterations.
In the subsequent discussion we will call
PseudoSub to the set of state action pairs that are quickly ‘prunned out’ by UCBVI-Shaped, the set of states only accessible through these state action pairs as PathPseudoSub and the set of state-action pairs in PseudoSub that are not in PathPseudoSub ⇥
A as BoundaryPseudoSub.
Once all states in
BoundaryPseudoSub have been visited enough, no states in PathPseudoSub will be visited again.
Provided this happens sufﬁciently fast, we can show a regret bound that is independent on the size of PathPseudoSub.
The subsequent discussion is aimed at formalizing this and culminates with our main result (Theorem 5.2).
Given a parameter D > 0, we say that an action a is D  pseudosuboptimal 4 for state s if V ?
h ( s )
D + e
Q uh ( s , a ) .
We denote the set of D  pseudosuboptimal state action pairs as:
[Equation]
The intuition we want to capture is that all states ˜ s that can be accessed only through traversing a state action pair in PseudoSub D , can be safely ignored because we can determine their suboptimality as soon as the state action pairs in PseudoSub D that lead to ˜ s are visited enough.
Now we deﬁne the set of D -path-pseudosuboptimal states as the states that can be reached only by traversing a D  pseudosuboptimal state action pair:
PathPseudoSub D = { s 2 S s.t.
all feasible paths from initial states to s intersect PseudoSub D } .
Notice that there may be state action pairs in PathPseudoSub D ⇥
A that may not be in PseudoSub D .
In fact, there may exist states s in PathPseudoSub D such that no ( s , a ) is in PseudoSub D for all a 2 A .
We will show that states in PathPseudoSub D will not be explored by UCBVI-Shaped after a few iterations.
Intuitively, this happens because the support of the state visitation distribution of the optimal policy does not contain any state in PathPseudoSub D , 8 D > 0 (or equivalently, Support ( d p ?
( s , a ))
\ PathPseudoSub D = /0 , 8 D > 0).
Hence, once the UCBVI-shaped identiﬁes some sub-optimal state action pairs, the algorithm will not visit these state-action pairs again.
For any state action pair ( s , a ) , deﬁne its set of neighboring states Neighbor ( s , a ) as the set of states with nonzero probability in P ?
( ·| s , a ) .
By deﬁnition of PathPseudoSub D , for all ( s , a [-) 
-][Equation]
[Equation] PathPseudoSub
[Equation]
[Equation]
, we have:
[Equation]
In other words, the neighborhood of any state action pair ( s , a ) whose state is not in PathPseudoSub D and such that ( s , a ) is not D  pseudo-suboptimal, are not in PathPseudoSub D .
We also introduce the notion of “boundary pseudosuboptimal” state action pairs to capture the set of state action pairs that are D  suboptimal but whose states are not in PathPseudoSub D .
[Equation]
Naturally, one of these states has to be traversed by any trajectory that contains any state in
PathPseudoSub D .
Although we only use e
Q u in the deﬁnition of PseudoSub D ,
PathPseudoSub D , and
BoundaryPseudoSub D , the size of these sets is modulated by the scale of b and the width of the intervals [Equation] [Equation] e
[Equation]
u (
[Equation]
[Equation]
) i .
We will show that (in the notation of Theorem 3.1) S pruned ⇡
S\ PathPseudoSub D .
With this notation, the formal version of our main results is stated as follows.
Theorem 5.2.
With probability at least 1  6 d , the regret of UCBVI-Shaped is upper bounded by
[Equation]
for all T 2 N . Where A ( D ) =
| S | 1 / 2 | A | 1 / 2 D and B ( D ) =
b
[Equation]
Theorem 5.2 instantiates the desiderata of Theorem 3.1.
Although this regret upper bound cannot be decomposed into a sum of two term as in Theorem 3.1.
For any ﬁxed D , the regret upper bound has two components, one where S remain can be identiﬁed with |S\ PathPseudoSub D | and a second one that scales logarithmically in T .
In the next section we ﬂesh out the steps in the proof of this result.
The full argument can be found in Appendix A.
The bonus scaling also allows us to ameliorate the horizon dependence of the upper bound.
Instead of obtaining a H 2 dependence as the bound of theorem 7.1 in [2], the ﬁrst term depends on H b e
V max .
Notice that the state dependence in the second term may be only be of order log ( |S| ) if there is a D such that | BoundaryPseudoSub D | ⌧ |S| (albeit at the cost of a quadratic dependence on 1 / D 2 ).
Moreover notice that | BoundaryPseudoSub D | could be much smaller than | PathPseudoSub D | (the set of states that are reachable only by visiting states in | PseudoSub D | ) thus showing that in some cases we can guarantee that even in the low order terms, the regret of UCBVI-Shaped that has polynomial dependence on an effective state space size that may be orders of magnitude smaller than the original one.
The full version of this bound, with all the low order terms we have omitted for the sake of readability can be found in Appendix B.1, Theorem B.11.
Note that Theorem 5.2 is a strict generalization to the UCBVI regret upper bounds, as setting D to a value that is smaller than the minimum gap recovers the exact result (Theorem 7.6in [2]).
2 Proof Intuitions and Sketch for Theorem [-5.
-]5.
Improved Horizon Dependence.
An empirical Bernstein bound shows that adding a bonus scaling (up to low order terms) with O
[Equation]
◆ is sufﬁcient to ensure optimism.
Since V ? is not known, this variance term can be substituted by the empirical second moment of b e
V h
+ 1 .
Finally, the scaling of these terms can be upper bounded by b e
V max .
Without knowledge of e
V , achieving this scaling would be challenging, since the only proxy for V ?
h + 1 available is b
V th + 1 or H both of which may vastly overestimate it.
State Pruning.
The value function clipping mechanism ensures that E s
⇠
[Equation]
b
P th ( ·| s , a ) he
V h
+ 1 ( s 0 )
i and therefore the empirical gap between b
Q th ( s , a ) and V ?
( s ) is decreasing at a rate of at most O ⇣ b e
V max / p
N th ( s , a ) ⌘ .
Since optimism ensures that
[Equation]
s ) , once b
Q ( s , a ) < V ?
( s ) , action a will not be chosen anymore.
Thus, for any D , the number of times any state action pair in PseudoSub D may be visited by UCBVI-Shaped is upper bounded by b⇣e
V max ⌘ 2 / D 2 .
Since any visit to a state action pair in PathPseudoSub D ⇥
A [ PseudoSub D requires a visit to a state in BoundaryPseudoSub D , which allows us to bound the total number of visits to a state (or a trajectory containing such a state) in PathPseudoSub D ⇥
A [ PseudoSub D by
H | BoundaryPseudoSub D | b⇣e
V max ⌘ 2 / D 2 .
Finally, we can split (a version of) the regret decomposition in Eqn.
3 into two sums, one over state action pairs in U bad =
PathPseudoSub D ⇥
A [ PseudoSub D (or trajectories intersecting U bad ) and a second one over state action pairs in U good = ( S ⇥ A ) \ ( PathPseudoSub D ⇥ A [ PseudoSub D ) (or trajectories without intersection with
U bad ).
We can then apply the upper bound on the visitation of state action pairs in PathPseudoSub D ⇥
A [ PseudoSub D to derive a regret upper bound over these states scaling with 1 / D .
Using the bound on the number of trajectories that intersect
PathPseudoSub D ⇥ A [ PseudoSub D , the trajectory decomposition yields the term scaling with 1 / D 2 (but having only logarithmic state dependence) in Theorem 5.2.
Recall that Eqn. 4 implies that the transition operators over state action pairs in U good have support only over S\ PathPseudoSub D .
Our proofs use this fact to prove a polynomial dependence on |S\ PathPseudoSub D | and not |S| in Theorem B.11 (see Appendix B.1), the full version of Theorem 5.2.
Detailed proofs are in Appendix A.
Practical Considerations: Online Model Selection
Now one may notice that UCBVI-Shaped requires knowledge of the scaling b is in order to actually perform the value projection. While this can be pre-provided by a user or set conservatively, in this section we discuss how this can be inferred by viewing this as an online model selection problem. In particular, given a set of N different values of b — [ b 1 , b 2 ,..., b N ] , each of which parameterizes a different setting of the learning algorithm UCBVI-Shaped( b ), an online model selection algorithm such as Stochastic CORRAL [1, 34] or RegretBalancing [33, 15] jointly infers the value of b and learns the appropriate value function online.
In Appendix B.4 we show how these techniques can yield meaningful regret guarantees and we provide pseudocode.
To show the practical relevance of our analysis on reward shaping we perform some numerical simulations on a family of maze environments with tabular state-action representations, as shown in
Fig.
These environments have deterministic dynamics and reward.
The reward is 0 at all states except a goal sink state, which has reward 1.
These simulations are aimed at studying the following questions: (1) Does reward shaping improve sample complexity in these types of maze environments over uninformed UCBVI?
What is the relative importance of the bonus reweighting and the e
V projection?
(3) How does the “suboptimality” of e
V impact the resulting sample complexity?
Does introducing decayed shaping actually allow for policy convergence?
In these experiments, e
V is constructed by scaling the optimal value function V ?
by per-state scaling factors sampled independently within the range b .
[Figure]
7.1 Does reward shaping help direct exploration over optimism under uncertainty?
We conducted numerical simulations on tabular environments to understand how reward shaping via
UCBVI-Shaped provides beneﬁts over standard UCBVI.
Fig.
4 shows cumulative regret accumulated with different variants of UCBVI-Shaped (with both projection and bonus scaling), UCBVI-Shaped-P (with only projection), UCBVI-Shaped-BS (with only bonus scaling), UCBVI (standard UCBVI without shaping, as described in [8]).
This is benchmarked across the three environments described in Fig.
3, with various levels of imperfect shaping applied by varying b = { 1 . 5 , 1 . 9 } .
As seen from
Fig. 4, across all environments UCBVI-shaped with projection and bonus scaling performs most favorably, followed typically by UCBVI-Shaped-P, followed by UCBVI-Shaped-BS and UCBVI, suggesting that reward shaping can signiﬁcantly help with learning efﬁciency.
[Figure]
7.2 How does the effectiveness of reward shaping vary across environments?
[Figure] [Table] We next conducted some numerical simulations across environments to understand how the nature of the environment itself affects the sample complexity of learning with shaping via UCBVI-Shaped.
As shown in Fig.
6, we see that UCBVI shaped can be very effective in environments with many irrelevant sub-optimal paths like the double corridor environment in Fig. 3, but is relatively less effective in environments where all exploration is directed the same way such as the single corridor.
Even incorrect but optimistic shaping will provide guidance towards the goal, making UCBVI relatively less dominant in the single corridor environment as compared to the double corridor. This suggests that in environments where
[Figure]
ruling out an entire part of the exploration space is easy from the shaping, we can expect to see larger beneﬁts.
7.3 How does the suboptimality of reward shaping affect learning?
We next compared how different levels of suboptimality of the b sandwich term in the reward shaping affect cumulative regret across environments.
As shown in Fig.
4 (a) and (d), we see that for environments with open paths (like the open gridworld), the shaping degradation has minimal negative effect until it gets very suboptimal.
On the other hand, for corridor and double corridor (Fig. 5 (b)), where there are only a few paths to the goal, suboptimal reward shaping along those paths signiﬁcantly hamper progress.
7.4 Is online UCBVI-Shaped able to infer b online without prior knowledge?
As described in Section 6, UCBVI-shaped can be freed of the assumption of b being known by performing online model selection of b and learning values jointly. In particular, we use the Stochastic CORRAL algorithm [34], a variant of the method introduced in [1] to perform online model selection, with the episodic return being the requisite criterion for updating the model selection distribution.
As we see in Fig 7, this scheme is able to show comparable results to when the actual b is known beforehand, only degrading as the value of b increases.
This suggests that online UCBVI-shaped can be practical in regimes with moderate levels of value corruption.
Discussion
In this work, we take a step towards formally analyzing the beneﬁts of reward shaping, proving that effective reward shaping can lead to more efﬁcient learning than uninformed exploration strategies.
In our analysis, we study an algorithm that incorporates reward shaping into a modiﬁed version of UCBVI, using it to modify bonuses and value function projection.
Our analysis shows that incorporating shaped rewards allows for pruning signiﬁcant parts of the state space and sharpening of optimism in a task directed way.
This reduces the dependence of the regret bound on the state space size and on the horizon, depending on the quality of the shaping term and parameters of the MDP.
This shows how reward shaping can direct exploration and provide signiﬁcant sample complexity beneﬁts while retaining asymptotic performance.
We hope that this work is a step towards moving sample complexity analysis away from being reward agnostic to actually considering reward shaping more formally in analysis.