Faster Reinforcement Learning with Value Target Lower Bounding
Anonymous Author(s)
AffiliationAddress email
Abstract
We show that an arbitrary lower bound of the maximum achievable value can beused to improve the Bellman value target during value learning.
In the tabular case,value learning using the lower bounded Bellman operator converges to the sameoptimal value as using the original Bellman operator, at a potentially faster speed.
In practice, discounted episodic return in episodic tasks and n-step bootstrappedreturn in continuing tasks can serve as lower bounds to improve the value target.
We experiment on Atari games, FetchEnv tasks and a challenging physicallysimulated car push and reach task.
We see large gains in sample efficiency aswell as converged performance over common baselines such as TD3, SAC and
Hindsight Experience Replay (HER) in most tasks, and observe a reliable andcompetitive performance against the stronger n-step methods such as td-lambda,
Retrace and optimality tightening.
Prior works have already successfully applied aspecial case of lower bounding (using episodic return), but are limited to a smallnumber of episodic tasks.
To the best of our knowledge, we are the first to proposethe general method of value target lower bounding (with possibly bootstrappedreturn), to demonstrate its optimality in theory, and effectiveness in a wide rangeof tasks over many strong baselines.
Introduction
The value function is a key concept in dynamic programming approaches to Reinforcement Learning(RL)
(Bellman, 1957).
It estimates the sum of all future rewards (usually time-discounted) of a givenstate.
In temporal difference (TD) learning, the value function is adjusted toward its Bellman targetwhich adds the reward of the current step with the (discounted) value of the next state (Sutton &
Barto, 2018).
This forms the basis of many state of the art RL algorithms such as DQN (Mnih et al.
,2013), DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018).
The value of the next state is typically estimated using a “bootstrapped value” based on the valuefunction itself, which is being actively learned during training.
The bootstrapped values can berandom and far from the optimal value, especially at the initial stage of training, or with sparse rewardtasks where rewards can only be achieved through a long sequence of actions.
Consequently, the
Bellman value targets as well as the learned values are usually far away from the optimal value (thevalue of the optimal policy).
Naturally, this leads to the following idea: If we can make the value target closer to the optimal value,we may speedup TD learning.
For example, we know that the optimal value is just the expecteddiscounted return of the optimal policy, which always upper bounds the expected return of any policy.
For episodic RL tasks, we could use the observed discounted return up to episode end from thetraining trajectories to lower bound the value target.
This makes the new value target closer to theoptimal value, when the empirical return is higher than the Bellman target.
Input: Finite MDP p ( s ′ , r | s, a ) , convergence threshold θ , a lower bound f ( s ) of the maximum achievable value ¯ G v ( s )
Output: State value v ( s ) v ( s ) ←repeat ∆ ←v p ( s ) ← v ( s ) for each state s do ˆ v ( s ) ← max a (cid:80) s ′ ,r p ( s ′ , r | s, a )[ r + γv p ( s ′ )]
ˆ v f ( s ) ← max( f ( s ) , ˆ v ( s )) v ( s ) ← ˆ v f ( s ) ∆ ← max(∆ , | v ( s ) − v p ( s ) | ) end for until ∆ < θ
The case for continuing or non-episodic tasks is less clear though.
When a continuing task canreturn negative rewards, any safe lower bound of the optimal value can be too low to be useful.
Onecould take the risk and use n-step bootstrapped return as a lower bound, which is unsafe becausebootstrapped return can overestimate and be greater than the optimal value.
Can we still use them aslower bounds to improve TD value targets?
Theoretical Results for the Tabular Case
Our results show that for the tabular case, arbitrary functions below a certain bootstrap bound can beused to lower bound the value target to still converge to the same optimal value.
2.1 Background
In finite MDPs with a limited number of states and actions, a table can keep track of the value ofeach state.
Using dynamic programming algorithms such as value iteration, values are guaranteed toconverge to the optimum through Bellman updates (Chapter 4.4 (Sutton & Barto, 2018)).
The core of the value iteration algorithm (Algorithm 1) is the Bellman update of the value function,
B ( v ) , where v ( s ′ ) is the bootstrapped value:
[Equation]
[Equation]
It is well known that the Bellman operator, B , is a contraction mapping over value functions (Denardo,1967).
That is, for any two value functions v 1 and v 2 , ||B ( v 1 ) − B ( v 2 ) || ∞
≤ γ || v 1 − v 2 || ∞ for thediscount factor γ ∈ [0 , 1) and || x || ∞ :
= max i | x i | (the L ∞ norm).
This guarantees that any valuefunction under the algorithm converges to the optimal value B ∞ ( v ) =
v ∗ .
2.2 Convergence of value target lower bounding
Definition 2.1.
The expected n-step bootstrapped return for a given policy π and value function v ( s )is defined as the expected bootstrapped return of taking n steps according to policy π :
[Equation]
Here, the step rewards r i and the resulting n-th step state s n are random variables, with the expectation 58 E π taken over all possible n-step trajectories under the policy π and the given MDP.
Definition 2.2.
Given the current learned value function v ( s ) , policy class Π , the maximum achievablevalue of a state s is defined as:
[Equation]
This is a more relaxed definition of maximum because for each state s , a different policy π ( s ) and adifferent number of steps n ( s ) can be used to achieve the maximum ¯ G v ( s ) .
And the theorem belowsays any function not exceeding the maximum achievable value can be used to lower bound the valuetarget, and still achieve the optimal value in convergence.
Theorem 2.3.
Under the same assumptions for Bellman value contraction, for any function f thatlower bounds the maximum achievable value, i.e. ∀ s, f ( s ) ≤ ¯ G v ( s ) , if we define the lower bounded
Bellman operator as B f ( v ) :
=
max( B ( v ) , f ) , then B ∞ f ( v ) =
B ∞ ( v ) .
Note, the value v ( s ) and the bootstrapped value can be inaccurate, and even above the optimal value.
As a consequence, when n is finite, the maximum achievable value ¯ G v ( s ) (and f ) can be above themaximum expected return (i.e. the optimal value).
On the other hand, when n is sufficiently large,the effect of the bootstrap value v ( s n ) diminishes (see Equation 2), and the maximum achievablevalue becomes the maximum expected return (i.e. the optimal value).
Therefore, ∀ s, ¯ G v ( s ) is nosmaller than the optimal value B ∞ ( v )( s ) .
As a special case of the theorem, as long as f is below theoptimal value, value target lower bounding converges correctly:
Corollary 2.4.
If function f lower bounds the optimal value, i.e. ∀ s, f ( s ) ≤ B ∞ ( v )( s ) , then
B ∞ f ( v ) =
B ∞ ( v ) .
A few things to note about the proof of Theorem 2.3 (included in Appendix 1.1).
First, this only proves convergence, not contraction under the original || v 1 − v 2 || ∞ metric.
In thecase of the Bellman operator, contraction shows that ∀ v 1 , v 2 value functions, ||B ( v 1 ) − B ( v 2 ) || ∞
≤γ ||
v 1 − v 2 || ∞ .
Here, for value target lower bounding, what’s proved is convergence to v ∗ at a rateof γ , not contraction.
There can be counter examples where the distance between v 1 and v 2 underone application of B f can increase in the original L ∞ metric space, even though v 1 and v 2 are bothgetting closer to v ∗ at a rate of γ .
One difficulty caused by convergence instead of contraction is thatthe stopping criterion in Algorithm 1 ( ∆ < θ ) no longer works, due to the inaccessible v ∗ duringlearning.
In practice, this may not be a serious concern, as people often train algorithms for a fixednumber of iterations or time steps.
Second, based on the proof, the new algorithm is at least as fast as the original.
When the lowerbound actually improves the value target, i.e. f ( s ) > B ( v 1 )( s ) , there is a chance for the convergenceto be faster.
Convergence is strictly faster when the lower bound f has an impact on the L ∞ distancebetween the current value and the optimal value, i.e. it increases the value target for the states wherethe differences between the current value and the optimal value are the largest.
Third, the lower bound function doesn’t have to be static during training.
As long as there is a singlef during each training update, convergence is preserved.
The following sections detail how to compute lower bounds of the maximum achievable value(Section 3), how to integrate the lower bounds into state of the art RL algorithms (Section 4), andprovide an illustration of how this method may benefit value learning in practice (Section 4.3).
Example Lower Bound Functions
We show a few cases where lower bound functions can be readily obtained from the trainingexperience.
Future work may investigate alternatives.
3.1 Episodic tasks
In episodic tasks, discounted return is accumulated up to the last step of an episode.
In this case, wecan wait until an episode ends, and compute future discounted returns of all time steps up to the endof the episode.
This episodic return is a lower bound of the optimal value when the environment is deterministic, because the reward sequence can be repeated using the same sequence of actions 2 .
Tomake training efficient, we can compute and store such discounted returns into the replay buffer foreach time step, and simply read them out during training, which adds very little computation to thebaseline one-step TD computation.
[Equation]
We call this variant “lb-DR”, short for lower bounding with discounted return.
3.1.1 Episodic with hindsight relabeled goals
In goal conditioned tasks, one helpful technique is hindsight goal relabeling (Andrychowicz et al.,2017).
It takes a future state that is d time steps away from the current state as the hindsight / relabeledgoal for the current state.
When the goal is reached, a reward of 0 is given, otherwise a -1 reward isgiven for each time step.
In this case, we know it took d steps to reach the hindsight goal, so the discounted future return is:
[Equation]
This calculation can be done on the fly as hindsight relabeling happens, requiring no extra space andvery little computation.
We call this variant “lb-GD”, short for lower bounding with goal distance based return.
Additionally, we can also apply lb-DR and lb-GD together, with discounted episodic return (lb-DR)on the original experience and goal distance based return (lb-GD) on the hindsight experience, givingthe “lb-DR+GD” variant, which was used in Fujita et al.
3.2 In general (including non-
episodic
tasks
If the task is continuing, without an episode end 3 , discounted return needs to be accumulated all theway to infinity.
When rewards are always non-negative, one can still use the accumulated discountedreward of the future n-steps to lower bound the value.
But accumulated n-step discounted reward is nolonger a lower bound when rewards can be negative, in which case, the more general lower boundingwith bootstrapped value can be used: given a trajectory of training experience τ := < s 0 , ..., s n > :
[Equation]
[Equation]
Assuming the rewards and the state s n can be repeated with the same action sequence, G vn ( τ ) lowerbounds the maximum achievable value ¯ G v ( s 0 )
(Equation 3).
Two variations are possible: Given a trajectory of length n ,1.
compute v ( s i ) for all i ∈ [1 , n ] and take the maximum of all G vi ( τ ) to obtain a tighter lowerbound.
We call this variant “lb-b- n step”:
[Equation]
2. only evaluate v on the last ( n th) step and use the n th-step bootstrapped return as the lowerbound, which involves less compute but results in a looser bound.
(When n is large enough,this becomes the lb-DR variant.)
We call this variant “lb-b- n step-only”.
[Equation]
Integration into RL algorithms
4.1 Background
The value target lower bounds can be readily plugged into RL algorithms that regresses value to atarget, e.g. DQN, DDPG or SAC.
In these algorithms, the action value q ( s, a ) is learned through a squared loss with the target value y .
In one step TD return, for a batch B of experience { s, a → r, s ′ } , the loss is:
L q := (cid:88) ( s
[Equation]
∈
[Equation]
In one step TD return, y is the one step TD return ˆ q ( s, a, r, s ′ ) :
[Equation]
Here, q ′ and µ ′ are the bootstrap value and policy functions, typically following the value and policyfunctions in a delayed schedule during training.
(They are also called “target value” and “targetpolicy”, and are very different from the “value target” y in this paper.)
4.2 Value target lower bounding
With lower bounding, we replace the value target y with the lower bounded target:
[Equation]
This way of lower bounding the value target is the same as was done by Fujita et al.
(2020) (confirmedvia personal communication), but is subtly and importantly different from lower bounding the qvalue directly (Oh et al., 2018; Tang, 2020): q ( s, a ) ← max( f, q ( s, a )) , which stays overestimated ifq ( s, a ) initially overestimates.
To the best of our knowledge, value target lower bounding with bootstrapped values is a novelcontribution of this work.
4.3 An Illustrative Example
Figure 1 includes a fairly general example showing how value target lower bounding would improvevalue learning.
Suppose we enhance an off policy algorithm such as DDPG with value target lowerbounding (lb-DR), when there is no training experience hitting the target state, no meaningful traininghappens for the baseline or lb-DR.
However, when there is one trajectory hitting the target state, allstates along the trajectory will soon be propagated with meaningful return, and nearby states will alsoenjoy faster learning.
As the state space becomes larger and the time horizon longer, a successfultrajectory will speed up learning quite a bit.
Experiments
The goal is to demonstrate the sample efficiency of lower bounding the value target over baseline suchas DDPG, TD3, SAC and HER.
Because the lower bounded value target can now look potentiallymany steps into the future, we suspect it to be best suited for long horizon, sparse reward tasks.
Hence, we choose to experiment on a sampled subset of Atari games, the goal conditioned FetchEnvtasks and the harder goal conditioned Pioneer Push and Reach tasks.
See details of the experimentsetup in Appendix 1.2.
5.1 Baselines
Baselines include DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al.,2018) and HER (Andrychowicz et al., 2017; Plappert et al., 2018).
Implementations are based on
S
T
Baseline:
S
T
Baseline:
S
T
Baseline:
S
T
Baseline:
S
T
Lowerbound
T S
Lowerbound
T S
Lowerbound
T S
Lowerbound open sourced repositories, and baseline performance is verified against published results under similarsettings.
The Appendix 1.6 and 1.5 include results on more baselines such as DDQN (van Hasseltet al., 2015), td-labmda (Sutton & Barto, 2018) and Retrace (Munos et al., 2016).
5.2 Hyperparameters
Value target lower bounding is applied on top of these baselines without any additional hyperparameter(Section 4).
The only hyperparameters come from the baselines.
These hyperparameters followpublished work as much as possible.
When baseline hyperparameters need to be tuned for anenvironment, e.g. Atari games or Pioneer tasks, we search for the best performance in total episodereward averaged across all tasks for that environment on one set of random seeds, then the optimalhyperparameters are fixed and evaluated on a separate set of random seeds never seen duringdevelopment.
Value target lower bounding simply uses the the parameter values optimal for thebaselines.
Details are in Appendix 1.3.
5.3 Results
We report results on both episodic and continuing/non-episodic tasks.
We report evaluation performance averaged across several runs of the algorithms (five for the less stable Atari games andthree for the others).
Each run uses a random seed never seen during development.
Due to spaceconstraints, the main paper only reports performance aggregated across all tasks for each environment.
During each run, we take one task and one random seed, run baseline and treatment algorithms, andrecord whether treatment agent evaluates strictly above the baseline agent as training progresses.
Weaverage across all the runs of the same environment, and plot the fraction of times where treatment isabove baseline and the standard deviation of that fraction in Figure 2.
Appendix 1.4 contains per taskevaluation curves.
Overall, value target lower bounding is a simple, effective, efficient, carefree (no hyperparameter)and theoretically justified approach.
Although the example lower bounds are limited to deterministicenvironments, the theory is generally applicable to stochastic environments. A similar prior workto compare would be Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), which issimple, effective, efficient, and also limited to deterministic environments (Blier & Ollivier, 2021).
However, unlike our work, HER relies on the task being goal conditioned with full knowledge of thereward function, has one hyperparameter to tune (the proportion of hindsight experience), and is notjustified in theory for stochastic environments. Our work shows further significant gains on top of
HER on hard continuous control tasks.
0.0 0.2 0.4 0.6 0.8 1.0 1.2 Environment [Figure] 1e7 0.
F r a c t i on o f r ando m t a sk t r i a l s
Fraction lb­DR (ours) above sac (a) Atari (lb-DR) 0.0 0.2 0.4 0.
0.8 1.0 1.2 Environment [Figure] 1e7 0.
F r a c t i on o f r ando m t a sk t r i a l s
[Figure] lb­b­3step (
[Figure] [Figure] 1e6 0.
F r a c t i on o f r ando m t a sk t r i a l s
[Figure] lb­DR (ours) [Figure] ddpg (c) FetchEnv
[Figure]
1e6 0.
F r a c t i on o f r ando m t a sk t r i a l s
[Figure]
lb­DR+GD ( [Figure] her ( [Figure]
[Figure]
SAC/DDPG/HER Figure 2 compares the lower bounding treatment with SAC, DDPG or HER baseline on 17 sampled Atari games, the FetchEnv tasks and the Pioneer tasks.
For all the environments, value target lowerbounding is not only more sample efficient, but also enjoys a higher converged performance. Aftertraining starts, it quickly gains ground and outperforms the baseline for 70% to 100% of the runs. Itkeeps that advantage even at the end of the training, outperforming baseline in converged performance. These plots show how frequently treatment is above baseline, but is insensitive to the magnitude ofchange.
Appendix 1.4 shows the magnitude of change with total episode return plotted for each task, oftenwith large gains in sample efficiency and sometimes much higher converged performance.
Among allthe 22 tasks, only one task (Atari Breakout) shows lb-DR underperforming the baseline.
Investigations show that the loss on Atari Breakout is likely due to the mismatch between trainingobjective and evaluation metric.
During training, raw rewards are clipped to [-1, 1] and step discountedat γ = 0 .
99 to compute value, while in evaluation, total reward is the unclipped and undiscountedcumulative sum of episode rewards.
The discount and clipping together severely penalizes largerewards earned later in the episode, which is what’s happening for Breakout, because hitting a toplayer block produces a reward of 7 while hitting a bottom layer block produces 1.
When we usenon-clipped rewards or a higher γ in training, the lower bounding method performs much better intotal reward.
Note, this train-test discrepancy as well as an additional training bias (Thomas, 2014) islikely present in all of the prior works using policy gradient methods on Atari games.
5.3.2 Value target improvement
The lb-DR method is mostly effective, but is it really due to improvements to the value targets?
Figure 3 looks at the fraction of training experience where lower bounding actually improves the Bellman value target over the course of training.
Overall, improved value target roughly coincideswith performance gains.
Appendix 1.4 shows the plots per each task.
The Appendix also has more results, comparing with baselines such as n-step methods, DDQN andoptimality tightening, and more analyses such as ablations and robustness to hyperparameter choices.
Related Work
Prior works (Fujita et al., 2020; Hoppe & Toussaint, 2020; He et al., 2017; Oh et al., 2018; Tang,2020) employed several different ways of computing future returns and using that as a lower boundto improve value learning.
It is quite easy to introduce biases and inefficiencies into the process andend up with a suboptimal or inefficient algorithm.
Our work is the first to propose the general form ofvalue target lower bounding (possibly with bootstrapping), to show its convergence to the optimalvalue in the tabular case, and to demonstrate its effectiveness in illustrative examples and extensiveexperiments on a wide range of tasks.
Fujita et al.
(2020)’s method is similar to a special case (the lb-DR+GD variant) of the generalmethod.
They used it as a part of a large system and showed that it improved sample efficiency for arobotic grasping task.
Hoppe & Toussaint (2020) also bounded the value target.
But instead of usingempirical return, they used a simplified MDP with a subset of actions.
Although without theoreticalproof and only experimented on a limited set of robotic manipulation tasks, both works show thatvalue target lower bounding increased sample efficiency.
This work, in addition to the theory and themore general method, shows that lower bounding improves both sample efficiency and convergedperformance in a wide range of tasks.
He et al.
(2017) used empirical return with bootstrap to improve value learning.
They formulated valuelearning as a constrained optimization problem with the empirical bootstrapped value being the lower(and upper) constraints of the value function.
In their experiments, the Lagrangian multiplier wasfixed, which would likely lead to suboptimal solutions.
Our lb-b- n step method also uses bootstrappedvalue.
But we lower bound the value target directly, which is simpler, more efficient, and likely moreoptimal.
Our work points out that for episodic tasks, even more efficient and effective methods likelb-DR exist.
Appendix 1.5 offers more discussion and results related to this.
Our work is subtly but importantly different from the prior works on lower bound Q learning or Self
Imitation Learning (SIL)
(Oh et al., 2018; Tang, 2020).
SIL uses empirical return R to lower boundthe value function itself (instead of the value target ).
This is achieved by adding an off policy valueloss during on-policy (AC or PPO) training ( L silvalue = 12 | v ( s ) − max( v ( s ) , R ) | 2 ).
When the valuefunction overestimates, the SIL value loss becomes zero, and keeps overestimating.
Mixing the SILloss with the loss from the baseline algorithms probably helped to correct the overestimation, but notheoretical guarantee was given.
In evaluation, SIL was often compared to on-policy Actor Critic or
PPO baselines, so it was not clear how much of the gain was due to lower bounding and how muchdue to off-policy value learning.
In this work, we bound the Bellman value target (Equation 11), sooverestimates are automatically corrected via Bellman updates, and convergence is guaranteed in thetabular case.
We also use off-policy algorithms as baselines for a cleaner comparison.
N-step return methods such as td-lambda (Sutton & Barto, 2018) and Retrace (Munos et al.
, 2016)also look a few steps ahead, but to obtain more accurate value of the behavior policy.
Traditionally,this requires careful off-policy correction, and the value can still be far from the optimal value dueto the often suboptimal behavior.
This work shows that value target lower bounding efficiently andeffectively looks ahead much further without the need for off-policy correction, due to aiming at theoptimal value.
Appendix 1.5 has more detailed observations and discussions.
Planning methods can look into the future to achieve higher value targets and better control.
Examplesinclude Monte Carlo Tree Search (MCTS) (Schrittwieser et al., 2019; Ye et al., 2021) and Model
Predictive Control (MPC) or receding horizon planning with raw actions (Chua et al., 2018; Hafneret al., 2019; Zhang et al., 2022), options (Silver & Ciosek, 2012), or subgoals (Nasiriany et al.
,2019; Nair & Finn, 2020;
Chane-Sane et al., 2021).
Planning methods use either a dynamics modeltogether with the learned value or just the learned value (in the case of goal conditioned tasks)(Nasiriany et al., 2019) to improve policy or value estimates.
Planning typically happens during rollout (Nasiriany et al., 2019), but can also be used to improve the value target, as in Reanalyze of
MuZero (Schrittwieser et al., 2019; Ye et al., 2021).
During value improvement, if planning takes the maximum over a set of possible future values (e.g. from different trajectories as in the caseof MPC), and if this set includes the one step Bellman value target, then the planner is essentiallyusing alternative trajectories and their values to lower bound the Bellman value target.
In this sense,the theory developed here can potentially justify and improve Reanalyze.
In general, planning isorthogonal to value target lower bounding, and typically requires additional components and a lotmore compute than the basic TD learning does.
Therefore, we leave it to future work to explore thesynergy between the two.
Interestingly, it is common practice to lower and upper bound the returns to the possible region,e.g. Andrychowicz et al.
(2017) bounds value between [ − 11 − γ , 0] .
Similar to lower bounding withepisodic return (Section 3.1), such strict bounds of the actual value can be thought of as admissibleheuristics (bounds) used during search of the optimal solution (Russell & Norvig, 2020).
What’s newin this work is that lower bounding with bootstrapped values (which can overestimate the value) stillconverges to the optimal value.
Kumar et al.
(DisCor) also recognized that bootstrapped value targets can be inaccurate.
Thisbias impacts learning adversely under function approximation.
DisCor uses distribution correction tosample experience with accurate bootstrap targets more frequently, while value target lower boundingaims to directly reduce the bias.
While in theory using empirical return to lower bound the value target is only correct for deterministicenvironments, in practice, it seems as long as the environment is not heavily impacted by random fluctuations, they still perform well.
In fact, with function approximation, the agents cannot distinguishbetween two slightly different states, making the problem partially observable (Sutton & Barto, 2018)and appear slightly random.
Prior methods such as SIL (Oh et al., 2018), Optimality Tightening (Heet al., 2017), and even Hindsight relabeling (Andrychowicz et al., 2017) and MuZero (Schrittwieseret al., 2019) require the environment to be deterministic.
Despite this theoretical limitation, the lowerbounding methods and the prior methods can still be very useful, outperforming baselines often bylarge margins and when deploying to the real world (Fujita et al., 2020).
Conclusions
We propose a general form of lower bounding the value target using possibly bootstrapped return.
Intheory, value target lower bounding converges to the same optimal solution as the original Bellmanoperator.
In practice, several ways of finding value lower bounds are examined.
For episodic tasks, discounted episodic return is an efficient and effective method involving verylittle extra computation.
Precomputing the episodic return and storing it into the replay bufferallows efficient lower bound computation.
It achieves much higher sample efficiency and convergedperformance than one-step baselines such as SAC, DDPG or TD3 in most tasks, and is competitiveamong n-step baselines.
Simple goal distance based return uses even less compute and achieves largegains in certain long horizon tasks over Hindsight relabeling (HER).
For non-episodic tasks or in general, lower bounding with n-step bootstrapped return outperformsone-step baselines and is a strong competitor to the n-step methods such as (truncated) td-lambda and
Retrace.
7.1 Future Work
There are probably better ways of finding value lower bounds that improve training even more.
Onedirection may be to use planning (e.g. Monte Carlo Tree Search, the Cross Entropy Method or usingsubgoals) to achieve tighter lower bounds given a model of the task.
Estimating value lower bound for stochastic tasks may be possible, e.g. by learning a rewardfunction and a dynamics model and using imagined rollouts to obtain bootstrapped returns withoutoverestimation.
Other ways of bounding the value target, e.g. upper bounding (He et al., 2017), may be worthinvestigating as well, e.g. to reduce overestimation in regions of poor reward.