O FFLINE R EINFORCEMENT L EARNING WITH V ALUEBASED E PISODIC M EMORY
A BSTRACT
Ofﬂine reinforcement learning (RL) shows promise of applying RL to real-world problems by effectively utilizing previously collected data.
Most existing ofﬂine
RL algorithms use regularization or constraints to suppress extrapolation error for actions outside the dataset.
In this paper, we adopt a different framework, which learns the V -function instead of the Q -function to naturally keep the learning procedure within the ofﬂine dataset.
To enable effective generalization while maintaining proper conservatism in ofﬂine learning, we propose Expectile V -Learning (EVL), which smoothly interpolates between the optimal value learning and behavior cloning.
Further, we introduce implicit planning along ofﬂine trajectories to enhance learned V -values and accelerate convergence.
Together, we present a new ofﬂine method called Value-based Episodic Memory (VEM).
We provide theoretical analysis for the convergence properties of our proposed VEM method, and empirical results in the D4RL benchmark show that our method achieves superior performance in most tasks, particularly in sparse-reward tasks.
Our code is public online at https://github.com/YiqinYang/VEM.
I NTRODUCTION
Despite the great success of deep reinforcement learning (RL) in various domains most current algorithms rely on interactions with the environment to learn through trial and error.
In real-world problems, particularly in risky and safety-crucial scenarios, interactions with the environment can be expensive and unsafe, and only ofﬂine collected datasets are available, such as the expert demonstration or previously logged data.
This growing demand has led to the emergence of ofﬂine reinforcement learning (ofﬂine RL) to conduct RL in a supervised manner.
The main challenge of ofﬂine RL comes from the actions out of the dataset’s support (Kumar et al., 2019; 2020).
The evaluation of these actions that do not appear in the dataset relies on the generalization of the value network, which may exhibit extrapolation error (Fujimoto et al., 2019).
This error can be magniﬁed through bootstrapping, leading to severe estimation errors.
A rapidly developing line of recent work (Fujimoto et al., 2019; Kumar et al., 2020; Ghasemipour et al., 2021; Yang et al., 2021) utilizes various methods to constrain optimistic estimation on unseen actions, such as restricting available actions with a learned behavior model (Fujimoto et al., 2019) or penalizing the unseen actions with additional regularization (Kumar et al., 2020).
However, conﬁning learning within the distribution of the dataset can be insufﬁcient for reducing extrapolation errors.
Another line of methods, on the contrary, uses the returns of the behavior policy as the signal for policy learning, as adopted in Wang et al.
; Peng et al.
; Chen et al.
By doing so, they keep the value learning procedure completely within the dataset .
However, the behavior policy of the dataset can be imperfect and insufﬁcient to guide policy learning.
To achieve a tradeoff between imitation learning and optimal value learning while conﬁnes learning within the dataset ,
[Figure]
[Figure]
we propose Expectile V -learning (EVL), which is based on a new expectile operator that smoothly interpolates between the Bellman expectation operator and optimality operator.
To better solve long-horizon and sparse-reward tasks, we further propose using value-based planning to improve the advantage estimation for policy learning.
We adopt an implicit memory-based planning scheme that strictly plans within ofﬂine trajectories to compute the advantages effectively, as proposed in recent advances in episodic memory-based methods (Hu et al., 2021).
Together, we present our novel framework for ofﬂine RL, Value-based Episodic Memory (VEM), which uses expectile V -learning to approximate the optimal value with ofﬂine data and conduct implicit memorybased planning to further enhance advantage estimation.
With the properly learned advantage function, VEM trains the policy network in a simple regression manner.
We demonstrate our algorithm in Figure 1, and a formal description of our algorithm is provided in Algorithm 1.
The contributions of this paper are threefold.
First, we present a new ofﬂine V -learning method,
EVL, and a novel ofﬂine RL framework, VEM.
EVL learns the value function through the trade-offs between imitation learning and optimal value learning.
VEM uses a memory-based planning scheme to enhance advantage estimation and conduct policy learning in a regression manner.
Second, we theoretically analyze our proposed algorithm’s convergence properties and the trade-off between contraction rate, ﬁxed-point bias, and variance.
Speciﬁcally, we show that VEM is provably convergent and enjoys a low concentration rate with a small ﬁxed-point bias.
Finally, we evaluate our method in the ofﬂine RL benchmark D4RL (Fu et al., 2020).
Comparing with other baselines, VEM achieves superior performance, especially in the sparse reward tasks like AntMaze and Adroit.
The ablation study shows that VEM yields accurate value estimates and is robust to extrapolation errors.
B ACKGROUND
Preliminaries .
We consider a Markov Decision Process (MDP) M deﬁned by a tuple ( S , A , P, r, γ ) , where S is the state space, A is the action space, P ( ·| s, a ) : S ×
A × S → R is the transition distri- bution function, r ( s, a ) :
S ×A → R is the reward function and γ ∈ [0 , 1) is the discount factor.
We say an environment is deterministic if P ( ·| s, a ) =
δ ( s (cid:48) ) for some s (cid:48) , where δ ( · ) is the Dirac function.
The goal of an RL agent is to learn a policy π : S ×
A → R , which maximizes the expectation of a discounted cumulative reward:
[Equation]
, where ρ 0 denotes the distribution of the initial states.
Value-based Ofﬂine Reinforcement Learning Methods .
Current ofﬂine RL methods can be roughly divided into two categories according to types of learned value function: Q -based and
V -based methods.
Q -based methods, such as BCQ (Fujimoto et al., 2019), learn Q -function for policy learning and avoid selecting unfamiliar actions via constraints or penalty.
On the contrary,
V -based methods (Peng et al., 2019; Siegel et al., 2020; Chen et al., 2020) learns the value of behavior policy V µ ( s ) with the trajectories in the ofﬂine dataset D and update policy as a regression problem.
Based on the learned V -function, V -based methods like AWR (Peng et al., 2019) updates the policy using advantage-weighted regression, where each state-action pair is weighted according to the exponentiated advantage:
[Equation]
Episodic Memory-Based Methods .
Inspired by psychobiology, episodic memory-based methods store experiences in a non-parametric table to fast retrieve past successful strategies when encountering similar states.
Model-free episodic control (Blundell et al., 2016a) updates the memory table by taking the maximum return R ( s, a ) among all rollouts starting from same state-action pair ( s, a ) .
Hu et al.
(2021) proposes Generalizable Episodic Memory , which extends this idea to the continuous domain, and proposes updating formula with a parametric memory Q EMθ .
M ETHOD
In this section, we describe our novel ofﬂine method, value-based episodic memory, as depicted in
Figure 1.
VEM uses expectile V -learning (EVL) to learn V -functions while conﬁnes value learning within the dataset to reduce extrapolation error.
EVL uses an expectile operator that interpolates between Bellman expectation operator and optimality operator to balance behavior cloning and optimal value learning.
Further, VEM integrates memory-based planning to improve the advantage estimation and accelerate the convergence of EVL.
Finally, generalized advantage-weighted learning is used for policy learning with enhanced advantage estimation.
A formal description for the
VEM algorithm is shown in Algorithm 1 in Appendix A.1.
3. E XPECTILE V-L EARNING
To achieve a balance between behavior cloning and optimal value learning, we consider the Bellman expectile operator deﬁned as follows:
[Equation]
where µ is the behavior policy, δ ( s, a )
=
E s (cid:48) ∼ P ( ·| s,a ) [ r ( s, a ) + γV ( s (cid:48) ) − v ] is the expected onestep TD error and [ · ] + = max( · , 0) .
This operator resembles the expectile statistics (Newey &
Powell, 1987; Rowland et al., 2019) and hence its name.
We can see that when τ = 1 / 2 , this operator is reduced to Bellman expectation operator, while when τ → 1 , this operator approaches
Bellman optimality operator, as depicted in Lemma 3.
[Figure]
We use the following toy example to further illustrate the trade-offs achieved by EVL.
Consider a random generated MDP.
When the operator can be applied exactly, the Bellman optimality operator is sufﬁcient to learn the optimal value V ∗ .
However, applying operators with an ofﬂine dataset raises a noise on the actual operator due to the estimation error with ﬁnite and biased data.
We simulate this effect by adding random Gaussian noise to the operator.
Applying the optimality operator on ofﬂine datasets can lead to severe overestimation due to the maximization bias and bootstrapping.
The value estimation learned by EVL, on the contrary, achieves a trade-off between learning optimal policy and behavior cloning and can be close to the optimal value with proper chosen τ , as depicted in Figure 2.
The noise upon the operator largely depends on the size of the dataset.
Estimation error can be signiﬁcant with insufﬁcent data.
In this case, we need a small τ to be conservative and be close to behavior cloning.
When the dataset is large and we are able to have an accurate estimation for the operator, we can use a larger τ to recover the optimal policy.
By adjusting τ , the expectile operator can accommodate variant types of datasets.
However, the expectile operator in Equation 2 does not have a closed-form solution.
In practice, we consider the one-step gradient expectile operator
[Equation]
where α is the step-size.
Please refer to Appendix B.1 for the detailed derivation.
For notational convenience, we use T µτ to denote the one-step gradient expectile operator ( T g ) µτ hereafter.
We consider the case where the dynamics are nearly-deterministic like robotic applications, and we remove the expectation over the next states in the operator.
This leads to a practical algorithm,
Expectile V -Learning, where we train the value network to minimize the following loss:
[Equation]
where ˆ V is the target value after applying one-step gradient expectile operator and δ ( s, a, s (cid:48) )
=
r ( s, a ) + γV θ (cid:48) ( s (cid:48) )
− V θ (cid:48) ( s ) .
V -function and the target
ˆ V -function are parameterized by θ and θ (cid:48) , respectively.
EVL is guaranteed to converge with concentration rate γ τ = 1 − 2(1 − γ )
α max { τ, 1 − τ } .
Please refer to Section 4 for a detailed analysis.
3. I MPLICIT M EMORY -B ASED P LANNING
Although EVL reduces the extrapolation error, it is still a challenging problem to bootstrap over long time horizons due to estimation errors with a ﬁxed dataset.
Therefore, we propose using valuebased planning to conduct bootstrapping more efﬁciently.
We adopt an implicit memory-based planning scheme that strictly plans within ofﬂine trajectories to avoid over-optimistic estimations in the planning phase.
This is aligned with recent advances in episodic memory-based methods (Hu et al., 2021), but we conduct this planning on expectile V -values rather than Q -values.
Speciﬁcally, we compare the best return so far along the trajectory with the value estimates ˆ V and takes the maximum between them to get the augmented return ˆ R t :
[Equation]
where t denotes steps along the trajectory, T is the episode length, and ˆ V is generalized from similar experiences.
This procedure is conducted recursively from the last step to the ﬁrst step along the trajectory, forming an implicit planning scheme within the dataset to aggregate experiences along and across trajectories.
Further, the back-propagation process in Equation 5 can be unrolled and rewritten as follows:
[Equation]
where n denotes different length of rollout steps and ˆ V t,n = 0 for n > T .
3. G ENERALIZED A DVANTAGE -W EIGHTED L EARNING
Based on ˆ R t calculated in Section 3.2, we can conduct policy learning in a regression form, as adopted in return-based ofﬂine RL methods (Nair et al., 2020; Siegel et al., 2020; Peng et al., 2019):
[Equation]
where ˆ A ( s t , a t ) =
ˆ R t − ˆ V ( s t ) and f is an increasing, non-negative function.
Please refer to Appendix C.1 for the detailed implementation of Equation 7.
Note that ˆ R t is not the vanilla returns in the dataset, but the enhanced estimation calculated by implicit planning from ˆ V t , as opposed with other return based methods.
Please refer to Algorithm 1 and Section 4 for implementation details and theoretical analysis.
T HEORETICAL A NALYSIS
In this section, we ﬁrst derive the convergence property of expectile V -Learning.
Then, we demonstrate that memory-based planning accelerates the convergence of the EVL.
Finally, we design a toy example to demonstrate these theoretical analyses empirically.
Please refer to Appendix B for the detailed proofs of the following analysis.
4. C ONVERGENCE P ROPERTY OF THE E XPECTILE V-L EARNING
In this section, we assume the environment is deterministic.
We derive the contraction property of
T µτ as the following statement:
Lemma 1.
For any τ ∈ (0 , 1) , T µτ is a γ τ -contraction, where γ τ = 1 − 2 α (1 − γ ) min { τ, 1 − τ } .
Proof.
We introduce two more operators to simplify the analysis:
[Equation]
Next we show that both operators are non-expansion (e.g.,
[Equation]
Finally, we rewrite T µτ based on T µ + and T µ −
and we prove that T µτ is a γ τ -contraction.
Please refer to Appendix B.2 for the complete proof.
Based on Lemma 1, we give a discussion about the step-size α and the fraction τ :
About the step-size α .
Generally, we always want a larger α .
However, α must satisfy that
V ( s )+2 ατδ ( s, a ) ≤ max { r ( s, a )+ γV ( s (cid:48) ) , V ( s ) } and V ( s )+2 α (1 − τ ) δ ( s, a ) ≥ min { r ( s, a )+ γV ( s (cid:48) ) , V ( s ) } , otherwise the V -value will be overestimated.
Thus, we must have 2 ατ ≤and 2 α (1 − τ ) ≤ 1 , which infers that α ≤ 1 2max { τ, 1 − τ } .
When α = 1 2max { τ, 1 − τ } , we have γ τ = 1 − 2 α min { τ, 1 − τ } (1 − γ )
= 1 − min { τ, 1 − τ } max { τ, 1 − τ } (1 − γ ) .
About the fraction τ .
It is easy to verify that γ τ approaches to 1 when τ → 0 or τ → 1 , which means that with a larger τ the contractive property is getting weaker.
The choice of τ makes a tradeoff between the learning stability and the optimality of values.
We further point out that when τ = 1 , the Expectile V -learning degrades as a special case of the generalized self-imitation learning (Tang, 2020), which losses the contractive property.
Next, we prove that T µτ is monotonous improving with respect to τ :
Lemma 2.
For any τ, τ (cid:48) ∈ (0 , 1) , if τ (cid:48) ≥ τ , we have T µτ (cid:48)
V ( s ) ≥ T µτ V ( s ) , ∀ s ∈ S .
Based on the Lemma 2, we derive that V ∗ τ is monotonous improving with respect to τ :
Proposition 1.
Let V ∗ τ denote the ﬁxed point of T µτ .
For any τ, τ (cid:48) ∈ (0 , 1) , if τ (cid:48) ≥ τ , we have
V ∗ τ (cid:48) ( s ) ≥ V ∗ τ ( s ) , ∀ s ∈ S .
Further, we derive that V ∗ τ gradually approaches V ∗ with respect to τ :
Lemma 3.
Let V ∗ denote the ﬁxed point of Bellman optimality operator T ∗ .
In the deterministic
MDP, we have lim τ
→ 1 V ∗ τ
= V ∗ .
Based on the above analysis, we have the following conclusion:
Remark 1.
By choosing a suitable τ , we can achieve the trade-off between the contraction rate and the ﬁxed point bias.
Particularly, a larger τ introduces a smaller ﬁxed point bias between V ∗ τ and
V ∗ , and produces a larger contraction rate γ τ simultaneously.
4. V ALUE - BASED E PISODIC M EMORY
In this part, we demonstrate that the memory-based planning effectively accelerates the convergence of the EVL.
We ﬁrst deﬁne the VEM operator as:
[Equation]
5 Bias 0 .0 .0 .0 .0 .0 .
C o n t r a c t i o n R a t e 0 .0 .0 .
9 Contraction Rate 0 .0 .0 .
V a r i a n ce (a)
The maximal rollout step n max .
2 Bias 0 .0 .0 .0 .
C o n t r a c t i o n R a t e 0 .0 .0 .0 .
50 Contraction Rate 0 .0 .0 .0 .0 .
V a r i a n ce (b) The different behavior policies.
where n max is the maximal rollout step for memory control.
Then, we derive that multi-step estimation operator T vem does not change the ﬁxed point and contraction property of T µτ :
Lemma 4.
Given τ ∈ (0 , 1) and n max ∈ N + , T vem is a γ τ -contraction.
If τ > 1 2 , T vem has the same ﬁxed point as
T µτ .
Next, we derive that the contraction rate of T vem depends on the dataset quality.
Further, we demonstrate that the convergence rate of T vem is quicker than T µτ even the behavior policy µ is random:
Lemma 5.
When the current value estimates V ( s ) are much lower than the value of behavior policy,
T vem provides an optimistic update.
Formally, we have
[Equation]
where n ∗ ( s ) = arg max 0 <n ≤ n max { ( T µ ) n − 1 T µτ V ( s ) } , V µn ∗ ,τ is the ﬁxed point of ( T µ ) n ∗ ( s ) − 1 T µτ
and it is the optimal rollout value starting from s .
This lemma demonstrates that T vem can provide an optimistic update for pessimistic value estimates.
Speciﬁcally, the scale of the update depends on the quality of the datasets.
If the behavior policy µ is expert, which means V µn ∗ ,τ is close to V ∗ τ .
Then, following the lemma, the contraction rate will be near to γ n ∗ ( s ) − 1 γ τ .
Moreover, if the initial value estimates are pessimistic (e.g., the initialized value function with zeros), we will have n ∗ ( s ) ≈ n max , indicating that the value update will be extremely fast towards a lower bound of V ∗ τ .
On the contrary, if µ is random, we have n ∗ ( s )
≈
and the value update will be slow towards V ∗ τ .
Remark 2.
By choosing a suitable n max , we can achieve the trade-off between the contraction rate and the estimation variance, i.e., a larger n max yields a fast update towards a lower bound of ﬁxed point and tolerable variances empirically.
Meanwhile, the choice of n max does not introduce additional bias, and the ﬁxed point bias is totally controlled by τ .
4. T OY E XAMPLE
We design a toy example in the random deterministic MDP to empirically demonstrate the above analysis.
Following (Rowland et al., 2020), we adopt three indicators, including update variance, ﬁxed-point bias, and contraction rate, which is shown in Figure 3.
Speciﬁcally, the contraction rate is sup V (cid:54)
=
V (cid:48)
(cid:107)T vem V − T vem V (cid:48) (cid:107) ∞
(cid:107) V − V (cid:48)
(cid:107) ∞ , the bias is (cid:107) V ∗ vem − V ∗ (cid:107) ∞ and the variance is
E (cid:104) (cid:107) ˆ T V − T vem V (cid:107)(cid:105) 12 , where ˆ
T vem is the stochastic approximation of T vem and V ∗ vem is the ﬁxed pointed of T vem .
First, the experimental results in Figure 3(a) demonstrate that the relationship of n -step estimation and τ .
Formally, the contraction rate decreases as n becomes larger, and the ﬁxed-point bias increases as τ becomes smaller, which are consistent with Lemma 1 and Lemma 2.
Figure 3(a) also shows that the variance is positively correlated with n .
Second, the experimental results in Figure 3(b) demonstrate that the relationship of dataset quality and τ .
The higher dataset quality corresponds to the lower contraction rate and variance, which is consistent with Lemma 5.
[Figure]
R ELATED W ORK
Ofﬂine Reinforcement Learning.
Ofﬂine RL methods (Kumar et al., 2019; Siegel et al., 2020;
Argenson & Dulac-Arnold, 2020; Wu et al., 2021; Dadashi et al., 2021; Kostrikov et al., 2021; Jin et al., 2021; Rashidinejad et al., 2021) can be roughly divided into policy constraint, pessimistic value estimation, and model-based methods.
Policy constraint methods aim to keep the policy to be close to the behavior under a probabilistic distance (Fujimoto et al., 2019; Peng et al., 2019; Nair et al., 2020).
Pessimistic value estimation methods like CQL (Kumar et al., 2020) enforces a regularization constraint on the critic loss to penalize overgeneralization.
Model-based methods attempt to learn a model from ofﬂine data, with minimal modiﬁcation to the policy learning (Kidambi et al., 2020; Yu et al., 2020; Janner et al., 2019).
However, these methods have to introduce additional behavioral policy models, dynamics models, or regularization terms (Zhang et al., 2020b;a; Lee et al., 2021).
Another line of methods uses empirical return as the signal for policy learning, which conﬁnes learning within the dataset but leads to limited performance (Levine et al., 2020; Geist et al., 2019; Wang et al., 2021).
Episodic Control.
Episodic control aims to store good past experiences in a non-parametric memory and rapidly latch into past successful policies when encountering similar states instead of waiting for many optimization steps (Blundell et al., 2016b).
Pritzel et al.
(2017) and Lin et al.
(2018) introduce a parametric memory, which enables better generalization through neural networks.
Our work is closely related to recent advances in Hu et al.
(2021), which adopts an implicit planning scheme to enable episodic memory updates in continuous domains.
Our method follows this implicit scheme, but conducts planning with expectile V -values to avoid overgeneralization on actions out of dataset support.
E XPERIMENTS
In our experiments, we aim to answer the following questions: 1) How does our method performe compared to state-of-the-art ofﬂine RL algorithms on the D4RL benchmark dataset? 2) How does implicit planning affect the performance on sparse reward tasks?
3) Can expectile V -Learning effectively reduces the extrapolation error compared with other ofﬂine methods? 4)
How does the critical parameter τ affect the performance of our method?
6. E VALUATION ENVIRONMENTS
We ran VEM on AntMaze, Adroit, and MuJoCo environments to evaluate its performance on various types of tasks.
Precisely, the AntMaze navigation tasks control an 8-DoF quadruped robot to reach a speciﬁc or randomly sampled goal in three types of maps.
The reward in the AntMaze domain is highly sparse.
The Adroit domain involves controlling a 24-DoF simulated hand tasked with hammering a nail, opening a door, twirling a pen, or picking up and moving a ball.
On the adroit tasks, these datasets are the following, “human”: transitions collected by a human operator,
[Table]
, averaged over three random seeds with ± standard deviation.
Scores range from 0 to 100, where 0 corresponds to a random policy performance, and 100 indicates an expert.
We use the results in Fu et al.
(2020) for AWR and BCQ, and use the results in Kumar et al.
(2020) for CQL.
The results of BAIL come from our implementation according to the ofﬁcial code (https://github.com/lanyavik/BAIL).
“cloned”: transitions collected by a policy trained with behavioral cloning interacting in the environment + initial demonstrations, “expert”: transitions collected by a ﬁne-tuned RL policy interacting in the environment.
As for the MuJoCo tasks, the datasets are “random”: transitions collected by a random policy,“medium”: transitions collected by a policy with suboptimal performance.
The complete implementation details are presented in Appendix C.
6. P ERFORMANCE ON D4RL TASKS
As shown in Table 1, VEM achieves state-of-the-art performance on most AntMaze tasks and has a signiﬁcant improvement over other methods on most Adroit tasks.
VEM also achieves good performances in MuJoCo domains.
We ﬁnd that VEM has low value estimation errors in all tasks, which promotes its superior performance.
However, as a similar training framework, BAIL only has reasonable performances on simple ofﬂine tasks, such as MuJoCo.
Please refer to Appendix D.2 for the complete training curves and value estimation error on D4RL.
To further analyze the superior performance of VEM in the sparse reward tasks, we visualize the learned value estimation in AntMaze tasks, which is shown in Figure 4.
Experimental results show that VEM has the higher value estimates on the critical place of the map (e.g., corners) since various trajectories in the datasets are connected.
The accurate value estimation leads to its success on complex sparse reward tasks.
6. A NALYSIS OF V ALUE E STIMATION
As both Expectile V -Learning (EVL) and Batch Constrained Q -Learning (BCQ) (Fujimoto et al., 2019) aim to avoid using the unseen state-action pairs to eliminate the extrapolation error, we replace EVL in VEM with BCQ (named BCQ-EM) to evaluate the effectiveness of the EVL module.
The experimental results in Figure 9 in Appendix D.1 indicate that the performance of BCQ-EM is mediocre, and BCQ reaches performance signiﬁcantly below VEM.
We observe a strong correlation between the training instability and the explosion of the value estimation.
This result should not come as a surprise since the Adroit tasks have a larger action space compared with MuJoCo domains and narrow human demonstrations.
Therefore, the generative model in BCQ cannot guarantee completely the unseen actions are avoided.
In contrast, VEM avoids fundamentally unseen actions by keeping the learning procedure within the support of an ofﬂine dataset, indicating the necessity of the EVL module.
Please refer to Appendix C for the implementation details.
We evaluate τ ∈ { 0 . 1 , 0 . 2 , ..., 0 . 9 } to investigate the effect of the critical hyper-parameter in EVL, which is shown in Figure 7 in Appendix D.1.
The experimental results demonstrate that the estimated value increases with a larger τ , which is consistent with the analysis in Section 4.1.
Moreover, we observe that τ is set at a low value in some complex high-dimensional robotic tasks or narrow human demonstrations, such as Adroit-cloned/human, to get the conservative value estimates.
However, if τ is set too high (e.g., τ = 0 .
9 in the pen-human task), the estimated value will explode and poor performance.
This is as expected since the over-large τ leads to the overestimation error caused by neural networks.
The experimental results demonstrate that we can balance behavior cloning and optimal value learning by choosing τ in terms of different tasks.
6. A BLATIONS
Episodic Memory Module.
Our ﬁrst study aims to answer the impact of memory-based planning on performance.
We replace the episodic memory module in VEM with standard n -step value estimation (named VEM-1step or VEM-nstep).
The experimental results in Figure 8 in Appendix D. indicate that implicit planning along ofﬂine trajectories effectively accelerates the convergence of
EVL.
Expectile Loss.
In addition to the Expectile loss, we explored other forms of loss.
Formally, we compare the Expectile loss and quantile loss, a popular form in Distributional RL algorithms (Dabney et al., 2018), which is shown in Figure 5 in Appendix D.1.
The experimental results indicate that the Expectile loss is better since it is more stable when dealing with extreme values.
C ONCLUSION
In this paper, we propose a novel ofﬂine RL method, VEM, based on a new V -learning algorithm,
EVL.
EVL naturally avoids actions outside the dataset and provides a smooth tradeoff between generalization and conversation for ofﬂine learning.
Further, VEM enables effective implicit planning along ofﬂine trajectories to accelerate the convergence of EVL and achieve better advantage estimation.
Unlike most existing ofﬂine RL methods, we keep the learning procedure totally within the dataset’s support without any auxiliary modular, such as environment model or behavior policy.
The experimental results demonstrate that VEM achieves superior performance in most D4RL tasks and learns the accurate values to guide policy learning, especially in sparse reward tasks.
We hope that
VEM will inspire more works on ofﬂine RL and promote practical RL methods in the future.
R EPRODUCIBILITY
To ensure our work is reproducible, we provide our code in the supplementary materials.
In the future, we will publish all source code on Github.
The detailed implementation of our algorithm is presented as follows.
The value network is trained according to Equation 4.
The actor-network is trained according to Equation 7.
The hyper-parameters and network structure used in VEM are shown in
Appendix C.3.
All experiments are run on the standard ofﬂine tasks, D4RL (https://github.com/railberkeley/d4rl/tree/master/d4rl).
V ALUE - BASED E PISODIC M EMORY C ONTROL
Initialize critic networks V θ 1 , V θ 2 and actor network π φ with random parameters θ 1 , θ 2 , φ
Initialize target networks θ (cid:48) 1 ← θ 1 , θ (cid:48)
2 ← θ 2 Initialize episodic memory
M for t = 1 to T do for i ∈ { 1 , 2 } do
Sample N transitions (cid:16) s t , a t , r t , s t , ˆ R ( i ) t (cid:17) from M
Update θ i ← min θ i N − 1 (cid:80) (cid:16) R ( i ) t − V θ i ( s t ) (cid:17)
Update φ ← max φ N − 1 (cid:80) ∇ log π φ ( a t
| s t ) · f (cid:16) min i ˆ R ( i ) t − mean i V θ
i ( s t ) (cid:17) end for if t mod u then θ (cid:48)
i ← κθ i + (1 − κ ) θ (cid:48)
i Update Memory end if end for for trajectories τ in buffer M do for s t , a t , r t , s t +1 in reversed ( τ ) do for i ∈ { 1 , 2 } do
Compute ˆ R ( i ) t with Equation 6 and save into buffer M end for end for end for
A. A N A PPROACH FOR A UTO - TUNING τ
When we have a good estimation of V ∗ , for example, when there is some expert data in the dataset, we can auto-tune τ such that the value learned by EVL is close to the estimation of V ∗ .
This can be done by calculating the Monte-Carlo return estimates of each state and selecting good return values as the estimation of optimal value ˜ V ∗ .
Based on this target, we develop a method for auto-tuning τ .
By parameterizing τ = sigmoid( ξ ) with a differentiable parameter ξ ∈ R , we can auto-tune τ by minimizing the following loss J ( ξ ) =
ξ ( E ˆ V ( s ) − ˜ V ∗ ) .
If ( E ˆ V ( s ) − ˜ V ∗ ) < 0 , the differentiable parameter ξ will become larger and the value estimation E
ˆ V ( s ) will become larger accordingly.
Similarly, ξ and E ˆ V ( s ) will become smaller if ( E ˆ V ( s ) − ˜ V ∗ ) > 0 .
The experimental results in
Figure 10 in Appendix D.1 show that auto-tuning can lead to similar performance compared with manual selection.
B T HEORETICAL A NALYSIS B. C OMPLETE DERIVATION .
The expectile regression loss (Rowland et al., 2019) is deﬁned as
[Equation]
where (cid:37) is the target distribution and the minimiser of this loss is called the τ -expectile of (cid:37) .
the corresponding loss in reinforcement learning is
[Equation]
Then
[Equation]
taking the gradient of the value objective:
[Equation]
Therefore,
[Equation]
B. P ROOF OF L EMMA
Lemma 1.
For any τ ∈ [0 , 1) , T µτ is a γ τ -contraction, where γ τ = 1 − 2 α (1 − γ ) min { τ, 1 − τ } .
Proof.
Note that T µ 1 / 2 is the standard policy evaluation Bellman operator for µ , whose ﬁxed point is
V µ .
We see that for any V 1 , V 2 ,
[Equation] [Equation]
We introduce two more operators to simplify the analysis:
[Equation]
Next we show that both operators are non-expansion (e
[Equation]
g., (cid:107)T µ + V 1 − T µ + V 2 (cid:107) ∞ ≤ (cid:107) V 1 − V 2 (cid:107) ∞ ).
For any V 1 , V 2 , we have
[Equation]
The relationship between [ δ 1 ( s, a )] + + V 1 ( s ) and [ δ 2 ( s, a )] + + V 2 ( s ) exists in four cases, which are
• δ 1 ≥ 0 , δ 2 ≥ 0 , then [ δ 1 ( s, a )] + + V 1 ( s ) − ([ δ 2 ( s, a )]
+ + V 2 ( s ))
=
γ ( V 1 ( s (cid:48) )
− V 2 ( s (cid:48) )) .
• δ 1 < 0 , δ 2 < 0 , then [ δ 1 ( s, a )] + + V 1 ( s ) − ([ δ 2 ( s, a )]
+ + V 2 ( s ))
=
V 1 ( s ) − V 2 ( s ) .
• δ 1 ≥ 0 , δ 2 < 0 , then
[Equation]
where the inequality comes from r ( s, a ) + γV 2 ( s (cid:48) )
<
V 2 ( s ) .
• δ 1 < 0 , δ 2 ≥
[Equation]
, then
[Equation]
) where the inequality comes from r ( s, a ) + γV 2 ( s
[Equation]
) ≥ V 2 ( s ) .
Therefore, we have T µ + V 1 ( s ) − T
[Equation]
+ V
[Equation]
s ) ≤
[Equation]
V 1 − V 2 (cid:107) ∞ .
With the T µ + , T µ − , we rewrite T µτ as
T
[Equation]
And
[Equation]
We ﬁrst focus on τ < 12 .
For any V 1 , V 2 , we have
T
[Equation]
Similarly, when τ > 1 / 2 , we have T µτ V 1 ( s ) −T µτ V
[Equation]
s
[Equation]
−
[Equation]
−
[Equation]
−
[Equation]
−
[Equation]
∞ .
B. P ROOF OF L EMMA
Lemma 2.
For any τ, τ (cid:48) ∈ (0 , 1) , if τ (cid:48) ≥ τ , we have T µτ (cid:48) ≥ T µτ , ∀ s ∈ S .
Proof.
Based on Equation 20, we have
[Equation]
B. P ROOF OF L EMMA
Lemma 3.
Let V ∗ denote the ﬁxed point of Bellman optimality operator T ∗ .
In the deterministic
MDP, we have lim τ
→ 1 V ∗ τ
= V ∗ .
Proof.
We ﬁrst show that V ∗ is also a ﬁxed point for T µ + .
Based on the deﬁnition of T ∗ , we have
V ∗ ( s ) = max a [ r ( s, a ) + γV ∗ ( s (cid:48) )] , which infers that δ ( s, a ) ≤ 0 , ∀ s ∈ S, a ∈ A .
Thus, we have
T µ + V ∗ ( s ) =
V ∗ ( s ) + E a ∼ µ [ δ ( s, a )]
+ = V ∗ ( s ) .
By setting (1 − τ ) → 0 , we eliminate the effect of T µ − .
Further by the contractive property of T µτ , we obtain the uniqueness of V ∗ τ .
The proof is completed.
B. P ROOF OF L EMMA
Lemma 4.
Given τ ∈ (0 , 1) and T ∈ N + , T vem is a γ τ -contraction.
If τ > 12 , T vem has the same ﬁxed point as
[Equation]
Proof.
We prove the contraction ﬁrst.
For any V 1 , V 2 , we have
[Equation]
Next we show that V ∗ τ , the ﬁxed point of T µτ , is also the ﬁxed point of T vem when τ > 12 .
By deﬁnition, we have V ∗ τ
= T µτ V ∗ τ .
Following Lemma 2, we have V ∗ τ
= T µτ V ∗ τ ≥ T µ 1 / 2 V ∗ τ = T µ V ∗ τ .
Repeatedly applying T µ and using its monotonicity, we have T µ V ∗ τ ≥ ( T µ ) n − 1 V ∗ τ , 1 ≤ n ≤ n max .
Thus, we have T vem
[Equation]
B. P ROOF OF L EMMA
Lemma 5.
When the current value estimates V ( s ) are much lower than the value of behavior policy,
T vem provides an optimistic update.
Formally, we have
[Equation]
where n ∗ ( s ) =
arg max 1 ≤ n ≤ T { ( T µ ) n − 1 T µτ V ( s ) } and V µn ∗ ,τ is the ﬁxed point of ( T µ ) n ∗ ( s ) − 1 T µτ .
Proof.
The lemma is a direct result of the triangle inequality.
We have
[Equation]
B. P ROOF OF P ROPOSITION
Proposition 1.
Let V ∗ τ denote the ﬁxed point of T µτ .
For any τ, τ (cid:48) ∈ (0 , 1) , if τ (cid:48) ≥ τ , we have
V ∗ τ (cid:48) ( s ) ≥ V ∗ τ ( s ) , ∀ s ∈ S .
Proof.
With the Lemma 2, we have T µτ (cid:48)
V ∗ τ ≥ T µτ V ∗ τ .
Since V ∗ τ is the ﬁxed point of T µτ , we have
T µτ V ∗ τ
= V ∗ τ .
Putting the results together, we obtain V ∗ τ
= T µτ V ∗ τ ≤ T µτ (cid:48)
V ∗ τ .
Repeatedly applying
T µτ (cid:48) and using its monotonicity, we have V ∗ τ ≤ T µτ (cid:48)
V
[Equation]
≤
[Equation]
T
[Equation]
C D ETAILED I MPLEMENTATION C. G ENERALIZED A DVANTAGE -W EIGHTED L EARNING
In practice, we adopt Leaky-ReLU or Softmax functions.
Leaky-ReLU:
[Equation]
where f ( ˆ A ( s, a ))
= (cid:40)
ˆ A ( s, a ) if ˆ A ( s, a ) > 0 ˆ
A ( s,a ) α if ˆ A ( s, a ) ≤(27)
[Equation]
C. BCQ-EM
The value network of BCQ-EM is trained by minimizing the following loss:
[Equation]
where ˆ a t corresponds to the perturbed actions, sampled from the generative model G w ( s t ) .
The perturbation network of BCQ-EM is trained by minimizing the following loss:
[Equation]
(31) where ξ φ ( s, a i , Φ) is a perturbation model, which outputs an adjustment to an action a in the range [ − Φ , Φ] .
We adopt conditional variational auto-encoder to represent the generative model G w ( s ) and it is trained to match the state-action pairs sampled from D by minimizing the cross-entropy loss-function.
[Equation]
H YPER -P ARAMETER AND N ETWORK S TRUCTURE
[Table] [Table]
We use a fully connected neural network as a function approximation with 256 hidden units and ReLU as an activation function
[Table]
The structure of the actor network is [( state dim , 256) , (256 , 256) , (256 , action dim )] .
The structure of the value network is [( state dim , 256) , (256 , 256) , (256 , 1)] .
D A DDITIONAL E XPERIMENTS ON D4RL D. A BLATION S TUDY
[Figure]
[Figure]
i [Figure] u
[Figure] [Figure]
s t i m a t i o n
[Figure] rr o r [Figure]
[Figure]
[Figure]
d
[Figure]
R
[Figure]
t u r n
[Figure] t i [Figure]
o
[Figure]
D. C OMPLETE TRAINING CURVES AND VALUE ESTIMATION ERROR
[Figure]
Comparison between ﬁxed τ (VEM) and auto-tuning τ (VEM(auto)) in the door-human task. [Figure] [Figure] i
[Figure]
d e R e t u r n
[Figure]
R
[Figure]
n
[Figure]
i s o d e R e t u r n
[Figure]
p i s o d e R e t u r n
[Figure]
[Figure]
s t
i m a t i o n
[Figure]
r
[Figure]
E s t i m a t i o n
E rr o r
[Figure]
i o n
E rr o r
[Figure]
t
i
[Figure]
n
[Figure]