K EEP D OING W HAT W ORKED : B EHAVIOR M ODELLING P RIORS FOR O FFLINE R EINFORCEMENT L EARNING
A BSTRACT
Off-policy reinforcement learning algorithms promise to be applicable in settings where only a ﬁxed data-set (batch) of environment interactions is available and no new experience can be acquired.
This property makes these algorithms appealing for real world problems such as robot control.
In practice, however, standard off-policy algorithms fail in the batch setting for continuous control.
In this paper, we propose a simple solution to this problem.
It admits the use of data generated by arbitrary behavior policies and uses a learned prior – the advantage-weighted behavior model (ABM) – to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task.
Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conﬂicting data-sources.
We ﬁnd improvements on competitive baselines in a variety of RL tasks – including standard continuous control benchmarks and multitask learning for simulated and real-world robots.
Videos are available at https: //sites.google.com/view/behavior-modelling-priors .
I NTRODUCTION
Batch reinforcement learning (RL) (Ernst et al., 2005; Lange et al., 2011) is the problem of learning a policy from a ﬁxed, previously recorded, dataset without the opportunity to collect new data through interaction with the environment.
This is in contrast to the typical RL setting which alternates between policy improvement and environment interaction (to acquire data for policy evaluation).
In many real world domains collecting new data is laborious and costly, both in terms of experimentation time and hardware availability but also in terms of the human labour involved in supervising experiments.
This is especially evident in robotics applications (see e.g. Riedmiller et al.
Haarnoja et al.
2018b;
Kalashnikov et al.
2018 for recent examples learning on robots).
In these settings where gathering new data is expensive compared to the cost of learning, batch RL promises to be a powerful solution.
There exist a wide class of off-policy algorithms for reinforcement learning designed to handle data generated by a behavior policy µ which might differ from π , the policy that we are interested in learning (see e.g. Sutton & Barto (2018) for an introduction).
One might thus expect solving batch
RL to be a straightforward application of these algorithms.
Surprisingly, for batch RL in continuous control domains, however, Fujimoto et al.
(2018) found that policies obtained via the naïve application of off-policy methods perform dramatically worse than the policy that was used to generate the data.
This result highlights the key challenge in batch RL: we need to exhaustively exploit the information that is in the data but avoid drawing conclusions for which there is no evidence (i.e. we need to avoid over-valuing state-action sequences not present in the training data).
As we will show in this paper, the problems with existing methods in the batch learning setting are further exacerbated when the provided data contains behavioral trajectories from different policies µ 1 , . . . , µ N which solve different tasks, or the same task in different ways (and thus potentially execute conﬂicting actions) that are not necessarily aligned with the target task that π should accomplish.
We empirically show that previously suggested adaptations for off-policy learning (Fujimoto et al., 2018; Kumar et al., 2019) can be led astray by behavioral patterns in the data that are consistent (i.e. policies that try to accomplish a different task or a subset of the goals for the target task) but not relevant for the task at hand.
This situation is more damaging than learning from noisy or random data where the behavior policy is sub-optimal but is not predictable, i.e. the randomness is not a correlated signal that will be picked up by the learning algorithm.
We propose to solve this problem by restricting our solutions to ‘stay close to the relevant data’.
This is done by: 1) learning a prior that gives information about which candidate policies are potentially supported by the data (while ensuring that the prior focuses on relevant trajectories), 2) enforcing the policy improvement step to stay close to the learned prior policy.
We propose a policy iteration algorithm in which the prior is learned to form an advantage-weighted model of the behavior data.
This prior biases the RL policy towards previously experienced actions that also have a high chance of being successful in the current task.
Our method enables stable learning from conﬂicting data sources and we show improvements on competitive baselines in a variety of RL tasks – including standard continuous control benchmarks and multi-task learning for simulated and real-world robots.
We also ﬁnd that utilizing an appropriate prior is sufﬁcient to stabilize learning; demonstrating that the policy evaluation step is implicitly stabilized when a policy iteration algorithm is used – as long as care is taken to faithfully evaluate the value function within temporal difference calculations.
This results in a simpler algorithm than in previous work (Fujimoto et al., 2018; Kumar et al., 2019).
B ACKGROUND AND N OTATION
In the following we consider the problem of reinforcement learning, modeling the environment as a markov decision process (MDP) consisting of the continuous states s ∈ S , actions a ∈
A , and transition probability distribution p ( s t +1 | s t , a t ) – describing the evolution of the system dynamics over time (e.g. probability of reaching s t +1 from state s t when executing action a t ) – together with the state-visitation distribution p ( s ) .
The goal of reinforcement learning is to ﬁnd a policy π ( a | s ) that maximizes the cumulative discounted return J ( π ) =
E π [ (cid:80) ∞ t =1 γ t r ( s t , a t ) | a t ∼ π ( ·| s t ) , s t +1 ∼ p ( ·| s t , a t ) , s 1 ∼ p ( · )] , for the reward function r ( s, a ) ∈ R .
We also deﬁne the state-action value function for taking action a t in state s t , and thereafter following π : Q π ( s t , a t ) =
E π [ (cid:80) ∞
i =
t γ i r
( s i , a i ) | a i ∼ π, s i +1 ∼ p ( ·| s i , a i )] , which we can relate to the objective J via J ( π ) =
E s ∼ p ( · ) [ E a ∼ π ( ·| s )
[ Q π ∗ ( s, a )]] , where π ∗ is the optimal policy.
We parameterize the policy π θ ( a | s ) by θ but we will omit this dependency where unambiguous.
In some of the experiments we will also consider a setting where we learn about multiple tasks k ∈ { 1 , . . .
K } , each with their own reward function r k ( s, a ) .
We condition the policy and Q-function on the task index k
(i.e. Q πk and π ( a | s, k ) ), changing the objective to maximize the sum of returns across all tasks.
For the batch RL setting we assume that we are given a dataset D µ containing trajectory snippets (i.e. sub-trajectories of length N )
τ = { ( s 0 , a 0 ) , · · · , ( s T , a T ) } , with τ ∈ D µ .
We assume access to the reward function r for the task of interest and can evaluate it for all transitions in D µ (for example, r may be some function of s ).
We further assume D µ was ﬁlled, prior to training, by following a set of arbitrary N behavior policies µ 1 , · · · , µ N .
Note that these behavior policies may try to accomplish the task we are interested in; or might indeed generate trajectories unrelated to the task at hand.
A LEARNED PRIOR FOR O FFLINE OFF - POLICY RL FROM IMPERFECT DATA
To stabilize off-policy RL from batch data, we want to restrict the learned policy to those parts of the state-action space supported by the batch.
In practice this means that we need to approximately restrict the policy to the support of the empirical state-conditional action distribution.
This prevents the policy from taking actions for which the Q-function cannot be trained and for which it might thus give erroneous, overly optimistic values (Fujimoto et al., 2018; Kumar et al., 2019).
In this paper we achieve this by adopting a policy iteration procedure – in which the policy is constrained in the improvement step.
As in standard policy iteration (Sutton & Barto, 2018), the procedure consists of two alternating steps.
First, starting with a given policy π i =
π θ i in iteration
i (with π θcorresponding to a randomly initialized policy distribution), we ﬁnd an approximate action-value function Q π i ( s, a ) ≈ ˆ Q ( s, a ; φ i ) , with parameters φ (Section 3.1) (as with the policy we will drop the dependence on φ i and write ˆ Q π i ( s, a ) where unambiguous).
Second, we optimize for π i +1 with respect to ˆ Q π i subject to a constraint that ensures closeness to the empirical state-conditional action distribution of the batch (Section 3.2).
Iterating these steps, overall, optimizes J ( π ) .
We realize both policy evaluation and improvement via a ﬁxed number of gradient descent steps – holding π and ˆ Q π i ﬁxed via the use of target networks (Mnih et al., 2015).
We refer to Algorithm 1 for details.
P OLICY E VALUATION
To learn the task action-value function in each iteration we minimize the squared temporal difference error for a given reward – note that when performing ofﬂine RL from a batch of data the reward r might be computed post-hoc and does not necessarily correspond to the reward optimized by the behavior policies µ 1 , · · · , µ N .
The result after iteration i is given as
[Equation]
with ˆ V π i ( s ) = E a ∼ π i ( [Equation] ˆ
[Equation]
[Equation]
[Equation]
We approximate the expectation required to calculate ˆ V π i ( s ) with M samples from π i , i.e. ˆ V π
i
( s ) ≈ 1 M [ (cid:80) M j =1 ˆ Q ( s, a j ; φ i − 1 )
| a j ∼ π i ( ·| s )] .
As further discussed in the related work Section 4, the use of policy evaluation is different from the Q-learning approach pursued in Fujimoto et al.
Kumar et al.
(2019), which requires a maximum over actions and may be more susceptible to overestimation of Q-values.
We ﬁnd that when enough samples are taken (we use M = 20 ) and the policy is appropriately regularized (see Section 3.2) learning is stable without additional modiﬁcations.
3. P RIOR L EARNING AND P OLICY I MPROVEMENT
In the policy improvement step we solve the following constrained optimization problem
[Equation]
where D µ is the behavior data, π the policy being learned, and π prior is the prior policy.
This is similar to the policy improvement step in Abdolmaleki et al.
but instead of enforcing closeness to the previous policy here the constraint is with respect to a separately learned “prior” policy, the behavior model .
The role of π prior in Equation 2 is to keep the policy close to the regime of the actions found in D µ .
We consider two different ways to express this idea by learning a prior alongside the policy optimization.
[-It should be noted that, if the prior itself is already good enough to solve the task we can set π i +
-]
For learning the prior, we ﬁrst consider simply modeling the raw behavior data.
This is similar to the approach of BCQ and BEAR-QL (Fujimoto et al., 2018; Kumar et al., 2019), but we use a parametric behavior model and measure distance by KL; we refer to the related work for a discussion.
The behavior model can be learned by maximizing the log likelihood of the observed data
[Equation]
where θ bm are the parameters of the behavior model prior.
Regularizing towards the behavior model can help to prevent the use of unobserved actions, but it may also prevent the policy from improving over the behavior in D µ .
In effect, the simple behavior prior in Equation 3 regularizes the new policy towards the empirical state-conditional action distribution in
D µ .
This may be acceptable for datasets dominated by successful trajectories for the task of interest or when the unsuccessful trajectories are not predictable (i.e. they correspond to random behaviour).
However, we here are interested in the case where D µ is collected from imperfect data and from multiple tasks.
In this case, D µ will contain a diverse set of trajectories – both (partially) successful and actively harmful for the target task.
With this in mind, we consider a second learned prior, the advantage-weighted behavior model, π abm , with which we can bias the RL policy to choose actions that are both supported by D µ and also good for the current task (i.e. keep doing actions that work).
We can formulate this as maximizing the following objective:
[Equation]
with R ( τ t : N ) =
γ N − t
ˆ V π
i ( s N [Equation] +
N − 1 (cid:88)
j = t γ j − t r ( s j , a j ) ,
[Equation]
[Equation]
where f is an increasing, non-negative function, and the difference R ( τ t : N ) − ˆ V π
i is akin to an n-step advantage function, but here calculated off-policy representing the “advantage” of the behavior snippet over the policy
π i .
This objective still tries to maximize the log likelihood of observed actions, and avoids taking actions not supported by data.
However, by “advantage weighting” we focus the model on “good” actions while ignoring poor actions.
We let f = 1 + (the unit step function with f ( x ) = 1 for x ≥ 0 and 0 otherwise) both for simplicty – to keep the number of hyperparameters to a minimum while keeping the prior broad – and because it has an intuitive interpretation: such a prior will start by covering the full data and, over time, ﬁlter out trajectories that would lead to worse performance than the current policy, until it eventually converges to the best trajectory snippets contained in the data.
We note that Equation 4 is similar to a policy gradient, though samples here stem from the buffer and it will thus not necessarily converge to the optimal policy in itself; π θ abm will instead only cover the best trajectories in the data due to no importance weighting being performed for off-policy data.
This bias is in fact desirable in a batch-RL setting; we want a broad prior that only considers actions present in the data.
We also note that we tried several different functions for f including exponentiation, e.g. f ( x ) = exp( x ) , but found that choice of function did not make a signiﬁcant difference in our experiments.
Using either π θ bm or π θ abm as π prior , Equation 2 can be solved with a variety of optimization schemes.
We experimented with an EM-style optimization following the derivations for the MPO algorithm (Abdolmaleki et al., 2018), as well as directly using the stochastic value gradient of ˆ Q π i wrt.
policy parameters (Heess et al., 2015).
It should be noted that, if the prior itself is already good enough to solve the task we can learn Q π prior – e.g. if the data stems from an expert or has sufﬁciently high quality.
In this case learning both Q π prior and π prior becomes independent of the RL policy improvement step; if we then set (cid:15) = 0 , skipping the policy improvement step, we obtain a further simpliﬁed algorithm consisting only of learning π θ abm and Q π θ abm (see Figure 9 for an ablation).
EM-style optimization We can optimize the objective from Equation 2 using a two-step procedure.
Following (Abdolmaleki et al., 2018), we ﬁrst notice that the optimal π for Equation 2 can be expressed as ˆ π ( a | s ) ∝ π prior ( a | s ) exp( ˆ Q πi ( s,a ) / η ) , where η is a temperature that depends on the (cid:15) used for the KL constraint and can be found automatically by a convex optimization (Appendix B.1).
Conveniently, we can sample from this distribution by querying ˆ Q π i using samples from π prior .
These samples can then be used to learn the parametric policy by minimizing the divergence KL (ˆ π (cid:107)
π θ i +1 ) , which is equivalent to maximizing the weighted log likelihood
[Equation]
which we optimize via gradient descent subject to an additional trust-region constraint on π θ given as
KL ( π θ i (cid:107) π θ ) < (cid:15) trust to ensure conservative updates (Appendix B.1).
Stochastic value gradient optimization Alternatively, we can use Langrangian relaxation to turn
Equation 2 into an objective amenable to gradient descent.
Inserting π θ for π and relaxing results in ( θ i +1 , η ) =
arg max θ min η
[Equation]
(6) for η > 0 and which we can optimize by alternating gradient descent steps on θ and η respectively, taking the stochastic gradient of the Q-value (Heess et al., 2015) through the sampling of a ∼ π θ ( · , s ) via re-parameterization.
See Appendix B.2 for a derivation of this gradient.
Full pseudocode for our approach can be found in Appendix A, Algorithm 1.
R ELATED W ORK
There exist a number of off-policy RL algorithms that have been developed since the inception of the RL paradigm (see e.g. Sutton & Barto (2018) for an overview).
Most relevant for our work, some of these have been studied in combination with function approximators (for estimating value functions and policies) with an eye on convergence properties in the batch RL setting.
In particular, several papers have theoretically analyzed the accumulation of bootstrapping errors in approximate dynamic programming (Bertsekas & Tsitsiklis, 1996; Munos, 2005) and approximate policy iteration (Farahmand et al., 2010;
Scherrer et al., 2015); for the latter of which there exist well known algorithms that are stable at least with linear function approximation (see e.g. Lagoudakis & Parr (2003)).
Work on RL with non-linear function approximators has mainly considered the “online” or “growing batch” settings, where additional exploration data is collected (Ernst et al., 2005;
Riedmiller, 2005; Ormoneit & Sen, 2002); though some success for batch RL in discrete domains has been reported (Agarwal et al., 2019).
For continuous action domains, however, off-policy algorithms that are commonly used with powerful function approximators fail in the ﬁxed batch setting.
Prior work has identiﬁed the cause of these failures as extrapolation or bootstrapping errors (Fujimoto et al., 2018; Kumar et al., 2019) which occur due to a failure to accurately estimate Q-values, especially for state-action pairs not present in the ﬁxed data set.
Greedy exploitation of such misleading Q-values (e.g. due to a max operation) can then cause further propagation of such errors in the Bellman backup, and to inappropriate action choices during policy execution (leading to suboptimal behavior).
In non-batch settings, new data gathered during exploration allows for the Q-function to be corrected.
In the batch setting, however, this feedback loop is broken, and correction never occurs.
To mitigate these problems, previous algorithms based on Q-learning identiﬁed two potential solutions: 1) correcting for overly optimistic Q-values in the Bellman update, and 2) restricting the policy from taking actions unlikely to occur in the data.
To address 1) prior work uses a Bellman backup operator in which the max operation is replaced by a generative model of actions (Fujimoto et al., 2018) which a learned policy is only allowed to minimally perturb; or via a maximum over actions sampled from a policy which is constrained to stay close to the data (Kumar et al., 2019) (implemented through a constraint on the distance to a model of the empirical data, measured either in terms of maximum mean discrepancy or relative entropy).
To further penalize uncertainty in the Q-values this can be combined with Clipped Double-Q learning (Fujimoto et al., 2018) or an ensemble of Q-networks (Kumar et al., 2019).
To address 2) prior work uses a similarly constrained max also during execution, by considering only actions sampled from the perturbed generative model (Fujimoto et al., 2018) or the constrained policy (Kumar et al., 2019), and choosing the best among them.
Our work is based on a policy iteration scheme instead of Q-learning – exchanging the max for an expectation.
Thus we directly learn a parametric policy that we also use for execution.
We estimate the Q-function as part of the policy evaluation step with standard TD-0 backups.
We ﬁnd that for an appropriately constrained policy no special treatment of the backup operator is necessary, and that it is sufﬁcient to simply use an adequate number of samples to approximate the expectation when estimating V (see Equation 1).
The only modiﬁcation required is in the policy improvement step where we constrain the policy to remain close to the adaptive prior in Equation 2.
As we demonstrate in the empirical evaluation it is the particular nature of the adaptive prior – which can adapt to the task at hand (see Equation 4) – that makes this constraint work well.
Additional measures to account for uncertainty in the Q values could also be integrated into our policy evaluation step but we did not ﬁnd it to be necessary for this work; we thus forego this in favor of our simpler procedure.
Our policy iteration scheme also bears similarity to previous works that use (relative) entropy regularized policy updates which implement constraints with respect to either a ﬁxed (e.g. uniform) policy (e.g. Haarnoja et al., 2018a) or, in a trust-region like scheme, to the previous policy (e.g.
Abdolmaleki et al., 2018).
Other work has also focused on policy priors that are optimized to be different from the actual policy but so far mainly in the multi-task or transfer-learning setup, i.e. to share knowledge across or to transfer knowledge to new tasks (Teh et al., 2017; Galashov et al., 2019;
Tirumala et al., 2019; Jaques et al., 2017).
The constrained updates are also related to trustregion optimization in action space, e.g. in TRPO / PPO (Schulman et al., 2015; 2017) and MPO (Abdolmaleki et al., 2018), which ensures stable learning in the standard RL setting by enforcing conservative updates.
The idea of conservative policy optimization can be traced back to Kakade &
Langford (2002).
Here we take a slightly different perspective: we enforce a trust region constraint
[Figure] [Figure]
not on the last policy in the policy optimization loop (conservative updates) but wrt. the advantage weighted behavior distribution.
E XPERIMENTS
We experiment with continuous control tasks in two different settings.
In a ﬁrst set of experiments we compare our algorithm to strong off-policy baselines on tasks from the DeepMind control suite (Tassa et al., 2018) – to give a reference point as to how our algorithm performs on common benchmarks.
We then turn to the more challenging setting of learning multiple tasks involving manipulation of blocks using a robot arm in simulation.
These span tasks from reaching toward a block to stacking one block on top of another.
Finally, we experiment with analogous tasks on a real robot.
We use the same networks for all algorithms that we compare, optimize parameters using Adam (Kingma & Ba, 2015), and utilize proprioceptive features (e.g. joint positions / velocities) together with task relevant information (mujoco state for the control suite, and position/velocity estimates of the blocks for the manipulation tasks).
All algorithms were implemented in the same framework, including our reproduction of BCQ and BEAR, and differ only in their update rules.
Note that for
BEAR we use a KL instead of the MMD as we found this to work well, see appendix.
In the multi-task setting (Section 5.1) we learn a task conditional policy π θ ( a | s, k ) and Q-function ˆ Q πφ ( s, a, k ) where k is a one-hot encoding of the task identiﬁer, that is provided as an additional network input.
We refer to the appendix for additional details.
5. C ONTROL S UITE E XPERIMENTS
We start by performing experiments on four tasks from the DeepMind control suite: Cheetah, Hopper,
Quadruped.
To obtain data for the ofﬂine learning experiments we ﬁrst generate a ﬁxed dataset via a standard learning run using MPO, storing all transitions generated; we repeat this with 5 seeds for each environment.
We then separate this collected data into two sets: for experiments in the high data regime, we use the ﬁrst 10,000 episodes generated from each seed.
For experiments with low-quality
[Figure] [Figure]
data we use the ﬁrst 2,000 episodes high policies are higher average A of the initial training seeds over episodes is given in the appendix, Figure 6. For our ofﬂine learning experiments, we reload this data into a replay buffer. The dataset is then ﬁxed and no new transitions are added; the ofﬂine learner never receives any data that any of its current or previous policies have generated.
We evaluate performance by concurrently testing the policy in the environment.
The results of this evaluation are shown in Figure 1.
As can be observed, standard off-policy RL algorithms (MPO / SVG) can learn some tasks ofﬂine with enough data, but learning is unstable even on these relatively simple control suite tasks – conﬁrming previous ﬁndings from (Fujimoto et al., 2018; Kumar et al., 2019).
In contrast the other methods learn stably in the high-data regime, with BCQ lagging behind in Hopper and Quadruped (sticking too close to the VAE prior actions, an effect already observed in (Kumar et al., 2019)).
Remarkably our simple method of combining a policy iteration loop with a behavior model prior (BM+MPO in the plot) performs as well or better than the more complex baselines (BEAR and BCQ from the literature).
Further improvement can be obtained using our advantage weighted behavior model even in some of these simple domains (ABM+SVG and ABM+MPO).
Comparing the performance of the priors (BM[prior] vs ABM[prior], dotted lines) on Hopper we can understand the advantage that ABM has over BM: the
BM prior performs well on simple tasks, but struggles when the data contains conﬂicting trajectories (as in Hopper, where some of the seeds learn sub-optimal jumping) leading to too hard constraints on the RL policy.
Interestingly, the ABM prior itself performs as well or better than the baseline methods for the control-suite domains.
Furthermore, in additional experiments presented in the appendix we ﬁnd that competitive performance can be achieved, in simple domains, when training only an ABM prior (effectively setting (cid:15) = 0 ); providing an even simpler method when one does not care about squeezing out every last bit of performance.
A test on lower quality data, Figure 2, shows similar trends, with our method learning to perform slightly better than the best trajectories in the data.
[Figure]
r a g
[Figure]
e t
[Figure]
r n
[Figure]
[Figure]
H o u r s t r a i n i n g / e x p l o r a t i o n
5. S IMULATED R OBOT E XPERIMENTS
We experiment with a Sawyer robot arm simulated in Mujoco (Todorov et al., 2012) in a multi-task setting – as described above.
The seven tasks are to manipulate blocks that are placed in the workspace of the robot.
They include: reaching for the green block (Reach), grasping any block (Grasp), lifting the green block (Lift), hovering the green block over the yellow block (Place Wide), hovering the green block over the center of the yellow block (Place Narrow), stacking the green block on top of yellow (Stack and Stack/Leave i.e. without gripper contact).
To generate the data for this experiment we again run MPO – here simultaneously learning all task-conditional policies for the full seven tasks. Data-was collected by randomly switching tasks after each episode (of 200 control steps) with random resets of the robot position every 20 episodes. As before, data from all executed tasks is collected in one big data-set annotating each trajectory snippet with all rewards (i.e. this is similar to the SAC-R setting from (Riedmiller et al., 2018). During ofﬂine learning we then compare the performance of MPO and RL with a behavior modelling prior (BM+MPO and ABM+MPO).
As shown in Figure 3, behavioral modelling priors improve performance across all tasks over standard MPO – which struggles in these more challenging tasks.
This is likely due to the sequential nature of the tasks: later tasks implicitly include earlier tasks but only a smaller fraction of trajectories achieve success on stack and leave (and the actions needed for stack conﬂict, e.g., with lifting), this causes the BM prior to be overly broad (see plots in appendix).
The ABM+MPO, on the other hand, achieves high performance across all tasks.
Interestingly, even with ABM in place, the RL policy learned using this prior still outperforms the prior, demonstrating that RL is still useful in this setting.
As an additional experiment we test whether we can learn new tasks entirely from previously recorded data.
Since our rewards are speciﬁed as functions of observations, we compute rewards for two new tasks (bringing the green block to the center and bringing it to the corner) for the entire dataset – we then test the resulting policy in the simulator.
As depicted in Figure 4, this is successful with
ABM+MPO, demonstrating that we can learn tasks which were not originally executed in the dataset (as long as trajectory snippets that lead to successful task execution are contained in the data).
5. R EAL R OBOT E XPERIMENTS
Finally, to validate that our approach is a feasible solution to performing fast learning for real-robot experiments, we perform an experiment using a real Sawyer arm and the same set of seven tasks (implemented on the real robot) from Section 5.2.
As before, data from all executed tasks is collected in one big data-set annotating each trajectory snippet with all rewards.
The full buffer after about two weeks of real robot training is used as the data for ofﬂine learning; which we here only performed with ABM+MPO due to the costly evaluation.
The goal is to re-learn all seven original tasks.
Figure 5 shows the results of this experiment – we ran an evaluation script on the robot, testing the ofﬂine learned policy, and stopped when there was no improvement in average reward (as measured over a window of 50 episodes).
As can be seen, ABM with MPO as the optimizer manages to reliably re-learn all seven tasks purely from the logged data in less than 12 hours.
All tasks can jointly be learned with only small differences in convergence time – while during the initial training run the harder tasks, of course, took the most time to learn.
This suggests that gathering large data-sets of experience from previous robot learning experiments, and then quickly extracting the skills of interest, might be a viable strategy for making progress in robotics.
C ONCLUSION
In this work, we considered the problem of stable learning from logged experience with off-policy
RL algorithms.
Our approach consists of using a learned prior that models the behavior distribution contained in the data (the advantage weighted behavior model) towards which the policy of an RL algorithm is regularized.
This allows us to avoid drawing conclusions for which there is no evidence in the data.
Our approach is robust to large amounts of sub-optimal data, and compares favourably to strong baselines on standard continuous control benchmarks.
We further demonstrate that our approach can work in challenging robot manipulation domains – learning some tasks without ever seeing a single trajectory for them.
A A LGORITHM
A full algorithm listing for our procedure is given in Algorithm 1.
RL with
Behavior Extraction
Input: N steps number of learning steps, N TU steps between target update, M number of action samples, KL regularization parameter (cid:15) , initial parameters for θ, η, α and φ initialize N = 0 , θ (cid:48)
= θ , φ (cid:48)
=
φ while i ≤ N steps do sample a batch B of trajectories τ from replay buffer D µ sample M actions from policies to estimate expectations below // compute gradient for prior model
For BM prior:
[Equation]
For ABM prior:
[Equation]
) // compute gradients for Q , π and η
[Equation]
For MPO:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
For Stochastic Value Gradient:
[Table]
used // apply gradient updates θ [-prior , η, α, φ ] , [ δ π , δ θ prior , δ η , δ α , δ φ ] ), i = i +// update target networks if i mod N TU = 
-][Equation] optimizer_ [Equation]
θ [Equation] [Equation] φ ] ), i = i +// update target networks if i mod N TU = 0 then θ (cid:48)
= θ , φ (cid:48)
=
φ end if end while
B D ETAILS ON P OLICY I MPROVEMENT
We here give additional details on the implementation of the policy improvement step in our algorithm.
Depending on the policy optimizer used (MPO or SVG) different update rules are used to maximize the objective given in Equation 4.
This is also outlined in Algorithm 1.
We here describe the general form for both algorithms, using π prior to represent the prior which then will be instantiated as either the ABM or BM prior.
B. MPO
For the EM-style optimization based on MPO we ﬁrst notice that the optimal non-parametric policy that respects the KL constraint wrt.
π prior is
[Equation]
with Z = (cid:82) π prior ( a | s ) exp( ˆ Q πi ( s,a ) / η ) da and where η is a temperature that depends on the desired constraint (cid:15) .
In practice we estimate
Z based on the M samples { a 1 , . .
. , a M } ∼ π prior ( a | s ) that we draw for each state to perform the optimization of π θ .
That is we set Z ≈ 1 / M (cid:80)
Mj =1 exp( ˆ Q πi [Equation] s,a j ) / η ) .
Using these samples we can then optimize for η in a way analogous to what is described in Abdolmaleki et al.
Speciﬁcally, we ﬁnd that the objective for ﬁnding η is
[Equation]
which can approximate based on a batch B of trajectories sampled from D µ (sampling M actions from π prior for each state therein) and corresponding action samples where we used the samples { a 1 , . .
. , a M } ∼ π prior ( a | s ) as
[Equation]
which can readily be differentiated wrt.
η .
We then use Adam (with standard settings and learning rate 2 e − 4 ) to take a gradient step in direction of ∇ η g ( η ) (we want to maximize g ( η ) ) for each batch.
We start our optimization with η = 3 to ensure stable optimization (i.e. to avoid large changes in the policy parameters in the beginning of optimization).
Further, after each gradient step, we project η to the positive numbers i.e. we set η = max( η, 0 . 001) , as η is required to be positive.
We ﬁnd that this procedure is capable of fulﬁlling the desired KL constraints well.
The same batch, and action samples, are then also used to take an optimization step for the policy parameters θ .
In particular we ﬁnd the parametric policy by minimizing the divergence KL (ˆ π (cid:107) π
θ i +1 ) , which is equivalent to maximizing the weighted log likelihood of sampled actions (Equation 5).
We only take M = 20 samples here, which is a relatively crude representation of the behavior model at state s ; therefore, to prevent the policy from converging too quickly it can be useful to employ an additional trust region constraint in this step that ensures slow convergence.
We do this by adjusting the maximum likelihood objective Equation 5 to contain an additional KL regularization towards the previous policy, yielding the following optimization problem using samples { a 1 , . . .
, a M } ∼ π prior ( a | s ) :
[Equation]
(10) for α > 0 , which is a Langrangian relaxation to the maximum likelihood problem under the additional constraint that KL ( π θ i (cid:107) π θ ) < (cid:15) trust , where α is the Langrange multiplier.
This objective can be differentiated wrt.
both θ and α
and we simply take alternating gradient descent steps (one per batch for both θ and α ) using Adam (Kingma & Ba, 2015) (starting with a random θ
and α = 1 ) and projecting α back to the positive regime if it becomes negative; i.e. we set α = max ( α, 0 . 001) .
B. SVG
To optimize the policy parameters θ via the stochastic value gradient (Heess et al., 2015) (under a KL constraints) we can directly calculate the derivative of the Langrangian relaxation from Equation 6.
In particular, again assuming that we have sampled a batch B , the gradient wrt.
θ can be obtained via the reparameterization trick (Kingma & Welling, 2014;
Rezende et al., 2014).
For this we ﬁrst require that a sample from our policy π θ ( a | s ) can be obtained via a deterministic function applied to a standard noise source.
We ﬁrst specify the policy class used in the paper to be that of Gaussian policies, parameterized as π θ ( a | s ) =
N ( a | µ θ ( s ) , Iσ 2 θ ( s )) , where N ( a | µ, Σ) denotes the pdf of a standard Normal distribution and where we assume µ θ is directly given as one output of the network whereas we parameterize the diagonal standard deviation as σ θ ( s )
=
softplus ( h θ ( s )) with h θ ( s ) being output by the neural network.
We can then obtain samples via the deterministic transformation f ( s, ξ ; θ ) = µ θ ( s ) + σ θ ( s ) ξ , where ξ ∼ N (0 , I ) ( I being the identity matrix).
Using this deﬁnition we can obtain the following expression for the value gradient:
[Equation]
where we use the Gaussian samples { ξ 1 , . .
. , ξ M } ∼ N (0 , I ) .
The gradient for the Langrangian multiplier is given as
[Equation]
where we dropped terms independent of η in the second line.
Following δ θ to maximize the objective, and conversely moving in the opposite direction of δ η to minimize the objective wrt.
η , can then be performed by taking alternating gradient steps.
We perform one step per batch for both η and θ via
Adam; starting from an initially random θ and η = 1 .
As in the MPO procedure we ensure that η is positive by projecting it to the positive regime after each gradient step.
C D ETAILS ON THE EXPERIMENTAL SETUP C. H YPERPARAMETERS AND NETWORK ARCHITECTURE
Hyperparameters for our MPO, SVG, and BCQ single-task experiments are shown in tables 1, 2, and 3, respectively.
For multitask experiments, we modify the parameters shown in 5.
C. D ETAILS ON BCQ AND BEAR
To provide strong off-policy learning baselines we re-implemented BCQ (Fujimoto et al., 2018) and
BEAR (Kumar et al., 2019) in the same framework that we used to implement our own algorithm.
As mentioned in the main paper we used the same network architecture for ll algorithms.
Algorithm speciﬁc hyperparameters where tuned via a coarse grid search on the control suite tasks; while following the advice from the original papers on good parameter ranges.
To avoid bias in our comparisons we did not utilize ensembles of Q-functions for any of the methods (e.g. we removed them from BEAR), we note that ensembling did not seem to have a major impact on performance (see appendix in (Kumar et al., 2019)).
Parameters for all methods where optimized with Adam.
Furthermore, to apply BEAR and BCQ in the multi-task setting we employed the same conditioning of the policy and Q-function on a one-hot task vector (which is used to select among multiple network "heads", yielding per task parameters, see description below).
For BCQ we used a range of [0 . 25 , 0 . 25] for the perturbative actions generated by the DDPG trained network, and chose a latent dimensionality of 64 for the VAE, see Table 3 for the full hyperparameters.
For BEAR we used a KL constraint rather than the maximum mean discrepancy.
This ensures comparability with our method and we did not see any issues with instability when using a KL.
To ensure good satisfaction of constraints we used the exact same optimization for the Langragian multiplier required in BEAR that was also used for our method – see description of SVG above.
The hyperparameters for the BEAR training run on the control suite are given in Table 4.
C. D ETAILS ON THE R OBOT E XPERIMENT S ETUP
The task setup for both the simulated and real robot experiments is described in the following.
A detailed description of the robot setup will be given in an accompanying paper.
We nonetheless give [Table] [Table] a description here for completeness.
We make no claim to have contributed these tasks speciﬁcally for this paper and merely use them as an evaluation test-bed [Table]
As the robot we utilize a Sawyer robotic arm mounted on a table and equipped with a Robotiq 2F-85 parallel gripper.
A basket is positioned in front of the robot which contains three cubes (the proportions of cubes and basket sizes are consistent between simulation and reality).
Three cameras on the basket track the cube using augmented reality tags.
To model the tasks as an MDP we provide both proprioceptive information from the robot sensors (joint positions, velocities and torques) and the tracked cube position, velocity (both in 3 dimensions) and orientation to the policy and Q-function.
Overall the observations provided to the robot are: Proprioception: Joint positions (7D, double), velocities (7D, double), torques (7D, double); wrist pose (7D, double), velocity (6D, double), force (6D, double); gripper ﬁnger angles (1D, int), velocity (1D, int), grasp ﬂag (1D, binary) and the object features: Object pose (7D, double) averaged over all cameras observing the object, Relative pose between object and gripper (7D, double).
In simulation the true object position and velocities / orientation are used instead of running a tracking algorithm.
We control both the robot and the gripper by commanding velocities in the 4-dimensional Cartesian space of the robot’s endeffector (speciﬁng three translational velocities plus the velocity for the wrists rotation) while the gripper control is 1-dimensional.
The action limits are [-0.07, 0.07] m/s [Table] [Table] for Cartesian actions, [-1, 1] rad/s for wrist rotation and [-255, 255] for ﬁnger velocity (for units see gripper speciﬁcations).
The control rate at which the actions are executed is 20 Hz.
Episodes are terminated if the wrist force exceeds 20 N on any axis, encouraging a gentle interaction of the robot with its environment.
For our experiment we use 7 different task to learn.
The ﬁrst 6 tasks are auxiliary tasks for better exploration that help to learn the ﬁnal task (Stack and Leave), i.e, stacking the green cube on top of the yellow cube.
The rewards functions for all tasks are given as:
Reach : stol ( d ( pos endeffector , pos green ) , 0 .
This function minimizes the distance d of the gripper to the green cube.
Grasp : Activate grasp sensor of gripper.
Lift : 0 if height of green block is less than 3 cm, if it is above 10 centimeters and ( height − 3 cm ) / (10 cm − 3 cm ) otherwise.
Place Wide : stol ( d ( pos green , pos yellow + [0 , 0 , 0 . 05]) , 0 . 01 , 0 . 20)
Move green cube to a position 5cm above the yellow cube.
Place Narrow : stol ( d ( pos green , pos yellow + [0 , 0 , 0 . 05]) , 0 . 00 , 0 . 01) :
Like Place
Wide but with more precision.
Stack : btol ( d xy ( pos green , pos yellow ) , 0 . 03) ∗ btol ( d z ( pos green , pos yellow ) + 0 . 05 , 0 . 01) ∗ (1 − Grasp )
A binary reward for stacking green cube on the yellow one and deactivating the grasp sensor.
[Table] 7. STACK_AND_LEAVE(G, Y) : stol ( d z ( pos endeffector , pos green ) + 0 . 10 , 0 . 03 , 0 . 10) ∗ Stack
Like STACK(G, Y), but it gets max reward if it moves the arm 10cm above the green cube.
Where d ( a, b ) denotes the euclidean distance between a and b. We also deﬁne two tolerance functions with outputs scaled between 0 and 1, i.e,
[Equation]
D A DDITIONAL EXPERIMENTAL RESULTS
We present additional plots that show some aspects of the developed algorithm in more detail.
D. E XPANDED PLOTS FOR CONTROL SUITE
We provide expanded plots for MPO on the control suite in Figure 7.
These show in detail that the learned advantage weighted behavior model (ABM, in red in the left column) is far superior to the standard behavior model prior, leading to less constrained RL policies.
D. E XPANDED PLOTS FOR R OBOT SIMULATION
Figure 8 shows full results for the simulated robot stacking task, including all 7 intentions as well as performance of the prior policies themselves during learning.
The task set is structured: earlier tasks like reaching and lifting are necessary to perform most other tasks, so the simple behavioral model performs well on these.
For the more difﬁcult stacking tasks, however, the presence of conﬂicting data means the simple behavioral model doesn’t achieve high reward, though it still signiﬁcantly improves performance of the regularized policy.
D. P ERFORMANCE TABLES FOR CONTROL SUITE AND ROBOT SIMULATION
Table 6 shows ﬁnal performance for all methods on control suite tasks.
Table 7 shows ﬁnal performance on simulated robotics tasks to make comparison between algorithms easier.
The episode returns are averaged over the ﬁnal 10% of episodes.
ABM provides a performance boost over BM alone, particularly for difﬁcult tasks such as block stacking and quadruped.
The RL policy further improves performance on the difﬁcult tasks.
As noted in the main paper, one additional option, to further simplify the algorithm is to omit the policy improvement step, setting π i +
=
π prior (i.e. considering the case where (cid:15) = 0 ) and, conversely learning the Q-values of the prior.
In additional experiments we have found that this procedure roughly recovers the performance of the ABM prior when trained together with MPO (ABM+MPO); i.e. this is an option for a simpler algorithm to [Figure] [Figure] [Figure]
When the value function is learned from the prior, the ABM prior no longer depends on the RL policy, and can be learned on its own. We can still perform MPO policy improvement using this value function and regularized to this new prior, shown on the right. Using a value function learned from the prior achieves similar performance, and may in fact perform slightly better on more difﬁcult tasks. [Table]
[Table]
implement at the cost of some performance loss (especially on the most complicated domains).
We included this setting as ABM ( (cid:15) = 0 , | τ | = 2) in the table – noting that in this case it is vital to choose short trajectory snippets in order to allow π prior to pick the best action in each state.