A DVANTAGE -W EIGHTED R EGRESSION : S IMPLE AND S CALABLE O FF -P OLICY R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines.
Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data.
Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy.
The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods.
We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay.
We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms.
AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions.
Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.
(Video 1 )
I NTRODUCTION
Model-free reinforcement learning can be a general and effective methodology for training agents to acquire sophisticated behaviors with minimal assumptions on the underlying task (Mnih et al., 2015; Heess et al., 2017; Pathak et al., 2017).
However, reinforcement learning algorithms can be substantially more complex to implement and tune than standard supervised learning methods.
Arguably the simplest reinforcement learning methods are policy gradient algorithms (Sutton et al., 2000), which directly differentiate the expected return and perform gradient ascent.
Unfortunately, these methods can be notoriously unstable and are typically on-policy (or nearly on-policy), often requiring a substantial number of samples to learn effective behaviors.
Our goal is to develop a reinforcement learning algorithm that is simple, easy to implement, and can readily incorporate off-policy experience data.
In this work, we propose advantage-weighted regression (AWR), a simple off-policy algorithm for model-free RL.
Each iteration of the AWR algorithm simply consists of two supervised regression steps: one for training a value function baseline via regression onto cumulative rewards, and another for training the policy via weighted regression.
The complete algorithm is shown in Algorithm 1.
AWR can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods.
Despite its simplicity, we ﬁnd that AWR achieves competitive results when compared to commonly used on-policy and off-policy RL algorithms, and can effectively incorporate fully off-policy data, which has been a challenge for other RL algorithms.
Our derivation presents an interpretation of AWR as a constrained policy optimization procedure, and provides a theoretical analysis of the use of off-policy data with experience replay.
We ﬁrst revisit the original formulation of reward-weighted regression, an on-policy RL method that utilizes supervised learning to perform policy updates, and then propose a number of new design decisions that signiﬁcantly improve performance on a suite of standard continuous control benchmark tasks.
We then provide a theoretical analysis of AWR, including the capability to incorporate off-policy data with experience replay.
Although the design of AWR involves only a few simple design decisions, we show experimentally that these additions provide for a large improvement over previous methods for regression-based policy search, such as reward-weighted regression (RWR) (Peters & Schaal, 2007), while also being substantially simpler than more modern methods, such as MPO (Abdolmaleki et al., 2018).
We show that AWR achieves competitive performance when compared to several well-established state-of-the-art on-policy and off-policy algorithms.
We further demonstrate our algorithm on challenging control tasks with complex simulated characters.
P RELIMINARIES
In reinforcement learning, the objective is to learn a control policy that enables an agent to maximize its expected return for a given task.
At each time step t , the agent observes the state of the environment s t ∈ S , and samples an action a t ∈
A from a policy a t ∼ π ( a t | s t ) .
The agent then applies that action, which results in a new state s t +1 and a scalar reward r t = r
( s t , a t ) .
The goal is to learn an optimal policy that maximizes the agent’s expected discounted return J ( π ) ,
[Equation]
where p π ( τ ) represents the likelihood of a trajectory τ
= { ( s 0 , a 0 , r 0 ) , ( s 1 , a 1 , r 1 ) , ... } under a policy π , and γ ∈ [0 , 1) is the discount factor.
d π ( s ) =
(cid:80) ∞ t =0 γ t p ( s t = s | π )
represents the unnormalized discounted state distribution induced by the policy
π (Sutton & Barto, 1998), and p ( s t = s | π ) is the likelihood of the agent being in state s after following π for t timesteps.
A popular class of algorithms for solving this problem is policy gradient (PG) methods, which directly estimates the gradient of the expected return with respect to the policy parameters ∇ π J ( π ) , and then updates the policy with gradient ascent. Basic PG algorithms are generally on-policy methods, which require the data to be sampled from the same policy that is being optimized.
This can result in poor sample efﬁciency, but PG algorithms can be modiﬁed to utilize off-policy data. An alternative class of RL methods is expectation-maximization algorithms. Instead of estimating the gradient of the expected return, EM algorithms ﬁrst construct an estimate of the optimal policy using a dataset of experiences (E-step), and then projects this estimate onto the space of parameterized policies (M-step). An early example of an EM-based RL algorithm is reward-weighted regression (RWR) (Peters et al., 2010).
At each iteration, the E-step constructs an estimate of the optimal policy according to π ∗ ( a | s ) ∝ π k ( a | s )exp ( R s , a /β ) , where π k represents the policy at the k th iteration of the algorithm, R s , a = (cid:80) ∞ t =
γ t r t is the return, and β > 0 is a temperature parameter.
Then the M-step projects π ∗ onto the space of parameterized policies by solving a supervised regression problem: [Equation] The RWR update can be interpreted as ﬁtting a new policy π k +1 to samples collected under the current policy π k , where the likelihood of each action is weighted by the exponentiated return received for that action.
Since EM algorithms do not directly estimate the gradient of the expected return with respect to the current policy, they can be more amenable to learning from off-policy data. 1: π 1 ← random policy 2: D ← ∅ 3: for iteration k = 1 , ..., k max do 4: add trajectories { τ i } sampled via π k to D 5: V D k ← arg min V E s , a ∼D (cid:104)(cid:12)(cid:12)(cid:12)(cid:12) R D s , a − V ( s ) (cid:12)(cid:12)(cid:12)(cid:12) 2 (cid:105) 6: π k +1 ← arg max π E s , a ∼D (cid:104) log π ( a | s ) exp (cid:16) 1 β (cid:0) R D s , a − V D k ( s ) (cid:1)(cid:17)(cid:105) 7: end for
A DVANTAGE -W EIGHTED R EGRESSION
In this work, we present advantage-weighted regression (AWR), a simple off-policy RL algorithm based on reward-weighted regression.
We ﬁrst provide an overview of the complete advantageweighted regression algorithm, and then describe its theoretical motivation and analyze its properties.
The complete AWR algorithm is summarized in Algorithm 1.
Each iteration k of AWR consists of the following simple steps.
First, the current policy π k ( a | s ) is used to sample a batch of trajectories { τ i } that are then stored in the replay buffer D , which is structured as a ﬁrst-in ﬁrst-out (FIFO) queue, as is common for off-policy reinforcement learning algorithms (Mnih et al., 2015; Lillicrap et al., 2016).
Then, the entire buffer D is used to ﬁt a value function V D k ( s ) to the trajectories in the replay buffer, which can be done with simple Monte Carlo return estimates R D s , a = (cid:80)
Tt =
γ t r t .
Finally, the same buffer is used to ﬁt a new policy using advantage-weighted regression, where each state-action pair in the buffer is weighted according to the exponentiated advantage exp( 1 β A D ( s , a )) , with the advantage given by A D ( s , a ) =
R D s , a − V D ( s ) and β is a hyperparameter.
AWR uses only supervised regression as learning subroutines, making the algorithm very simple to implement.
In the following subsections, we ﬁrst motivate the algorithm as an approximation to a constrained policy search problem, and then extend our analysis to incorporate experience replay.
3. D ERIVATION
In this section, we derive the AWR algorithm as an approximate optimization of a constrained policy search problem.
Our goal is to ﬁnd a policy that maximizes the expected improvement η ( π ) =
J ( π ) − J ( µ ) over a sampling policy µ ( a | s ) .
We ﬁrst derive AWR for the setting where the sampling policy is a single Markovian policy.
Then, in the next section, we extend our result to the setting where the data is collected from multiple policies, as in the case of experience replay that we use in practice.
The expected improvement η ( π ) can be expressed in terms of the advantage
A µ ( s , a ) =
R µ s , a − V µ ( s ) with respect to the sampling policy µ (Kakade & Langford, 2002;
Schulman et al., 2015):
[Equation]
where R µ s , a denotes the return obtained by performing action a in state s and following µ for the following timesteps, and V µ ( s )
= (cid:82) a µ ( a | s ) R as d a corresponds to the value function of µ .
This objective differs from the ones used in the derivations of related algorithms, such as RWR and
REPS (Peters & Schaal, 2007; Peters et al., 2010; Abdolmaleki et al., 2018), which maximize the expected return J ( π ) instead of the expected improvement.
The expected improvement directly gives rise to an objective that involves the advantage.
We will see later that this yields weights for the policy update that differ in a subtle but important way from standard reward-weighted regression.
As we show in our experiments, this difference results in a large empirical improvement.
The objective in Equation 3 can be difﬁcult to optimize due to the dependency between d π ( s ) and π , as well as the need to collect samples from π .
Following Schulman et al.
, we can instead optimize an approximation ˆ η ( π ) of η ( π ) using the state distribution of µ :
[Equation]
Here, ˆ η ( π ) matches η ( π ) to ﬁrst order (Kakade & Langford, 2002), and provides a good estimate of η if π and µ are close in terms of the KL-divergence (Schulman et al., 2015).
Using this objective, we can formulate the following constrained policy search problem:
[Equation]
The constraint in Equation 6 ensures that the new policy π is close to the data distribution of µ , and therefore the surrogate objective ˆ η ( π ) remains a reasonable approximation to η ( π ) .
We refer the reader to Schulman et al.
(2015) for a detailed derivation and an error bound
[Equation] We can derive AWR as an approximate solution to this constrained optimization.
This derivation follows a similar procedure as Peters et al.
(2010), and begins by forming the Langrangian of the constrained optimization problem presented above,
[Equation]
where β is a Lagrange multiplier.
Differentiating L ( π, β ) with respect to π ( a | s ) and solving for the optimal policy π ∗ results in the following expression for the optimal policy
[Equation]
with Z ( s ) being the partition function.
A detailed derivation is available in Appendix A.
If π is represented by a function approximator (e.g., a neural network), a new policy can be obtained by projecting π ∗ onto the manifold of parameterized policies,
[Equation]
While this derivation for AWR largely follows the derivations used in prior work (Peters et al., 2010; Abdolmaleki et al., 2018), our expected improvement objective introduces a baseline V µ ( s ) to the policy update, which as we show in our experiments, is a crucial component for an effective algorithm. A similar advantage-weighting scheme has been previously used for ﬁtted Q-iteration (Neumann & Peters, 2009), where the policy is given by π ( a | s ) = 1 Z ( s ) exp (cid:0)(cid:0) R µ s , a − V µ
[Equation]
s )
[Equation] β (cid:1) . In this deﬁnition, the likelihood of an action does not depend on its likelihood under the sampling distribution, and therefore does not enforce a trust region with respect to the sampling distribution. Next, we extend AWR to incorporate experience replay for off-policy training, where the sampling policy is no longer a single policy, but rather a mixture of policies from past iterations.
3. E XPERIENCE R EPLAY AND O FF -P OLICY L EARNING
A crucial design decision of AWR is the choice of sampling policy µ ( a | s ) .
Standard implementations of RWR typically follow an on-policy approach, where the sampling policy is selected to be the current policy µ ( a | s ) =
π k ( a | s ) at iteration k .
This can be sample inefﬁcient, as data collected at each iteration of the algorithms are discarded after a single update iteration.
Importance sampling can be incorporated into RWR to reuse data from previous iterations, but at the cost of larger variance from the importance sampling estimator (Kober & Peters, 2009).
Instead, we can improve sample efﬁciency of AWR by incorporating experience replay and explicitly accounting for training data from a mixture of multiple prior policies.
As described in Algorithm 1, at each iteration, AWR collects a batch of data using the latest policy π k , and then stores this data in a replay buffer D , which also contains data collected from previous policies { π 1 , · · · , π k } .
The value function and policy are then updated using samples drawn from D .
This replay strategy is analogous to modeling the sampling policy as a mixture of policies from previous iterations µ k ( τ )
= (cid:80) ki
=1 w i
π
i ( τ ) , where π i ( τ ) =
p ( τ | π i ) represents the likelihood of a trajectory τ under a policy
π i from the i th iteration, and the weights (cid:80) i w i = 1 specify the probabilities of selecting each policy π i .
We now extend the derivation from the previous section to the off-policy setting with experience replay, and show that Algorithm 1 indeed optimizes the expected improvement over a sampling policy modeled by the replay buffer.
Given a replay buffer consisting of trajectories from past policies, the joint state-action distribution of µ is given by µ ( s , a ) = (cid:80) ki
=1 w i d
π i ( s ) π i ( a | s ) , and similarly for the marginal state distribution d µ ( s ) = (cid:80) ki
=1 w i d
π i ( s ) .
The expected improvement can now be expressed with respect to the set of sampling policies in the replay buffer: η ( π ) =
J ( π ) − (cid:80) i
w i J ( π i ) .
Similar to Equation 3, η ( π ) can be expressed in terms of the advantage
A π i ( s , a ) =
R π i s , a −
V π
i ( s ) of each sampling policies,
[Equation]
As before, we can optimize an approximation ˆ η ( π ) of η ( π ) using the state distribution of µ ,
[Equation]
In Appendix B, we show that the update procedure in Algorithm 1 optimizes the following objective:
[Equation]
where µ
[Equation]
a
[Equation]
represents the conditional action distribution deﬁned by the replay buffer.
This objective can be solved via the Lagrangian to yield the following update:
[Equation]
where the expectations can be approximated by simply sampling from D following Line 6 of Algorithm 1.
A detailed derivation is available in Appendix B. Note, the baseline in the exponent now consists of an average of the value functions of the different policies.
One approach for estimating this quantity would be to ﬁt separate value functions V π
i for each policy.
However, if only a small amount of data is available from each policy, then V π i could be highly inaccurate (Fu et al., 2019).
Therefore, instead of learning separate value functions, we ﬁt a single mean value function ¯ V ( s ) that directly estimates the weighted average of V
π
i ’s,
[Equation]
This loss can also be approximated by simply sampling from the replay buffer following Line 5 of
Algorithm 1.
The optimal solution ¯ V
[Equation]
[Table]
is exactly the baseline in Equation 15.
3. I MPLEMENTATION D ETAILS
Finally, we discuss several design decisions that are important for a practical implementation of
AWR.
An overview of AWR is provided in Algorithm 1.
The policy update in Equation 10 requires sampling states from the discounted state distribution d µ ( s ) .
However, we found that simply sampling states uniformly from D was also effective, and simpler to implement.
This is a common strategy used in standard implementations of RL algorithms (Dhariwal et al., 2017).
When updating the value function and policy, Monte Carlo estimates can be used to approximate the expected return R D s , a of samples in D , but this can result in a high-variance estimate.
Instead, we opt to approximate R D s , a using TD( λ ) to obtain a lower-variance estimate (Sutton & Barto, 1998).
TD( λ ) is applied by bootstrapping with the value function V D k − 1 ( s ) from the previous iteration.
A simple
Monte Carlo return estimator can also be used though, as shown in our experiments, but this produces somewhat worse results.
To further simplify the algorithm, instead of adaptively updating the
Lagrange multiplier β , as is done in previous methods (Peters & Schaal, 2007; Peters et al., 2010;
Abdolmaleki et al., 2018), we ﬁnd that simply using a ﬁxed constant for β is also effective.
The weights ω D s , a = exp (cid:16) 1 β (cid:0) R D s , a − V D ( s ) (cid:1)(cid:17) used to update the policy can occasionally assume excessively large values, which can cause gradients to explode.
We therefore apply weight clipping ˆ ω D s , a = min (cid:0) ω D s , a , ω max (cid:1) with a threshold ω max to mitigate issues due to exploding weights.
R ELATED W ORK
Existing RL methods can be broadly categorized into on-policy and off-policy algorithms (Sutton &
Barto, 1998).
On-policy algorithms generally update the policy using data collected from the same policy.
A popular class of on-policy algorithms is policy gradient methods (Williams, 1992; Sutton et al., 2000), which have been shown to be effective for a diverse array of complex tasks (Heess et al., 2017; Pathak et al., 2017; Peng et al., 2018; Rajeswaran et al., 2018).
However, on-policy algorithms are typically data inefﬁcient, requiring a large number of interactions with the environment.
Offpolicy algorithms improve sample efﬁciency by enabling a policy to be trained using data from other sources, such as data collected from different agents or data from previous iterations of the algorithm.
Importance sampling is a simple strategy for incorporating off-policy data (Sutton & Barto, 1998; Meuleau et al., 2000; Hachiya et al., 2009), but can introduce optimization instabilities due to the potentially large variance of the importance sampling estimator.
Dynamic programming methods based on Q-function learning can also leverage off-policy data (Precup et al., 2001;
Mnih et al., 2015; Lillicrap et al., 2016;
Gu et al., 2016; Haarnoja et al., 2018b).
But these methods can be notoriously unstable, and in practice, require a variety of stabilization techniques to ensure more consistent performance (Hasselt et al., 2016;
Wang et al., 2016; Munos et al., 2016; Hessel et al., 2017; Fujimoto et al., 2018; Nachum et al., 2018; Fu et al., 2019).
Furthermore, it can be difﬁcult to apply these methods to learn from fully off-policy data, where an agent is unable to collect additional environmental interactions (Fujimoto et al., 2019; Kumar et al., 2019).
Alternatively, policy search can also be formulated under an expectation-maximization framework.
This approach has lead to a variety of EM-based RL algorithms (Peters et al., 2010; Neumann, 2011; Abdolmaleki et al., 2018), an early example of which is reward-weighted regression (RWR) (Peters & Schaal, 2007).
RWR presents a simple on-policy RL algorithm that casts policy search as a supervised regression problem.
A similar algorithm, relative entropy policy search (REPS) (Peters et al., 2010), can also be derived from the dual formulation of a constrained policy search problem.
RWR has a number appealing properties: it has a very simple update rule, and since each iteration corresponds to supervised learning, it can be more stable and easier to implement than many of the previously mentioned RL methods.
Despite these advantages, RWR has not been shown to be an effective RL algorithm when combined with neural network function approximators, as demonstrated in prior work and our own experiments (Schulman et al., 2015; Duan et al., 2016).
In this work, we propose a number of modiﬁcations to the formulation of RWR to produce an effective off-policy deep RL algorithm, while still retaining much of the simplicity of previous methods.
Policy updates using supervised regression have been used in a number of prior work.
The optimization problem being solved in REPS is similar to AWR (Peters et al., 2010), but REPS optimizes the expected return instead of the expected improvement.
The weights in REPS also contains a Bellman error term that superﬁcially resembles advantages, but are computed using a linear value function derived from a feature matching constraint.
Learning the REPS value function requires minimization of a dual function, which is a complex function of the Bellman error, while the value function in AWR can be learned with simple supervised regression.
More recently, Abdolmaleki et al.
(2018) proposed MPO, a deep RL variant of REPS, which applies a partial EM algorithm for policy optimization.
The method ﬁrst ﬁts a Q-function of the current policy via bootstrapping, and then performs a policy improvement step with respect to this Q-function under a trust region constraint that penalizes large policy changes.
MPO uses off-policy data for training a Q-function critic via bootstrapping and employs Retrace( λ ) for off-policy correction (Munos et al., 2016).
In contrast, AWR is substantially simpler, as it can simply ﬁt a value function to the observed returns in a replay buffer, and performs weighted supervised regression on the actions to ﬁt the policy. Oh et al.
(2018) proposed self-imitation learning (SIL), which augments policy gradient algorithms with an auxiliary behaviour cloning loss to reuse samples from past experiences.
In contrast to SIL,
AWR is a standalone algorithm, and does not need to be combined with an auxiliary RL algorithm.
Neumann & Peters (2009) proposed LAWER, a kernel-based ﬁtted Q-iteration algorithm where the Bellman error is weighted by the normalized advantage of each state-action pair.
This was then followed by a soft-policy improvement step.
Similar to Neumann & Peters (2009), our method also uses exponentiated advantages during policy updates, but their deﬁnition of the policy is different from the one in AWR and does not enforce a trust region constraint.
Furthermore, AWR does not perform ﬁtted Q-iteration, and instead utilizes off-policy data in a simple constrained policy search procedure.
Wang et al.
(2018) applied a similar advantage-weighting scheme for imitation learning, but the method was not demonstrated for the RL setting.
In this work, we propose several design [Figure] decisions that are vital for an effective RL algorithm.
We also provide a theoretical analysis of AWR when combined with experience replay, and show that the algorithm indeed optimizes the expected improvement with respect to a trajectory-level mixture of past policies modeled by a replay buffer.
E XPERIMENTS
Our experiments aim to comparatively evaluate the performance of AWR to commonly used onpolicy and off-policy deep RL algorithms.
We evaluate our method on the OpenAI Gym benchmarks (Brockman et al., 2016), consisting of discrete and continuous control tasks.
We also evaluate our method on complex motion imitation tasks with high-dimensional simulated characters, including a 34 DoF humanoid and 64 DoF dog (Peng et al., 2018).
We then demonstrate the effectiveness of AWR on fully off-policy learning, by training on static datasets of demonstrations collected from demo policies.
Behaviors learned by the policies are best seen in the supplementary video 1 .
Code for our implementation of AWR is available at sites.google.com/view/awr-supp/. At each iteration, the agent collects a batch of approximately 2000 samples, which are stored in the replay buffer D along with samples from previous iterations.
The replay buffer stores 50k of the most recent samples.
Updates to the value function and policy are performed by uniformly sampling minibatches of 256 samples from D .
The value function is updated with 200 gradient steps per iteration, and the policy is updated with 1000 steps.
Detailed hyperparameter settings are provided in Appendix C.
5. B ENCHMARKS
We compare AWR to a number of state-of-the-art RL algorithms, including on-policy algorithms, such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017), off-policy algorithms, such as DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018a), as well as RWR (Peters & Schaal, 2007), which we include for comparison due to its similarity to
AWR.
2 TRPO and PPO use the implementations from OpenAI baselines (Dhariwal et al., 2017).
DDPG, TD3, and SAC uses the implementations from RLkit (Pong, 2019).
RWR is a custom implementation following the algorithm described by Peters & Schaal (2007).
Snapshots of the AWR policies are shown in Figure 2.
Learning curves comparing the different algorithms on the OpenAI Gym benchmarks are shown in Figure 3, and Table 1 summarizes the average returns of the ﬁnal policies across 5 training runs initialized with different random seeds.
Overall,
AWR shows competitive performance with the state-of-the-art deep RL algorithms.
It signiﬁcantly outperforms on-policy methods such as PPO and TRPO in both sample efﬁciency and asymptotic performance.
While it is not yet as sample efﬁcient as current state-of-the-art off-policy methods, such SAC and TD3, it is generally able to achieve a similar asymptotic performance, despite using only simple supervised regression for both policy and value function updates.
The complex
Humanoid-V2 task proved to be the most challenging case for AWR, and its performance still lags well behind SAC.
Note that RWR generally does not perform well on any of these tasks.
This sug- [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
gests that, although AWR is simple and easy to implement, the particular modiﬁcations it makes compared to standard RWR are critical for effective performance.
To illustrate AWR’s generality on tasks with discrete actions, we compare AWR to TRPO, PPO
[Figure]
and RWR on LunarLander-v2.
DDPG, TD
[Figure]
and SAC are not easily applicable to discrete action spaces due to their need to backpropagate from the Q-function to the policy.
On this discrete control task, AWR also shows strong performance compared to the other algorithms [Figure]
5. A BLATION E XPERIMENTS
To determine the effects of various design decisions, we evaluate the performance of AWR when key components of the algorithm have been removed.
The experiments include an on-policy version of
AWR (On-Policy), where only data collected from the latest policy is used to perform updates.
We also compare with a version of AWR without the baseline V ( s ) (No Baseline), which corresponds to using the standard RWR weights ω s , a = exp( 1 β R s , a ) , and another version that uses Monte Carlo return estimates instead of TD( λ ) (No TD( λ )).
The effects of these components are illustrated in
Figure 4.
Overall, these design decisions appear to be vital for an effective algorithm, with the most crucial components being the use of experience replay and a baseline.
Updates using only on-policy data can lead to instabilities and result in noticeable degradation in performance, which may be due to overﬁtting on a smaller dataset.
This issue might be mitigated by collecting a larger batch of on-policy data per iteration, but this can also negatively impact sample efﬁciency.
Removing the baseline also noticeably hampers performance.
Using simple Monte Carlo return estimates instead of TD( λ ) seems to be a viable alternative, and the algorithm still achieves competitive performance on some tasks.
When combined, these different components yield substantial performance gains over standard RWR.
To better evaluate the effect of experience replay on AWR, we compare the performance of policies trained with different capacities for the replay buffer.
Figure 4 illustrates the learning curves for buffers of size 5k, 20k, 50k, 100k, and 500k, with 50k being the default buffer size in our experiments.
The size of the replay buffer appears to have a signiﬁcant impact on overall performance.
Smaller buffer sizes can result in instabilities during training, which again may be an effect of overﬁtting to a smaller dataset.
As the buffer size increases, AWR remains stable even when the dataset is dominated by off-policy data from previous iterations.
In fact, performance over the course of training appears more stable with larger replay buffers, but progress can also become slower.
Since the sampling policy µ ( a | s ) is modeled by the replay buffer, a larger buffer can limit the rate at which µ changes by maintaining older data for more iterations.
Due to the trust region penalty in Equation 7, a slower changing µ also prevents the policy π from changing quickly.
The replay buffer therefore provides a simple mechanism to trade-off between stability and learning speed.
[Figure]
5. M OTION I MITATION
The Gym benchmarks present relatively low-dimensional tasks.
In this section, we study how AWR can solve higher-dimensional tasks with complex simulated characters, including a 34 DoF humanoid and 64 DoF dog.
The objective of the tasks is to imitate reference motion clips recorded using motion capture from real world subjects.
The experimental setup follows the motion imitation framework proposed by Peng et al.
Motion clips are collected from publicly available datasets (CMU; SFU; Zhang et al., 2018).
The skills include highly dynamics motions, such as spinkicks and canters (i.e. running), and motions that requires more coordinated movements of the character’s body, such as a cartwheel.
Snapshots of the behaviors learned by the AWR policies are available in Figure 5.
Table 2 compares the performance of AWR to RWR and the highly-tuned PPO implementation from Peng et al.
Learning curves for the different algorithms are shown in
Figure 6.
AWR performs well across the set of challenging skills, consistently achieving comparable or better performance than PPO.
RWR struggles with controlling the humanoid, but exhibits stronger performance on the dog.
This performance difference may be due to the more dynamic and acrobatic skills of the humanoid, compared to the more standard locomotion skills of the dog.
5. O FF -P OLICY L EARNING WITH S TATIC D ATASETS
Since AWR is an off-policy RL algorithm, it has the advantage of being able to leverage data from other sources.
This not only accelerates the learning process on standard tasks, as discussed above, but also allows us to apply AWR in a fully off-policy setting, where the algorithm is provided with a static dataset of transitions, and then tasked with learning the best possible policy.
To evaluate our method in this setting, we use the off-policy tasks proposed by Kumar et al.
The objective of these tasks is to learn policies solely from static datasets, without collecting any additional data from the policy that is being trained.
The dataset consists of trajectories τ
= { ( s 0 , a 0 , r 0 ) , ( s 1 , a 1 , r 1 ) , ... } from rollouts of a demo policy.
Unlike standard imitation learning tasks, which only observes the states and actions from the demo policy, the dataset also provides the reward received by the demo policy at each step.
The demo policies are trained using SAC on various OpenAI Gym tasks.
A dataset of 1 million timesteps is collected for each task.
[Table]
For AWR, we simply treat the dataset as the replay buffer D and directly apply the algorithm without additional modiﬁcations.
Figure 7 compares AWR with other algorithms when applied to the datasets.
We include comparisons to the performance of the original demo policy used to generate the dataset (Demo) and a behavioral cloning policy (BC).
The comparisons also include recent off-policy methods: batch-constrained Q-learning (BCQ) (Fujimoto et al., 2019) and bootstrapping error accumulation reduction (BEAR)
(Kumar et al., 2019), which have shown strong performance on off-policy learning with static datasets.
Note that both of these prior methods are modiﬁcations to existing off-policy RL methods, such as TD3 and SAC, which are already quite complex.
In contrast, AWR is simple and requires no modiﬁcations for the fully off-policy setting.
Despite not collecting any additional data, AWR is able to learn effective policies from these fully off-policy datasets, achieving comparable or better performance than the original demo policies.
On-policy methods, such as PPO performs poorly in this off-policy setting.
Q-function based methods, such as
TD3 and SAC, can in principle handle off-policy data but, as discussed in prior work, tend to struggle in this setting in practice (Fujimoto et al., 2019; Kumar et al., 2019).
Indeed, standard behavioral cloning (BC) often outperforms these standard RL methods.
In this fully off-policy setting, AWR can be interpreted as an advantage-weighted form of behavioral cloning, which assigns higher likelihoods to demonstration actions that receive higher advantages.
Unlike Q-function based methods,
AWR is less susceptible to issues from out-of-distribution actions as the policy is always trained on observed actions from the behaviour data (Kumar et al., 2019).
AWR also shows comparable performance to BEAR and BCQ, which are speciﬁcally designed for this off-policy setting and introduce considerable algorithmic overhead.
D ISCUSSION AND F UTURE W ORK
We presented advantage-weighted regression, a simple off-policy reinforcement learning algorithm, where policy updates are performed using standard supervised learning methods.
Despite its simplicity, our algorithm is able to solve challenging control tasks with complex simulated agents, and achieve competitive performance on standard benchmarks compared to a number of well-established
RL algorithms.
Our derivation introduces several new design decisions, and our experiments verify the importance of these components.
AWR is also able to learn from fully off-policy datasets, demonstrating comparable performance to state-of-the-art off-policy methods.
While AWR is effective for a diverse suite of tasks, it is not yet as sample efﬁcient as the most efﬁcient off-policy algorithms.
We believe that exploring techniques for improving sample efﬁciency and performance on fully off-policy learning can open opportunities to deploy these methods in real world domains.
We are also interested in exploring applications that are particularly suitable for these regression-based
RL algorithms, as compared to other classes of RL techniques.
A better theoretical understanding of the convergence properties of these algorithms, especially when combined with experience replay, could also be valuable for the development of future algorithms.
A AWR D ERIVATION
In this section, we derive the AWR algorithm as an approximate optimization of a constrained policy search problem.
Our goal is to ﬁnd a policy that maximize the expected improvement η ( π ) =
J ( π ) − J ( µ ) over a sampling policy µ ( a | s ) .
We start with a lemma from Kakade & Langford (2002), which shows that the expected improvement can be expressed in terms of the advantage
A µ ( s , a ) =
R µ s , a − V µ ( s ) with respect to the sampling policy µ , where R µ s , a denotes the return obtained by performing action a in state s and following µ for the following timesteps, and V µ ( s )
= (cid:82) a µ ( a | s ) R as d a corresponds to the value function of µ ,
[Equation]
We can rewrite Equation 22 with an expectation over states instead of trajectories:
[Equation]
where d π ( s ) =
(cid:80) ∞ t =0 γ t p ( s t = s | π )
represents the unnormalized discounted state distribution induced by the policy
π (Sutton & Barto, 1998), and p ( s t = s | π ) is the likelihood of the agent being in state s after following π for t timesteps.
The objective in Equation 25 can be difﬁcult to optimize due to the dependency between d π ( s ) and π , as well as the need to collect samples from π .
Following Schulman et al.
(2015), we can optimize an approximation ˆ η ( π ) of η ( π ) using the state distribution of µ ,
[Equation]
(26) ˆ η ( π ) matches η ( π ) to ﬁrst order (Kakade & Langford, 2002), and provides a reasonable estimate of η if π and µ are similar.
Using this objective, we can formulate the following constrained policy search problem:
[Equation] [Equation]
Since enforcing the pointwise KL constraint in Equation 28 at all states is intractable, we relax the constraint by enforcing it only in expectation (cid:82) s d µ ( s )D KL ( π ( ·| s ) || µ ( ·| s ))
d s ≤ (cid:15) .
To further simplify the optimization problem, we relax the hard KL constraint by converting it into a soft constraint with coefﬁcient β ,
[Equation]
Next we form the Lagrangian,
[Equation]
with β and α = { α s | ∀ s ∈ S} corresponding to the Lagrange multipliers.
Differentiating L ( π, β, α ) with respect to π ( a | s ) results in ∂
[Equation]
Setting to zero and solving for π ( a | s ) gives
[Equation]
Since (cid:82) a π ( a | s ) d a = 1 , the second exponential term is the partition function Z ( s ) that normalizes the conditional action distribution,
[Equation] [Equation]
The optimal policy is therefore given by,
[Equation]
If π is represented by a function approximator
[Equation]
the optimal policy π ∗ can be projected onto the manifold of parameterized policies by solving the following supervised regression problem
[Equation]
[Equation]
B AWR D ERIVATION WITH E XPERIENCE R EPLAY
In this section, we extend the derivation presented in Appendix A to incorporate experience replay using a replay buffer containing data from previous policies.
To recap, the sampling distribution is a mixture of k past policies { π 1 , · · · , π k } , where the mixture is performed at the trajectory level.
First, we deﬁne the trajectory distribution µ ( τ ) , marginal state-action distribution µ ( s , a ) , and marginal state distribution d µ ( s ) of the replay buffer according to:
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
w
[Equation]
d
[Equation]
( s ) (40) where the weights (cid:80) i w i = 1 specify the probabilities of selecting each policy π i .
The conditional action distribution µ ( a | s ) induced by the replay buffer is given by:
[Equation]
Next, using Lemma 6.1 from Kakade & Langford (2002) (also derived in Appendix A), the expected improvement of π over each policy π i satisﬁes
[Equation]
The expected improvement over the mixture can then be expressed with respect to the individual policies,
[Equation] [Equation]
In order to ensure that the policy π is similar to the past policies
[Equation]
we constrain π against the conditional action distributions of the replay buffer,
[Equation] [Equation]
Note that constraining π against µ ( a | s ) has a number of desirable properties.
First, the constraint prevents the policy π from choosing actions that are vastly different from all of the policies { π 1 , · · · , π k } .
Second, the mixture weight assigned to each π i in the deﬁnition of µ depends on the marginal state density d π i ( s ) for the particular policy.
This property is desirable as the policy π is now constrained to be similar to π i only at states that are likely to be visited by π i .
This then yields the following constrained objective:
[Equation]
The Lagrangian of the above objective is given by:
[Equation]
Solving the Lagrangian following the same procedure as Appendix A leads to an optimal policy of the following form:
[Equation]
Finally, if π is represented by a function approximator, the optimal policy π ∗ can be projected onto the manifold of parameterized policies by solving the following supervised regression problem
[Equation]
One of the challenges of optimizing the objective in Equation 54 is that computing the expected return in the exponent requires rolling out multiple policies starting from the same state, which would require the environment to be resettable to any given state.
Therefore, to obtain a more practical objective, we approximate the expected return across policies using a single rollout from the replay buffer,
[Equation]
This single-sample estimator results in a biased estimate of the exponentiated advantage, because the expectation with respect to the mixture weights appears in the exponent.
But in practice, we ﬁnd this biased estimator to be effective for our experiments.
Therefore, the objective used in practice is given by:
[Equation]
where the expectations can be approximated by simply sampling from D following Line 6 of Algorithm 1.
Note, the baseline in the exponent now consists of an average of the value functions of the different policies.
One approach for estimating this quantity would be to ﬁt separate value functions
V π i for each policy.
However, if only a small amount of data is available from each policy, then V π i could be highly inaccurate.
Therefore, instead of learning separate value functions, we ﬁt a single mean value function ¯ V ( s ) that directly estimates the weighted average of V π
i ’s,
[Equation]
This loss can also be approximated by simply sampling from the replay buffer following Line 5 of
Algorithm 1.
The optimal solution ¯ V ( s ) =
[Equation]
is exactly the baseline in Equation 56.
C E XPERIMENTAL S ETUP
In our experiments, the policy is represented by a fully-connected network with 2 hidden layers consisting of 128 and 64 ReLU units respectively (Nair & Hinton, 2010), followed by a linear output layer.
The value function is modeled by a separate network with a similar architecture, but consists of a single linear output unit for the value.
Stochastic gradient descent with momentum is used to update both the policy and value function.
The stepsize of the policy and value function are 5 × 10 − 5 and 1 × 10 − 4 respectively, and a momentum of 0.9 is used for both.
The temperature is set to β = 0 .
for all experiments, and λ = 0 .
95 is used for TD( λ ).
The weight clipping threshold ω max is set to 20.
At each iteration, the agent collects a batch of approximately 2000 samples, which are stored in the replay buffer D along with samples from previous iterations.
The replay buffer stores 50k of the most recent samples.
Updates to the value function and policy are performed by uniformly sampling minibatches of 256 samples from D .
The value function is updated withgradient steps per iteration, and the policy is updated with 1000 gradient steps.
D S IMILARITIES TO P OLICY G RADIENTS
On the surface, the AWR policy update bears striking similarities to a conventional policy gradient (PG) update (Sutton et al., 2000):
[Equation]
However, there are a number of subtle but important differences between the two.
First, basic policy gradient algorithms are on-policy methods, which requires the data to be sampled from the same policy π that is being optimized s
∼ d π ( s ) and a ∼ π ( a | s ) , whereas AWR can in principle learn using data from any sampling distribution µ .
This requirement for policy gradient methods is because
PG directly differentiates through the sampling distribution to compute the gradient of the expected return with respect to the policy parameters.
But with AWR and other EM algorithms, they ﬁrst construct an estimate of the optimal action distribution at each state, and then projects that action distribution onto the space of parameterized policies.
Therefore AWR does not need to differentiate through the sampling distribution, which is a critical feature for settings such as batch RL, where the sampling distribution (e.g. demo policy) may not be available to the agent.
In AWR, the log probability of an action log π ( a | s ) is weighted by the exponentiated advantage exp (cid:16) 1 β
( R s , a − V ( s )) (cid:17) , while in PG the log probability is weighted just by the advantage ( R s , a − V ( s )) without the exponential.
Since the exponentiated advantage is non-negative, the objective used in the AWR update is a maximum likelihood objective that tries to maximize the likelihood of all actions, but to varying amounts depending on the exponentiated advantage.
In the case of PG, the advantage can be both positive and negative, therefore PG updates decrease the likelihood of actions with negative advantages, and thus it is not a conventional maximum likelihood objective.
In practice, negative
TD updates are often a source of instability when applying PG to off-policy data.
E A DDITIONAL E XPERIMENTS
A comprehensive comparison of AWR with prior methods on all of the tasks considered are available in Figure 8 and 9.
E. W EIGHT C LIPPING
To analyze the effects of weight clipping on the stability of AWR we compare learning curves of policies trained with weight clipping using a threshold of ω max = and policies trained without weight clipping.
Figure 10 compares the learning curves with and without clipping.
5 separate AWR runs with different random seeds are visualized separately.
With weight clipping, performance remains stable throughout training.
Policies trained without weight clipping are substantially more unstable, exhibiting drastic ﬂuctuations in performance as a result of exploding gradients from excessively large weights.
Some training runs without clipping are terminated early due to exploding gradients causing the networks to output NaNs.
These experiments suggest that weight clipping is vital for ensuring stable training with AWR.