D EPLOYMENT -E FFICIENT R EINFORCEMENT L EARNING VIA M ODEL -B ASED O FFLINE O PTIMIZATION
A BSTRACT
Most reinforcement learning (RL) algorithms assume online access to the environment, in which one may readily interleave updates to the policy with experience collection using that policy.
However, in many real-world applications such as health, education, dialogue agents, and robotics, the cost or potential risk of deploying a new data-collection policy is high, to the point that it can become prohibitive to update the data-collection policy more than a few times during learning.
With this view, we propose a novel concept of deployment efﬁciency , measuring the number of distinct data-collection policies that are used during policy learning.
We observe that naïvely applying existing model-free ofﬂine RL algorithms recursively does not lead to a practical deployment-efﬁcient and sample-efﬁcient algorithm.
We propose a novel model-based algorithm, Behavior-Regularized
Model-ENsemble (BREMEN), that not only performs better than or comparably as the state-of-the-art dynamic-programming-based and concurrently-proposed model-based ofﬂine approaches on existing benchmarks, but can also effectively optimize a policy ofﬂine using 10-20 times fewer data than prior works.
Furthermore, the recursive application of BREMEN achieves impressive deployment efﬁciency while maintaining the same or better sample efﬁciency, learning successful policies from scratch on simulated robotic environments with only 5-10 deployments, compared to typical values of hundreds to millions in standard RL baselines.
I NTRODUCTION
Reinforcement learning (RL) algorithms have recently demonstrated impressive success in learning behaviors for a variety of sequential decision-making tasks (Barth-Maron et al., 2018; Hessel et al., 2018; Nachum et al., 2019).
Virtually all of these demonstrations have relied on highly-frequent online access to the environment, with the RL algorithms often interleaving each update to the policy with additional experience collection of that policy acting in the environment.
However, in many real-world applications of RL, such as health (Murphy et al., 2001), education (Mandel et al., 2014), dialog agents (Jaques et al., 2019), and robotics (Gu et al., 2017a;
Kalashnikov et al., 2018), the deployment of a new data-collection policy may be associated with a number of costs and risks.
If we can learn tasks with a small number of data collection policies, we can substantially reduce them.
Based on this idea, we propose a novel measure of RL algorithm performance, namely deployment efﬁciency , which counts the number of changes in the data-collection policy during learning, as illustrated in Figure 1.
This concept may be seen in contrast to sample efﬁciency or data efﬁciency (Precup et al., 2001; Degris et al., 2012;
Gu et al., 2017b; Haarnoja et al., 2018; Lillicrap et al., 2016; Nachum et al., 2018), which measures the amount of environment interactions incurred during training, without regard to how many distinct policies were deployed to perform those interactions.
Even when the [Figure] data efﬁciency is high, the deployment efﬁciency could be low, since many on-policy and off-policy algorithms alternate data collection with each policy update (Schulman et al., 2015; Lillicrap et al., 2016;
Gu et al., 2016; Haarnoja et al., 2018).
Such dependence on high-frequency policy deployments is best illustrated in the recent works in ofﬂine RL (Fujimoto et al., 2019; Jaques et al., 2019; Kumar et al., 2019; Levine et al., 2020; Wu et al., 2019) where baseline off-policy algorithms exhibited poor performance when trained on a static dataset.
These ofﬂine RL works, however, limit their study to a single deployment, which is enough for achieving high performance with data collected from a sub-optimal behavior policy, but often not from a random policy.
In contrast to those prior works, we aim to learn successful policies from scratch in a manner that is both sample and deployment-efﬁcient.
Many existing model-free ofﬂine RL algorithms (Levine et al., 2020) are tuned and evaluated on massive datasets (e.g., one million transitions).
In order to develop an algorithm that is both sample and deployment-efﬁcient, each iteration of the algorithm between successive deployments has to work effectively on much smaller dataset sizes.
We believe model-based RL is better suited to this setting due to its higher demonstrated sample efﬁciency than model-free RL (Kurutach et al., 2018; Nagabandi et al., 2018).
Although the combination of model-based RL and ofﬂine or limiteddeployment settings seems straight-forward, we ﬁnd this naïve approach leads to poor performance.
This problem can be attributed to extrapolation errors (Fujimoto et al., 2019) similar to those observed in model-free methods.
Speciﬁcally, the learned policy may choose sequences of actions which lead it to regions of the state space where the dynamics model cannot predict properly, due to poor coverage of the dataset.
This can lead the policy to exploit approximation errors of the dynamics model and be disastrous for learning.
In model-free settings, similar data distribution shift problems are typically remedied by regularizing policy updates explicitly with a divergence from the observed data distribution (Jaques et al., 2019; Kumar et al., 2019;
Wu et al., 2019), which, however, can overly limit policies’ expressivity (Sohn et al., 2020).
In order to better approach these problems arising in limited deployment settings, we propose
Behavior-Regularized Model-ENsemble (BREMEN), which learns an ensemble of dynamics models in conjunction with a policy using imaginary rollouts while implicitly regularizing the learned policy via appropriate parameter initialization and conservative trust-region learning updates.
We evaluate
BREMEN on standard ofﬂine RL benchmarks of high-dimensional continuous control tasks, where only a single static dataset is used.
In this ﬁxed-batch setting, our experiments show that BREMEN can not only achieve performance competitive with state-of-the-art when using standard dataset sizes but also learn with 10-20 times smaller datasets, which previous methods are unable to attain.
Enabled by such stable and sample-efﬁcient ofﬂine learning, we show that BREMEN can learn successful policies with only 5-10 deployments in the online setting, signiﬁcantly outperforming existing off-policy and ofﬂine RL algorithms in deployment efﬁciency while keeping sample efﬁciency.
P RELIMINARIES
We consider a Markov Decision Process (MDP) setting, characterized by the tuple M =
( S , A , p, r, γ ) , where S is the state space,
A is the action space, p ( s (cid:48) | s, a ) is the transition probability distribution or dynamics, r ( s ) is the reward function and γ ∈ (0 , 1) is the discount factor.
A policy π is a function that determines the agent behavior, mapping from states to probability distributions over actions.
The goal is to obtain the optimal policy π ∗ , which maximizes the expectation of discounted sum of rewards.
The transition probability p ( s (cid:48) | s, a ) is usually unknown, and estimated with a parameterized dynamics model f φ (e.g. a neural network) in model-based RL.
For simplicity, we assume that the reward function r ( s ) is known, and the reward can be computed for any arbitrary state, but we may extend to the unknown setting and predict it using a parameterized function.
On-policy vs Off-policy, Online vs Ofﬂine At a high-level, most RL algorithms alternate many times between collecting a batch of transitions (deployments) and optimizing the policy (learning).
If the algorithms discard data after each policy update, they are on-policy (Schulman et al., 2015; 2017), while if they accumulate data in a buffer D , i.e. experience replay (Lin, 1992), they are off-policy (Mnih et al., 2015;
Lillicrap et al., 2016;
Gu et al., 2016; 2017b; Haarnoja et al., 2018;
Fujimoto et al., 2019; Fakoor et al., 2019) because not all the data in buffer comes from the current policy.
However, we consider all these algorithms to be online RL algorithms, since they involve many deployments during learning, ranging from hundreds to millions.
On the other hand, in pure ofﬂine RL, one does not assume direct interaction and learns a policy from only a ﬁxed dataset, which effectively corresponds to a single deployment allowed for learning.
Classically, interpolating these two extremes were semi-batch RL algorithms (Lange et al., 2012;
Singh et al., 1995), which improve the policy through repetitions of collecting a large batch of transitions
D = { ( s, a, s (cid:48) , r ) } and performing many or full policy updates.
While these semi-batch RL also realize good deployment efﬁciency, they have not been extensively studied with neural network function approximators or in off-policy settings with experience replay for scalable sample-efﬁcient learning.
In our work, we aim to have both high deployment efﬁciency and sample efﬁciency by developing an algorithm that can solve the tasks with minimal policy deployments as well as transition samples.
D EPLOYMENT E FFICIENCY
Deploying a new policy for data collection can be associated with a number of costs and risks for many real-world applications like medicine, dialogue systems, or robotic control (Murphy et al., 2001;
Mandel et al., 2014; Gu et al., 2017a;
Kalashnikov et al., 2018; Nachum et al., 2019; Jaques et al., 2019).
While there are abundant works on safety for RL (Chow et al., 2015; Eysenbach et al., 2018;
Chow et al., 2018; Ray et al., 2019; Chow et al., 2019), they often do not provide guarantees in practice when combined with neural networks and stochastic optimization.
It is therefore necessary to verify each policy before deployment (e.g. measuring the variance of rewards or checking out-of-bounds actions).
Due to such costs associated with each deployment, it is desirable to minimize the number of distinct deployments needed during the learning process.
Even ignoring safety considerations, frequent updates to a deployed policy can exacerbate communication bottlenecks in large-scale distributed RL systems, which are becoming more prevalent (Nair et al., 2015; Espeholt et al., 2018; 2019).
We additionally discuss on the importance of the deployment efﬁciency in real-world applications.
See Appendix C.
In order to focus research on these practical bottlenecks, we propose a novel measure of RL algorithms, namely, deployment efﬁciency , which counts how many times the data-collection policy has been changed during improvement from random policy to solve the task.
For example, if an RL algorithm operates by using its learned policy to collect transitions from the environment I times, each time collecting a batch of B new transitions, then the number of deployments is I , while the total number of samples collected is I × B .
The lower I is, the more deployment-efﬁcient the algorithm is; in contrast, sample efﬁciency looks at I × B .
Online RL algorithms, whether they are on-policy or off-policy, typically update the policy and acquire new transitions by deploying the newly updated policy at every iteration.
This corresponds to performing hundreds to millions of deployments during learning on standard benchmarks (Haarnoja et al., 2018), which is severely deployment inefﬁcient.
On the other hand, ofﬂine RL literature only studies the case of 1 deployment.
A deployment-efﬁcient algorithm would stand in the middle of these two extremes and ideally learn a successful policy from scratch while deploying only a few distinct policies, as illustrated in Figure 1.
Recent deep RL literature seldom emphasizes deployment efﬁciency, with few exceptions in speciﬁc applications (Kalashnikov et al., 2018) where such a learning procedure is necessary.
Deploymentinefﬁcient algorithms will fail in scenarios where the deployment of each new policy is exorbitantly expensive, such as safety-critical robotics or user-facing products.
Although current state-of-the-art algorithms on continuous control have substantially improved sample or data efﬁciency, they have not optimized for deployment efﬁciency.
For example, SAC (Haarnoja et al., 2018), an efﬁcient model-free off-policy algorithm, performs half a million to one million policy deployments during learning on MuJoCo (Todorov et al., 2012) benchmarks.
ME-TRPO (Kurutach et al., 2018), a model-based algorithm, performs a much lower 100-300 policy deployments, although this is still relatively high for practical settings.
In our work, we demonstrate successful learning on standard benchmark environments with only 5-10 deployments.
B EHAVIOR -R EGULARIZED M ODEL -E NSEMBLE
To achieve a favorable combination of both high deployment and sample efﬁciency, we propose
Behavior-Regularized Model-ENsemble (BREMEN).
BREMEN incorporates Dyna-style (Sutton, 1991; Kurutach et al., 2018) model-based RL, learning an ensemble of dynamics models in conjunction with a policy using imaginary rollouts and behavior regularization via conservative trust-region updates.
4. I MAGINARY R OLLOUT FROM M ODEL E NSEMBLE
As in recent Dyna-style model-based RL methods (Kurutach et al., 2018; Wang et al., 2019), BREMEN uses an ensemble of K deterministic dynamics models ˆ f φ = (cid:110)
ˆ f φ 1 , . .
ˆ f φ K (cid:111) to alleviate the problem of model bias.
Each model
ˆ f φ i is parameterized by φ i and trained by the following objective, which minimizes mean squared error between the prediction of next state
ˆ f φ i ( s t , a t ) and true next state s t +1 over a dataset D :
[Equation]
During training of a policy π θ , imagined trajectories of states and actions are generated sequentially, using a dynamics model
ˆ f φ i that is randomly selected at each time step:
[Equation]
4. P OLICY U PDATE WITH B EHAVIOR R EGULARIZATION
In order to manage the discrepancy between the true dynamics and the learned model caused by the distribution shift in batch settings, we propose to use iterative policy updates via a trust-region constraint, re-initialized with a behavior-cloned policy after every deployment.
Speciﬁcally, after each deployment, we are given an updated dataset of experience transitions D .
With this dataset, we approximate the true behavior policy π b through behavior cloning (BC), utilizing a neural network ˆ π β parameterized by β , where we implicitly assume a ﬁxed variance, a common practice in
BC (Rajeswaran et al., 2017):
[Equation]
After obtaining the estimated behavior policy, we initialize the target policy π θ as a Gaussian policy with mean from ˆ π β and standard deviation of 1 .
This BC initialization in conjunction with gradient descent based optimization may be seen as implicitly biasing the optimized π θ to be close to the data-collection policy (Nagarajan & Kolter, 2019), and thus works as a remedy for the distribution shift problem (Ross et al
To further bias the learned policy to be close to the data-collection [Table] [Table] policy, we opt to use a KL-based trust-region optimization (Schulman et al., 2015).
Therefore, the optimization of BREMEN becomes
[Equation]
where A π θk ( s, a ) is the advantage of π θ k computed using model-based rollouts in the learned dynamics model and δ is the maximum step size.
The combination of BC for initialization and ﬁnite iterative trust-region updates serves as an implicit
KL regularization.
This is in contrast to many previous ofﬂine RL algorithms that augment the value function with a penalty of explicit KL divergence (Siegel et al., 2020; Wu et al., 2019) or maximum mean discrepancy (Kumar et al., 2019).
Empirically, we found that our regularization technique outperforms the explicit KL penalty (Section 5.3).
Furthermore, we provide a mathematical intuition explaining how our methods works as an implicit regularization of distributional shift in Appendix A.
By recursively performing ofﬂine procedure, BREMEN can be used for deployment-efﬁcient learning as shown in Algorithm 1, starting from a randomly initialized policy, collecting experience data, and performing ofﬂine policy updates.
E XPERIMENTS
In order to realize a deployment-efﬁcient RL algorithm, the batch policy optimizer has to be stable and sample-efﬁcient.
We ﬁrst evaluate BREMEN in the ofﬂine setting, where the algorithm learns the policy from a static dataset.
Standard benchmarks of MuJoCo physics simulator shown in (Wu et al., 2019) and more recent datasets (Fu et al., 2020) are used in the evaluation, and we compared the asymptotic performance of BREMEN with other ofﬂine RL methods including the concurrent model-based approaches.
We then tested the sample-efﬁciency of ofﬂine algorithms using smaller datasets.
We lastly extend the experiment to deployment-efﬁcient settings, where the algorithms learn their policies from scratch via a limited number of deployments and perform some ablations to see how components in BREMEN affect performance.
See Appendix F for further details.
5. E VALUATING O FFLINE RL P ERFORMANCES
Standard Benchmarks We evaluate BREMEN on standard ofﬂine RL benchmarks following and identical protocol as in Wu et al.
We ﬁrst train online SAC to a certain cumulative reward threshold, 4,000 in HalfCheetah, 1,000 in Ant, Hopper, and Walker2d, and collect ofﬂine datasets.
We evaluate agents with the ofﬂine dataset of one million (1M) transitions, which is standard for BCQ and
BRAC.
Table 1 (top) shows that BREMEN can achieve performance competitive with state-of-the-art model-free ofﬂine RL algorithms when using the standard dataset size of 1M. We also test BREMEN with more recent benchmarks of D4RL (Fu et al., 2020) and compared the performance with the existing model-free and model-based methods.
See Appendix D for the results.
[Table] [Table] Evaluating Sample-Efﬁciency We then evaluate the sample-efﬁciency by making much smaller datasets of 50k and 100k transitions (5 ∼ 10 % of Wu et al.
Surprisingly, Table 1 (middle and bottom) shows that BREMEN can also learn with smaller datasets [Table] where BCQ and BRAC are unable to exceed even BC baseline.
This is a novel evaluation protocol we proposed
[Table]
and our BREMEN’s superior performance here is exactly what enables recursive BREMEN in the next section to be an effective algorithm in deployment-constrained settings.
5. E VALUATING D EPLOYMENT E FFICIENCY IN O NLINE RL B ENCHMARKS
We compare BREMEN to ME-TRPO, SAC, BCQ
[Table]
and BRAC applied to limited deployment settings. To adapt ofﬂine methods (BCQ, BRAC) to this setting, we simply apply them in a recursive fashion;at each deployment iteration, we collect a batch of data with the most recent policy and then run the ofﬂine update with this dataset.
As for SAC, we simply change the replay buffer to update only at speciﬁc deployment intervals.
For the sake of comparison, we align the number of deployments and the amount of data collection at each deployment (either 100k or 200k) for all methods.
Figure 2 shows the results with 200k (top) and 100k (bottom) batched transitions per deployment.
Regardless of the environments and the batch size per update, BREMEN achieves remarkable performance while existing online and ofﬂine RL methods struggle to make any progress.
As a point of comparison, we also include results for online SAC and ME-TRPO without deployment-limits but using the same number of transitions.
We additionally compare BREMEN to the model-based ofﬂine
RL methods with uncertainty-based penalties.
See Appendix E for further details.
Following the motivation of deployment efﬁciency, obtaining a successful policy under data-collection constraint conditions in the real application, we extensively evaluate our algorithm on more realistic robotics environments in OpenAI Gym. The experimental procedure is the same as above, while [Figure] required for learning.
Interestingly, BREMEN achieves even better performance than the original ME-TRPO(online), suggesting the effectiveness of implicit behavior regularization.
For SAC and ME-TRPO under deployment-constrained evaluation, their batch size between policy deployments differs substantially from their standard settings, and therefore we performed extensive hyper-parameter search on the relevant parameters such as the number of policy updates between deployments, as discussed in Appendix F.2.1.
[Figure] we limit the batch size at each deployment as only 25k.
Figure 3 presents the reaching tasks with
Fetch robot and 20-DoF shadow hand (Plappert et al., 2018), and the experimental results in both environments.
Only BREMEN shows stable improvement and high performance, while other ofﬂine and online algorithms fail to learn.
These results suggest that a model-based method is a desirable approach for satisfying practical requirements in robotics, i.e. sample and deployment efﬁciency.
5. A BLATION : E VALUATING E FFECTIVENESS OF I MPLICIT KL C ONTROL
In this section, we present an experiment to better understand the effect of BREMEN’s implicit regularization.
Figure 4 shows the KL divergence of learned policies from the last deployed policy.
We compare BREMEN to variants of BREMEN that use an explicit KL penalty on value instead of BC initialization (conservative KL trust-region updates are still used).
We ﬁnd that the explicit KL without behavior initialization variants learn policies that move farther away from the last deployed policy than behavior initialized policies.
This suggests that the implicit behavior regularization employed by
BREMEN is more effective as a conservative policy learning protocol.
In addition, to assess the effect of repeated behavior cloning initialization, we also evaluate a variant of BREMEN without behavior cloning re-initialization (grey).
This variant works in easier environments (Ant, Halfcheetah), but does not show remarkable progress in more challenging ones with termination (Hopper, Walker2d). [Figure] = [Figure]
[Figure]
This result empirically supports the need for repeated behavior initialization after each deployment.
The results of further experiments are shown in Appendix G
[Figure]
R ELATED W ORK
Deployment Efﬁciency and Ofﬂine RL Although we are not aware of any previous works which explicitly proposed the concept of deployment efﬁciency, its necessity in many real-world applications has been generally known.
One may consider previously proposed semi-batch RL algorithms (Ernst et al., 2005; Lange et al., 2012; Singh et al., 1994; Roux, 2016) or theoretical analysis of switching cost under the tabular PAC-MDP settings (Bai et al., 2019; Guo & Brunskill, 2015) as approaching this issue.
More recently, a related but distinct problem known as ofﬂine RL has gained popularity (Levine et al., 2020; Wu et al., 2019; Agarwal et al., 2019; Kumar et al., 2020).
These works consider an extreme version of 1 deployment, and typically collect the static batch with a partially trained policy rather than a random policy.
While ofﬂine RL has shown promising results for a variety of real-world applications, such as robotics (Mandlekar et al., 2019), dialogue systems (Jaques et al., 2019), or medical treatments (Gottesman et al., 2018), these algorithms struggle when learning a policy from scratch or when the dataset is small.
Nevertheless, common themes of many ofﬂine RL algorithms – regularizing the learned policy to the behavior policy (Fujimoto et al., 2019; Kumar et al., 2019;
Siegel et al., 2020; Wu et al., 2019) and utilizing ensembles to handle uncertainty (Kumar et al., 2019; Wu et al., 2019) – served as inspirations for the proposed our algorithm.
A major difference of BREMEN from prior works is that the target policy is not explicitly forced to stick close to the estimated behavior policy through the policy update except for the initial iteration.
Rather,
BREMEN employs a more implicit regularization by initializing the learned policy with a behavior cloned policy and then applying conservative trust-region updates.
Another major difference is the application of model-based approaches to fully ofﬂine settings, which has not been extensively studied in prior works (Levine et al., 2020), except the two concurrent works (Kidambi et al., 2020;
Yu et al., 2020) that study pessimistic or uncertainty penalized MDPs with guarantees – closely related to Liu et al.
By contrast, our work shows that a simple technique can already enable model-based ofﬂine algorithms to signiﬁcantly outperform the prior model-free methods, and is, to the best of our knowledge, the ﬁrst to deﬁne and extensively evaluate deployment efﬁciency with recursive experiments.
Model-Based RL There are many types of model-based RL algorithms (Sutton, 1991; Deisenroth & Rasmussen, 2011; Heess et al., 2015).
A simple algorithmic choice is Dyna-style (Sutton, 1991), which uses a parameterized model to estimate the true MDP transition function, stochastically mapping states and actions to next states.
The dynamics model can then serve as a simulator of the environment during policy updates.
Dyna-style algorithms often suffer from the distributional shift, also known as model bias, which leads RL agents to exploit regions where the data is insufﬁcient, and signiﬁcant performance degradation.
A variety of remedies have been proposed to relieve the issue of model bias, such as the use of multiple dynamics models as an ensemble (Chua et al., 2018;
Kurutach et al., 2018; Janner et al., 2019), meta-learning (Clavera et al., 2018), energy-based regularizer (Boney et al., 2019), game-theoretic framework (Rajeswaran et al., 2020), and explicit penalty for unknown states (Kidambi et al., 2020; Yu et al., 2020).
Notably, we have employed a subset of these remedies – model ensembles and trust-region updates (Kurutach et al., 2018) – for BREMEN.
Compared to prior works, our work is notable for using BC initialization in conjunction with trust-region updates to alleviate the distribution shift of the learned policy from the dataset used to train the dynamics model.
C ONCLUSION
In this work, we introduced deployment efﬁciency , a novel measure for RL performance that counts the number of changes in the data-collection policy during learning.
To enhance deployment efﬁciency, we proposed a novel model-based ofﬂine algorithm, Behavior-Regularized Model-ENsemble (BREMEN), combining model-ensembles with trust region updates from model-based RL literature (Kurutach et al., 2018), and policy initialization with behavior cloning from ofﬂine RL literature (Fujimoto et al., 2019; Wu et al., 2019).
Crucially, BREMEN can improve policies ofﬂine sample-efﬁciently even when the batch size is 10-20 times smaller than prior works, allowing BREMEN to achieve impressive results in limited deployment settings, obtaining successful policies from scratch in only 5-10 deployments.
Not only can this help alleviate costs and risks in real-world applications, but it can also reduce the amount of communication required during distributed learning and could form the basis for communication-efﬁcient large-scale RL in contrast to prior works (Nair et al., 2015; Espeholt et al., 2018; 2019).
Most critically, we show that under deployment efﬁciency constraints, most prior algorithms – model-free or model-based, online or ofﬂine – fail to achieve successful learning.
One possible direction for future work is to incorporate veriﬁcation efﬁciency into consideration, since a stochastic multi-modal policy could collect more diverse transitions while it takes more trajectories to be veriﬁed for safety than a uni-modal policy.
While we presented promising results on some realistic simulated environments, validating BREMEN on real robots is another direction.
We hope our work can gear the research community to value deployment efﬁciency as an important criterion for RL algorithms, and to eventually achieve similar sample efﬁciency and asymptotic performance as the state-of-the-art algorithms like SAC (Haarnoja et al., 2018) while having the deployment efﬁciency well-suited for safe and practical real-world reinforcement learning.
A PPENDIX A I MPLICIT KL C ONTROL FROM A M ATHEMATICAL P ERSPECTIVE
We can intuitively understand that behavior cloning initialization with trust-region updates works as a regularization of distributional shift, and this can be supported by theory.
Following the notation of Janner et al.
(2019), we denote the generalization error of a dynamics model on the state distribution under the true behavior policy as (cid:15) m = max t E s ∼ d πbt D TV ( p ( s t +1 | s t , a t )
||
p φ ( s t +1 | s t , a t )) , where
D TV represents the total variation distance between true dynamics p and learned model p φ .
We also denote the distribution shift on the target policy as max s D TV ( π b || π ) ≤ (cid:15) π .
A bound relating the true returns η
[ π ] and the model returns ˆ η [ π ] on the target policy is given in Janner et al.
(2019) as,
[Equation]
[Equation]
This bound guarantees the improvement under the true returns as long as the improvement under the model returns increases by more than the slack in the bound due to (cid:15) m , (cid:15) π (Janner et al.,
[Equation]
Levine et al., 2020).
We may relate this bound to the speciﬁc learning employed by BREMEN, which includes dynamics model learning, behavior cloning policy initialization, and conservative KL-based trust-region policy updates.
To do so, we consider an idealized version of BREMEN, where the expectations over states in equations Equation 1, 3, 4 are replaced with supremums and the dynamics model is set to have unit variance.
Proposition 1 (Policy and model error bound) .
Suppose we apply the idealized BREMEN on a dataset D , and deﬁne (cid:15) β , (cid:15) φ in terms of the behavior cloning and dynamics model losses as,
[Equation]
where H denotes the Shannon entropy.
If one then applies T KL-based trust-region steps of step size δ (Equation 4) using stochastic dynamics models with mean ˆ f φ and standard deviation 1, then
[Equation]
where d a and d s denotes the dimension of action and state space.
Proof.
We ﬁrst consider (cid:15) π .
The behavior cloning objective in its supremum form is,
[Equation]
We apply Pinsker’s inequality to the true and estimated behavior policy to yield
[Equation]
By the same Pinsker’s inequality, we have,
[Equation]
Therefore, by triangle inequality, we have
[Equation]
as desired.
We perform similarly for (cid:15) m .
The model dynamics loss is
[Equation]
We apply Pinsker’s inequality to the true dynamics and learned model to yield
[Equation]
as desired.
B T RADE - OFF BETWEEN S AMPLE AND D EPLOYMENT E FFICIENCY
An important aspect of deployment efﬁciency is the trade-off between sample and deployment efﬁciency.
To collect multiple data points per experiment and show this trade-off, we run recursive
BREMEN with different batch sizes, and record how many samples are required to cross different reward thresholds.
HalfCheetah (Reward 7,000 result) and other results from Figure 5 generally show that high deployment efﬁciency lowers sample efﬁciency, conﬁrming the inherent trade-off.
However, in rare cases, e.g. Ant (Reward 5,000 result), it could be possible to achieve both high deployment efﬁciency and high sample efﬁciency through the right choice of the batch size hyper-parameter.
[Figure]
C D ISCUSSION : I MPORTANCE OF D EPLOYMENT E FFICIENCY IN R EAL -W ORLD A PPLICATIONS
Our notion of deployment-efﬁciency is necessitated by cost and safety constraints typical in many real world scenarios.
Namely, a common approach to real-world applications (Cabi et al., 2020; DulacArnold et al., 2019; Kalashnikov et al., 2018) is the following iterative training and data-collection paradigm:
Aggregate past previous dataset from worker(s) 2.
Update policy based on the collected data 3.
Deploy the policy to the worker(s) 4.
Monitor the policy works as expected e.g. checking if it does not violate safety criterion (this safety veriﬁcation step may alternatively happen before step 3) 5.
Let the worker(s) collect experiences with the latest policy.
It is easy to see that the number of deployments is a critical bottleneck, as it involves both monitoring of the policy (Step 4) and communication to the workers (Step 3), and both of these steps can incur signiﬁcant cost.
Speciﬁcally, Step 4 requires evaluating the policy for safety, and often requires human monitors (Atkeson et al., 2015).
As for Step 3, communication to workers can also be a bottleneck, especially in highly-parallelized distributed RL systems (Nair et al., 2015; Espeholt et al., 2018; 2019).
Every policy deployment requires a potentially expensive communication between different machines/processes, and this can be a bottleneck on the whole system.
As a concrete example of the necessity of good deployment efﬁciency, consider optimization of personalization in web apps or recommender systems (Abel et al., 2017).
Once a policy is learned on a batch of past experiences, it is deployed to a collection of web-hosting servers.
In this scenario, both safety and communication concerns are relevant: Safety of the new policy is typically ensured by initially deploying the policy to a small percentage of users; after monitoring the results for some length of time (e.g. the newly deployed policy does not deteriorate user experiences), one can expand the target user set.
As for communication, deploying a new policy to web-hosting servers can be time intensive, especially in large-scale web applications where the policy must be deployed to a network of servers around the world.
Thus, in this setting, it is clear that online updating of the policy is infeasible due to both safety and communication constraints.
Accordingly, the deployment-efﬁciency of any candidate RL algorithm is of tantamount importance.
The safe exploration might be mentioned as a potential alternative to deployment-efﬁciency.
While safe exploration can arguably tackle the ﬁrst concern above (safety risks of the policy), it does nothing to mitigate the latter (the engineering or communication costs associated with online deployment of a policy).
Furthermore, this still ignores the fact that in many scenarios the ability to do safe exploration is not a given.
While some safe RL algorithms can provide guarantees in tabular cases, these guarantees no longer hold when using function approximation with neural networks (Chow et al., 2018).
In these cases, it can be much more difﬁcult to perform “safe exploration” than it is to develop a deployment-efﬁcient algorithm.
D E VALUATING O FFLINE P ERFORMANCES ON D4RL D ATASETS
We compare BREMEN to MOPO (Yu et al., 2020), concurrently proposed model-based ofﬂine methods penalized by model epistemic uncertainty, and state-of-the-art model-free ofﬂine algorithms, namely, CQL (Kumar et al., 2020), BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019), AWR (Peng et al., 2019) and BCQ (Fujimoto et al., 2019), on the D4RL MuJoCo locomotion datasets (Fu et al., 2020), used as standard ofﬂine RL benchmarks (Kumar et al., 2020; Nair et al., 2020).
They have several types of ofﬂine data collected with different strategies.
We choose the hyper parameters of
BREMEN in Section 5.1 and F.2.2.
Table 2 shows BREMEN beats recent state-of-the-art algorithms with the highest normalized score (around 100 corresponds to an expert) in several tasks, while none of the methods consistently achieves the best performance.
This result suggests that the implicit regularization with the model-based method performs surprisingly well in ofﬂine settings despite of its simplicity.
[Table]
E I NCORPORATING P ESSIMISTIC M ODEL - BASED O FFLINE M ETHODS INTO BREMEN
The concurrent model-based ofﬂine RL methods prescribe the use of uncertainty-based penalties (Kidambi et al., 2020; Yu et al., 2020), which can be incorporated into BREMEN.
We therefore augmented BREMEN with either a hard (MOReL-like, green) or soft (MOPO-like, orange) reward penalty according to model uncertainty.
MOReL quantiﬁes the uncertainty measuring the maximum discrepancy of the prediction across the ensembles of the models and receives constant negative reward (-5.0 in our experiments) if the discrepancy is larger than the threshold (we set 3.0).
MOPO measures the uncertainty by the maximum standard deviation of the model ensembles and uses this as a reward penalty with a coefﬁcient (0.1 in our experiments).
Evaluations in Figure 6 reveal that the soft reward penalty has notable results in Hopper and Walker2d, where model uncertainty is more crucial due to the episode’s termination.
Hard reward penalty seems overly pessimistic in deployment-efﬁcient settings.
[Figure]
F D ETAILS OF E XPERIMENTAL S ETTINGS F. I MPLEMENTATION D ETAILS
For our baseline methods, we use the open-source implementations of SAC, BC, BCQ, and BRAC published in Wu et al.
SAC and BRAC have (300, 300) Q-Network and (200, 200) policy network.
BC has (200, 200) policy network, and BCQ has (300, 300) Q-Network, (300, 300) policy network, and (750, 750) conditional VAE.
As for online ME-TRPO, we utilize the codebase of model-based RL benchmark (Wang et al., 2019).
BREMEN and online ME-TRPO use the policy consisting of two hidden layers with 200 units.
The dynamics model also consists of two hidden layers with 1,024 units.
We use Adam (Kingma & Ba, 2014) as the optimizer with the learning rate of 0.001 for the dynamics model, and 0.0005 for behavior cloning in BREMEN.
Especially in
BREMEN and online ME-TRPO, we adopt a linear feature value function to stabilize the training.
BREMEN in deployment-efﬁcient settings takes about two or three hours per deployment on an
NVIDIA TITAN V.
To leverage neural networks as Dyna-style (Sutton, 1991) dynamics models, we modify reward and termination function so that they are not dependent on the internal physics engine for calculation, following model-based benchmark codebase (Wang et al., 2019); see Table 3.
Note that the score of baselines (e.g., BCQ, BRAC) is slightly different from Wu et al.
due to this modiﬁcation of the reward function.
We re-run each algorithm in our environments and got appropriate convergence.
The maximum length of one episode is 1,000 steps without any termination in Ant and HalfCheetah; however, termination function is enabled in Hopper and Walker2d.
The batch size of transitions for policy update is 50,000 in BREMEN and ME-TRPO, following Kurutach et al.
The batch size of BC and BRAC is 256, and BCQ is 100, also following Wu et al.
[Figure]
[Table]
F. H YPER P ARAMETERS
In this section, we describe the hyper-parameters in both deployment-efﬁcient RL (Section F.2.1) and ofﬂine RL (Section F.2.2) settings.
We run all of our experiments with ﬁve random seed, and the results are averaged.
F.2. D EPLOYMENT -E FFICIENT RL
Table 4 shows the hyper-parameters of BREMEN.
The rollout length is searched from {250, 500, 1000}
[Table]
and max step size δ is searched from {0.001, 0.01, 0.05, 0.1, 1.0}.
As for the discount factor γ and GAE λ , we follow Wang et al.
[Table]
Number of Iterations for Policy Optimization To achieve high deployment efﬁciency, the number of iterations for policy optimization between deployments is one of the important hyper-parameters for fast convergence.
In the existing methods (BCQ, BRAC, SAC), we search over three values: {10,000, 50,000, 100,000}
[Table]
and choose 10,000 in BCQ and BRAC, and 100,000 in SAC (Figure 8).
For BREMEN, we also search over three values: {2,000, 4,000, 6,000}.
Figure 9 shows the results of iteration search, and we choose 2,000 in Ant, HalfCheetah, and Walker2d, and 6,000 in Hopper
[Figure]
[Figure]
Stationary Noise in BREMEN To achieve effective exploration, the stochastic Gaussian policy is a good choice.
We found that adding stationary Gaussian noise to the policy in the imaginary trajectories and data collection led to the notable improvement.
Stationary Gaussian policy is written as,
[Equation]
Another choice is a learned Gaussian policy, which parameterizes not only µ θ but also σ θ .
Learned gaussian policy is also written as,
[Equation]
We utilize the zero-mean Gaussian N (0 , σ 2 ) , and tune up σ in Figure 10 with HalfCheetah, comparing stationary and learned strategies.
From this experiment, we found that the stationary noise, the scale of 0.1, consistently performs well, and therefore we used it for all our experiments.
[Figure]
Other Hyper-parameters in the Existing Methods As for online ME-TRPO, we collect 3, steps through online interaction with the environment per 25 iterations and split these transitions into a 2-to-1 ratio of training and validation dataset for learning dynamics models.
In batch size 100, settings, we collect 2,000 steps and split with 1-to-1 ratio.
Totally, we iterate 12,500 times policy optimization, which is equivalent to 500 deployments of the policy.
Note that we carefully tune up the hyper-parameters of online ME-TRPO, and the performance is improved from Wang et al.
Table 5 and Table 6 shows the tunable hyper-parameters of BCQ and BRAC, respectively.
We refer Wu et al.
to choose these values.
In this work, BRAC applies a primal form of KL value penalty, and BRAC (max Q) means sampling multiple actions and taking the maximum according to the learned Q function.
[Table]
[Table]
F.2. O FFLINE RL
In the ofﬂine experiments, we apply the same hyper-parameters as in the deployment-efﬁcient settings described above, except for the iteration per batch.
Algorithm 2 is pseudocode for BREMEN in ofﬂine RL settings where policies are updated only with one ﬁxed batch dataset.
The number of iteration T is set to 6,250 in BREMEN, and 500,000 in BC, BCQ, and BRAC.
The datasets for 50k or 100k experiments are sliced from the beginning of the 1M batched datasets without shufﬂing, but we observed that the distribution of rewards in 50k or 100k is not different from 1M.
[Table]
G A DDITIONAL E XPERIMENTAL R ESULTS G. P ERFORMANCE ON THE D ATASET WITH D IFFERENT N OISE
Following Wu et al.
(2019) and Kidambi et al.
, we additionally compare BREMEN in ofﬂine settings to the other baselines (BC, BCQ, BRAC) with ﬁve datasets of different exploration noise.
Each dataset has also one million transitions.
• eps1 : 40 % of the dataset is collected by data-collection policy (partially trained SAC policy) π b , 40 % of the dataset is collected by epsilon greedy policy with (cid:15)
= 0 .
1 to take a random action, and 20 % of dataset is collected by an uniformly random policy.
• eps3 : Same as eps1, 40 % of the dataset is collected by π b , 40 % is collected by epsilon greedy policy with (cid:15) = 0 . 3 , and 20 % is collected by an uniformly random policy.
• gaussian1 : 40 % of the dataset is collected by data-collection policy π b , 40 % is collected by the policy with adding zero-mean Gaussian noise N (0 , 0 .
1 2 ) to each action sampled from π b , and 20 % is collected by an uniformly random policy.
• gaussian3 : 40 % of the dataset is collected by data-collection policy π b , 40 % is collected by the policy with zero-mean Gaussian noise N (0 , 0 .
3 2 ) , and 20 % is collected by an uniformly random policy.
• random :
All of the dataset is collected by an uniformly random policy.
Table 7 shows that BREMEN can also achieve performance competitive with state-of-the-art modelfree ofﬂine RL algorithm even with noisy datasets.
The training curves of each experiment are shown in Appendix G.
. G. C OMPARISON AMONG D IFFERENT N UMBER OF E NSEMBLES
To deal with the distribution shift during policy optimization, also known as model bias, we introduce the dynamics model ensembles.
We validate the performance of BREMEN with a different number of dynamics models K .
Figure 11 and Figure 12 show the performance of BREMEN with the different number of ensembles in deployment-efﬁcient and ofﬂine settings.
Ensembles with more dynamics models resulted in better performance due to the mitigation of distributional shift except for K = 10 , and then we choose K =
[Figure] [Figure]
[Table]
G. I MPLICIT KL C ONTROL IN O FFLINE S ETTINGS
Similar to Section 5.3, we present ofﬂine RL experiments to better understand the effect of implicit
KL regularization.
In contrast to the implicit KL regularization with Equation 4, the optimization of
BREMEN with explicit KL value penalty becomes
[Equation]
where A π θk ( s, a ) is the advantage of π θ k computed using imaginary rollouts with the learned dynamics model and δ is the maximum step size.
Note that BREMEN with explicit KL penalty does not utilize behavior cloning initialization.
We empirically conclude that the explicit constraint − αD KL ( π θ ( ·| s ) (cid:107) ˆ π β ( ·| s )) is unnecessary and just TRPO update with behavior-initialization as implicit regularization is sufﬁcient in BREMEN algorithm.
Figure 13 shows the KL divergence between learned policies and the last deployed policies (top row) and model errors measured by a mean squared error of predicted next state from the true state (second row).
We ﬁnd that behavior initialized policy with conservative KL trust-region updates well stuck to the last deployed policy during improvement without explicit KL penalty.
The policy initialized with behavior cloning also tended to suppress the increase of model error, which implies that behavior initialization alleviates the effect of the distribution shift.
In Walker2d, the model error of BREMEN is relatively large, which may relate to the poor performance with noisy datasets in Section G.1.
[Figure]
i v
[Figure] [Figure]
G. T RAINING C URVES FOR O FFLINE RL WITH D IFFERENT N OISES
In this section, we present training curves of our all experiments in ofﬂine settings.
Figure 14 shows the results in Section 5.1.
Figure 15, 16, 17,
[Figure]
and 19 also show the results in Section G.
[Figure] [Figure]
v
[Figure]
[Figure] [Figure] [Figure]
[Figure]
G. D EPLOYMENT -E FFICIENT RL E XPERIMENT WITH D IFFERENT R EWARD F UNCTION
In addition to the main results in Section 5.2 (Figure 2), we also evaluate BREMEN in deploymentefﬁcient setting with different reward function.
We modiﬁed HalfCheetah environment into the one similar to cheetah-run task in Deep Mind Control Suite.
The reward function is deﬁned as r t =
[Equation]
[Equation]
x
[Equation]
> 10) , and the termination is turned off.
Figure 20 shows the performance of BREMEN and existing methods.
BREMEN also shows better deployment efﬁciency than other existing ofﬂine methods and online ME-TRPO, except for SAC, which is the same trend as that of main results.
[Figure]
c h = ,
[Figure]
c u m u l a t i v e r e w a r d s
[Figure]