P ARETO P OLICY P OOL FOR M ODEL - BASED O FFLINE R EINFORCEMENT L EARNING
A BSTRACT
Online reinforcement learning (RL) can suffer from poor exploration, sparse reward, insufﬁcient data, and overhead caused by inefﬁcient interactions between an immature policy and a complicated environment.
Model-based ofﬂine RL instead trains an environment model using a dataset of pre-collected experiences so online
RL methods can learn in an ofﬂine manner by solely interacting with the model.
However, the uncertainty and accuracy of the environment model can drastically vary across different state-action pairs, so the RL agent may achieve a high model return but perform poorly in the true environment.
Unlike previous works that need to carefully tune the trade-off between the model return and uncertainty in a single objective, we study a bi-objective formulation for model-based ofﬂine RL that aims at producing a pool of diverse policies on the Pareto front performing different levels of trade-offs, which provides the ﬂexibility to select the best policy for each realistic environment from the pool.
Our method, “Pareto policy pool (P3)”, does not need to tune the trade-off weight but can produce policies allocated at different regions of the Pareto front.
For this purpose, we develop an efﬁcient algorithm that solves multiple bi-objective optimization problems with distinct constraints deﬁned by reference vectors targeting diverse regions of the Pareto front.
We theoretically prove that our algorithm can converge to the targeted regions.
In order to obtain more Pareto optimal policies without linearly increasing the cost, we leverage the achieved policies as initialization to ﬁnd more Pareto optimal policies in their neighborhoods.
On the D4RL benchmark for ofﬂine RL, P3 substantially outperforms several recent baseline methods over multiple tasks, especially when the quality of pre-collected experiences is low.
I NTRODUCTION
Ofﬂine reinforcement learning (ofﬂine RL) (Levine et al., 2020) or batch RL (Lange et al., 2012) can train an agent without interacting with the environment by instead using pre-collected experiences on other agents/policies.
Recently, ofﬂine RL has been attracting growing interest due to the availability of large-scale datasets of diverse experiences in many RL applications, e.g., autonomous driving (Shin & Kim, 2019), healthcare (Yu et al., 2019), robot control (Schulman et al., 2016), etc.
However,
RL algorithms developed for the online/interactive setting usually perform poorly in the ofﬂine setting (Fujimoto et al., 2019; Janner et al., 2019) due to the data distribution shift caused by (1) the difference between the policy-in-training and the behavior policies used to collect the data; and (2) the difference between the realistic environment in which we will deploy the policy and the environments used to collect the data.
These differences can result in function approximation error and biased policy learning (Levine et al., 2020).
To address these challenges, model-based RL approaches (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021; Yu et al., 2021) ﬁrstly learn an environment model from a dataset of logged experiences using supervised learning and then conduct online RL interacting with the model.
The learned environment model fully exploits the pre-collected experiences and can avoid/reduce the costly interactions with the realistic environment required by
RL, hence improving the sample efﬁciency.
That being said, due to the large size of state-action space, model-based ofﬂine RL approaches can still suffer from “model exploitation” (Levine et al., 2020): when the dataset does not contain sufﬁcient samples for some state-action pairs, the epistemic uncertainty of the model on these “outof-distribution” pairs can lead to a poor policy in ofﬂine RL.
For example, an RL agent may easily exploit the model by repeatedly visiting the out-of-distribution states where the model erroneously issues higher rewards than the true environment dynamics, and thus the RL objective is biased by overly optimistic evaluation of the agent’s performance.
A widely-studied strategy to tackle this problem is applying regularization to the reward function, which keeps the model uncertainty small when maximizing the reward issued by the model (Yu et al., 2020; Kidambi et al., 2020;
Rafailov et al., 2021).
By carefully tuning the regularization weight, it can achieve preferable trade-off between the model reward and model uncertainty.
However, it is usually challenging to tune the weight without access to the realistic environment that the learned policy will be deployed to.
Moreover, even with access to the realistic environment, hyperparameter tuning methods such as Bayesian optimization (Frazier, 2018) need to run multiple instances of ofﬂine RL and can be computationally expensive.
In addition, the ofﬂine-trained policy may not be directly generalized to different environments.
In this paper, we address a challenging problem, i.e., how to balance between the model reward and model uncertainty during ofﬂine RL so it can produce diverse policies that can be adapted to different realistic environments?
Instead of formulating the problem as a single-objective optimization with regularization as in previous works, a primary contribution of this paper is to treat the model reward/return and model uncertainty as two separate objectives and develop an efﬁcient bi-objective optimization method producing a pool of diverse policies on the Pareto front, which correspond to different trade-offs between the two objectives.
Therefore, when deployed to a new realistic environment, we can accordingly choose the best policy from the pool.
Our method is called “Pareto policy pool (P3),” and an example of the policies achieved by P3 is shown in Fig.
When deployed to the realistic environment, Pareto policy 1 is overly optimal on the model return, so it runs fast at the beginning but quickly falls to the ground due to the aforementioned “model exploitation”.
On the contrary,
Pareto policy 4 favoring small model uncertainty is overly conservative and keeps standing still for 1000 time-steps because it avoids taking exploratory actions that potentially increase the uncertainty.
As expected, Pareto policy 2&3 with the more balanced trade-off between the model return and uncertainty perform better and achieve higher returns in the test environment.
These results imply that model-based ofﬂine RL’s performance signiﬁcantly relies on the trade-off between model return and uncertainty in the optimization objectives.
However, it is usually challenging or intractable to determine the optimal trade-off before deployment and control it during training.
Moreover, for existing methods adopting a regularized single-objective, i.e., scalarization (Boyd & Vandenberghe, 2004, Chapter 4.7), even trying all possible regularization weights cannot fully recover the Pareto front and thus cannot guarantee to ﬁnd the optimal trade-off.
For example, in Fig. 1, by running multiple instances with different regularization weights, MOPO (Yu et al., 2020) and MOReL (Kidambi et al., 2020) can only generate a few separated solutions, and it is difﬁcult to ﬁnd one with advantageous trade-off among them.
In contrast, our method aims at efﬁciently generating a rich pool of diverse and representative policies covering the entire Pareto front without tuning the trade-off during training.
Thereby, when deployed to a new realistic environment, the best policy for the new environment can be efﬁciently selected from the pool.
Hence, P3 provides a simple and principal approach that addresses the two major challenges in model-based ofﬂine RL, i.e., “model exploitation” and generalization to different unseen states in order to achieve high returns.
Due to the complicated shape of the Pareto front that is unknown during training, ﬁnding a pool of diverse policies covering the whole Pareto front raises several non-trivial algorithmic challenges: (1) How to ﬁnd policies located at different regions of the Pareto front associated with different levels of model return-uncertainty trade-off?
(2) How to avoid training each policy from scratch and reduce the computational cost linearly increasing with the number of policies?
Inspired by recent works in multi-objective optimization (Cheng et al., 2016; Ma et al., 2020; Xu et al., 2020), we explore different regions of the Pareto front by generating multiple diverse reference vectors in the bi-objective space, each deﬁning a constrained bi-objective optimization whose solution resides in a local region of the Pareto front.
By solving these constrained bi-objective optimization problems, we can obtain a diverse set of policies covering the whole Pareto front.
For solving each problem, we extend MGDA algorithm (D ´ esid ´ eri, 2012) to be a two-stage gradient-based method that provably converges to the Pareto front region targeted by the reference vector.
In order to achieve more policies on the Pareto front without linearly increasing the cost, we start from the previously obtained policies for initialization and explore their neighborhoods on the Pareto front by perturbing their corresponding reference vectors, resulting in a dense set of Pareto policies in each local region.
In experiments, we evaluate P3 and compare it with several state-of-the-art model-based/free ofﬂine
RL methods on the standard D4RL Gym benchmark (Fu et al., 2020).
P3 achieves the highest average-score over all datasets and signiﬁcantly outperforms the baseline methods in 5 out of thelow/medium-quality datasets, showing the advantages of P3 on learning from non-expert experiences.
We also present a thorough ablation study to identify the most important components in P3.
R ELATED W ORK
Due to lack of space, we focus our discussion here on directly related works and present a more detailed overview of related work in Appendix A.2.
To address the aforementioned “model exploitation”, recent works rely on applying uncertainty regularization to the model return (Yu et al., 2020;
Kidambi et al., 2020; Rafailov et al., 2021) which can be difﬁcult and costly to tune the weight of regularization.
In contrast, our work reframes the policy learning under the environment model as a bi-objective optimization problem and produces a diverse set of policies on the Pareto front performing different levels of model return-uncertainty trade-off, which provides ﬂexibility to select the best policy in the inference stage.
P RELIMINARIES
We consider an episodic Markov Decision Process (MDP) M = {S , A , P, r, H, ρ 0 } , where S is the state space, A is the space of actions, P is the transition probability: S ×
A → ∆( S ) where ∆( S ) is a probability distribution over S , r : S ×
A → R is the reward function so r
( s h , a h ) is the immediate reward for taking action a h at state s h , H is the horizon of the process, and ρ 0 is the distribution of the initial state s 0 .
RL aims at learning a policy π : S
→ A maximizing the expected return in
Eq.
(1), where π in this paper takes a deterministic action a h based on the state s h .
[Equation]
In model-based ofﬂine RL, the agent instead interacts with an environment model rather than the realistic one.
We train an environment model (cid:99) M = { ˆ S , A , ˆ P, ˆ r, H, ˆ ρ 0 } using a pre-collected dataset
D (cid:44) { ( s h , a h , s h
+1 , r h ) | π b } of experiences by behavior policies, hand-designed controllers, or human demonstrators.
By interacting with the model, online RL methods can learn in an ofﬂine manner.
However, when D does not contain sufﬁcient samples for some state-action pairs, the epistemic uncertainty of the model on these “out-of-distribution (OOD)” pairs can result in a poor policy.
For example, an RL agent may easily exploit the model by repeatedly visiting the OOD states where the model erroneously issues ˆ r higher than the true environment reward r and thus the RL objective is biased by overly optimistic evaluation of the agent’s performance.
A widely-studied strategy to tackle this problem is applying regularization to ˆ r , which keeps the model uncertainty small while maximizing the model reward (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021).
A practical implementation of the regularized reward function is developed by (Yu et al., 2020):
˜ r h
=
ˆ r h − λu (ˆ s h , a h ) , where u (ˆ s h , a h ) denotes the estimation of the model uncertainty at the state-action pair (ˆ s h , a h ) and λ controls the trade-off between ˆ r and u , which has to be carefully tuned in practice.
Based on the regularized reward function, Yu et al.
(2020) proposed a modiﬁed policy optimization objective:
[Equation]
Despite being intuitive, this method’s performance is sensitive to the regularization weight
λ (Kidambi et al., 2020; Yu et al., 2021), and tuning λ is usually challenging without access to the realistic environment that the learned policy will be deployed to.
Moreover, even granted the access, hyperparameter tuning methods such as Bayesian optimization (Frazier, 2018) require running many instances of ofﬂine RL, which can be computationally prohibitive.
Instead of optimizing a single regularized objective, our “Pareto policy pool (P3)” proposed later treats ˆ r and u as two separate objectives and develop an efﬁcient bi-objective optimization method that does not need to tune the trade-off deliberately but produces a pool of diverse policies, which are learned under different trade-offs between the two objectives.
Thereby, when deployed to a realistic environment, the best policy can be chosen from the pool.
P ARETO P OLICY P OOL FOR M ODEL - BASED O FFLINE RL 4. P ROBLEM F ORMULATION
In order to estimate the model uncertainty accurately and alleviate the model exploitation problem, we follow previous works (Janner et al., 2019; Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021;
Yu et al., 2021) and construct a bootstrap ensemble of K environment models { (cid:99)
M i } Ki
=1 .
Each model (cid:99) M
i
=
{ ˆ S , A ,
ˆ P i , ˆ r i , H, ˆ ρ 0 } is a two-head feed-forward neural network that takes a state-action pair (ˆ s h , a h ) as input and outputs the mean µ i and standard deviation σ i of [ˆ s h
+1 , ˆ r h ] , i.e., the next state concatenated with the reward.
More details about our model are given in Appendix A.5.
As demonstrated in Yu et al.
(2020), this ensemble is effective in estimating the model uncertainty as the maximal standard deviation over all models, i.e., u (ˆ s h , a h ) = max i ∈
[ K ] (cid:107) σ i (ˆ s h , a h ) (cid:107) 2 .
Moreover, by randomly selecting a model in each step to provide the reward ˆ r (ˆ s h , a h ) =
ˆ r h , we can effectively mitigate the model exploitation problem.
Unlike previous works combining ˆ r and u as a single objective, we treat them as two separate objectives and aims at solving the bi-objective optimization below.
[Equation]
where κ is a temperature applied to u (ˆ s h , a h ) .
For simplicity, in the rest of this paper, we remove ˆ ρand (cid:99) M .
In Eq.
(3), J ˆ r aims to maximize the expected model return, and J u is designed to minimize the expected cumulative model uncertainty.
However, the bi-objective optimization naturally has multiple (up to inﬁnite) optimal solutions instead of one single policy and each optimal policy performs a different level of trade-off between the two objectives.
For example, a policy favoring small model uncertainty maybe overly conservative and avoids taking exploratory actions so its model return can be low.
In contrast, a policy pursuing high model return might fail in realistic environments at a state that the model is highly uncertain about.
Examples of these policies are given in Fig.
Formally, for any two policies π i and π j , π i dominates π j if and only if J ( π i ) ≥ J ( π j ) and J ( π i ) (cid:54) =
J ( π j ) .
A policy π ∗ is Pareto optimal if no any policy dominates π ∗ in R d , i.e., no objective can be improved without detriment to another objective at π ∗ .
All Pareto optimal policies constitute the Pareto set, and the
Pareto front is the image of the Pareto set in the space spanned by all the objectives.
Unfortunately, it is almost infeasible in practice to ﬁnd the whole Pareto set and the Pareto front’s shape is also unknown.
As discussed in Sec. 3, it is difﬁcult to determine the optimal trade-off between J ˆ r and J u because we cannot access the realistic environment during training.
It is also challenging and costly to control the trade-off.
Hence, a straightforward strategy is to ﬁnd a diverse set of policies on the Pareto front performing different levels of trade-off and select the best one when deployed to a realistic environment.
However, how to ﬁnd these Pareto optimal policies is still an open challenge.
In addition, it is expensive to train each Pareto policy from scratch.
After allocating a few diverse policies on the Pareto front, can we start from them to ﬁnd more Pareto policies so we can avoid linearly increasing the computational costs?
To overcome these challenges, we develop “Pareto policy pool (P3)”, which can efﬁciently and precisely ﬁnd a diverse set of policies on the Pareto front of return-uncertainty bi-objective optimization.
[Figure]
Fig.
2 illustrates the main idea of “Pareto policy pool (P3)”, whose detailed procedure are given in Alg.
In order to ﬁnd diverse Pareto policies and precisely control the trade-off of each policy, P3 generates multiple reference vectors { v i } ni =1 in the objective space, each forming a constraint to the bi-objective optimization in Eq.
(3) and targeting a different region on the Pareto front (Sec. 4.2.1).
Thereby, the
Pareto policies { π i } ni =1 achieved by solving the n constrained bi-objective optimizations are diverse in terms of the objective trade-off.
For solving each of them, we develop an efﬁcient two-stage optimization algorithm, i.e., Alg. 2, which starts from an initial point and moves to a region fulﬁlling the constraint (correction stage), and then apply a multi-objective optimization algorithm to ﬁnd a Pareto policy in the targeted region.
To avoid training each Pareto policy from scratch, which leads to linearly increasing costs for ≥ n policies, we develop a local extension method on the Pareto front in Sec.
This “Pareto Extension” ﬁrstly generates more reference vectors by perturbing each vector in { v i } ni =
and then uses their associated Pareto policies { π i } ni =1 to warm start the bi-objective optimization constrained by the new vectors (Line 9-13 in Alg. 1).
These locally extended policies, together with the n policies, compose a pool of diverse policies on the Pareto front, from which we can select the best policy when deploying them to realistic environments.
. D IVERSE P ARETO P OLICIES BY R EFERENCE V ECTORS
Inspired by recent works in multi-objective optimization (Cheng et al., 2016; Lin et al., 2019; Ma et al., 2020; Xu et al., 2020), we explore the Pareto front by generating a diverse set of reference vectors deﬁning multiple constrained bi-objective optimization problems.
As shown in Fig.
2, we generate n uniformly distributed reference vectors { v i } ni =1 in a 0-1 normalized objective space by Eq.
[Equation] i.e., [Equation]
where τ a and τ b control the range covered by reference vectors.
Each v i =
( v 1 i , v 2 i ) deﬁnes a constrained bi-objective optimization problem based on Eq.
(3) whose solution resides in the targeted region of the Pareto front:
[Equation]
where Ψ( π θ , v i ) deﬁnes a similarity metric between the reference vector v i and the objective vector
J ( π θ ) > 0 .
When ψ is large, J ( π θ ) is constrained to be close to v i , and the targeted region on the
Pareto front is small.
Hence, solving the constrained bi-objective optimization for the diverse set of reference vectors produces a diverse set of Pareto policies associated with different trade-offs between J ˆ r and J u .
In the following, we develop an efﬁcient two-stage algorithm for solving this optimization problem in Eq.
1: input: π θ t , v i , ψ 2: if Ψ( π θ t , v i ) <
ψ then (cid:46) Correction stage 3: if J ˆ r ( π θ t ) /J u
( π θ t ) <
v 1 i /v
i then 4:
Compute ∇ θ J ˆ r ( π θ t ) ; 5: θ t +1 = θ t + η ∇ θ J ˆ r ( π θ t ) ; 6: else 7:
Compute ∇ θ J u
( π θ t ) ; 8: θ t +1 =
θ t + η ∇ θ J u
( π θ t ) ; 9: else (cid:46)
Ascending stage 10:
Compute ∇ θ F ( π θ t ) ; 11:
Find α ∗ t to Eq.
(7); 12: θ t +1 =
θ t + η α ∗ t ∇ θ F ( π θ t ) ; 13: t ← t +14: output: π θ t
To solve the bi-objective optimization with an inequality constraint in Eq.
(6), we propose a two-stage gradient-based method in Alg.
It ﬁrst ﬁnds a solution meeting the constraint by alternating between optimizing the two objectives (correction stage) and then treats the constraint as the third objective and applies an existing multi-objective optimization algorithm to ﬁnd a Pareto policy in the region targeted by the constraint (ascending stage).
In the ﬁrst stage, the algorithm checks how the constraint is violated (line 3), e.g., whether J ˆ r or J u is too small, and then accordingly chooses one to apply gradient ascent towards meeting the constraint.
Once it ﬁnds a feasible solution fulﬁlling the constraint, it switches to the second stage, which reframes the problem as a triobjective optimization by turning the constraint to be the third objective, i.e., max θ F ( π θ ) (cid:44) ( J ˆ r ( π θ ) , J u
( π θ ) , Ψ( π θ , v i ))
T , and applies MGDA (D ´ esid ´ eri, 2012) to ﬁnd a Pareto solution for this problem.
Each step of MGDA aims at ﬁnding a convex combination α t ∇ θ F ( π θ t ) of all objectives’ gradients ∇ θ F ( π θ t ) (cid:44) ( ∇ θ J ˆ r ( π θ t ) , ∇ θ J u
( π θ t )
, ∇ θ Ψ( π θ t , v i ))
T such that no objective is decreasing on the combined direction.
This is equal to solving the following min-norm problem.
[Equation]
which can be efﬁciently solved by Frank-Wolfe’s algorithm
(Jaggi, 2013).
Then MGDA takes one step along the direction to update the policy θ t , i.e., θ t +
=
θ t + η α t ∇ θ F ( π θ t ) .
Since MGDA always improves each objective, the constraint will not be violated, and the algorithm will ﬁnally ﬁnd a Pareto policy in the targeted region.
In practice, we use OpenAI’s ES (Salimans et al., 2017) to estimate the gradient of each objective (see Appendix A.3) for its efﬁciency and stability.
Alg.
2 provably converges to the Pareto front region targeted by the reference vector with approximate gradients.
Assumption 1.
Suppose m objectives { f i } mi =1 of a multi-objective function are differentiable and that their gradients are Lipschitz continuous with constant L i > 0 .
Assumption 2.
For the ES gradient ∇ f i,ν ( x t ) , we have E ε t ∼N (0 ,I ) [ ∇ f i,ν ( x t )]
= ∇ f i ( x t ) .
Suppose
Var( ∇ f i,ν ( x t )) ≤ σ 2 .
Lemma 1.
∀ mutually independent objectives f i ≥ 0 , which satisﬁes Assumption 1 & 2, has
E ε t [ f i ( x t +1 )] − f i ( x t ) ≤ − ( η − L i η 2 2 ) (cid:107) ¯ d t (cid:107) 2 + L i η 2 2 σ 2 where ¯ d t = (cid:80) mi =1 α i,t ∇ f i ( x t ) .
Lemma 1 implies that when η <
[Equation]
L i , the ascending stage leads to a monotonically nonincreasing sequence of the objectives.
Theorem 1 (Non-convex convergence rate) .
Let Assumption 1 & 2 hold, ∆ =
f i ( x 0 ) − f i ( x ∗ ) , β = η − L i η 2 2 , and γ =
L i η 2 2 .
For an arbitrary objective f i , given any (cid:15) > 0 , after T
=
O ( ∆ β(cid:15)
− γσ 2 ) iterations of ascending stage, we have 1 T (cid:80) T − 1 t =
E x t [ (cid:107) ¯ d t (cid:107) 2 ] ≤ (cid:15) .
Complete proofs are provided in Appendix A.1.
Theorem 1 provides the convergence rate of the ascending stage for non-convex objectives.
For the correction stage that performs a single-objective
ES gradient ascent, Nesterov & Spokoiny (2017) have proved its convergence rate to a stationary point of the non-convex objective.
4.2. L OCAL E XTENSION OF P ARETO F RONT
Although solving Eq.
6 for a diverse set of reference vectors using the algorithm in Sec.
4.2.1 can produce a diverse set of Pareto policies, the computational cost linearly increases with the number of policies.
In practice, a few Pareto policies cannot cover all possible trade-offs and thus may lead to sub-optimal choices of policy in deployment.
In order to efﬁciently obtain more policies with different ﬁne-grained levels of model return-uncertainty trade-off, we propose a “Pareto Extension” that starts from the diverse set of Pareto policies and locally searches for more policies near them on the Pareto front.
This warm-start strategy avoids training more policies from scratch and practically saves a signiﬁcant amount of computation.
As illustrated in Fig.
2 and Alg.
1, we perturb each reference vector v i for more reference vectors v
+ i and v − i , i.e., v
+ i = v i + (cid:15) , v − i
=
v i − (cid:15) , (cid:15) (cid:44) ( (cid:15), − (cid:15) ) , 0 < (cid:15) <
τ a .
These new vectors create more constrained bi-objective optimization problems in the same form as Eq.
Instead of using a random initialization, we start from and ﬁne-tune { π i } ni =1 for a few iterations to achieve the Pareto policies of these new optimization problems.
These policies and { π i } ni =1 together constitute a pool of Pareto policies.
When deployed to a realistic environment, we ﬁrst evaluate these policies for a few steps and then select the one achieving the highest return to deploy.
More details about our selection strategy are given in Appendix A.4.
As long as we include sufﬁcient diverse policies in the pool, we can ﬁnd a promising policy that signiﬁcantly outperforms the policies trained by other model-based ofﬂine RL methods.
E XPERIMENTS
This section aims to answer the following questions by evaluating P3 with other ofﬂine RL methods on the datasets from the D4RL Gym benchmark (Fu et al., 2020).
(1) Comparison to prior work :
Does P3 outperform other state-of-the-art ofﬂine RL methods?
Moreover, when the dataset does not contain high-quality samples for some state-action pairs, can the policies generated by P3 generalize to unseen states in order to achieve high returns?
(2) Ablation study :
In the experiments, we apply several techniques used in previous work (see Appendix A.6 for more details).
How do they affect performance?
(3) Effectiveness of P3 : Why is it challenging to ﬁnd the optimal trade-off between the model return and its uncertainty?
We empirically explain how P3 efﬁciently alleviate this problem by generating a rich pool of diverse and representative policies.
To answer question (1) , we compare P3’s performance with the state-of-the-art ofﬂine RL algorithms, including BCQ (Fujimoto et al., 2019), BEAR (Kumar et al., 2019), CQL (Kumar et al., 2020),
UWAC (Wu et al., 2021), TD3+BC (Fujimoto & Gu, 2021), MOPO (Yu et al., 2020), MOReL (Kidambi et al., 2020), and COMBO (Yu et al., 2021).
For fairness of comparison, we re-run these algorithms using author-provided implementations 1 and train each algorithm for 1000 epochs.
We also carefully tune the hyperparameters of baselines such as BCQ, CQL, TD3+BC, MOPO, and
MOReL by grid search and choose the best ones for each benchmark dataset.
Most of them achieve higher scores than those previous versions.
For other baselines, such as UWAC and COMBO, we adopt the hyperparameters in their original papers, assuming they have chosen the best hyperparameters they can ﬁnd.
More details on the experimental setting can be found in Appendix A.6.
Our experiment results are provided in Table 1.
It is obvious that P3 achieves the highest average-score across all datasets, and signiﬁcantly outperforms the baseline methods in 5 out of the 9 low/mediumquality datasets.
Moreover, we ﬁnd that the low/medium-quality datasets in the D4RL benchmark are “imbalanced”.
As illustrated in Fig.
8, there are a large number of bad samples, some mediocre
[Table]
[Figure] samples, and a few good samples, causing problems with learning accurate behavior policies or generalizable environment models (Buckman et al., 2021; Zhang et al., 2021).
Therefore, many ofﬂine RL algorithms, especially model-free algorithms that heavily rely on the accurate recovery of behavior policy (Fujimoto et al., 2019; Kumar et al., 2020; Fujimoto & Gu, 2021), perform poorly on these datasets.
According to the results in Table 1, P3 achieves SoTA performance and outperforms other baselines by a large margin on random, medium, and medium-replay datasets, indicating the advantages of P3 on learning from low-quality experiences.
We use online policy evaluation to select the best policy during the test phase of P3, which can be computationally intensive or cause overheads when deployed to realistic environments.
To overcome this drawback, we replace the online evaluation with Fitted Q Evaluation (FQE)
(Le et al., 2019), an ofﬂine policy evaluation method, which (approximately) evaluates policies using the ofﬂine data only.
The implementation details and experimental results are reported in Appendix A.4 and Table 1, respectively.
We surprisingly ﬁnd that “P3+FQE (ofﬂine policy evaluation)” only slightly degrades from “P3+online policy evaluation” on the performance, but still outperforms all baselines on the low/medium datasets, suggesting that FQE enables more efﬁcient inference for P
so it has exactly the same inference cost as other baselines.
To answer question (2) , we conduct a thorough ablation study toward ﬁve variants of P3, each removing/changing one component used in P3.
Table 2 reports their results on the D4RL Gym benchmark with 9 representative environment-dataset combinations.
In Fig.
9 in Appendix, we visualize the Pareto policies obtained by P3 to highlight the effectiveness and superiority of our method.
Among the ﬁve variants of P3, “scalarization” replaces our porposed Alg. 2 with the scalarization method (Boyd & Vandenberghe, 2004, Chapter 4.7); “no StateNorm” removes the state normalization (Mania et al., 2018; Fujimoto & Gu, 2021); “no RankShaping” removes the rank-based scale shaping (Wierstra et al., 2014); “no ParetoExtension” removes the Pareto extension proposed in
Sec.
4.2.2; “no BehaviorCloning” removes the behavior cloning initialization (Kumar et al., 2020;
Kidambi et al., 2020).
More details on these variants are provided in Appendix A.9.
According to the results in Table 2 and Fig.
9, we give the following conclusions: (1) except “scalarization” and “no ParetoExtension”, other variants perform comparably to our P3 while outperforming the previous results achieved by model-based (MOPO) and model-free (TD3+BC) RL algorithms on
[Table]
98.9 ± 3.
MOPO 35.9 ± 2.
[Table]
4.2 ± 5.7 69.2 ± 1.
[Table]
32.7 ± 9.
73.7 ± 9.
70.3 ± 21.
60.6 ± 32.
77.4 ± 27.9 TD [Table] 10.6 ± 1.
8.6 ±
[Table] [Table] [Table] [Table]
[Table] ± [Table]
[Table]
[Table] [Table] [Table]
the low/medium-quality datasets, reﬂecting that these widely-used techniques can improve P3’s performance but are not crucial to our appealing results.
(2) “scalarization” shows noticeable degradation in performance and cannot obtain a dense set of diverse policies, as shown in Fig.
The results can be explained as follows: the scalarization method only ﬁnds a few separated policies, and it is difﬁcult to ﬁnd one with advantageous trade-off among them.
In addition, we remark that the computational cost of multiple training with different weight assignments is similar to the cost of running P3.
(3) “no ParetoExtension” degrades P3’s performance on all 9 environment-dataset combinations, corroborating that a dense set of policies on the Pareto front is essential to our results.
To answer question (3) , in Fig.
4, we study how the P3 policies with different uncertainty-return trade-off perform in the deployed environment.
For low/medium-quality datasets (the left three plots in Fig. 4), the optimal policies with high realistic returns (bright areas in the heatmap) spread across almost the whole Pareto front.
Therefore, to ﬁnd the best policy, it is essential to explore the whole
Pareto front and select one from a diverse set of Pareto optimal/stationary policies as P3 does.
This explains why P3 performs the best on all the low/medium datasets.
On the contrary, for high-quality datasets (the right two plots in Fig. 4), the optimal policies with high realistic returns gather within a small region of the Pareto front and associate to one trade-off level.
Therefore, by carefully tuning the trade-off weight, previous methods can still ﬁnd the optimal policy without visiting the whole
Pareto front.
Hence, we observe less advantages of P3 on the high-quality datasets.
The reason behind is that the MDP models are very conﬁdent on high-return (realistic) state-action pairs if most samples in the training data are with high-return (high-quality), while they can be uncertain about many high-return pairs if the training data only cover a few high-return samples (low/medium quality).
It is worth noting that collecting high-quality datasets is usually expensive or infeasible in practice and many applications lack sufﬁcient high-quality data.
In these imperfect but practical scenarios, P3 performs signiﬁcantly better and more stably than existing model-based ofﬂine RL methods that only learns one single policy.
C ONCLUSION
In this paper, we ﬁnd that model-based ofﬂine RL’s performance signiﬁcantly relies on the trade-off between model return and its uncertainty, while determining the optimal trade-off is challenging without access to the realistic environment.
To address the problem, we study a bi-objective formulation for model-based ofﬂine RL and develop an efﬁcient method that produces a pool of diverse policies on the Pareto front performing different levels of trade-offs, which provides ﬂexibility to select the best policy in the inference stage.
We extensively validate the efﬁcacy of our method on the D4RL benchmark, where ours largely outperforms several recent baselines and exhibits promising results on low-quality datasets. A CKNOWLEDGMENTS
Yijun Yang and Yuhui Shi are supported in part by the Shenzhen Fundamental Research Program under
Grant No.
JCYJ20200109141235597, the National Science Foundation of China under Grant No. 61761136008, the Shenzhen Peacock Plan under Grant No. KQTD2016112514355531, the Program for Guangdong Introducing Innovative and Entrepreneurial Teams under Grant No. 2017ZT07X386. R EPRODUCIBILITY S TATEMENTS
Code is available at https://github.com/OverEuro/P3 .
We provide a full description of all our experiments in Section 5 and Appendix A.
A A PPENDIX A. C ONVERGENCE A NALYSIS OF A LG .
We provide a proof for Lemma
[Equation]
Proof.
For α t obtained by solving min α t (cid:107) (cid:80) mi =1 α i,t ∇ f i,ν ( x t ) (cid:107) 2 , s.t.
[Equation]
α it
[Equation]
1 , α it ≥ 0 , let
[Equation]
According to assumption 1:
[Equation]
Take expectation on both sides:
[Equation]
Hence, we have E ε t
[ f i ( x t +1 )]
− f i ( x t ) ≤ − ( η − L i η 2 2 ) (cid:107) ¯ d t
[Equation]
+ L i
[Equation]
σ 2 .
Let the right term:
[Equation]
We have:
[Equation]
When η t <
[Equation]
η
[Equation]
d
[Equation]
2 + L
i [Equation] 2 t 2 σ 2 < 0 , then E ε t [ f i ( x t +1 )] − f i ( x t ) ≤ 0 .
Hence, the ascending stage leads to a monotonically non-increase sequence, convergence proved.
We provide a proof for Theorem 1.
Proof.
According to Lemma 1, we have:
[Equation]
Take expectation with respect to the solution x t and do telescoping:
[Equation]
Note that E x T ,ε T [ f i ( x T +1 )]
=
E x T +
[ f i ( x T +1 )] , hence
[Equation]
According to Eq.
16, we can obtain
[Equation]
[Equation]
Let
[Equation]
Rearranging terms, we have
[Equation]
By the condition that β(cid:15) − γσ 2 > 0 , we have the following:
[Equation]
Let η < (cid:15) (cid:15) + σ 2 2 L i , given (cid:15) > 0 , after T
= O (
[Equation]
A. D ETAILED R ELATED W ORK
Model-based Ofﬂine RL.
Model-based RL methods are promising candidates for sequential decisionmaking problems due to their high sample-efﬁciency and compact modeling of a dynamic environment.
Time-dependent linear models and Gaussian processes provide excellent performance in the low-data and low-dimensional scenario.
High-capacity models, e.g., deep neural networks, are more successful because they beneﬁt from powerful supervised learning techniques that allow the leverage of large-scale datasets, even in domains with high-dimensional image observations.
Although the convenience of working with large-scale datasets, these methods still suffer from the effects of distribution shift and error accumulation of the model predictions (Levine et al., 2020), especially in the ofﬂine setting.
Existing works explore several classical methods to solve the problem.
Dyna-style algorithms (Sutton, 1991) utilize a technique named branched rollout to collect short-horizon model rollouts, which alleviates accumulated extrapolation errors.
Another method introduces ensemble learning (Chua et al., 2018), i.e., the leverage of multiple dynamics models, against model errors.
Other works also incorporate novel concepts into model-based RL, e.g., an energy-based model regularization (Boney et al., 2019), a game-theoretic framework for model-based RL (Rajeswaran et al., 2020), meta-learning (Clavera et al., 2018), policy regularization (Berkenkamp et al., 2017), and generative temporal difference learning (Janner et al., 2020).
In the ofﬂine setting, since the learned model cannot be corrected with additional data collection and training, it is crucial to prevent the policy from visiting “out-of-distribution” states.
Recent two works propose uncertainty-regularized policy optimization algorithms for this purpose (Yu et al., 2020; Kidambi et al., 2020).
One adds a soft uncertainty penalty associated with a user-chosen weight to the reward issued by the model, and the other constructs a pessimistic MDP model associated with a hard threshold for terminating the interaction between the policy and the model when the model prediction is inaccurate.
(Matsushima et al., 2021) presents a deployment-efﬁcient RL framework to avoid the ofﬂine distribution shift issue.
(Rafailov et al., 2021) extends an existing model-based ofﬂine RL method to image-based tasks.
More recently, conservative ofﬂine model-based policy optimization (COMBO)
(Yu et al., 2021) regularizes the value function on out-of-distribution states yielded via interacting with the learned model, which leads to a conservative estimate of value function for these state-action pairs, without requiring explicit uncertainty penalty.
Moreover, a recent method named MuZero Unplugged focuses on using the model-based planning directly for policy improvement (Schrittwieser et al., 2021).
Multi-objective Optimization.
Multi-objective optimization (MOO), also known as Pareto optimization, aims to optimize more than one objective function.
For a nontrivial MOO problem, no single solution exists that optimizes all objectives simultaneously.
In this case, these objective functions are said to be conﬂicting, and we can ﬁnd a set of Pareto optimal solutions given different trade-offs among objectives.
Existing methods in MOO can be divided into two categories: gradient-based methods and heuristic methods.
Multiple gradient descent algorithm (MGDA) (D´esid´eri, 2012) is an extension of the classical gradient descent algorithm to multiple objectives.
It uses an adaptive weighting aggregation of sub-objective gradients to compute the descent direction for all objectives.
Because of the ability of fast convergence with theoretical guarantee, MGDA-based methods have been widely used to solve multi-task learning (Sener & Koltun, 2018; Lin et al., 2019) and multiobjective recommendation (Milojkovic et al., 2020).
However, MGDA-based methods generate only one solution or a ﬁnite set of sparse solutions with different trade-offs.
In contrast, heuristic methods usually ﬁnd a Pareto set by various evolutionary algorithms such as the genetic algorithm (NSGA-II) (Deb et al., 2002) and the decomposition-base evolution algorithm (MOEA/D) (Zhang & Li, 2007).
They can ﬁnd a dense set of Pareto near-optimal solutions by only one time running.
But in practice, these black-box optimization methods usually take unacceptable run time when the parameter space is extremely large, e.g., the number of parameters for a deep neural network, or when the function evaluation is computationally expensive, e.g., population-based training for RL agents.
In this paper, our approach borrows the ideas from the heuristic methods to achieve a dense set of policies.
Moreover, we replace inefﬁcient evolutionary operators with a gradient-based method, reducing the training cost signiﬁcantly.
Model-free Ofﬂine RL.
Model-free RL directly learn a policy without requiring the learning of an environment model.
A direct extension of model-free RL methods to ofﬂine setting (Fujimoto et al., 2019) usually perform poorly due to the data distribution shift.
To address this issue, prior model-free ofﬂine RL methods regularize the learned policy to be “close” to the behavior policy either implicitly by conservative Q learning (Riedmiller, 2005; Fujimoto et al., 2019; Kumar et al., 2020; Zhang et al., 2021; Fujimoto & Gu, 2021), or explicitly by direct constraints in state or action spaces (Kumar et al., 2019; Liu et al., 2020;
Zhou et al., 2020).
Compared with model-based ofﬂine RL methods, these model-free methods behave more conservatively and lack broader generalization when the distribution of pre-collected experiences is narrow.
Instead of formulating as a single-objective optimization with regularization, a concurrent work (DiME) by Abdolmaleki et al.
(2021) takes a multi-objective perspective for model-free ofﬂine RL.
It adopts a modiﬁed scalarization method to achieve multiple policies on the Pareto front.
However, scalarization cannot ﬁnd any solution on the non-concave parts of Pareto front and thus may lead to sub-optimal performance.
A. E VOLUTION S TRATEGY
For black-box or noisy objective functions such as RL’s policy optimization, computing accurate the gradient is usually challenging.
Hence, as a derivative-free optimization method, evolution strategy (ES) has seen a recent revival in the RL community (Salimans et al., 2017; Mania et al., 2018).
Instead of solving the complicated objective function F ( θ ) directly, ES solves its Gaussian smoothing version: F ν ( θ ) =
E ε ∼N (0 ,I )
[ F ( θ + νε )] , where ν > 0 denotes a smoothing parameter.
When ν is small, the smoothed version F ν ( θ ) is close to the original objective (Nesterov & Spokoiny, 2017), and its gradient with respect to policy parameters θ is deﬁned by ∇ θ F ν ( θ )
= (2 π ) − d/ 2 (cid:82) R d F ( θ + νε ) e − 12 (cid:107) ε (cid:107) 22 ε d ε .
Although the gradient is intractable, it can be estimated by a standard Monte
Carlo method: ∇ θ F ν ( θ ) = 1 kν (cid:80)
ki
=1 F ( θ + νε i ) ε i .
The Monte Carlo estimation has no bias but high variance.
Many following algorithms were proposed to reduce the variance while keeping the bias unchanged.
In this paper, we adopt an antithetic estimator (Mania et al., 2018), as shown in
Eq.
(21), which uses the symmetric difference between a perturbation ε i ∼ N ( 0 , I )
and its antithetic counterpart − ε i .
[Equation]
where
J ˆ r ( θ ) , J u ( θ ) and Ψ( θ, v i ) are results achieved by the policy π θ on one trajectory generated from the environment model.
Despite the simplicity, ES achieves competitive performance compared to policy gradient methods (Choromanski et al., 2020).
A. S ELECTION OF P ARETO P OLICIES BY F ITTED Q E VALUATION (FQE)
Given a pool of Pareto policies, how do we determine the best one for a realistic environment?
A straightforward method is online policy selection that evaluates the performance of every Pareto policy in the real environment, and then selects the one with the highest return.
Most existing ofﬂine RL methods (Kidambi et al., 2020; Kumar et al., 2020; Matsushima et al., 2021; Zhang et al., 2021) adopt this method, which however requires many steps of online interactions with the realistic environment, especially when the number of policies to be evaluated is large.
We also follow these works and use online policy evaluation to achieve the results in Table 1.
Probably a more efﬁcient alternative is ofﬂine policy selection (Voloshin et al., 2019), which is proposed very recently, and aims at choosing the best policy from a set of policies, given only ofﬂine data.
However, ofﬂine policy selection/evaluation is an open problem (Levine et al., 2020), and a number of recent works in RL (Paine et al., 2020; Zhang & Jiang, 2021; Fu et al., 2021) are still exploring new strategies for it while combining our method with ofﬂine policy evaluation methods can potentially improve the efﬁciency of P3.
To evaluate this strategy, we use FQE (Le et al., 2019) to estimate the performance of each policy.
Although there exist various ofﬂine policy evaluation methods, we choose FQE for its simplicity and stability in practice.
By contrast, other methods have to solve complex estimation problems, such as learning a transition model from visual inputs or estimating the importance weights in a continuous action space (Fu et al., 2021).
In our experiments, both P3 and FQE have access to the same ofﬂine data.
On an ofﬂine dataset D , FQE algorithm trains a critic Q φ i for each Pareto policy π i generated by P3, and then uses the critic to estimate the expected value of π i w.r.t.
the initial state s 0 , i.e., ˆ
V π i =
E s 0 ∼ D
[ Q φ i ( s 0 , π i ( s 0 ))] , which reasonably quantiﬁes the performance of π i when deployed to a realistic environment.
The pseudo-code for FQE algorithm can be found in Alg.
FQE can efﬁciently ﬁnd a near-optimal Pareto policy after only hundreds of epochs, which is demonstrated by the results in Appendix A.7.
A. E NVIRONMENT M ODEL
We train an ensemble of N models and pick the best K models based on their prediction error on a hold-out set.
In the training phase, each model is optimized independently using the maximum likelihood estimation with mini-batch stochastic gradient descent.
In the inference phase, we randomly select one of K models and draw a state-reward concatenation from the resulting distribution, allowing for different transitions along a single episode to be sampled from different dynamics models.
The method makes our model more uncertain and noisier than the realistic environment, resulting in a more challenging MDP problem.
Prior work (Ha & Schmidhuber, 2018; Janner et al., 2019; Yu et al., 2020) demonstrated that the method effectively alleviates the model exploitation issue (Levine et al., 2020), especially in the ofﬂine setting.
As shown in Table 3, we list the hyperparameters of environment model for D4RL Gym experiments.
A. D4RL G YM E XPERIMENTS
D4RL Gym Datasets.
D4RL is a widely-used benchmark for evaluating ofﬂine RL algorithms.
It provides a variety of environments, tasks, and corresponding datasets containing samples of multiple trajectories generated via behavior policies, hand-designed controllers, or human demonstrators (Fu et al., 2020).
In the continuous control domain based on the MuJoCo simulator (Todorov et al., 2012), we apply a subset of datasets, including three environments (halfcheetah, hopper, and walker2d) and ﬁve dataset types (random, medium, medium-replay, expert, and medium-expert), to yield a total of 15 benchmark problems, in which random contains 1M samples from a random policy, medium contains 1M samples from a policy trained to approximately 1/3 of the performance of the expert, expert contains 1M samples from a policy trained to the performance of the expert, medium-replay contains the whole replay buffer of a policy trained up to the performance of the medium agent, and medium-expert contains a 50-50 split of medium and expert dataset (2M samples).
Practical Modiﬁcations.
We list the techniques adopted by P3 for D4RL Gym experiments.
(1) P3 applies a simple behavior cloning approach (Kidambi et al., 2020; Matsushima et al., 2021;
Kumar et al., 2020) to estimate the initial policy π i .
The technique effectively alleviates the data distribution shift caused by the difference between the policy-in-training and the behavior policies used to collect the data.
(2) We notice that the states of the high-dimensional complex tasks take the values in a broad range, causing the policies only pay attention to particular features of these states.
Therefore, we normalize the received state before feeding it to the policy network: a h = π ( σ − 1 s (ˆ s h − µ s )) where µ s and σ s are the mean and standard deviation of states, respectively.
In the ofﬂine setting, we can easily obtain µ s and σ s by computing the mean and standard deviation of all the states in the dataset, similar to the feature normalization in supervised learning.
Previous work (Mania et al., 2018; Fujimoto & Gu, 2021) demonstrated that the technique makes the policies more robust to multiple-scale state inputs.
(3) P3 applies the rank-based scale shaping (Wierstra et al., 2014) to compute the ES gradients, which transforms the objective function value via a rank-based score function, suppressing the inﬂuence of outlier perturbations.
Hyperparameters.
We outline the hyperparameters used in the experiments
[Table] [Table]
A. R ANK C ORRELATION BETWEEN OFFLINE AND ONLINE POLICY EVALUATION IN P3 ON D4RL G YM E XPERIMENTS
In Section A.4, in order to remove the extra computation required by online policy evaluation during the test, we instead use Fitted Q Evaluation (Alg. 3) to provide metrics for policy selection.
Here we take a closer look at the correlation between the policy ranking by the online evaluation and the policy ranking by FQE.
First, we rank all the policies in the P3 pool by the two metrics and show their generated rank orders (in an ascending order) for every policy in Fig.
On three datasets, the two rank orders show a strong correlation, which indicates that the FQE based ofﬂine policy evaluation provides an accurate approximation to the realistic return of P3 policies, and thus can be used to improve the test efﬁciency of P3.
In addition, we report the Spearman’s rank correlation (“Spearman’s ρ ”) and Kendall rank correlation (“Kendall’s τ ”) to quantitatively measure the correlation between the two rank orders.
We report the results in Fig.
6, which demonstrates a strong correlation coefﬁcient close to 1 .
for all datasets.
Therefore, replacing the online evaluation with FQE is an efﬁcient solution preserving the original rank orders of P3 policies.
[Figure]
k o f po li c
i e s a cc o r
d
i ng t o t r ue v a l ue s
[Figure]
an k o f
[Figure]
li c i e s a cc o r d
i
[Figure]
t o t r ue v a l ue s
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
A. R ESULTS ON THE D4RL G YM E XPERIMENTS
A. A BLATION S TUDY
For clarity in ablation study, we provide implementation details on the scalarization method (Boyd & Vandenberghe, 2004, Chapter 4.7).
The method maximizes a weighted sum of objectives: max ω J ( π θ ) where ω ∈ [0 , 1] denotes the weight assignment.
By solving multiple single-objective optimization with different weight assignments, we can obtain multiple policies with different tradeoffs, as shown in Fig.
For the experimental results in Table 2 and Fig.
9, we train 5 policies from scratch using the scalarization method with random weights and report the best score among them.
A. E FFECTIVENESS OF P ARETO POLICY POOL