DROP: C ONSERVATIVE M ODEL - BASED O PTIMIZATION FOR O FFLINE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we decouple the iterative (bi-level) ofﬂine RL from the ofﬂine training phase, forming a non-iterative bi-level paradigm that avoids the iterative error propagation over two levels.
Speciﬁcally, this non-iterative paradigm allows us to conduct inner-level optimization in training ( i.e. , employing policy/value regularization), while performing outer-level optimization in testing ( i.e. , conducting policy inference).
Naturally, such paradigm raises three core questions (that are not fully answered by prior non-iterative ofﬂine RL counterparts like rewardconditioned policy): ( Q1 )
What information should we transfer from the innerlevel to the outer-level?
( Q2 )
What should we pay attention to when exploiting the transferred information for the outer-level optimization?
( Q3 )
What are the beneﬁts of concurrently conducting outer-level optimization during testing?
Motivated by model-based optimization (MBO), we proposed DROP, which fully answers the above three questions.
Particularly, in the inner-level, DROP decomposes ofﬂine data into multiple subsets, and learns a MBO score model ( A1 ).
To keep safe exploitation to the score model in the outer-level, we explicitly learn a behavior embedding and introduce a conservative regularization ( A2 ).
During testing, we show that DROP permits deployment adaptation, enabling an adaptive inference across states ( A3 ).
Empirically, we evaluate DROP on various tasks, showing that
DROP gains comparable or better performance compared to prior methods.
I NTRODUCTION
Ofﬂine reinforcement learning (RL) (Lange et al., 2012; Levine et al., 2020) describes a task of learning a policy from previously collected static data.
Due to the overestimation of values at out-ofdistribution (OOD) state-actions, recent iterative ofﬂine RL methods introduce various policy/value regularization to avoid deviating from the ofﬂine data distribution (or support) in the training phase.
Then, these methods directly deploy the learned policy in an online environment to test the performance.
To unfold our following analysis, we term this kind of learning procedure as iterative bi-level ofﬂine RL (Figure 1 left), wherein the inner-level optimization refers to trying to eliminate the OOD issue by constraining the policy/value function, the outer-level optimization refers to trying to learn a better policy that will be employed at testing.
Here, we use the “iterative” term to emphasize that the inner-level and outer-level are iteratively optimized in the training phase.
However, without enough inner-level optimization (OOD regularization), there is still a distribution shift between the behavior policy and the policy to be evaluated.
Further, due to the iterative error exploitation and propagation (Brandfonbrener et al., 2021) over the two levels, performing such an iterative bi-level optimization completely in training often struggles to learn a stable policy/value function.
In this work, we thus advocate for non-iterative bi-level optimization (Figure 1 right) that decouples the bi-level optimization from the training phase, namely, performing inner-level optimization (eliminating OOD) in training and performing outer-level optimization (updating policy) in testing.
Intuitively, incorporating the outer-level optimization into the testing phase can eliminate the iterative error propagation over the two levels.
Then, three core questions 1 are: ( Q1 )
What information (“ ”) should we transfer from the inner-level to the outer-level?
( Q2 )
What should we pay special attention to when we exploit “ ” for outer-level optimization?
( Q3 )
Notice that the outer-level optimization and the online rollout test form a new loop (“ ”), what new beneﬁt does this give us?
[Figure]
(Online rollout )
[Figure]
Outer
[Figure]
Training phase
Testing phase (Online rollout) 𝜋 ∗ (𝑎 𝑡 |𝑠 𝑡 ) 𝑠 𝑡
Intriguingly, prior works under such a non-iterative framework have proposed to transfer (as “ ” in Q1 ) ﬁltered trajectories (Chen et al., 2021), a reward-conditioned policy (Emmons et al., 2021;
Kumar et al., 2019b), and the Q-value estimation of the behavior policy (Brandfonbrener et al., 2021; Gulcehre et al., 2021), all of which, however, partially address the aforementioned questions (we will elaborate these works in Table 1).
In this work, we propose a new alternative method that transfers an embedding-conditioned (Q-value) score model and we will show that this method sufﬁciently answers the above questions and beneﬁts most from
-iterative framework
Before introducing our method, we introduce a conceptually similar task (to the non-iterative bilevel optimization) — ofﬂine model-based optimization (MBO, Trabucco et al.
(2021)) 2 , which aims to discover, from static input-score pairs, a new design input that will lead to the highest score.
Typically, ofﬂine MBO ﬁrst learns a score model that maps the input to its score via supervised regression (corresponding to inner-level optimization), and then performs inference with the learned score model (as “ ”), for instance, by optimizing the input against the learned score model via gradient ascent (corresponding to the outer-level).
To enable this MBO implementation in ofﬂine RL, we are required to decompose an ofﬂine RL task into multiple sub-tasks, each of which thus corresponds to a behavior policy-return (parameters-return) pair.
However, there are practical optimization difﬁculties when learning the score model (inner-level) and performing inference (outer-level) on high-dimensional policy’s parameter space (as input for the score model).
At inference, directly extrapolating the learned score model (“ ”) also tends to drive the high-dimensional candidate policy (parameters) towards out-of-distribution, invalid, and low-scoring parameters (Kumar & Levine, 2020), as these are falsely and over-optimistically scored by the learned score model.
To tackle these problems, we suggest ( A1 ) learning low-dimensional embeddings for these sub-tasks decomposed in the MBO implementation, over which we estimate an embedding-conditioned Qvalue as the MBO score model (“ ” in Q1 ), and ( A2 ) introduce a conservative regularization, which pushes down the predicted scores on OOD embeddings, so as to avoid over-optimistic exploitation and protect against producing unconﬁdent embeddings when conducting outer-level optimization (policy/embedding inference).
Meanwhile, ( A3 ) learning embedding permits deployment adaptation, which means we can dynamically adjust inferred embeddings across different states in testing (aka test-time adaptation).
We name our method DROP (Design fROm Policies).
Compared with standard ofﬂine MBO for parameter design (Trabucco et al., 2021), deployment adaptation in DROP leverages the MDP structure of RL tasks, rather than simply conducting inference at the beginning of test rollout.
Empirically, we demonstrate that DROP can effectively extrapolate a better policy that beneﬁts from the non-iterative framework by answering the above three questions, and can achieve comparable or better performance compared to many prior ofﬂine RL algorithms.
P RELIMINARIES 2. R EINFORCEMENT LEARNING AND OFFLINE REINFORCEMENT LEARNING
We model the interaction between agent and environment as a Markov Decision Process (MDP) (Sutton & Barto, 2018), denoted by the tuple ( S , A , R, P, µ ) , where S is the state space,
A is the action space, P : S ×
A × S → [0 , 1] is the transition kernel, R : S ×
A → R is the reward function, and p 0 : S → [0 , 1] is the initial state distribution.
Let π ∈ Π :=
{ π : S × A → [0 , 1] } denotes a policy.
In RL, we aim to ﬁnd a stationary policy that maximizes the expected discounted return J ( π ) :
=
E τ ∼ π
[ (cid:80) ∞ t =
γ t R ( s t , a t )] in the environment, where τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , . . . ) , r t =
R ( s t , a t ) , is a sample trajectory and γ ∈ (0 , 1) is the discount factor.
We also deﬁne the state-action value function Q π ( s , a ) :
=
E τ ∼ π
[ (cid:80) ∞ t =
γ t R ( s t , a t ) | s 0 =
s , a 0 = a ] , which describes the expected discounted return starting from state s and action a and following π afterwards, and the state value function V π ( s ) =
E a ∼ π ( a | s )
[ Q π ( s , a )] .
To maximize J ( π ) , actor-critic algorithm alternates between policy evaluation and improvement.
Speciﬁcally, given initial Q 0 and π 0 , it iterates
[Table] [Table]
[Equation]
where the value function (critic) Q ( s , a ) is updated by minimizing the mean squared Bellman error with an experience replay dataset D + and, following the deterministic policy gradient theorem (Silver et al., 2014), the policy (actor) π ( a | s ) is updated to maximize the estimated Q k +1 ( s , π ( a | s )) .
In ofﬂine RL (Levine et al., 2020), the agent is provided with a static data D = { τ } which consists of trajectories collected by running some data-generating policies.
Note that here we denote static ofﬂine data D , distinguishing from the experience replay D + in online setting.
Unlike the online RL problem, where the experience D + in Equation 1 can be dynamically updated, the agent, in ofﬂine
RL, is not allowed to interact with the environment to collect new experience data.
As a result, naively performing policy improvement as in Equation 2 may evaluate the estimated Q k ( s (cid:48) , a (cid:48) ) on actions that lie far outside of the static ofﬂine data D , resulting in pathological values Q k +1 ( s , a ) that incur large error.
Further, iterating policy evaluation and improvement will cause the learned policy π k +1 ( a | s ) to be biased towards out-of-distribution actions with erroneously overestimated values.
2. O FFLINE MODEL - BASED OPTIMIZATION
Model-based optimization (MBO) (Trabucco et al., 2022) aims to ﬁnd an optimal design input x ∗ with a given score function f ∗ :
X → Y ⊂ R , i.e. , x ∗ = arg max x f ∗ ( x ) .
Typically, we can repeatedly query the oracle score model f ∗ for new candidate design, until it produces the best design.
However, we often do not have the oracle score function f ∗ , but are provided with a static ofﬂine dataset { ( x , y ) } of labeled input-score pairs.
To track such ofﬂine MBO question, we can ﬁt a parametric model f to the oracle score function f ∗ via the empirical risk minimization (ERM), f ← arg min f E x ,y (cid:104)
( f ( x ) − y ) 2 (cid:105) .
Then, starting from the best point in the dataset, we can perform gradient ascent on the design input and set the learned optimal design x ∗ = x ◦ K :=
GradAscent f
( x ◦ 0 , K ) (for simplicity, next we will omit subscript f in
GradAscent f ), where
[Equation]
Since the aim is to ﬁnd a better design input beyond all the designs in the dataset and while directly optimizing score model f with ERM can not ensure new candidates (out-of-distribution design inputs) receive correct scores, thus one crucial requirement is to conduct conﬁdent extrapolation.
DROP: DESIGN FROM POLICIES
We present our framework in Figure 2.
In Sections 3.1 and 3.2, we will answer questions Q1 and
Q2 , setting a learned MBO score model as “ ” ( A1 ) and introducing a conservative regularization over the score model ( A2 ).
In Section 3.3, we will answer Q3 , where we show that we can conduct outer-level optimization during testing, enabling an adaptive embedding inference across states ( A3 ).
3. T ASK DECOMPOSITION
Our core idea is to explore MBO in the non-iterative bi-level ofﬂine RL framework (Figure 1 right), while capturing the structural characteristics of RL tasks and answering the raised questions ( Q1 ,
Q2 , and Q3 )
in the introduction section.
To begin with, we ﬁrst decompose the ofﬂine data D into
N ofﬂine subsets D [ N ] := {D 1 , . . .
, D N } .
In other words, we decompose an ofﬂine task, learning 𝛽(a|𝑠, 𝑧) 𝑓(𝑠, a, 𝑧) 𝛽(a|𝑠, 𝑧 ∗ ) 𝑧 1 𝑧𝑧 3 𝑧𝜇(𝑧) offline data (task) sub-tasks task embedding & deployment conservative MBO adaptation 𝒟 𝑠 a 𝒟𝒟𝒟𝒟𝜙(𝑧|𝑛) ⇌ ↦ training phase (inner-level ) testing phase 𝛽 𝑓 outer-level 𝑧 ∗
[Equation]
Such a decomposition also comes with an additional beneﬁt that it provides an avenue to exploit the hybrid modes in ofﬂine data D , because that D is often collected using hybrid data-generating behavior policies (Fu et al., 2020), which suggests that ﬁtting a single behavior policy may not be optimal to model the multiple modes of the ofﬂine data distribution (see Appendix C.1 for the empirical evidence).
Thus, to encourage the emergence of diverse sub-tasks which capture distinct behavior modes in D (this is not our focus in this work 3 ), we simply perform task decomposition according to the returns of trajectories in D , heuristically ensuring that trajectories in the same subtask share similar returns and trajectories from different sub-tasks have distinct returns.
3. T ASK EMBEDDING AND CONSERVATIVE MODEL - BASED OPTIMIZATION
Naive model-based optimization (MBO) over behavior policies.
Beneﬁting from the above ofﬂine task decomposition, we can conduct MBO over a set of input-score ( x , y ) pairs, where we model the (parameters of) behavior policies β n as the design inputs x and the corresponding expected returns at initial state J ( β n ) as scores y .
Note that, ideally, evaluating behavior policy β n , i.e. , calculating J ( β n ) ≡ E s 0 (cid:2) V β n ( s 0 ) (cid:3) , with subset D n will never trigger the overestimation of values in the inner-level optimization.
By introducing a score model f : Π → R (as the transferred information “ ” in Q1 ), we can then perform outer-level policy inference with
[Equation]
, where
[Equation]
However, directly performing optimization-based inference (outer-level optimization), max π f ( π ) , will quickly ﬁnd an invalid input for which the learned score model f outputs erroneously large values ( Q2 ).
Furthermore, it is particularly severe if we perform the inference directly over the parameters of policies, accounting for the fact that the parameters of input behavior policies lie on a narrow manifold in a high-dimensional parametric space (Kumar & Levine, 2020).
Task embedding.
To enable feasible policy inference, we propose to decouple the MBO techniques from the high-dimensional space of policy parameters.
We achieve this by learning a latent embedding space Z with an information bottleneck ( dim( Z ) (cid:28) min( N, dim(Π)) ), conditioned on the sub-task id n , from which the high-dimensional parameters of behavior policies can be inferred
We can thus use the embedding z ∈ Z to represent sub-tasks (or the corresponding behavior policies).
Formally, we learn a task embedding 4 φ : R N × Z → [0 , 1] and a contextual behavior policy β : S ×
Z × A → [0 , 1] , which replaces N separate behavior policies in Equation 4:
[Equation]
Conservative model-based optimization.
In principle, by substituting the learned task embedding φ ( z | n ) and the contextual behavior policy β ( a | s , z ) into the original objective in Equation 5, we can then conduct MBO over the embedding space: learning f : Z → R with min f E n,φ ( z | n ) (cid:104)
( f ( z ) − J ( β n )) 2 (cid:105) , and setting the optimal embedding z ∗ =
arg max z f ( z ) and the corresponding policy π ∗ ( a | s ) =
β ( a | s , z ∗ ) , where z ∗ can be inferred with gradient ascent as in
Equation 3.
However, we must deliberate a new distribution shift in the Z -space, stemming from the original distribution shift in the parametric space when directly optimizing Equation 5.
Motivated by the energy model (LeCun et al., 2006) and the conservative regularization (CQL, Kumar et al.
, we introduce the conservative score model learning, additionally regularizing the scores of out-of-distribution embeddings µ ( z ) :
[Equation]
Intuitively, as long as the scores of out-of-distribution embeddings E µ ( z ) [ f ( z )] is lower than that of in-distribution embeddings E n,φ ( z | n ) [ f ( z )] (up to a threshold η ), conducting embedding inference with z ∗ =
arg max z f ( z ) would produce the best and conﬁdent solution, avoiding towards embeddings that are far away from the training set { φ ( z | n ) , n = 1 , . .
. , N } .
Now that we have reframed the non-iterative bi-level ofﬂine RL problem as one of ofﬂine MBO: in the inner-level optimization ( Q1 ), we set the practical choice for “ ” as the learned score model f ( A1 ); in the outer-level optimization ( Q2 ), we introduce task embedding and conservative regularization to avoid over-optimistic exploitation when exploiting f for policy/embedding inference ( A2 ).
In next section, we will show how to slightly change the form of the score model f , so as to leverage the (MDP) structural characteristic (loop “ ”) of RL tasks and answer the left Q3 .
3. D EPLOYMENT ADAPTATION
Recalling that we update f ( z ) to regress the value at initial state E s 0 (cid:2) V β n ( s 0 ) (cid:3) in Equation 7, we then conduct outer-level inference with z ∗ = arg max z f ( z ) and rollout the z ∗ -conditioned policy π ∗ ( a | s ) :
= β ( a | s , z ∗ ) until the end of rollout episode at deployment (testing).
In essence, this inference produces an extrapolation over the distribution of the behavior policies (corresponding to embeddings).
Going beyond the (outer-level) inference only at the initial state, we propose that a implementation can beneﬁt by performing inference at any rollout state in testing ( A3 ).
To enable deployment adaptation, we model the score model with f : S×A×Z → R , taking a stateaction as extra input.
Then, we encourage the score model to regress the values of behavior policies over all state-action pairs in each sub-task, min f E
[Equation]
For simplicity, instead of learning an additional value function Q β n for each behavior policy, we learn the score model directly with the TD-error used for learning the value function Q β n ( s , a ) as in Equation 1, together with the conservative regularization in Equation 7:
[Equation]
where ¯ f denotes a target network and we update the target ¯ f with soft updates: ¯ f = (1 − υ ) ¯ f + υf .
In testing, we thus can dynamically adapt the outer-level optimization, setting policy inference with π ∗ ( a | s ) =
β ( a | s , z ∗ ( s )) , where z ∗ ( s ) =
arg max z f (cid:0) s , β ( a | s , z ) , z (cid:1) .
Speciﬁcally, at any state s in the deployment phase, we perform gradient ascent to ﬁnd the optimal behavior embedding z ∗ ( s ) =
z ◦ K ( s ) :
=
GradAscent( s , z ◦ 0 , K ) , where z ◦ 0 is the starting point and
[Equation]
[Table]
3. C ONNECTION TO PRIOR NON - ITERATIVE OFFLINE COUNTERPARTS
In Table
[Table]
we summarize the comparison with prior representative non-iterative ofﬂine RL methods.
Intuitively, our DROP (leveraging returns to decompose D ) is similar in spirit to F-BC and
RvS-R
(Chen et al., 2021; Emmons et al., 2021), both of which use return R ( τ ) to guide the inner-level optimization.
However, both F-BC and RvS-R leave Q2 unanswered.
In outer-level, F-BC can not enable policy extrapolation, which heavily relies on the data quality in ofﬂine tasks, and RvS-R needs to handcraft a target return (as the contextual variable for π ( a | s , · ) ), which also probably triggers the potential distribution shift between the hand-crafted contextual variable and that used for learning the contextual policy (see examples in Figure 6 of Emmons et al.
Diving deeper into the bi-level optimization, we can also ﬁnd DROP combines the advantages of Onestep (Brandfonbrener et al., 2021) and COMs (Trabucco et al., 2021), where Onestep performs outer-level optimization in action space ( arg max a ), COMs performs that in parameter space ( arg max β ), while our DROP performs that in embedding space ( arg max z ).
As a result, the choice of f ( s , a , z ) in DROP allows us to conduct safe exploitation over “ ” in outer-level ( Q2 ) and leverage the structural characteristic of RL task (the loop “ ” in Q3 ), rather than simply conducting outer-level optimization at initial states as in COMs (corresponding to the objective in Equation 5).
3. P RACTICAL I MPLEMENTATION
Algorithm
Training)
Require:
Dataset of trajectories, D = { τ } .
1: Initialize φ z
, β s
2: Decompose D into N sub-sets D [ N
3: while converged do 4:
Learn
6: end while
Return: β ( a | s , z ) and f ( s , a , z ) .
Algorithm
Require: Env, β ( a | s , z ) , and f ( s , a , z ) .
1: s
= Env.
Reset().
2: while not done do 3:
Inference (deployment adaptation): z ∗ ( s t ) =
arg max z f ( s t , β ( a t | s t , z ) , z ) .
Sample action: a t ∼ β ( a t | s t , z ∗ ( s t )) .
Step Env: s t +1 ∼ P
( s t +1 | s t , a t ) .
6: end while
We now summarize the DROP
algorithm (see Algorithm 1 for the training phase and Algorithmfor the testing phase).
During training (inner-level optimization), we alternate between updating φ ( z | n ) , β ( a | s , z ) , and f ( s , a , z ) , wherein we update φ with both maximum likelihood loss and
TD-error loss in Equations 6 and 8.
During testing (outer-level optimization), for each state s , we use the gradient ascent in Equation 9 to choose the optimal embedding z ∗ ( s ) .
Instead of simply sampling a single starting point z ◦ 0 , we choose N starting points corresponding to all the embeddings { z n | n = 1 , . . .
, N } of sub-tasks, and then choose the optimal z ∗ ( s ) from those updated embeddings for which the learned f outputs the highest score: z ∗ ( s ) =
arg max z f ( s , β ( a | s , z ) , z ) s.t. z ∈ { GradAscent( s , z n , K ) | n = 1 , . .
. , N } .
Then, we sample action from π ∗ ( a | s ) :
= β ( a | s , z ∗ ( s )) .
For more optimization (training/testing) details, we refer the reader to Appendix D.
R ELATED W ORK
Ofﬂine RL.
In ofﬂine RL, learning with static ofﬂine data is prone to exploiting out-of-distribution (OOD) state-action pairs and producing over-estimation of values, which makes vanilla iterative policy learning and value optimization challenging (Rashidinejad et al., 2021).
To eliminate the problem, a number of methods have been explored, in essence, by either introducing a policy/value regularization in the iterative loop or trying to eliminate the iterative procedure itself .
Iterative methods: Sticking with the normal iterative updates in RL, ofﬂine policy regularization methods aim to keep the learning policy to be close to the behavior policy under a probabilistic distance (Cang et al., 2021; Fujimoto & Gu, 2021; Kostrikov et al., 2021a; Kumar et al., 2019a;
Liu et al., 2022; Nair et al., 2020a; Peng et al., 2019; Siegel et al., 2020; Wu et al., 2019; Zhang et al., 2021).
Some works also conduct implicit policy regularization with variants of importance sampling Lee et al.
Liu et al.
; Nachum et al.
Besides regularizing policy, it is also feasible to constrain the substitute value function in the iterative loop.
Methods constraining the value function aim at mitigating the over-estimation, which typically introduces pessimism to the prediction of the Q-values (Chebotar et al., 2021;
Jin et al., 2021; Kumar et al., 2020; Li et al., 2022; Ma et al., 2021a;b) or penalizes the value with an uncertainty quantiﬁcation (An et al., 2021;
Bai et al., 2022; Rezaeifar et al., 2021; Wu et al., 2021), making the value for out-of-distribution state-actions more conservative.
Similarly, another branch of model-based methods (Kidambi et al., 2020; Yu et al., 2020; 2021b; Rigter et al., 2022) also perform iterative bi-level updates, alternating between regularized evaluation and improvement.
Different from these works, DROP only evaluates values of behavior policies in the inner-level, avoiding error propagation between two levels.
Non-iterative methods: Another complementary line of work studies how to eliminate the iterative updates, which simply casts RL as a weighted or conditional imitation learning problem ( Q1 ).
Derived from the behavior-regularization RL (Geist et al., 2019; Vieillard et al., 2020), the former conducts weighted behavior cloning: ﬁrst learn a value function for the behavior policy, then weigh the state-action pairs with the learned values or advantages (Abdolmaleki et al., 2018; Chen et al., 2019; Wang et al., 2020; Peng et al., 2019).
Besides, some works also propose implicitly behavior policy regularization that also avoids estimating the value of new candidate policies, initializing the learning policy with a behavior policy Matsushima et al.
or performing only a “onestep” update (policy improvement) over the behavior policy (Fujimoto et al., 2019; Gulcehre et al., 2021).
For the latter , this branch method typically builds upon the hindsight information matching (Andrychowicz et al., 2017;
Eysenbach et al., 2020; Pong et al., 2018; Wan et al., 2021), assuming that the future trajectory information can be useful to infer the middle decision that leads to the future and thus relabeling the trajectory with the reached states or returns.
Due to the simplicity and stability, RvS-based methods thus advocate for learning a goal-conditioned or reward-conditioned policy (Chen et al., 2021;
Ding et al., 2019; Emmons et al., 2021; Furuta et al., 2021; Janner et al., 2021; Lin et al., 2022; Srivastava et al., 2019; Yang et al., 2022) with supervised learning.
However, these works do not fully exploit the non-iterative bi-level framework and fail to answer the proposed questions, which either does not regularize the inner-level optimization before exploiting “ ” in the outer-level ( Q2 ), or does not support the deployment adaptation in testing ( Q3 ).
Ofﬂine model-based optimization (MBO).
Similar to ofﬂine RL, the main challenge of MBO is to reason about uncertainty and OOD values (Brookes et al., 2019; Fannjiang & Listgarten, 2020), since a direct gradient-ascent against the learned score model can easily produce invalid inputs that are falsely and highly scored.
To counteract the effect of model exploitation, prior works introduce various techniques, including normalized maximum likelihood estimation (Fu & Levine, 2021), model inversion networks (Kumar & Levine, 2020), local smoothness prior (Yu et al., 2021a), and conservative objective models (COMs) (Trabucco et al., 2021).
Compared to COMs, DROP shares similarity with the conservative model, but instantiates on the embedding space instead of the parameter space.
Such difference is nontrivial, not only because DROP allows OOD sampling (aimed at pessimism) directly in embedding space, avoiding an adversarial training as in COMs, but also because DROP allows deployment adaptation, enabling dynamical inference across states in testing.
E XPERIMENTS
In this section, we present our empirical results.
We ﬁrst give examples to illustrate the deployment adaptation.
Then we evaluate DROP against prior ofﬂine RL algorithms on D4RL benchmark.
Finally, we compare DROP with prior (ofﬂine) latent-based baselines.
For more ablation studies wrt to the decomposition rules and the conservative regularization, we refer readers to the appendix.
Illustration of deployment adaptation.
To better understand the deployment adaptation of DROP, we include four comparisons that exhibit different embedding inference rules at testing:
[Figure]
choose the from those
At we conduct inference ascent with z
= max z f ( s β ( a 0 | s
z ) , z ) , and keep this embedding
ﬁxed
throughout
the rollout
(3) DROP-Best-Ada: We adapt the contextual policy by setting π ∗ ( a t | s t ) =
β ( a t | s t , z ∗ 0 ( s t )) ,
where we choose
the best embedding z ∗ 0 ( s t ) directly from
those
embeddings
of
behavior policies
for
which
the score model
outputs the highest
score
, i
e. , z ∗
( s
t
) =
arg max
z
f ( s
t
, β ( a
t
| s t , z ) , z ) s.t.
z ∈ z [ N ] .
DROP-Grad-Ada (gradient-based adaptation as described
in Section
We set
π ∗
( a t | s t )
=
β ( a t | s t , z ∗ ( s t )) and choose the best embedding from those updated embeddings of behavior policies, i.e. , z ∗ ( s t ) =
arg max z f ( s t , β ( a t | s t , z ) , z ) s.t. z ∈ { GradAscent( s t , z n , K ) | n = 1 , . . .
, N } .
In Figure 3, we visualize the four different inference rules and report the corresponding performance in the halfcheetah-medium-expert task (Fu et al., 2020).
In Figure 3 (a), we set the starting point as the best embedding z ∗ 0 ( s 0 ) in z [ N ] , and perform gradient ascent to ﬁnd the optimal z ∗ 0 ( s 0 ) for
DROP-Grad.
In Figure 3 (b), we can ﬁnd that at different time steps, DROP-Best-Ada chooses different embeddings (as contextual variables for β ( a t
| s t , · ) ).
At a high level, performing such dynamical inference enables us to combine different embeddings, switching behavior policies at different states.
Further, in Figure 3 (c1, c2), we ﬁnd that the additional inference (with gradient ascent) in DROP-Grad-Ada allows to extrapolate beyond the embeddings of behavior policies, and thus results in sequential composition of new embeddings (policies) across different states.
For practical impacts of these different inference rules, we provide the performance comparison in Figure 3 (d), where we can ﬁnd that performing gradient-based optimization (*-Grad-*) outperforms the natural selection among these embeddings of behavior policies in sub-tasks (*-Best-*), and rollout with adaptive embedding inference (DROP-*-Ada) outperforms that with ﬁxed embeddings (DROP-*).
Empirical performance on benchmark tasks.
We evaluate DROP on a number of tasks from the
D4RL dataset and make comparison with both prior iterative and non-iterative ofﬂine algorithms 5 .
Considering that DROP follows the non-iterative ofﬂine RL paradigm, we compare DROP with prior non-iterative ofﬂine baselines (BC, F-BC, DT (Chen et al., 2021), RvS-R, Onestep, and COMs) in the main paper.
Compared with our DROP, these baselines do not fully answer the raised questions (see Table 1), which either does not regularize the inner-level optimization before exploiting “ ” in outer-level ( Q2 ), or does not support the deployment adaptation in testing ( Q3 ).
Moreover, we also provide the comparison with CQL, which inspires us to design the conservation in Equation 7.
In
Table 2, we show the evaluation results for AntMaze-* and Gym-*-medium-* tasks in D4RL *-v2, where we can ﬁnd DROP (-Grad-Ada) achieves better performance than these non-iterative ofﬂine
RL baselines overall.
Compared with CQL, DROP shows superior performance in AntMaze-large-* and Gym-*-medium-expert (m.-exp.) tasks, while leads inferior performance in AntMaze-medium- * and Gym-*-medium-replay (m.-rep.) tasks.
As an extension, we also design DROP+CVAE implementation (see motivation in next paragraph and details in Appendix C.4), which further improves
DROP’s performance and retains superior/comparable performance in all tasks.
Comparison with
latent policy methods.
Note that one additional merit of DROP is that it naturally accounts for hybrid modes in D by conducting task decomposition in inner-level, we thus
[Table] [Table]
compare DROP to latent policy methods (PLAS (Zhou et al., 2020) and LAPO (Chen et al., 2022))
that use conditional variational autoencoder (CVAE) to model ofﬂine data and also account for multi-modes in ofﬂine data.
Essentially, both our DROP and baselines (PLAS and LAPO) learn a latent policy in the inner-level optimization, except that we adopt the non-iterative bi-level learning while baselines are instantiated under the iterative paradigm.
By answering Q3 , DROP permits deployment adaptation, enabling us to dynamically switch/stitch “skills” (latentpolicy/behaviors as shown in Figure 3) and encouraging high-level abstract exploration in testing.
However, the aim of introducing the latent policy in PLAS and LAPO is to regularize the inner-level optimization, which fairly answers Q2 in the iterative ofﬂine counterpart but can not provide the potential beneﬁt (deployment adaptation) by answering Q3 in the non-iterative paradigm.
We provide the comparison results in Table 3.
We can observe that DROP (-Best-Ada) and DROP (-Grad-Ada) consistently achieves better performance than PLAS on AntMaze-*-v2 tasks.
On the
Gym *-medium domain, DROP (-Grad-Ada) also performs better than LAPO.
However, there is a big performance gap between DROP and LAPO on the *-random domain.
We speculate that it is mainly caused by the decomposition rule.
In our DROP implementation, we heuristically use return to conduct task decomposition (motivated by RvS-R (Emmons et al., 2021)), while LAPO and PLAS conduct decomposition (learning latent policy) automatically.
Similarly, to bridge the gap, we also adopt CVAE to model the ofﬂine data and afterwards take the learned latent embedding in CVAE as the embedding of behaviors, instead of conducting return-guided task decomposition.
We provide implementation details (DROP+CVAE) in Appendix C.4 and new results in Tables 2 and 3 (DROP+), where we can see such CVAE-based DROP implementation can bring a substantial performance improvement.
Further, in Table 6 (Appendix C.4), we compare DROP+CVAE to IQL (Kostrikov et al., 2021b), consistently demonstrating the competitive empirical performance of DROP approach against state-of-art ofﬂine iterative/non-iterative baselines.
C ONCLUSION
In this work, we introduce non-iterative bi-level ofﬂine RL and based on this paradigm, we raise three questions ( Q1 , Q2 , and Q3 ).
To answer that, we reframe the ofﬂine RL problem as one of MBO and learn a score model ( A1 ), introduce embedding learning and conservative regularization ( A2 ), and propose deployment adaptation in testing ( A3 ).
We evaluate DROP on various tasks, showing that DROP gains comparable or better performance compared to prior methods.
A A PPENDIX
Here we provide the content of the appendix:
B Discussion and future work
C More
experiments
C.1 Single behavior policy vs. multiple behavior policies C.2 Decomposition rules C.
Online ﬁne-tuning C.4 DROP +
CVAE C.
Ablation study D Implementation details E
Additional
results
B D ISCUSSION AND FUTURE WORK
Limitations.
DROP also has several limitations.
First, the ofﬂine data decomposition dominates the following bi-level optimization, and thus choosing a suitable decomposition rule is a crucial requirement for policy inference (see experimental analysis in Appendix C.2).
An exciting direction for future work is to study generalized task decomposition rules (Rao et al., 2021).
In Appendix C.4, we also exhibit a potential of such generalized task decomposition by introducing CVAE into DROP’s implementation, and ﬁnd such a combination (DROP + CVAE) can bring practical performance improvement.
Second, we ﬁnd that when the number of sub-tasks is too large, the inference is unstable, where adjacent checkpoint models exhibit larger variance in performance (such instability also exists in prior ofﬂine RL methods, discovered by Fujimoto & Gu (2021)).
One natural approach to this instability is conducting online ﬁne-tuning (see Appendix C.3 for our empirical studies).
Going forward, we believe our work suggests a feasible alternative for generalizable ofﬂine robotic learning: by decomposing a single robotic dataset into multiple subsets, ofﬂine policy inference can beneﬁt from performing model-based optimization (MBO) and the joint deployment adaptation procedure.
Social impact: Beyond a general ofﬂine RL improvement, the authors do not foresee negative social impacts.
C M ORE EXPERIMENTS C. S INGLE BEHAVIOR POLICY VS . MULTIPLE BEHAVIOR POLICIES
[Figure]
In Figure 4, we provide empirical evidence that learning a single behavior policy (using BC) is not sufﬁcient to characterize the whole ofﬂine dataset, and multiple behavior policies (conducting task decomposition) deliver better resilience to characterize the ofﬂine data than a single behavior policy.
[Figure]
and Rank decomposition rules respectively.
We can ﬁnd across 18 tasks (AntMaze and MuJoCoGym suites) and 3 embedding inference methods (DROP-Grad, DROP-Best-Ada, and DROP-GradAda), Rank is more stable and yields better performance compared with the other two decomposition rules.
C. D ECOMPOSITION RULES
In DROP algorithm, we explicitly decompose an ofﬂine task into multiple sub-tasks, over which we then reframe the ofﬂine policy learning problem as one of ofﬂine model-based optimization.
In this section, we discuss three different designs for the task decomposition rule.
Random ( N, M ) :
We decomposition ofﬂine dataset D := { τ } into N subsets, each of which contains at most M trajectories that are randomly sampled from the ofﬂine dataset.
Quantization ( N, M ) : Leveraging the returns of trajectories in ofﬂine data, we ﬁrst quantize ofﬂine trajectories into N bins, and then randomly sample at most M trajectories (as a sub-task) from each bin.
Speciﬁcally, in the i -th bin, the quantized trajectories { τ i } satisfy R min +∆ ∗ i < Return( τ i ) ≤
R min + ∆ ∗ ( i + 1) , where ∆ =
(R max − R min ) N , Return( τ i ) denotes the return of trajectory τ i , and R max and R min denote the maximum and minimum trajectory returns in the ofﬂine dataset respectively.
Rank ( N, M ) :
We ﬁrst rank the ofﬂine trajectories descendingly based on their returns, and then sequentially sample M trajectories for each subset.
( We adopt this decomposition rule in main paper. )
In Figure 5, we provide the comparison of the above three decomposition rules (see the selected number of sub-tasks and the number of trajectories in each sub-task in Table 9).
We can ﬁnd that across a verity of tasks, decomposition rule has a fundamental impact on the subsequent modelbased optimization.
Across different tasks and different embedding inference rules, Random and
Quantization decomposition rules tend to exhibit large performance ﬂuctuations, which reveals the importance of choosing a suitable task decomposition rule.
In our paper, we adopt the Rank decomposition rule, as it demonstrates a more robust performance shown in Figure 5.
In Appendix C.4, we adopt the conditional variational auto-encoder (CVAE) to conduct automatic task decomposi- cloning (F-BC) on D4RL AntMaze and MuJoCo suites (*-v2).
We take the baseline results of BC and F-BC from Emmons et al.
(2021), where F-BC is trained over the top 10% trajectories, ordered by the returns.
Our DROP results are computed over 5 seeds and 10 episodes for each seed.
[Table]
tion (treating each trajectory in ofﬂine dataset as an individual task) and we ﬁnd such implementation (DROP+CVAE) can further improve DROP’s performance.
In future work
[Table]
we also encourage better decomposition rules to decompose ofﬂine tasks so as to enable more effective model-based optimization for ofﬂine RL tasks.
Comparison with ﬁltered behavior cloning.
We also note that the Rank decomposition rule leverages more high-quality trajectories than the other two decomposition rules (Random and Quantization).
Thus, a natural question to ask is, is the performance of Rank better than that of Random and Quantization due to the presence of more high-quality trajectories in the decomposed sub-tasks?
That is, whether DROP (using the Rank decomposition rule) only conducts behavioral cloning over those high-quality trajectories, thus leading to better performance.
To answer the above question
[Table]
we compare DROP (using the Rank decomposition rule) with the ﬁltered behavior cloning (F-BC), where the latter (F-BC) performs behavior cloning after ﬁltering for trajectories with highest returns.
We provide the comparison results in Table 4.
We can ﬁnd that in AntMaze tasks, the overall performance of DROP is higher than that of F-BC.
For the MuJoCoGym suite, DROP-based methods outperforms F-BC on these ofﬂine tasks that contain a plenty of sub-optimal trajectories, including the random, medium, and medium-replay domains.
This result indicates that DROP can leverage the sort of embedding inference (extrapolation) to ﬁnd a better policy beyond all the behavior policies in sub-tasks, which is more effective than simply performing imitation learning on a subset of the dataset.
C. O NLINE FINE - TUNING
Online ﬁne-tuning (checkpoint-level).
In Figure 6, we show the learning curves of DROP-Best on four DR4L tasks.
We can ﬁnd that DROP exhibits a high-variance (in performance) across
[Figure]
training steps 6 , which means the performance of the agent may be dependent on the speciﬁc stopping point chosen for evaluation (such instability also exists in prior ofﬂine RL methods (Fujimoto & Gu, 2021)).
To choose a suitable stopping checkpoint over which we perform the DROP inference (DROP-Grad,
DROP-Best-Ada and DROP-Grad-Ada)
, we propose to conduct checkpoint-level online ﬁne-tuning (see Algorithm 3 in Section D for more details): we evaluate each of the latest T checkpoint models and choose the best one that leads to the highest episode return.
In Figure 7, we show the total normalized returns across all the tasks in each suite (including
Maze2d, AntMaze, and MuJoCo-Gym).
We can ﬁnd that in most tasks, ﬁne-tuning (FT) can guarantee a performance improvement.
However, we also ﬁnd such ﬁne-tuning causes negative impacts in performance in AntMaze(*-v0) suite.
The main reason is that, in this checkpoint-level ﬁne-tuning, we choose the “suitable” checkpoint model using the DROP-Best embedding inference rule, while we adopt the other three embedding inference rules (DROP-Grad, DROP-Best-Ada and
DROP-Grad-Ada) at the test time.
Such ﬁnding also implies that the success of DROP’s deployment adaptation is not entirely dependent on the best embedding across sub-tasks 7 ( i
[Table]
e. , the best embedding z ∗ 0 ( s 0 ) in DROP-Best), but requires switching between some “suboptimal” embeddings (using
DROP-Best-Ada) or extrapolating new embeddings (using DROP-Grad-Ada).
[Table]
B e s t - A d a F T
D R O PG
r a d - A d a
D R O PG
r a d - A d a F T
AntMaze(-v2)
D R O PG
r a d
D R O PG
r a d F
T
D R O PB e s t - A d a
D R O PB e s t - A d a F T
D R O PG
r a d - A d a
D R O PG
r a d - A d a F T
MuJoCo-
[Figure] [Figure]
Online ﬁne-tuning (embedding-level).
Beyond the above checkpoint-level ﬁne-tuning procedure, we can also conduct embedding-level online ﬁne-tuning: we aim to choose a suitable gradient update step for the gradient-based embedding inference rules (including DROP-Grad and DROP-GradAda).
Similar to the checkpoint-level ﬁne-tuning, we ﬁrst conduct the deployment adaptation pro- cedure (DROP-Grad and DROP-Grad-Ada) over a set of gradient update steps, and then choose the best step that leads to the highest episode return (see Algorithm 4 in Section D for more details).
In Table 5, we compare our DROP (DROP-Grad and DROP-Grad-Ada) to three ofﬂine RL methods (AWAC (Nair et al., 2020b), CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2021b)), reporting the initial performance and the performance after online ﬁne-tuning.
We can ﬁnd that the embedding-level ﬁne-tuning (0.3M) enables a signiﬁcant improvement in performance.
The ﬁnetuned DROP-Grad-Ada (0.3M) outperforms the AWAC and CQL counterparts in most tasks, even though we take less rollout steps to conduct the online ﬁne-tuning (baselines take 1 M online rollout steps, while DROP-based ﬁne-tuning takes 0 .
3 M steps).
However, there is still a big gap between the ﬁne-tuned IQL and the embedding-level ﬁne-tuned DROP (0.3M).
Considering that there remains 0 .
7 M online steps in the comparison, we further conduct “parametric-level” ﬁne-tuning (updating the parameters of the policy network) for our DROP-Grad-Ada on medium-* and large-* tasks, we can ﬁnd which achieves competitive ﬁne-tuning performance even compared with IQL.
The baseline results of AWAC, CQL, and IQL are taken from Kostrikov et al.
(2021b), where they run 1 M online steps to ﬁne-tune the learned policy.
For our DROP method (DROP-Grad and
DROP-Grad-Ada), we run 0 .
3 M ( = 6 checkpoint × 50 K max × 1000 steps per episode ) online steps to ﬁne-tune (embedding-level) the policy, i.e. , aiming to ﬁnd the optimal gradient ascent step that is used to infer the contextual embedding z ∗ ( s 0 ) or z ∗ ( s t ) for π ∗ ( a t | s t ) :
= β ( a t | s t , · ) (see Algorithm 4 for the details).
Moreover, for medium-* and large-* tasks, we conduct additional parametric-level ﬁnetuning, with 0 .
7 M online steps to update the policy’s parameters.
Our DROP results are computed over 5 seeds and 10 episodes for each seed
[Table]
C. DROP + CVAE
CVAE-based embedding learning.
Similar to LAPO (Chen et al., 2022) and PLAS (Zhou et al., 2020), we adopt the conditional variational auto-encoder (CVAE) to model ofﬂine data.
Speciﬁcally, we learn the contextual policy and behavior embedding:
[Equation]
Then, we learn the score model f with the TD-error and the conservative regularization:
[Equation]
where ¯ f denotes a target network and µ ( z ) denotes the uniform distribution over the Z -space.
In testing, we also dynamically adapt the outer-level optimization, setting policy inference with π ∗ ( a | s ) =
β ( a | s , z ∗ ( s )) , where z ∗ ( s ) =
arg max z f (cid:0) s , β ( a | s , z ) , z (cid:1) .
In Table 6, we compare DROP+CVAE (-Grad-Ada) with LAPO (Chen et al., 2022), PLAS (Zhou et al., 2020), CQL Kumar et al.
(2020), IQL Kostrikov et al.
(2021b)
and the naive implementation of
DROP(-Grad-Ada) (conducting return-guided task decomposition and afterward learning behavior embedding as in Equation 6).
We highlight that even there is a big performance gap between DROP and baselines (LAPO and PLAS) in Gym-MuJoCo *-random tasks, our CVAE-based implementation (DROP+CVAE) can bridge such performance gap.
Further, in *-medium tasks, DROP+CVAE
[Table]
can further improve the performance (of DROP), surpassing both LAPO and PLAS that similarly adopt the CVAE to model the ofﬂine data.
Besides, we compare DROP+CVAE to baselines CQL and IQL, where we see DROP+CVAE can also achieve a competitive performance.
C. A BLATION STUDY
[Figure]
Note that our embedding inference depends on the learned score model f .
Without proper regularization, such inference will lead to out-of-distribution embeddings that are erroneously high scored ( Q2 ).
Here we conduct an ablation study to examine the impact of the conservative regularization used for learning the score model.
In Figure 8, we compare DROP-Grad and
DROP-Grad-Ada to their naive implementation ( w.o. Reg ) that ablates the regularization on halfcheetah-medium-expert.
We can ﬁnd that removing the conservative regularization leads to unstable performance when changing the update steps of gradient-based optimization.
However, we empirically ﬁnd that in some tasks such naive implementation ( w/o Reg ) does not necessarily bring unstable inference (Appendix E).
Although improper gradient update step leads to faraway embeddings, to some extent, embedding-conditioned behavior policy can correct such deviation.
D I MPLEMENTATION DETAILS
For the practical implementation of DROP, we parameterize the task embedding function φ ( z | n ) ,
the contextual behavior policy β ( a | s , z ) and the score model f
( s , a , z ) with neural networks (see
Appendix D for speciﬁc architectures).
For Equation 8, we construct a Lagrangian and solve the optimization through primal-dual gradient descent.
For the choice of µ ( z ) , we simply set µ ( z ) to be the uniform distribution over the Z -space and empirically ﬁnd that such uniform sampling can effectively avoid the out-of-distribution extrapolation at inference.
Lagrangian Relaxation.
To optimize the constrained objective in Equation 8 in the main paper, we construct a Lagrangian and solve the optimization through primal-dual gradient descent,
[Equation]
This unconstrained objective implies that if the expected difference in scores of out-of-distribution embeddings and in-distribution embeddings is less than a threshold η , λ is going to be adjusted to 0 , on the contrary, λ is likely to take a larger value, used to punish the over-estimated value function.
This objective encourages that out-of-distribution embeddings score lower than in-distribution embeddings, thus performing embedding inference will not lead to these out-of-distribution embeddings that are falsely and over-optimistically scored by the learned score model.
In our experiments, we tried ﬁve different values for the Lagrange threshold η ( 1 . 0 , 2 . 0 , 3 . 0 , 4 . 0 and 5 . 0 ).
We did not observe a signiﬁcant difference in performance across these values.
Therefore, we simply set η = 2 . 0 .
Hyper-parameters.
In Figure 9, we provide the network architecture of the task embedding φ ( z | s ) , the contextual behavior policy β ( a | s , z ) , and the score function f ( s , a , z ) , where the corresponding hyper-parameters are provided in Table 7.
For the gradient ascent update steps (used for embedding inference), we set K = 100 for all the embedding inference rules in experiments.
[Table]
In Table 9, we provide the number of sub-tasks, the number of trajectories in each sub-task, and the dimension of the embedding for each sub-task (behavior policy).
The selection of hyperparameter
N is based on two evaluation metrics: (1) the ﬁtting loss of the decomposed behavioral policies to the ofﬂine data, and (2) the testing performance of DROP.
Speciﬁcally,
• (Step1) Over a hyperparameter (the number of sub-tasks) set, we conduct the hyperparameter search using the ﬁtting loss of behavior policies, then we choose/ﬁlter the four best hyperparameters; • (Step2)
We follow the normal practice of hyperparameter selection and tune the four hypermeters selected in Step1 by interacting with the simulator to estimate the performance of DROP under each hyperparameter setting.
Require: Env, last T checkpoint models: β t ( a | s , z ) and f t ( s , a , z ) ( t = 1 , · · · , T ).
1: R MAX =
−∞ .
2: β best ← None.
3: f best ← None.
4: while t = 1 , · · · , T do 5: s
= Env.
Reset().
6: z ∗ 0 ( s 0 )
← Conduct embedding inference with DROP-Best.
Return ← Evaluate β t and f t on Env, setting π ∗ ( a | s ) =
β ( a | s , z ∗ 0 ( s 0 )) .
8: if R MAX <
Return then 9:
Update the best checkpoint models: β best ← β t , f best ← f t .
Update the optimal return: R MAX ← Return.
11: end if 12: end while
Return: β best and f best .
[Table]
We provide the hyperparameter sets in Table 8.
In Step2, we tune the (ﬁltered) hyperparameters using 1 seed, then evaluate the best hyperparameter by training on an additional 4 seeds and ﬁnally report the results on thetotal seeds (see next “evaluation protocol”).
In Antmaze domain, a single ﬁxed N works well for many tasks; while in Gym-mujoco and Adroit domains, we did not ﬁnd a ﬁxed N that provides good results for all tasks in the corresponding domain in D4RL, thus we use the above hyperparameter selection rules (Step1 and Step2) to choose the number N.
Environment details.
For the comparison of our method to prior iterative ofﬂine RL methods, we consider the v0 versions of the datasets in D4RL 8 .
We take the baseline results of BEAR, BCQ,
CQL, and BRAC-p from the D4RL paper (Fu et al., 2020), and take the results of TD3+BC from their origin paper (Fujimoto & Gu, 2021).
For the comparison of our method to prior non-iterative ofﬂine RL method, we use the v2 versions of the dataset in D4RL.
All the baseline results of behavior cloning (BC), Decision Transform (DT), RvS-R, and Onestep are taken from Emmons et al.
In our implementation of COMs, we take the parameters (neural network weights) of behavior policies as the design input for the score model; and during testing, we conduct parameters inference (outer-level optimization) with 200 steps gradient ascent over the learned score function, then the rollout policy is initialized with the inferred parameters.
For the speciﬁc architecture, we instance the policy network with dim( S ) input units, two layers with 64 hidden units, and a ﬁnal output layer with dim( A ) .
Evaluation protocol.
We evaluate our results over 5 seeds.
For each seed, instead of taking the ﬁnal checkpoint model produced by a training loop, we take the last T ( T = 6 in our experiments) checkpoint models, and evaluate them over 10 episodes for each checkpoint.
That is to say, we report the average of the evaluation scores over 5 seed × 6 checkpoint × 10 episode rollouts.
Online ﬁne-tuning (checkpoint-level): Instead of re-training the learned (ﬁnal) policy with online rollouts, we ﬁne-tune our policy with enumerated trail-and-error over the last T checkpoint models (Algorithm 3).
Speciﬁcally, for each seed, we run the last T checkpoint models in environment over one episode for each checkpoint.
The checkpoint model which achieves the maximum episode return is returned.
In essence, this ﬁne-tuning procedure imitates the online RL evaluation protocol: if the current policy is unsatisfactory, we can use checkpoints of previous iterations of the policy.
Online ﬁne-tuning (embedding-level): The embedding-level ﬁne-tuning aims to ﬁnd a suitable gradient ascent step that is used to conduct the embedding inference in DROP-Grad or DROP-Grad-Ada.
Require: Env, last T checkpoint models: β t ( a | s , z ) and f t ( s , a , z ) ( t = 1 , · · · , T ).
None.
5: while t = 1 , · ·
, T do 6: while k = 1 , · · · , K max do 7: s
= Env.
Reset().
# Conduct embedding inference with DROP-Grad or DROP-Grad-Ada 8:
Return ← Evaluate β t and f t on Env, setting π ∗ ( a | s ) =
β ( a | s , z ∗ ( s 0 )) or β ( a | s , z ∗ ( s )) , where we conduct k gradient ascent steps to obtain z ∗ ( s 0 ) or z ∗ ( s ) .
9: if R MAX < Return then 10:
Update the best checkpoint models: β best ← β t , f best ← f t .
Update the best gradient update step: k best ← k .
Update the optimal return: R MAX ← Return.
13: end if 14: end while 15: end while
Return: β best , f best and k best .
Thus, we enumerate a list of gradient update steps and pick the best update step (according to the episode returns).
Codebase.
Our code is based on d3rlpy: https://github.com/takuseno/d3rlpy .
We provide our source code in the supplementary material.
Computational resources.
The experiments were run on a computational cluster with 22x
GeForce RTX 2080 Ti, and 4x NVIDIA Tesla V100 32GB for 20 days.
[Table]
E A DDITIONAL R ESULTS
Comparison with iterative ofﬂine RL baselines.
Here, we compare the performance of DROP (Grad, Best-Ada, and Grad-Ada ) to iterative ofﬂine RL baselines (BEAR (Kumar et al., 2019a),
BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), BRAC-p (Wu et al., 2019), and
TD3+BC (Fujimoto & Gu, 2021)) that perform iterative bi-level ofﬂine RL paradigm with (explicit or implicit) value/policy regularization in inner-level.
In Table 10, we present the results for the
Maze2D, AntMaze, Gym-MuJoCo, and Adroit suites in standard D4RL benchmark (*-v0), where we can ﬁnd that DROP-Grad-Ada performs comparably or surpasses prior iterative bi-level works on most tasks: outperforming (or comparing) these policy regularized methods (BRAC-p and TD3+BC) on 25 out of 33 tasks and outperforming (or comparing) these value regularized algorithms (BEAR,
BCQ, and CQL) on 19 out of 33 tasks.
Comparison with RvS baselines.
As a complement to the results shown in Table 2 in the main paper, we provide the performance comparison for more tasks (Gym-MuJoCo and AntMaze *-v suites in D4RL) in Figure 11.
We can ﬁnd similar results as presented in our main paper: DROP consistently outperforms baselines in AntMaze tasks (the last three rows of sub-ﬁgures in Figure 11) and reaches comparable results on most tasks in Gym-MuJoCo suite.
Across different environments, we also ﬁnd DROP exhibits more robust performance.
Although baseline Onestep shows impressive performance in Gym-MuJoCo tasks, we can see that Onestep fails to make progress in AntMaze-medium-* and AntMaze-large-* tasks.
However, we ﬁnd that
DROP-based methods exhibit a signiﬁcant performance improvement in this AntMaze suite.
We attribute the success of DROP outperforming Onestep (conducting only behavior policy improvement) to three advantages: (1) DROP learns multiple behavior policies; (2) DROP conducts policy improvement (corresponding to the embedding inference procedure) over multiple behavior policies; (3) DROP permits deployment adaptation , enabling the agent to “switch” behavior policies.
[Figure]
Ablation studies.
In Figure 10, we provide more results for the ablation of the conservative regularization term in Equation 8 in the main paper.
We can ﬁnd that for the halfcheetah-medium and hopper-medium tasks, the performance of DROP-Grad-Ada w/o Reg depends on the choice of the gradient update steps, showing that too small or too large number of gradient update step deteriorates the performance.
Such result is also consistent with COMs (Trabucco et al., 2021), which also observes the sensitivity of naive gradient update ( i.e. , w/o Reg) to number of update steps used for design input inference.
By comparison, conservative score model learned with DROP-Grad-Ada exhibits more stable and robust to the gradient update steps.
Further, we also ﬁnd that in walker2d-medium and walker2d-medium-expert tasks, the naive gradient update ( w/o Reg ) does not affect performance signiﬁcantly across a wide range of gradient update steps.
The main reason is that although the excessive gradient updates lead to faraway embeddings, conditioned on the inferred embeddings, the learned contextual behavior policy can safeguard against the embeddings distribution shift.
Compared to prior model-based optimization that conducts direct gradient optimization (inference) over the design input itself, such “self-safeguard” is a special merit in the ofﬂine RL domain as long as we reframe the ofﬂine RL problem as one of model-based optimization and conduct inference over the embedding space.
Thus, we encourage the research community to pursue further study to this model-based optimization view for the ofﬂine
RL problem.
DROP results.
In Table 11, we provide our complete results (including the variance) on all tasks in the paper.
[Table]
[Figure]
T a b l e 11 :
N o r m a li
ze
d r e t u r n o f
D R O
P - G r a d ,
D R O
P - B e s tA d a , a nd
D R O
P - G r a dA d a acc r o ss
M aze 2d ,
A n t
M aze ,
M u J o
C o - G y m , a nd
[Table]
o
[Table]
u it
[Table]
s .
[Table]
O
P
[Table]
a d
[Table]
O
P
[Table]
e s t -
[Table]
a
[Table]
P
[Table]
O
P
[Table] [Table]
M
[Table]
s e 54 .±
20 .68 .±
7 .66 .±
10 .48 .±
.62 .±
16 .66 .±
12 .a n t m aze - m e d i u m - p l a y 20 .± 6 .26 .± 18 .30 .± 6 .24 .± 10 .34 .± 12 .30 .± 21 .a
n t m aze - m e d i u m - d i v e r s e 12 .± 16 .20 .±
11 .22 .±
14 .20 .± 19 .24 .± 12 .30 .± 16 .a
n t m aze - l a r g ep l a y 16 .±
18 .20 .±
.16 .± 13 .24 .± 8 .36 .±
.42 .± 17 .a
n t m aze - l a r g ed i v e r s e 20 .± 8 .22 .±
24 .22 .±
.14 .± 8 .20 .±
.26 .± 13 .
M u J o
C o - G y m h a l f c h ee t a h -r a ndo m 2 .±
.2 .±
.2 .± 
 .2 .±
.2 .± 
 .2 .±
.h
a l f c h ee t a h - m e d i u m 36 .±
.36 .±
.37 .±
.42 .±
.42 .± 0 .
[Table]
h a l f c h ee t a h - m e d i u m - e xp e r t 85 .± 15 .89 .± 11 .96 .± 7 .
[Table]
[Table] [Table]
h a l f c h ee t a h - m e d i u m -r e p l a y 32 .± 8 .36 .± 3 .
[Table]
33 .± 6 .
[Table]
hopp e r-r a ndo m 11 .±
.11 .±
.11 .± 0 .5 .± 0 .
[Table]
[Table]
hopp e r- m e d i u m 45 .± 14 .
[Table]
±
[Table] [Table]
±
[Table] [Table]
[Table]
hopp e r- m e d i u m - e xp e r t 111 .± 1 .112 .±
.112 .± 0 .
[Table] [Table]
[Table]
hopp e r- m e d i u m -r e p l a y 31 .± 6 .32 .±
3 .32 .±
4 .48 .± 17 .
[Table]
w a l k e r 2d -r a ndo m 3 .±
.±
.4 .±
.2 .±
1 .3 .± 1 .3 .±
1 .w
a l k e r 2d - m e d i u m 19 .± 13 .26 .± 8 .24 .± 9 .76 .±
.75 .±
3 .79 .±
1 .w
a l k e r 2d - m e d i u m - e xp e r t 84 .±
.91 .± 10 .91 .± 5 .107 .±
2 .106 .± 3 .106 .± 3 .w
a l k e r 2d - m e d i u m -r e p l a y 13 .± 4 .13 .±
.16 .±
7 .37 .± 13 .60 .±
7 .61 .± 2 .
A d r o it doo r-
c l on e d 0 .±
.2 .± 2 .2 .±
3 .doo r-
e xp e r t 98 .± 5 .102 .± 2 .102 .± 1 .doo
rhu m a n 3 .± 4 .1 .± 2 .3 .±
2 .h a mm e rc l on e d 0 .± 0 .0 .± 0 .0 .±
.h
a mm e re xp e r t 65 .± 26 .73 .±
.77 .± 25 .h
a mm e rhu m a n 1 .± 1 .0 .±
.2 .±
3 .p
e nc l on e d 76 .± 15 .77 .±
.82 .±
36 .p
e ne xp e r t 113 .± 10 .118 .± 13 .116 .±
23 .p
e nhu m a n 71 .± 34 .85 .± 13 .81 .± 16 .r
e l o ca t ec l on e d 0 .± 0 .0 .± 0 .0 .± 0 .r
e l o ca t ee xp e r t 2 .± 3 .6 .±
4 .5 .±
3 .r
e l o ca t ehu m a n 0 .± 0 .0 .± 0 .0 .± 0 .