DARA: D YNAMICS -A WARE R EWARD A UGMENTATION IN O FFLINE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Ofﬂine reinforcement learning algorithms promise to be applicable in settings where a ﬁxed dataset is available and no new experiment can be acquired.
However, such formulation is inevitably ofﬂine-data-hungry and, in practice, collecting a large ofﬂine dataset for one speciﬁc task over one speciﬁc environment is also costly and laborious.
In this paper, we thus 1) formulate the ofﬂine dynamics adaptation by using (source) ofﬂine data collected from another dynamics to relax the requirement for the extensive (target) ofﬂine data, 2) characterize the dynamics shift problem in which prior ofﬂine methods do not scale well, and 3) derive a simple dynamics-aware reward augmentation (DARA) framework from both modelfree and model-based ofﬂine settings.
Speciﬁcally, DARA emphasizes learning from those source transition pairs that are adaptive for the target environment and mitigates the ofﬂine dynamics shift by characterizing state-action-next-state pairs instead of the typical state-action distribution sketched by prior ofﬂine RL methods.
The experimental evaluation demonstrates that DARA, by augmenting rewards in the source ofﬂine dataset, can acquire an adaptive policy for the target environment and yet signiﬁcantly reduce the requirement of target ofﬂine data.
With only modest amounts of target ofﬂine data, our performance consistently outperforms the prior ofﬂine RL methods in both simulated and real-world tasks.
I NTRODUCTION
[Figure]
Ofﬂine reinforcement learning (RL) (Levine et al., 2020; Lange et al., 2012), the task of learning from the previously collected dataset, holds the promise of acquiring policies without any costly active interaction required in the standard online RL paradigm.
However, we note that although the active trail-and-error (online exploration) is eliminated, the performance of ofﬂine RL method heavily relies on the amount of ofﬂine data that is used for training.
As shown in Figure 1, the performance deteriorates dramatically as the amount of ofﬂine data decreases.
A natural question therefore arises: can we reduce the amount of the (target) ofﬂine data without signiﬁcantly affecting the ﬁnal performance for the target task?
Bringing the idea from the transfer learning (Pan & Yang, 2010), we assume that we have access to another (source) ofﬂine dataset, hoping that we can leverage this dataset to compensate for the performance degradation caused by the reduced (target) ofﬂine dataset.
In the ofﬂine setting, previous work (Siegel et al., 2020; Chebotar et al., 2021) has characterized the reward (goal) difference between the source and target, relying on the ”conﬂicting” or multi-goal ofﬂine dataset (Fu et al., 2020), while we focus on the relatively unexplored transition dynamics difference between the source dataset and the target environment.
Meanwhile, we believe that this dynamics shift is not arbitrary in reality: in healthcare treatment, ofﬂine data for a particular patient is often limited, whereas we can obtain diagnostic data from other patients with the same case (same reward/goal) and there often exist individual differences between patients (source dataset with different transition dynamics).
Careful treatment with respect to the individual differences is thus a crucial requirement.
Given source ofﬂine data, the main challenge is to cope with the transition dynamics difference, i.e. , strictly tracking the state-action supported by the source ofﬂine data can not guarantee that the same transition (state-action-next-state) can be achieved in the target environment.
However, in the ofﬂine setting, such dynamics shift is not explicitly characterized by the previous ofﬂine RL methods, where they typically attribute the difﬁculty of learning from ofﬂine data to the state-action distribution shift (Chen & Jiang, 2019; Liu et al., 2018).
The corresponding algorithms (Fujimoto et al., 2019;
Abdolmaleki et al., 2018; Yu et al., 2020) that model the support of state-action distribution induced by the learned policy, will inevitably suffer from the transfer problem where dynamics shift happens.
Our approach is motivated by the well established connection between reward modiﬁcation and dynamics adaptation (Kumar et al., 2020b; Eysenbach & Levine, 2019; Eysenbach et al., 2021), which indicates that, by modifying rewards, one can train a policy in one environment and make the learned policy to be suitable for another environment (with different dynamics).
Thus, we propose to exploit the joint distribution of state-action-next-state : besides characterizing the state-action distribution shift, we additionally identify the dynamics ( i.e. , conditional distribution of next-state given state-action pair) shift and penalize the agent with a dynamics-aware reward modiﬁcation.
This modiﬁcation aims to discourage the learning from these ofﬂine transitions that are likely in source but are unlikely in target.
Unlike the concurrent work (Ball et al., 2021; Mitchell et al., 2021) paying attention to the ofﬂine domain generalization, we explicitly focus on the ofﬂine domain (dynamics) adaptation.
Our principal contribution in this work is the characterization of the dynamics shift in ofﬂine RL and the derivation of dynamics-aware reward augmentation (DARA) framework built on prior modelfree and model-based formulations.
DARA is simple and general, can accommodate various ofﬂine
RL methods, and can be implemented in just a few lines of code on top of dataloader at training.
In our ofﬂine dynamics adaptation setting, we also release a dataset, including the Gym-MuJoCo tasks (Walker2d , Hopper and HalfCheetah), with dynamics (mass, joint) shift compared to D4RL, and a 12-DoF quadruped robot in both simulator and real-world.
With only modest amounts of target ofﬂine data, we show that DARA-based ofﬂine methods can acquire an adaptive policy for the target tasks and achieve better performance compared to baselines in both simulated and real-world tasks.  R ELATED W ORK
Ofﬂine RL describes the setting in which a learner has access to only a ﬁxed dataset of experience, while no interactive data collection is allowed during policy learning (Levine et al., 2020).
Prior work commonly assumes that the ofﬂine experience is collected by some behavior policies on the same environment that the learned policy be deployed on.
Thus, the main difﬁculty of such ofﬂine setting is the state-action distribution shift (Fujimoto et al., 2019; Liu et al., 2018). Algorithms address this issue by following the two main directions: the model-free and model-based ofﬂine RL. Model-free methods for such setting typically fall under three categories: 1) Typical methods mitigate this problem by explicitly (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019) or implicitly (Siegel et al., 2020; Peng et al., 2019; Abdolmaleki et al., 2018) constraining the learned policy away from OOD state-action pairs.
Conservative estimation based methods learn pessimistic value functions to prevent the overestimation (Kumar et al., 2020a;
Xu et al., 2021).
Importance sampling based methods directly estimate the state-marginal importance ratio and obtain an unbiased value estimation (Zhang et al., 2020; Nachum & Dai, 2020; Nachum et al., 2019b).
Model-based methods typically eliminate the state-action distribution shift by incorporating a reward penalty, which relies on the uncertainty quantiﬁcation of the learned dynamics (Kidambi et al., 2020;
Yu et al., 2020).
To remove this uncertainty estimation, Yu et al.
(2021) learns conservative critic function by penalizing the values of the generated state-action pairs that are not in the ofﬂinedataset.
These methods, however, deﬁne their objective based on the state-action distribution shift, and ignore the potential dynamics shift between the ﬁxed ofﬂine data and the target MDP.
In contrast, we account for dynamics (state-action-next-state) shift and explicitly propose the dynamics aware reward augmentation.
A counterpart, close to our work, is off-dynamics RL (Eysenbach et al., 2021), where they set up dynamics shift in the interactive environment while we focus on the ofﬂine setting.
P RELIMINARIES
We study RL in the framework of Markov decision processes (MDPs) speciﬁed by the tuple M :=
( S , A , r, T, ρ 0 , γ ) , where S and A denote the state and action spaces, r ( s , a ) ∈ [ − R max , R max ] is the reward function, T ( s (cid:48) | s , a ) is the transition dynamics, ρ 0 ( s ) is the initial state distribution, and γ is the discount factor.
The goal in RL is to optimize a policy π ( a | s ) that maximizes the expected discounted return η M ( π ) :
=
E τ ∼ p πM ( τ ) [ (cid:80) ∞ t =
γ t r ( s t , a t )] , where τ := ( s 0 , a 0 , s 1 , a 1 , ... ) .
We also deﬁne Q-values Q ( s , a ) :
=
E τ ∼ p πM ( τ ) [ (cid:80) ∞ t =
γ t r ( s t , a t ) | s 0 =
s , a 0 = a ] , V-values V ( s ) :=
E a ∼ π ( a | s )
[ Q ( s , a )] , and the (unnormalized) state visitation distribution d πM ( s ) :
= (cid:80) ∞
t =0 γ t P ( s | π,
M, t ) , where P ( s | π, M, t ) denotes the probability of reach state s at time t by running π in M .
In the ofﬂine RL problem, we are provided with a static dataset D := { ( s , a , r, s (cid:48) ) } , which consists of transition tuples from trajectories collected by running one or more behavioral policies, denoted by π b , on MDP M .
With a slight abuse of notation, we write D = { ( s , a , r, s (cid:48) )
∼ d D ( s ) π b ( a | s ) r
( s , a ) T ( s (cid:48) | s , a ) } , where the d D ( s ) denotes state-marginal distribution in D .
In the ofﬂine setting, the goal is typically to learn the best possible policy using the ﬁxed ofﬂine dataset.
Model-free RL algorithms based on dynamic programming typically perform policy iteration to ﬁnd the optimal policy.
Such methods iteratively conduct 1) policy improvement with G M Q := arg max π E s ∼ d πM ( s ) , a ∼ π ( a | s )
[ Q ( s , a )] and 2) policy evaluation by iterating the Bellman equation
Q ( s , a ) =
B πM Q ( s , a ) :
= r ( s , a ) + γ E s (cid:48) ∼ T ( s (cid:48) | s , a ) , a (cid:48) ∼ π ( a (cid:48) | s (cid:48) )
[ Q ( s (cid:48) , a (cid:48) )] over d πM ( s ) π ( a | s ) .
Given off-policy data D , we resort to 1) improvement with G D Q := arg max π E s ∼ d D ( s ) , a ∼ π ( a | s )
[ Q ( s , a )] and 2) evaluation by iterating Q ( s , a )
=
B π D Q ( s , a ) :
= r ( s , a )+ γ E s (cid:48) ∼ T ( s (cid:48) | s , a ) , a (cid:48) ∼ π ( a (cid:48) | s (cid:48) )
[ Q ( s (cid:48) , a (cid:48) )] over all ( s , a ) in D .
Speciﬁcally, given any initial Q 0 , it iterates
Policy improvement: π k +1 =
G D Q k ,
Policy [-evaluation: Q k +
-]evaluation
[Equation]
Model-free ofﬂine RL based on the above iteration suffers from the state-action distribution shift, i.e. , policy evaluation B π k D Q k − 1 may encounter unfamiliar state action regime that is not covered by the ﬁxed ofﬂine dataset D , causing erroneous estimation of Q k . Policy improvement G D Q k further exaggerates such error, biasing policy π k +1 towards out-of-distribution (OOD) actions with erroneously high Q-values.
To address this distribution shift, prior works 1) explicitly constrain policy to be close to the behavior policy (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019;
Ghasemipour et al., 2021), introducing penalty αD ( π ( a | s ) , π b ( a | s )) into G D or B π D in Equation 1:
[Equation]
where D is a divergence function between distributions over actions ( e.g. , MMD or KL divergence), or 2) train pessimistic value functions (Kumar et al., 2020a
; Yu et al., 2021; Xu et al., 2021), penalizing Q-values at states in the ofﬂine dataset D for actions generated by the current policy π :
[Equation]
Model-based RL algorithms iteratively 1) model the transition dynamics T ( s (cid:48) | s , a ) , using the data collected in M : max ˆ T E s , a , s (cid:48)
∼ d π M ( s ) π ( a | s ) T ( s (cid:48) | s , a ) [log ˆ T ( s (cid:48) | s , a )] , and 2) infer a policy π from the modeled ˆ M = ( S , A , r, ˆ T, ρ 0 , γ ) , where we assume that r and ρ 0 are known, maximizing η ˆ M ( π ) with a planner or the Dyna-style algorithms (Sutton, 1990).
In this paper, we focus on the latter.
Model-based ofﬂine RL algorithms similarly suffer from OOD state-action (Kidambi et al., 2020;
Cang et al., 2021) if we directly apply policy iteration over ˆ T :=
max ˆ T E s , a , s (cid:48)
∼D [log ˆ T ( s (cid:48) | s , a )] .
Like the conservative estimation approach described in Equation 3, recent conservative model-based ofﬂine RL methods provide the policy with a penalty for visiting states under the estimated ˆ T where ˆ T is likely to be incorrect.
Taking u ( s , a ) as the oracle uncertainty (Yu et al., 2020) that provides a consistent estimate of the accuracy of model ˆ T at ( s , a ) , we can modify the reward function to obtain a conservative MDP: ˆ M c = ( S , A , r − αu, ˆ T, ρ 0 , γ ) , then learn a policy π by maximizing η ˆ M c ( π ) .
P ROBLEM F ORMULATION
In standard ofﬂine RL problem, the static ofﬂine dataset D consists of samples { ( s , a , r, s (cid:48) ) ∼ d D ( s ) π b ( a | s ) r ( s , a ) T ( s (cid:48) | s , a ) } .
Although ofﬂine RL methods learn policy for the target MDP
Q k ( s ,
) − Q ( s ,
M :=
( S , A , r, T, ρ 0 , γ ) without (costly) online data, as we shown in Figure 1, it requires a fair amount of (target) ofﬂine data D collected on M .
Suppose we have another (source) ofﬂine dataset
D (cid:48) , consisting of samples { ( s , a , r, s (cid:48) ) ∼ d D (cid:48) ( s ) π b (cid:48) ( a | s ) r ( s , a ) T (cid:48) ( s (cid:48) | s , a ) } collected by the behavior policy π b (cid:48) on MDP M (cid:48) :
= ( S , A , r, T (cid:48) , ρ 0 , γ ) , then we hope the transfer of knowledge between ofﬂine dataset {D (cid:48) ∪D} can reduce the data requirements on D for learning policy for the target M .
4. D YNAMICS S HIFT IN O FFLINE RL
Although ofﬂine RL methods in Section 3 have incorporate the state-action distribution constrained backups (policy constraints or conservative estimation), they also fail to learn an adaptive policy for the target MDP M with the mixed datasets {D (cid:48) ∪ D} , as we show in Figure 4 (Appendix).
We attribute this failure to the dynamics shift (Deﬁnition 2) between D (cid:48) and M in this adaptation setting.
Deﬁnition 1 (Empirical MDP)
An empirical MDP estimated from D is ˆ M :=
( S , A , r, ˆ T, ρ 0 , γ ) where ˆ T = max ˆ T E s , a , s (cid:48)
∼D [log ˆ T ( s (cid:48) | s , a )] and ˆ T ( s (cid:48) | s , a ) = [-
-]0 for all ( s , a , s (cid:48) ) not in dataset D .
Deﬁnition 2 (Dynamics shift) Let ˆ M :=
( S , A , r, ˆ T, ρ 0 , γ ) be the empirical MDP estimated from
D .
To evaluate a policy π for M :
= ( S , A , r, T, ρ 0 , γ ) with ofﬂine dataset D , we say that the dynamics shift (between D and M ) in ofﬂine RL happens if there exists at least one transition pair ( s , a , s (cid:48) )
∈ { ( s , a , s (cid:48) ) :
d π ˆ M ( s ) π ( a | s ) ˆ T ( s (cid:48) | s , a ) > 0 } such that ˆ T ( s (cid:48) | s , a ) (cid:54) =
T ( s (cid:48) | s , a ) .
In practice, for a stochastic M and any ﬁnite ofﬂine data D collected in M , there always exists the dynamics shift.
The main concern is that ﬁnite samples are always not sufﬁcient to exactly model stochastic dynamics.
Following Fujimoto et al.
, we thus assume both MDPs M and M (cid:48) are deterministic, which means the empirical ˆ M and ˆ M (cid:48) are both also deterministic.
More importantly, such assumption enables us to explicitly characterize the dynamics shift under ﬁnite ofﬂine samples.
Lemma 1 Under deterministic transition dynamics, there is no dynamics shift between D and M .
For ofﬂine RL tasks, prior methods generally apply B π D Q along with the state-action distribution correction (Equations 2 and 3), which overlooks the potential dynamics shift between the (source) ofﬂine dataset and the target MDP ( e.g. , D (cid:48)
→ M ).
As a result, these methods do not scale well to the setting in which dynamics shift happens, e.g. , learning an adaptive policy for M with (source) D (cid:48) .
4. D YNAMICS S HIFT IN M ODEL - FREE AND M ODEL - BASED O FFLINE F ORMULATIONS
From the model-free (policy iteration) view, an exact policy evaluation on M is characterized by iterating Q ( s , a ) =
B πM Q ( s , a ) for all ( s , a ) such that d πM ( s ) π ( a | s ) > 0 .
Thus, to formalize the policy evaluation with ofﬂine D or D (cid:48) (for an adaptive π on target M ), we require that Bellman operator
B π D Q ( s , a ) or B π D (cid:48) Q ( s , a ) approximates the oracle B πM Q ( s , a ) for all ( s , a ) in S π or S (cid:48) π , where S π and S (cid:48)
π denote the sets { ( s , a ) :
d D ( s ) π ( a | s ) > 0 } and { ( s , a ) : d D (cid:48) ( s ) π ( a | s ) > 0 } respectively.
1) To evaluate a policy π for M with D ( i.e. , calling the Bellman operator B π D ), notable modelfree ofﬂine method BCQ (Fujimoto et al., 2019) translates the requirement of B π D = B πM into the requirement of ˆ T ( s (cid:48) | s , a )
=
T ( s (cid:48) | s , a ) .
Note that under deterministic environments, we have the property that for all ( s , a , s (cid:48) ) in ofﬂine data D , ˆ T ( s (cid:48) | s , a )
=
T ( s (cid:48) | s , a ) (Lemma 1).
As a result, such property permits BCQ to evaluate a policy π by calling B π D , replacing the oracle B πM , meanwhile constraining S π to be a subset of the support of d D ( s ) π b ( a | s ) .
This means a policy π which only traverses transitions contained in (target) ofﬂine data D , can be evaluated on D without error.
2) To evaluate a policy π for M with D (cid:48) ( i.e. , calling the Bellman operator B π D (cid:48) ), we have lemma 2:
Lemma 2 Dynamics shift produces that B π D (cid:48) Q ( s , a ) (cid:54) =
B πM Q ( s , a ) for some ( s , a ) in S (cid:48) π .
With the ofﬂine data D (cid:48) , lemma 2 suggests that the above requirement B π D (cid:48)
=
B πM becomes infeasible, which limits the practical applicability of prior ofﬂine RL methods under the dynamics shift.
To be speciﬁc, characterizing an adaptive policy for target MDP M with D (cid:48) moves beyond the reach of the off-policy evaluation based on iterating Q = B π D (cid:48)
Q (Equations 2 and 3).
Such iteration may cause the evaluated Q (or learned policy π ) overﬁts to ˆ T (cid:48) and struggle to adapt to the target T .
To overcome the dynamics shift, we would like to resort an additional compensation ∆ ˆ T (cid:48) ,T such that
[Equation]
4) for all ( s , a ) in S (cid:48) π .
Thus, we can apply B π D (cid:48) Q + ∆ ˆ T (cid:48) ,T to act as a substitute for the oracle B π M Q .
From the model-based view, the oracle η M ( π ) (calling the Bellman operator B πM on the target M ) and the viable η ˆ M (cid:48) ( π ) (calling B π ˆ M (cid:48) on the estimated ˆ M (cid:48) from source D (cid:48) ) have the followinglemma.
Lemma 3 Let B πM V ( s ) =
E a ∼ π ( a | s ) [Equation] r
[Equation] s , a ) + γ E s
[Equation]
∼ T ( s
[Equation]
| s , a ) [ V [Equation] s [Equation]
[Equation]
For any π , we have:
[Equation]
Lemma 3 states that if we maximize η ˆ M (cid:48) ( π ) subject to | E s ∼ d π ˆ M [Equation] s [Equation]
[ B π ˆ M
[Equation]
V M ( s ) − B πM V M ( s )]
| ≤ (cid:15) , η M ( π ) will be improved.
If F is a set of functions f : S → R that contains V M , then we have
[Equation]
E
[Equation]
where d F ( ˆ T (cid:48) ( s (cid:48)
[Equation]
[Equation]
[Equation]
s
[Equation]
| s ,
[Equation]
[ f
[Equation]
s
[Equation]
)] | , which is the integral probability metric (IPM).
Note that if we directly follow the admissible error assumption in MOPO (Yu et al., 2020) i.e. , assuming d F ( ˆ T (cid:48) ( s (cid:48) | s , a ) , T ( s (cid:48) | s , a )) ≤ u ( s , a ) for all ( s , a ) , this would be too restrictive: given that ˆ T (cid:48) is estimated from the source ofﬂine samples collected under
T (cid:48) , not the target T , thus such error would not decrease as the source data increases.
Further, we ﬁnd
[Equation]
Thus, we can bound the d F ( ˆ T (cid:48) , T ) term with the admissible error assumption over d F ( ˆ T, T ) , as in
MOPO, and the auxiliary constraints d F ( ˆ T (cid:48) , ˆ T ) .
See next section for the detailed implementation.
In summary , we show that both prior ofﬂine model-free and model-based formulations suffer from the dynamics shift, which also suggests we to learn a modiﬁcation ( ∆ or d F ) to eliminate this shift.
D YNAMICS -A WARE R EWARD A UGMENTATION
In this section, we propose the dynamics-aware reward augmentation (DARA), a simple data augmentation procedure based on prior (model-free and model-based) ofﬂine RL methods.
We ﬁrst provide an overview of our ofﬂine reward augmentation motivated by the compensation ∆ ˆ T (cid:48) ,T in
Equation 4 and the auxiliary constraints d F ( ˆ T (cid:48) , ˆ T ) in Equation 6, and then describe its theoretical derivation in both model-free and model-based formulations.
With the (reduced) target ofﬂine data
D and the source ofﬂine data D (cid:48) , we summarize the overall DARA framework in Algorithm 1. Require: Target ofﬂine [Table] D
[Table]
and source ofﬂine
[Table] D [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
5. D YNAMICS -A WARE R EWARD A UGMENTATION IN M ODEL - FREE F ORMULATION
Motivated by the well established connection of RL and probabilistic inference (Levine, 2018), we ﬁrst cast the model-free RL problem as that of inference in a particular probabilistic model.
Speciﬁcally, we introduce the binary random variable O that denotes whether the trajectory τ :=
( s 0 , a 0 , s 1 , ... ) is optimal ( O = 1 ) or not ( O = 0 ).
The likelihood of a trajectory can then be modeled as p ( O = 1 | τ ) =
exp ( (cid:80) t r t /η ) , where r t :=
r
( s t , a t ) and η > 0 is a temperature parameter.
(Reward Augmentation with Explicit Policy/Value Constraints)
We now introduce a variational distribution p π ˆ M (cid:48)
( τ ) =
p ( s 0 ) (cid:81) t =1 ˆ T (cid:48) ( s t +1 | s t , a t ) π ( a t | s t ) to approximate the posterior distribution p π M ( τ |O = 1) , which leads to the evidence lower bound of log p π M ( O = 1) :
[Equation]
Since we are interested in inﬁnite horizon problems, we introduce the discount factor γ and take the limit of steps in each rollout, i.e. , H → ∞ .
Thus, the RL problem on the MDP M , cast as the inference problem arg max π log p πM ( O = 1) , can be stated as a maximum of the lower bound
E τ ∼ p
π ˆ M (cid:48) ( τ ) (cid:104)(cid:80) ∞ t =
γ t (cid:16) r t − η log ˆ T (cid:48)
( s t +1 | s t , a t ) T ( s t +1 | s t , a t ) (cid:17)(cid:105) .
This is equivalent to an RL problem on ˆ M (cid:48) with the augmented reward r ← r ( s , a ) − η log ˆ T (cid:48) ( s (cid:48) | s , a ) T ( s (cid:48) | s , a ) .
Intuitively, the − η log ˆ T (cid:48)
( s (cid:48) | s , a ) T ( s (cid:48) | s , a ) term discourages transitions (state-action-next-state) in D (cid:48) that have low transition probability in the target M .
In the model-free ofﬂine setting, we can add the explicit policy or Q-value constraints (Equations2 and 3) to mitigate the OOD state-actions.
Thus, such formulation allows the oracle B πM to be reexpressed by B π D (cid:48) and
the modiﬁcation log ˆ T (cid:48) T , which makes the motivation in Equation 4 practical.
(Reward Augmentation with Implicit Policy Constraints)
If we introduce the variational distribution p π (cid:48)
ˆ M (cid:48)
( τ ) :
= p ( s 0 ) (cid:81) t =1 ˆ T (cid:48) ( s t +1 | s t , a t ) π (cid:48) ( a t | s t ) , we can recover the weighted-regressionstyle (Wang et al., 2020; Peng et al., 2019; Abdolmaleki et al., 2018; Peters et al., 2010) objective by maximizing
[-E τ ∼ p π (cid:48) ˆ M (cid:48) ( τ ) (cid:104)(cid:80) ∞ t =
 γ t (cid:16) r t − η log ˆ T (cid:48) ( s t +1 | s t , a t ) T ( s t +1 | s t , a t ) − η log π (cid:48) ( a t | s t ) π-][Equation]
(lower bound of log p πM ( O = 1) ).
Following the Expectation Maximization (EM) algorithm, we can maximize J ( π (cid:48) , π ) by iteratively (E-step) improving J ( π (cid:48) , · ) w.r.t.
π (cid:48) and (M-step) updating π w.r.t.
π (cid:48) .
(E-step)
We deﬁne ˜ Q ( s , a , s (cid:48) )
=
E τ ∼ p π
[Equation]
M
[Equation]
s
[Equation]
[-(cid:48) ( s (cid:48) | s , a ) T ( s (cid:48) | s , a ) | s 
-]
[-= s , a 0 = a , s 
-]
= s (cid:48) (cid:105) .
Then, given ofﬂine data D (cid:48) , we can rewrite J ( π (cid:48) , · ) as a constrained objective (Abdolmaleki et al., 2018): max
[Equation]
When considering a ﬁxed π , the above optimization over π (cid:48) can be solved analytically (Vieillard et al., 2020;
Geist et al., 2019; Peng et al., 2019).
The optimal π (cid:48)∗ is then given by π (cid:48)∗ ( a | s ) ∝ π ( a | s ) exp ( Q ( s , a )) exp( − η ˜ Q
( s , a , ˆ T (cid:48)
( s (cid:48) | s , a ))) .
As the policy evaluation in Equation 1 (Footnote2), we estimate Q ( s , a ) and ˜ Q ( s , a , s (cid:48) ) by minimizing the Bellman error with ofﬂine samples in D (cid:48) .
(M-step)
Then, we can project π (cid:48)∗ onto the manifold of the parameterized π :
[Equation]
From the regression view, prior work MPO (Abdolmaleki et al., 2018) infers actions with Q-value weighted regression, progressive approach compared to behavior cloning, however, such paradigm lacks the ability to capture transition dynamics.
We explicitly introduce the exp( − η ˜ Q ( s , a , s (cid:48) )) term, which as we show in experiments, is a crucial component for eliminating the dynamics shift.
Implementation:
In practice, we adopt ofﬂine samples in D to approximate the true dynamics T of
M , and introduce a pair of binary classiﬁers, q sas ( ·| s , a , s (cid:48) ) and q sa ( ·| s , a ) , to replace log ˆ T (cid:48) ( s (cid:48) | s , a ) T ( s (cid:48) | s , a ) as in Eysenbach et al.
(2021): log ˆ T (cid:48)
( s (cid:48) | s , a ) T ( s (cid:48) | s , a ) = log q sas ( source | s , a , s (cid:48) )
q sas ( target | s , a , s (cid:48) )
− log q sa ( source | s , a ) q sa ( target | s , a ) .
(See AppendixA.1.3 for details).
Although the amount of data D sampled from the target M is reduced in our problem setup, we experimentally ﬁnd that such classiﬁers are sufﬁcient to achieve good performance.
5. D YNAMICS -A WARE R EWARD A UGMENTATION IN M ODEL - BASED F ORMULATION
Following Equation 6, we then characterize the dynamics shift compensation term as in the above model-free analysis in the model-based ofﬂine formulation.
We will ﬁnd that across different derivations, our reward augmentation ∆ r has always maintained the functional consistency and simplicity.
Following MOPO, we assume F = { f : (cid:107) f (cid:107) ∞ ≤ 1 } , then we have d F ( ˆ T (cid:48)
( s (cid:48) | s , a ) , ˆ T ( s (cid:48) | s , a )) =
D TV ( ˆ T (cid:48) ( s (cid:48) | s , a ) , ˆ T ( s (cid:48) | s , a )) ≤ ( D KL ( ˆ T (cid:48)
( s (cid:48) | s , a ) , ˆ T ( s (cid:48) | s , a )) / 2) 12 , where D TV is the total variance distance.
Then we introduce the admissible error u ( s , a ) such that d F ( ˆ T ( s (cid:48) | s , a ) , T ( s (cid:48) | s , a )) ≤ u ( s , a ) for all ( s , a ) , and η and δ such that ( D KL ( ˆ T (cid:48) , ˆ T ) / 2) 12 ≤ ηD KL ( ˆ T (cid:48) , ˆ T ) + δ .
Following
Lemma 3, we thus can maximize the following lower bound with the samples in ˆ M (cid:48)
( λ := γR max 1 − γ ):
[Equation]
Implementation:
We model the dynamics ˆ T (cid:48) and ˆ T with an ensemble of 2*N parameterized Gaussian distributions: N i ˆ T (cid:48)
( µ θ (cid:48) ( s , a ) , Σ φ (cid:48) ( s , a )) and N i ˆ T ( µ θ ( s , a ) , Σ φ ( s , a )) , where i ∈ [1 , N ] .
We approximate u with the maximum standard deviation of the learned models in the ensemble: u ( s , a ) = max Ni =1 (cid:107) Σ φ ( s , a ) (cid:107) F , omit the training-independent δ , and treat λ as a hyperparameter as in MOPO.
For the log ˆ T (cid:48)
ˆ T term, we resort to the above classiﬁers ( q sas and q sa ) in model-free setting.
(See Appendix-A.3.2 for comparison between using classiﬁers and estimated-dynamics ratio.)
5. R EGULARIZATION V IEW OF D YNAMICS -A WARE R EWARD A UGMENTATION
Here we shortly characterize our dynamics-aware reward augmentation from the density regularization.
Note the standard max-return objective η M ( π ) in RL can be written exclusively in terms of the on-policy distribution d πM ( s ) π ( a | s ) .
To introduce an off-policy distribution d D ( s ) π b ( a | s ) in the objective, prior works often incorporate a regularization (penalty): D ( d πM ( s ) π ( a | s ) (cid:107) d D ( s ) π b ( a | s )) , as in Equations 2 and 3.
However, facing dynamics shift, such regularization should take into account the transition dynamics, which is penalizing D ( d π M ( s ) π ( a | s ) T ( s (cid:48) | s , a ) (cid:107) d D (cid:48)
( s ) π b (cid:48) ( a | s ) ˆ T (cid:48)
( s (cid:48) | s , a )) .
From this view, it is also straightforward to derive the reward modiﬁcation built on prior regularized off-policy max-return objective 2 e.g. , the off-policy approach AlgaeDICE (Nachum et al., 2019b).
E XPERIMENTS
We present empirical demonstrations of our dynamics-aware reward augmentation (DARA) in a variety of settings.
We start with two simple control experiments that illustrate the signiﬁcance of
DARA under the domain (dynamics) adaptation setting.
Then we incorporate DARA into state-ofthe-art (model-free and model-based) ofﬂine RL methods and evaluate the performance on the D4RL tasks.
Finally, we compare our framework to several cross-domain-based baselines on simulated and real-world tasks.
Note that for the dynamics adaptation, we also release a (source) dataset as a complement to D4RL, along with the quadruped robot dataset in simulator (source) and real (target).
6. H OW DOES DARA HANDLE THE DYNAMICS SHIFT IN OFFLINE SETTING ?
[Figure]
the tra- that are generated by the learned policy
[Figure]
Here we characterize both external and internal dynamics shifts:
In Map tasks (Figure 2 left), the source dataset D (cid:48) is collected in a 2D map and the target D is collected in the same environment but with an obstacle (the dashed line);
In Ant tasks (Figure 3 left), the source dataset D (cid:48) is collected using the Mujoco Ant and the target D is collected with the same Ant but one joint of which is restricted.
[Table]
Using MPO, as an example of ofﬂine RL method, we train a policy on dataset {D (cid:48) ∪ D} and deploy the acquired policy in both source and target MDPs.
As shown in Figure 2 (middle-top, w/o Aug. ), such training paradigm does not produce an adaptive policy for the target.
By modifying rewards in source D (cid:48) , we show that applying the same training paradigm on the reward augmented data exhibits a positive transfer ability in Figure 2 (middle-bottom, DARA ).
In Figure 2 (right), we show that
our
DARA produces low Q-values on the obstructive state-action pairs (in left) compared to the vanilla
MPO, which thus prevents the Q-value weighted-regression on these unproductive state-action pairs.
More generally, we illustrate how DARA can handle the dynamics adaptation from the reward modiﬁcation view.
In Figure 3 (right), the learned reward modiﬁcation − ∆ r (dashed blue line) clearly produces a penalty (red slices) on these state-action pairs (in source) that produce infeasible nextstate transitions in the target MDP.
If we directly apply prior ofﬂine RL methods, these transitions that are beyond reach in target and yet are high valued, would yield a negative transfer.
Thus, we can think of DARA as ﬁnding out these transitions that exhibit dynamics shifts and enabling dynamics adaptation with reward modiﬁcations, e.g. , penalizing transitions covered by red slices ( − ∆ r < 0 ).
6. C AN DARA ENABLE AN ADAPTIVE POLICY WITH REDUCED OFFLINE DATA IN TARGET ?
To characterize the ofﬂine dynamics shift, we consider the Hopper, Walker2d and Halfcheetah from the Gym-MuJoCo environment, using ofﬂine samples from D4RL as our target ofﬂine dataset.
For the source dataset, we change the body mass of agents or add joint noise to the motion, and, similar to D4RL, collect the Random, Medium, Medium-R and Medium-E ofﬂine datasets for the three environments.
Based on various ofﬂine RL algorithms (BEAR, BRAC-p, BCQ, CQL, AWR, MOPO), we perform the following comparisons: 1) employing the 100% of D4RL data ( 10T ), 2) employing only 10% of the D4RL data ( 1T ), 3) employing 10% of the D4RL data and 100% of our collected source ofﬂine data ( 1T+10S w/o Aug. ), and 4) employing 10% of the D4RL data and 100% of our collected source ofﬂine data along with our reward augmentation ( 1T+10S DARA ).
Due to page limit, here we focus on the dynamics shift concerning the body mass on Walker2d and Hopper.
We refer the reader to appendix for more experimental details, tasks, and more baselines (BC, COMBO).
As shown in Table 1, in most of the tasks, the performance degrades substantially when we decrease the amount of target ofﬂine data, i.e. , 10T → 1T .
Training with additional ten times source ofﬂine data ( 1T+10S w/o Aug. ) also does not bring substantial improvement (compensating for the reduced data in target), which even degrades the performance in some tasks.
We believe that such degradation (compared to 10T ) is caused by the lack of target ofﬂine data as well as the dynamics shift (induced
[Table]
by the source data).
Incorporating our reward augmentation, we observe that compared to 1T and 1T+10S w/o Aug. that both use 10% of the target ofﬂine data, our 1T+10S DARA signiﬁcantly improves the performance across a majority of tasks.
Moreover, DARA can achieve comparable or better performance compared to baseline 10T that training with ten times as much target ofﬂine data.
6. C AN DARA PERFORM BETTER THAN CROSS - DOMAIN BASELINES ?
In Section 6.2, 1T+10S w/o Aug. does not explicitly learn policy for the target dynamics, thus one proposal ( 1T+10S ﬁne-tune ) for adapting the target dynamics is ﬁne-tuning the model that learned with source ofﬂine data, using the (reduced) target ofﬂine data.
Moreover, we also compare DARA with the recently proposed MABE (Cang et al., 2021), which is suitable well for our cross-dynamics setting by introducing behavioral priors π p in the model-based ofﬂine setting.
Thus, we implement two baselines: 1) 1T+10S
MABE
π
p ˆ T and 2)
1T+10S MABE ˆ
Tπ p , which denotes 1) learning π p with target domain data and ˆ T with source domain data, and 2) learning π p with source domain data and ˆ T with target domain data, respectively.
We show the results for the Walker (with body mass shift) in Table 2, and more experiments in Appendix A.3.5.
Our results show that DARA achieves signiﬁcantly better performance than the na ¨ ıve ﬁne-tune-based approaches in a majority of tasks (67 ” ↑ ” vs. 13 ” ↓ ”, including results in appendix).
On twelve out of the sixteen tasks (including results in appendix), DARA-based methods outperform the MABE-based methods.
We attribute
MABE’s failure to the difﬁculty of the reduced target ofﬂine data, which limits the generalization of the learned π p or ˆ T under such data.
However, such reduced data ( 10% of target) is sufﬁcient to modify rewards in the source ofﬂine data, which thus encourages better performance for our DARA.
[Table]
For real-world tasks , we also test DARA in a new ofﬂine dataset on the quadruped robot (see appendix for details).
Note that we can not access the privileged information ( e.g. , coordinate) in real robot, thus the target ofﬂine data (collected in real-world) does not contain rewards.
This means that prior ﬁne-tune-based and MABE-based methods become unavailable.
However, our reward augmentation frees us from the requisite of rewards in target domain.
We can freely perform ofﬂine training only using the augmented source ofﬂine data as long as the learned ∆ r is sufﬁcient.
For comparison, we also employ a baseline ( w/o Aug. ): directly deploying the learned policy with source data into the (target) real-world.
We present the results (deployed in real with obstructive stairs) in Table 3 and videos in supplementary material.
We can observe that training with our reward augmentation, this performance can be substantially improved.
Due to page limit, we refer readers to Appendix A.3.6 for more experimental results and discussion.
C ONCLUSION
In this paper, we formulate the dynamics shift in ofﬂine RL.
Based on prior model-based and modelfree ofﬂine algorithms, we propose the dynamics-award reward augmentation (DARA) framework that characterizes constraints over state-action-next-state distributions.
Empirically we demonstrate
DARA can eliminate the dynamics shift and outperform baselines in simulated and real-world tasks.
A A PPENDIX A. D ERIVATION A.1. P ROOF OF L EMMA
Let B πM V ( s ) =
E a ∼ π ( a | s ) (cid:2) r
( s , a ) + γ E s (cid:48) ∼ T ( s (cid:48) | s , a )
[ V ( s (cid:48) )] (cid:3) and r ( s ) =
E a ∼ π ( a | s )
[ r ( s , a )] .
Then, we have
[Equation]
A.1. M ODEL - BASED FORMULATION
Here we provide detailed derivation of the lower bound in Equation 9 in the main text.
Assumption 1 Assume a scale c and a function class F such that V M ∈ c F .
Following MOPO
(Yu et al., 2020), we set F = { f : (cid:107) f (cid:107) ∞ ≤ 1 } .
In Section Preliminaries, we have that the reward function is bounded: r ( s , a ) ∈
[ − R max , R max ] .
Thus, we have (cid:107) V M (cid:107) ∞ ≤ (cid:80) ∞
t =
γ t R max =
R max 1 − γ and hence the scale c
=
R max 1 − γ .
As a direct corollary of Assumption 1 and Equation 5, we have
[Equation]
Further, we ﬁnd
[Equation]
For the ﬁrst term d F ( ˆ T (cid:48) ( s (cid:48) | s , a ) , ˆ T ( s (cid:48) | s , a )) in Equation 12, through Pinsker’s inequality, we have
[Equation]
To keep consistent with the DARA-based method-free ofﬂine methods, we introduce scale η and bias δ to eliminate the square root in Equation 12.
To be speciﬁc, we assume 3 scale η and bias δ such that (cid:113) 1 2 D KL ( ˆ T (cid:48) , ˆ T ) ≤ ηD KL ( ˆ T (cid:48) , ˆ T )
+ δ .
Thus, we obtain d
[Equation]
[Equation]
[Equation]
For the second term d F ( ˆ T ( s (cid:48) | s , a ) , T ( s (cid:48) | s , a )) in Equation 11, we assume that we have access to an oracle uncertainty qualiﬁcation module that provides an upper bound on the error of the estimated empirical MDP ˆ M := {S , A , r, ˆ T, ρ 0 , γ } .
Assumption 2 Let F be the function class in Assumption 1.
We say u : S ×A → R is an admissible error estimator for ˆ T if d F ( ˆ T ( s (cid:48) | s , a ) , T ( s (cid:48) | s , a )) ≤ u ( s , a ) for all ( s , a ) .
Thus, we have
[Equation] [Equation]
Bring Inequations 10, 11, 13, and 14 into Lemma 3, we thus have
[Equation]
A.1. L EARNING C LASSIFIERS
Applying Bayes’ rule, we have
[Equation]
a
[Equation]
s
[Equation]
s
[Equation]
p
[Equation]
target
[Equation]
p ( target
[Equation]
s
[Equation]
a
[Equation]
(cid:48)
[Equation]
p ( target | s
[Equation]
Then we parameterize p ( ·| s , a , s (cid:48) ) and p ( ·| s , a ) with the two classiﬁers q sas and q sa respectively.
Using the standard cross-entropy loss, we learn q sas and q sa with the following optimization objective:
[Equation]
With the trained q sas and q sa , we have
[Equation]
In our implementation, we also clip the above reward modiﬁcation between − 10 and 10 .
A. R EGULARIZATION V IEW
In Table 4, we provide some related works with respect to the (state-action pair)
d D ( s ) π b ( a | s ) regularization and the (state-action-next-state pair)
d D (cid:48)
( s )
[Table]
regularization .
More papers with respect to unsupervised
, inverse
(imitation
), meta
RL, multi-agent RL, and hierarchical are not included
[Table]
[Table]
[Table]
[Table]
Eysenbach
[Table]
Ofﬂine (off-policy evaluation):
Fujimoto et al
[Table]
Kumar
[Table]
Wu
[Table]
Abdolmaleki
[Table] Kumar [Table] Kostrikov
[Table]
Nachum
[Table]
Nachum
[Table]
Zhang
[Table]
Nachum &
[Table]
Jiang &
[Table]
Uehara et
[Table]
Yu [Table]
Kidambi et
[Table]
MOReL
[Table]
Yu
[Table]
A. M ORE E XPERIMENTS A.3. T RAINING WITH {D (cid:48) ∪ D}
As we show in Figure 1 in Section Introduction
[Figure]
the performance of prior ofﬂine RL methods deteriorates dramatically as the amount of (target) ofﬂine data D decreases.
In Figure [Figure] we show that directly training with the mixed dataset {D (cid:48) ∪ D} will not compensate for the deteriorated performance caused by the reduced target ofﬂine data [Figure]
and training with such additional source ofﬂine data can even lead the performance degradation in some tasks
[Figure]
of target ofﬂine data D . We can observe that 1) the performance deteriorates dramatically as the amount of (target) ofﬂine data decreases ( 100% D (red line) → 50% D (orange bar) → 20% D (orange bar) → 10% D (orange bar) → 5% D (orange bar)), 2) after training with the additional 100% of source ofﬂine data, { 100% D (cid:48) ∪ x % D} , the ﬁnal performance is improved in most tasks, but most of the improvement is a pittance compared to the original performance degradation (compared to that training with the 100% of target ofﬂine data, i.e. , the red lines). A.3. C OMPARISON BETWEEN LEARNING CLASSIFIERS AND LEARNING DYNAMICS ( FOR THE REWARD MODIFICATION )
[Table]
In Table [Table] we show the comparison between learning classiﬁers and learning dynamics (for our reward modiﬁcation) in the Hopper tasks. We can observe that the two schemes for estimating the reward modiﬁcation have similar performance.
Thus, for simplicity and following Eysenbach et al.
[Table]
we adopt the classiﬁers to modify rewards in the source ofﬂine data in our experiments
. A.3. M ORE E XAMPLES WITH RESPECT TO THE REWARD AUGMENTATION
[Figure]
In Figure 5, we provide more examples with respect to the reward augmentation in the Ant task in
Figure 3 (left).
A.3. C OMPARISON BETWEEN 10T , 1T , 1T+10S w/o Aug. , AND 1T+10S DARA
Based on various ofﬂine RL algorithms (BEAR (Kumar et al., 2019), BRAC-p (Wu et al., 2019), BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020a), AWR (Peng et al., 2019), MOPO (Yu et al., 2020), BC (behavior cloning), COMBO (Yu et al., 2021)), we provide the additional results in Tables 6, 7, 8, 9, and 10. [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
A.3. C OMPARISON WITH THE CROSS - DOMAIN BASED BASELINES
In Tables 11 and
[Table]
we provide the comparison between our DARA-based methods, ﬁne-tune based methods [Table] and MABE-based methods in Hopper and Walker2d tasks, over the dynamics shift concerning the joint noise of motion. We can observe that in a majority of tasks
[Table]
our DARAbased methods outperforms the ﬁne-tune-based method (67 ” ↑ ” vs. 13 ” ↓ ”, including the results in the main text). Moreover, our DARA can achieve comparable or better performance compared to MABE-based baselines on eleven out of sixteen tasks (including the results in the main text) [Table]
[Table]
[Table]
A.3. A DDITIONAL RESULTS ON THE QUADRUPED ROBOT
In this ofﬂine sim2real setting, we collect the source ofﬂine data in the simulator ( 10 6 or 2 ∗ 10steps) and target ofﬂine data in the real world ( 3 ∗ 10 4 steps).
See Appendix A.4 for details.
For testing, we directly deploy the learned policy in the real (ﬂat or obstructive) environment and adopt the average distance covered in an episode (300 steps) as our evaluation metrics.
[Table]
[Table]
(Flat and static environment)
We ﬁrst deploy our learned policy in the ﬂat and static environment.
The results (distance covered in an episode) are provided in Table 13.
1) BCQ (Figure 7):
We ﬁnd that with Medium-R ofﬂine data, w/o Aug. BCQ and DARA BCQ both could not acquire the locomotion skills, which we think is caused by the lack of high-quality ofﬂine data.
{+With more ”expert” data (Medium-R → Medium → Medium-E, or Medium-R → MediumR-E), w/o-Aug. BCQ allows for progressive performance ( 0 . 
+}
{+→ 1 . 
+}
{+→ 2 . 16 , or 0 . 
+}
→ 1 . 69 in BCQ), but with our reward augmentation, such performance can be further improved (with average improvement 13 . 6% ).
2) CQL (Figure 8):
We ﬁnd that with Medium-R or Medium-R-E ofﬂine data, w/o Aug. CQL and DARA CQL both could not learn the locomotion skills, which we think is caused by the low- quality ”Replay” ofﬂine data. With Medium or Medium-E ofﬂine data, w/o Aug. CQL and DARA CQL acquire similar performance on this ﬂat and static environment. MOPO:
We ﬁnd that the model-based MOPO (both w/o Aug. and DARA ) could hardly learn the locomotion skill under the provided ofﬂine data.
[Table]
(Obstructive and dynamic environment)
We then deploy our learned policy in the obstructive and dynamic environment.
The results (distance covered in an episode) are provided in Table 14.
1) BCQ (Figure 9):
In this obstructive environment, we can obtain similar results as in the ﬂat environment.
{+With more ”expert” data (Medium → Medium-E → Medium-R-E), w/o Aug. BCQ allows for progressive performance ( 0 . 
+}
→ 1 . 15 → 1 . 27 ), and with our reward augmentation, such performance can be further improved (with average improvement 25 . 9% ).
At the same time, we can also ﬁnd that due to the presence of environmental obstacles, the performance of both w/o Aug. BCQ and w/o Aug. DARA is decreased compared to the deployment on the ﬂat environment. However, we ﬁnd that our DARA exhibits greater average performance improvement under this obstructive environment ( 13 . 6% → 25.9% ) compared to that in the ﬂat environment. These results demonstrate that our DARA can learn an adaptive policy for the target environment and thus show a greater advantage over w/o-Aug. in more complex environments.
2) CQL (Figure 10): Similar to BCQ, our DARA CQL exhibits a greater performance improvement over baseline in the obstructive and dynamic environment ( 0 .
2% → 30.9% ) compared to that in the ﬂat and static environment.
In summary , The results in the quadruped robot tasks support our conclusion in the main text regarding the dynamics shift problem in ofﬂine RL — with only modest amounts of target ofﬂine data ( 3 ∗ 10 4 steps), DARA-based methods can acquire an adaptive policy for the (both ﬂat and obstructive) target environment and exhibit better performance compared to baselines under the dynamics adaptation setting.
A.3. A BLATION STUDY WITH RESPECT TO THE AMOUNT OF TARGET OFFLINE DATA
To see whether the amount of target ofﬂine data can be further reduced
[Table]
we show the results of the ablation study with respect to the amount of target ofﬂine data in Tables 15 and
[Table]
[Table]
[Table]
[Figure] E NVIRONMENTS AND DATASET In this work, the tasks include Hopper, Walker2d, HalfCheetah, simulated and real-world quadruped robot, which are illustrated in Figure 11. [Equation]
[Table]
In the Hopper, Walker2d and HalfCheetah dynamics adaptation setting, we set the D4RL (Fu et al., 2020) dataset as our target domain.
For the source dynamics, we change the body mass (body mass shift) or add noises to joints (joint noise shift) of the agents (see Table 17 for the details) and then collect the source ofﬂine dataset in the changed environment.
Following Fu et al.
(2020), on the changed source environment, we collect the 1) ”Random” ofﬂine data, generated by unrolling a randomly initialized policy, 2) ”Medium” ofﬂine data, generated by a trained policy with the “medium” level of performance in the source environment, 3) ”Medium-Replay” (Medium-R) ofﬂine data, consisting of recording all samples in the replay buffer observed during training until the policy reaches the “medium” level of performance, 4) ”Medium-Expert” (Medium-E) ofﬂine data, mixing equal amounts of expert demonstrations and ”medium” data in the source environment.
In the sim2real setting (for the quadruped robot), we use the
A1 dog from Unitree (Wang, 2020).
We collect the target ofﬂine data using ﬁve target behavior policies in the real-world with changing terrains, as shown in Figure 12, and collect the ”Medium”, ”Medium-Replay” (Medium-R), ”MediumExpert” (Medium-E), ”Medium-Replay-Expert” (Medium-RE) source ofﬂine data in the simulator, where ”MediumReplay-Expert” denotes mixing equal amounts of ”MediumReplay” data and expert demonstrations in the simulator.
In
Section A.5, we provide the details of how to obtain the target and source behavior policy, so as to collect our target and source ofﬂine data.
We list our tasks properties in Table 18 and provide our collected dataset in supplementary material.
In implementation, we set η = 0 .
1 for all simulated tasks and set η = 0 .
01 for the sim2real task.
In
Table 23, we also report the sensitivity of DARA on the hyper-parameters η .
A. T RAINING THE ( TARGET AND SOURCE ) BEHAVIOR POLICY FOR THE QUADRUPED ROBOT
To obtain a behavior policy that can be deployed in simulator (for collecting the source ofﬂine data) or real-world (for collecting the target ofﬂine data), we introduce the prior knowledge
[Table]
Prior Knowledge: To reduce the impact of the foot at the moment of touching the ground during the robot locomotion, we designed a compound cycloid trajectory (Sakakibara et al., 1990) as prior knowledge.
In our implementation for the foot trajectory, four aspects are mainly considered: 1)
The robot walks stably without obvious shaking; 2)
The joint impact of the robot during the locomotion is small; 3) The joint speed and acceleration of the robot during the locomotion are continuous and smooth; 4) The feet of the robot will not slide when they are in contact with the ground.
Similar to Lee et al.
(2020), we deﬁne a periodic phase variable φ i ∈ [0 . 0 , 0 . 6) , i = 1 , 2 , 3 , 4 for each leg, which represents swing phase if φ i ∈ [0 . 0 , 0 . 3) and contact phase if φ i ∈ [0 . 3 , 0 . 6) .
At every time step t , φ i =
{+( t ∗ f 0 + φ 
+}
[ i ] + φ offset [ i ])
( mod 2 T m ) where T m = 0 .
3 , and f 0 = 1 .
1 is the base frequency, and φ 0 =
3 , 0 . 3 , 0] is the initial phase.
φ offset is part of the output of the controller.
The trajectory of the swing leg is:
[Equation]
where [Equation] and [Equation]
The trajectory of the standing leg is:
[Equation]
where S =
[Equation]
14 m, H = 0 .
18 m are the maximum foot length and height.
S= [0 . 17 , 0 . 17 , − 0 .
2 , − 0 .
2] , Y 0 =
[ − 0 .
13 , 0 . 13 , − 0 .
13] , Z 0 =
[ − 0 .
32 , − 0 .
32 , − 0 .
32 , − 0 .
32] are the default target foot position in body frame.
Domain Randomization: To encourage the policy to be robust to variations in the dynamics, we incorporate the domain randomization.
In Table 19, we provide the dynamics parameters and their respective range of values.
[Table]
State Space, Action Space and Reward Function: The action is a 16-dimensional vector consisting of leg phase and target foot position residuals in the body frame. The design of state space and reward function mainly follows the prior work Lee et al. In Table 20, we provide the state representation.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
r [Table] [Table] −
[Table]
[Table]
−
[Table]
The reward function is deﬁned as
[Equation] The individual terms are deﬁned as follows. 1) Linear velocity reward [Equation]
[Equation] where v pr = v xy · ˆ v xy is the base linear velocity projected onto the command direction. [Equation] where y and ˆ y is the yaw and desired yaw angle.
3) Roll and pitch reward
[Equation]
where φ are the roll and pitch angle. P xz = P 1 − P 4 or P xz = P 2 − P 3 , P i , i ∈ [1 , 4] are the foot position in world frame.
The advantage of designing the target pitch angle in this way is to ensure that the body of the robot is parallel to the supporting surface of the stand legs, thereby ensuring that the robot can smoothly over challenge terrain, such as upward stairs.
[Equation] motion [Equation] ) where ω xy are the roll and pitch rates.
[Equation]
Body collision
[Equation]
where I body and I foot are the contact numbers of robot’s body parts and foot with the terrain, respectively.
6) Target smooth reward
[Equation]
where f d,i ( i = t, t − 1 , t − 2) are the target foot positions in the time-step t , t − 1 and t − 2 .
7) Torqure reward [Equation] ) where τ i is the joint torques. Training Details: Both the behavior policy and value networks are Multilayer Perceptron (MLP) with 3 hidden layers, which have 256, 128 and 64 nodes.
The activation function is the Tanh function, and the optimizer is Adam .
With the above prior knowledge, domain randomization and reward function, we train our behavior policy with SAC (Haarnoja et al., 2018) in PyBullet (Coumans &
Bai, 2016–2021).
A. A DDITIONAL R ESULTS
Here we provide additional results regarding 1) the error bars (Tables 21 and 22), 2) hyper-parameter η
(Table 23), 3) the comparison with importance sampling based dynamics correction (Table 24), and 4) the illustration of whether the learned policy is limited to the source ofﬂine data (Figure 13).
[Figure]
path without obstacle) there transition of through obstacles source data performing DARA with {D
∪ D} enables us to acquire the behavior of moving through obstacles. As we increase the number of target ofﬂine data D , training with {D (cid:48) ∪ D} can gradually acquire optimal trajectories. T a b l e 21 : N o r m a li
ze d s c o r e s f o r t h e D 4 R
L t a s k s ( w it h body m a ss s h i f t ) . W e t a k e t h e b a s e li n e r e s u lt s (f o r 10 T ) o f M O
P O fr o m t h e i r o r i g i n a l p a p e r s a nd t h a t o f t h e o t h e r m od e lfr
ee m e t hod s ( B E A R , BR A C - p , A
W R ,
BC
Q a nd C Q L ) fr o m t
h e D 4 R L p a p e r ( F u e t a l ., ) .
B ody M a ss S h i f t 10 T 1 T 1 T + 10 S 1 T + 10 S 10 T 1 T 1 T + 10 S 1 T + 10 S 10 T 1 T 1 T + 10 S 1 T + 10 S w / o
A ug .
( DA R A )
w / o A ug .
( D A R A ) w / o A ug .
( DA R A )
Hopper B E A R
BR A C - p A W R
R a ndo m 11 .1 ± 0 .4 .± 3 .8 .± 1 . 10 .± 0 .9 .±
{+± 
+}
{+.10 .10 .± 
 .3 .± 
 .4 .± 0 .+}
M e d i u m 52 .0 .±0 .± 0 .1 .±32 . ± 6 .29 .± 2 .
[Table]
±
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
M e d i u mR 33 .1 .± 1 .18 .±34 .± 5 .0 .5 .± 3 .20 .± 4 .
[Table] [Table]
[Table]
M e d i u mE 96 .0 .± 0 .0 .±1 .±
[Table]
±
[Table]
[Table]
±
[Table]
[Table] [Table] [Table]
[Table]
[Table] [Table]
±
[Table]
Hopper BC Q C Q
L M O P O R a ndo m 10 {+.10 .± 0 .8 .± 0 .9 .± 
+}
{+.10 .10 .± 
 .10 .± 
 .10 .± 
 .11 .4 .±+}
2 .2 ± 2 .
[Table] [Table]
± 1 .
M e d i u m 54 .37 .± 6 .25 .± 5 .38 .±  ± 9 .44 .±
[-1) BCQ (Figure 7): We ﬁnd that with Medium-R ofﬂine data, w/o Aug. BCQ and DARA BCQ both could not acquire the locomotion skills, which we think is caused by the lack of high-quality ofﬂine data. With more ”expert” data (Medium-R → Medium → Medium-E, or Medium-R → MediumR-E), w/o-Aug. BCQ allows for progressive performance ( 0 . 
 → 1 . 
-]2 .59 .±
4 .±5 ± 5 .10 .± 5 . M e d i u mR 33 .9 .± 4 .28 .± 1 .32 .±
.48 .9 .±
{+5 .1 .± 
+}
.3 .±
1 .67 .1 ± 0 .5 .±
4 .8 .±
M e d i u mE 110 . ± 16 .75 .±
7 .84 .±
9 .98 .59 .±
{+34 .53 .± 21 .99 .± 
+}
{+.23 .1 .± 
+}
.4 .±
2 .5 .±
Walker2d
B E A
R
BR
A C - p
{+A W R R a ndo m 7 .1 .± 
 .3 .± 
+}
.3 .±
{+0 .- 0 .0 ± 0 .1 .± 
+}
.3 .±
{+2 .1 .1 .± 
 .2 ± 2 .± 0 . M e d i u m 59 .- 0 .± 0 .0 .± 0 .0 .± 
+}
.77 .6 .±
{+9 . ± 10 . ± 3 .17 .14 .± 2 .17 .± 
 .17 .± 0 .+}
{+M e d i u mR 19 .0 .± 
 .6 .± 5 .7 .±+}
1 .- 0 .8 .± 2 .9 .±18 .±
6 .15 .7 .±
2 .1 .± 0 .1 .± 0 . M e d i u mE 40 .- 0 .±
.1 .± 2 .2 .± 2 .76 .20 .± 16 .64 .±
{+10 .77 .± 
+}
.35 .± 10 .52 .± 1 .53 .± 0 .
Walker2d
BC Q C Q L
M O P O
R a ndo m 4 {+.1 .± 
 .4 .± 
 .4 .± 0 . 1 .± 1 .3 .±+}
{+1 .3 .± 1 .13 .- 0 .± 0 .- 0 .± 0 .- 0 .± 0 . M e d i u m 53 .32 .± 8 .50 .± 4 .52 .± 1 .
+}
.42 .± 24 .
{+± 1 .81 .± 3 .17 .7 ± 3 .5 .± 4 . ± 4 . M e d i u mR 6 .± 
 .14 .± 
 .15 .± 
+}
.26 .4 .±
{+3 .0 .± 
 .2 ± 1 . 5 .± 5 .3 .±+}
2 .14 .± 4 .
M e d i u mE 57 .32 .±
9 .55 .± 3 .57 .± 0 .
49 .±
26 .63 .±
22 .93 .± 8 .44 .
[Table]
T a b l e 22 : N o r m a li
ze d s c
o r e s f o r t
h e
D 4 R L t a s k s ( w it h j o i n t no i s e s h
i f t ) .
W e t a k e t
h e b a s e li n e r e s u lt s (f o r 10 T ) o f
M O
P O fr o m t h e i r o r i g i n a l p a p e r s a nd t h a t o f t
h e o t
[-At every time step t , φ i = ( t ∗ f 0 + φ 
 [ i ] + φ offset [ i ])-]h e r m od e lfr
ee m e t hod s ( B E
A R ,
BR
A C - p , A W R , BC Q a nd
C Q
L ) fr o m t
h e
D 4 R
L p a p e r (
F u e t a l ., ) .
J o i n t
N o i s e S
h
i f t 10 T 1 T 1 T + 10 S 1 T + 10 S 10 T 1 T 1 T + 10 S 1 T + 10 S 10 T 1 T 1 T + 10 S 1 T + 10 S w / o
A ug .
( DA
R A )
w / o
A ug .
( DA
R A )
w / o
A ug .
( DA
R A )
Hopper
B E A
R
BR
A C - p
A W
R
{+R a ndo m 11 .0 .±7 .± 
+}
.4 .±
10 .± 0 .
{+± 
+}
{+.10 .±
+}
{+.10 .±
+}
.±4 ± 0 .
{+M e d i u m 52 .0 .±
+}
± 2 ± 0 .
[Table] [Table]
±
[Table]
[Table]
[Table]
[Table]
[Table]
± 5 . M e d i u mR 33 .2 .± 1 .3 .± 0 .9 .± [Table] ±
[Table] [Table] [Table]
M
[Table]
d i
[Table]
m [Table] E [Table]
[Table]
[Table]
[Table] [Table] [Table] ±
[Table]
[Table]
[Table]
[Table]
±
[Table] [Table] [Table] [Table]
Hopper
BC Q [Table] a ndo m
[Table] [Table] [Table]
0 .7 ± 9 .± [Table] [Table] ± 0 .
[Table] [Table] [Table] M e d i u m
[Table]
[Table]
± 2 . ± 1 .54 .± 0 . 46 .
[Table]
± 11 .
[Table]
[Table]
[Table]
± 2 .
[Table]
[Table]
[Table]
± 3 .
M e d i u mR 33 .
±23 .±
{+3 . ± 0 .48 .13 .± 6 .2 .± 
+}
{+.3 .± 
 .67 .0 .± 
 .2 .± [Table]+}
[Table]
M e d i u mE 110 .44 .± 18 . ± 0 .
± 0 .98 .50 .±
26 .73 .±
{+1 .108 .± 
+}
.23 .1 ± 0 .6 .±
1 .7 .± 0 .
Walker2d
B E A
R
BR
A C - p
A W
R
{+R a ndo m 7 .2 .± 0 .0 .± 
+}
.2 .± 0 .- 0 .2 .±
2 .3 .±
{+2 .8 .±1 .
+}
{+.± 
+}
.1 .±
[Table]
[Table]
[Table]
M e d i u m 59 .- 0 {+.± 0 .0 .±0 .± .77 .28 .± 28 .55 .± 15 .72 .± 
 .12 .± 
+}
{+.17 .± 
+}
.17 .± 0 .
{+M e d i u mR 19 .0 .± 
 .4 ± 0 .10 .± 2 .-+} 0 .6 .± 1 .32 .±
11 .34 .±
10 .15 .6 ±
[Table] ±2 .± 0 . {+M e d i u mE 40 .- 0 .± 0 .0 .± 0 .0 .± 
 .76 .21 .±+}
18 .62 .± 13 .74 .± 1 .53 .40 .± 12 . ± 0 .53 .± Walker2d
BC
Q
C Q L M O P O
{+R a ndo m 4 .3 .± 1 .3 .± 
+}
.5 .± 0 . {+
+}
{+.±2 .± 
+}
{+.6 .± 
+}
{+.13 .- 0 .± 0 .- 
+}
.±- 0 .± 0 .
M e d i u m 53 . ± 8 .44 .±
{+3 .52 .± 
+}
{+.79 .43 .± 
+}
{+.73 .± 
+}
.81 .± 1 .17 .5 .±
5 .7 .±
6 .12 .±
M e d i u mR 5 .± 0 .9 .±
{+5 .14 .± 
+}
.26 .1 .±
{+1 .1 .± 0 .1 .± 0 . 
+}
{+.± 
+}
.9 .±
5 .16 .± 4 . M e d i u mE 57 .44 .± 3 .40 .± 16 .57 .± 0 .
46 .±
109 .± 4 .116 .± 9 .44 .2 .±
3 .15 .±
12 .26 .± 18 .
[Table] [Table]