D EMO DICE: O FFLINE I MITATION L EARNING WITH S UPPLEMENTARY I MPERFECT D EMONSTRATIONS
A BSTRACT
We consider ofﬂine imitation learning (IL), which aims to mimic the expert’s behavior from its demonstration without further interaction with the environment.
One of the main challenges in ofﬂine IL is to deal with the narrow support of the data distribution exhibited by the expert demonstrations that cover only a small fraction of the state and the action spaces.
As a result, ofﬂine IL algorithms that rely only on expert demonstrations are very unstable since the situation easily deviates from those in the expert demonstrations.
In this paper, we assume additional demonstration data of unknown degrees of optimality, which we call imperfect demonstrations.
Compared with the recent IL algorithms that adopt adversarial minimax training objectives, we substantially stabilize overall learning process by reducing minimax optimization to a direct convex optimization in a principled manner.
Using extensive tasks, we show that DemoDICE achieves promising results in the ofﬂine IL from expert and imperfect demonstrations.
I NTRODUCTION
Reinforcement learning (RL) (Sutton et al., 1998) aims to learn a intelligent behavioral strategy based on reward feedback.
Although RL has achieved remarkable success in many challenging domains, its practicality and applicability are still limited in two respects.
First, we need to specify the reward function, which may be non-trivial to do so in many real-world problems that require complex decision making.
Second, the standard RL setting assumes online interaction with the environment during the intermediate stages of learning, which is infeasible for mission-critical tasks.
Imitation learning (IL) (Pomerleau, 1991; Ng & Russell, 2000) addresses the ﬁrst limitation of
RL, where the agent is trained to mimic the expert from demonstration instead of specifying the reward function.
It is well known that adopting supervised learning for training the imitating agent, commonly referred to as behavioral cloning (BC), is vulnerable to the distribution drift (Ross et al., 2011).
Thus, most of the successful IL algorithms rely on online experiences collected from the environment by executing intermediate policies during training.
Recent progress made by adversarial imitation learning (AIL) (Ho & Ermon, 2016; Ke et al., 2019; Kostrikov et al., 2020) achieving stateof-the-art results on challenging imitation tasks still relies on such an online training paradigm.
Unfortunately, in many realistic tasks such as robotic manipulation and autonomous driving, online interactions are either costly or dangerous.
Ofﬂine RL (Fujimoto et al., 2019; Kumar et al., 2019; 2020; Levine et al., 2020; Wang et al., 2020; Lee et al., 2021; Kostrikov et al., 2021) aims to address these concerns by training the agent from the pre-collected set of experiences without online interactions.
To prevent an issue caused by the distributional shift, ofﬂine RL algorithms mitigate the phenomenon by constraining the shift or making a conservative evaluation of the policy being learned.
In this paper, we are concerned with ofﬂine IL problems.
Finding an effective algorithm for these problems is tricky.
For instance, naively extending the ofﬂine RL algorithms (which assume a reward function) to the ofﬂine IL setting does not work.
In practice, expert demonstrations are scarce due to the high cost of obtaining them.
Thus, they typically cover only a small fraction of the state and action spaces, which in turn makes the distribution drift issue even more stand out compared with the standard ofﬂine RL setting with a reward function.
We mitigate this issue by assuming a large number of supplementary imperfect demonstrations, without requiring any level of optimality for these imperfect demonstrations; they may contain expert or near-expert trajectories (Wu et al., 2019;
Wang et al., 2021) as well as non-expert ones all together.
This generality covers the situations from real-world applications, but at the same time, it poses a signiﬁcant challenge for the design of a successful ofﬂine IL algorithm.
In this paper, we propose DemoDICE, a novel model-free algorithm for ofﬂine IL from expert and imperfect demonstrations.
We formulate an ofﬂine IL objective which not only mitigates distribution shift from the demonstration-data distribution but also naturally utilizes imperfect demonstrations.
Our new formulation allows us to compute a closed form solution, which learns a policy in the space of stationary distributions but suffers from the instability issue in practice.
We tackle the issue by proposing an alternative objective, which leads to a stable algorithm in practice while keeping the optimal stationary distribution.
Finally, we introduce a method to extract the expert policy from a learned stationary distribution in a simple yet effective way.
Our extensive evaluations show that
DemoDICE achieves performance competitive to or better than a state-of-the-art off-policy IL algorithm in the ofﬂine-IL tasks with expert and imperfect demonstrations.
The the code to reproduce our results is available at our GitHub repository 1 .
P RELIMINARIES 2. M ARKOV D ECISION P ROCESS (MDP)
We assume an environment modeled as a Markov Decision Process (MDP), deﬁned by tuple M = (cid:104) S, A, T, R, p 0 , γ (cid:105) , where S is the set of states, A is the set of actions, T : S
×
A → ∆( S ) is the probability p ( s t +1 | s t , a t ) of making transition from state s t to state s t +1 by executing action a t at timestep t , R : S ×
A → R is the reward function, p 0 ∈ ∆( S ) is the distribution of the initial state s 0 , and γ ∈ [0 , 1] is the discount factor.
A policy π : S → ∆( A ) of MDP M is a mapping from states of M to distributions over actions.
For the given policy π , the stationary distribution d π is deﬁned as follows:
[Equation]
We assume a precollected dataset D E of ( s, a, s (cid:48) ) tuples generated by the expert, and a precollected imperfect dataset D I (generated by unknown degrees of optimality).
More precisely, for the (underlying) expert policy’s stationary distribution d E ( s, a ) , we assume that ( s, a, s (cid:48) )
∈
D E is sampled as ( s, a ) ∼ d E , s (cid:48) ∼ T ( ·| s, a ) .
We deﬁne D U
= D E ∪ D I , the union of two datasets, and denote the corresponding stationary distribution of the dataset D U as d U .
We denote the trajectories generated by expert policy as expert trajectories, and trajectories generated by non-expert policies as non-expert trajectories.
The expert demonstrations consist only of expert trajectories and imperfect demonstrations consist of a mixture of expert and non-expert trajectories.
In this paper, we assume that the quality of imperfect demonstrations is unknown.
2. I MITATION L EARNING
Behavior cloning (BC) is a classical IL approach, which attempts to ﬁnd a function that maps s to a via supervised learning.
The standard BC ﬁnds a policy π by minimizing the negative log-likelihood:
[Equation]
However, it is known to be brittle (Ross et al., 2011) when the interaction with the environment deviates from the scarce trajectories in D E .
In such cases, BC fails to recover expert policies.
One of the notable approaches for IL is to formulate the problem as distribution matching (Ho &
Ermon, 2016; Ke et al., 2019; Kostrikov et al., 2020).
When instantiated with the KL divergence widely used in previous IL works (Ke et al., 2019; Kostrikov et al., 2020), the approach amounts to ﬁnding a policy π by optimizing the following objective:
[Equation]
Since we cannot directly access the exact value of d E ( s, a ) and d π ( s, a ) , we estimate their ratio using samples from d E ( s, a ) and d π ( s, a ) , given as follows:
[Equation]
Here, the optimal discriminator c ∗ recovers log c ∗ ( s, a ) − log(1 − c ∗
[Equation]
= log d E
[Equation]
s
[Equation]
a
[Equation]
d π
[Equation]
s
[Equation]
a ) . Based on this connection between generative adversarial networks (GANs) and IL, AIL algorithms focus on recovering the expert policy (Ho & Ermon, 2016; Kostrikov et al., 2019).
However, the agent obtain samples from d π through interaction with environment, which is impossible in ofﬂine setting.
Therefore, to tackle the ofﬂine IL problems, we should derive an alternative estimation without on-policy samples.
D EMO DICE
In this section, we present a novel model-free ofﬂine IL algorithm named ofﬂine imitation learning using additional imperfect Demonstrations via stationary DIstribution Correction Estimation (DemoDICE).
Starting from a regularized ofﬂine IL objective which accords with ofﬂine RL algorithms, we present a formulation that does not require on-policy samples.
Such formulation allows us to construct a nested maximin optimization for ofﬂine IL from expert and imperfect demonstrations (Section 3.1).
Then, we derive the closed-form solution to the sub-problem of the aforementioned optimization and obtain a simple convex optimization objective (Section 3.2).
Since the objective is unstable in practice, we transform the objective to an alternative yet still convex objective (Section 3.3).
Finally, we show how to extract the policy from the learned correction term (Section 3.4).
3. T RANSFORM CONSTRAINED OPTIMIZATION INTO NESTED OPTIMIZATION
In the context of ofﬂine RL, most works use expected return maximization with some regularization to overcome the extrapolation error in ofﬂine settings (Fujimoto et al., 2019; Kumar et al., 2019;
Nachum et al., 2019b; Kumar et al., 2020; Lee et al., 2021).
In this work, we use KL divergence minimization between d π and d E with KL-regularization:
[Equation]
where α ≥ 0 is a hyperparameter that controls the balance between minimizing KL divergence with d E and preventing deviation of d π from d U .
Many online AIL algorithms estimate divergence between expert and current policy using on-policy samples, which is not available in ofﬂine scenario.
In contrast, to construct a tractable optimization problem in the ofﬂine setting, we consider a problem equivalent to Equation 4 in terms of stationary distribution d :
[Equation]
[Equation]
The constraints (6-7) are called the Bellman ﬂow constraints.
The Lagrangian of the above constrained optimization problem is
[Equation]
where ν ( s ) are the Lagrange multipliers
[Equation]
( B ∗ d )( s ) :
= (cid:80) a d ( s, a ) is the marginalization operator, and ( T ∗ d )( s ) :
= (cid:80) ¯ s, ¯ a T ( s | ¯ s, ¯ a ) d (¯ s, ¯ a ) is the transposed Bellman operator.
We introduce following derivations for the optimization (8) to obtain tractable optimization in the ofﬂine setting:
[Equation]
the equality in Equation 9 holds from the following properties of transpose operators:
[Equation]
where ( B ν )( s, a ) =
ν ( s ) , ( T ν )( s, a ) =
(cid:80) s (cid:48) T ( s (cid:48) | s, a ) ν
( s (cid:48) ) , with assumption d E ( s, a ) > 0 when d ( s, a ) >
(Nachum et al., 2019a).
We introduce another log ratio, denoted by r ( s, a ) in Equation 10 to avoid using log d E ( s,a ) d ( s,a ) , which requires on-policy samples to estimate.
Unlike log d E ( s,a ) d ( s,a ) , we can estimate r ( s, a ) in the ofﬂine setting using d E and d U , as we will discuss in the next section in detail.
We change the distribution used in the expectation of Equation 10 from d to d U by following the standard trick of importance sampling as follows:
[Equation]
As an alternative, one can convert expectation of d to d E instead of d U in Equation 10 by the similar application of the trick of importance sampling.
[Equation]
In practice, due to the small number of demonstrations in d E , there may be a lack of diversity.
To release this practical issue, we prefer to use d U instead of d E .
In summary, DemoDICE solves the following maximin optimization:
[Equation]
where r is trained by using precollected datasets.
Note that the solution w ∗ of Equation 12 with the ground-truth ratio r is the ratio of two distributions, the stationary distribution d π ∗ of the expert policy π ∗ and the stationary distribution d U of union of expert and imperfect demonstrations.
3. P RETRAINED STATIONARY DISTRIBUTION RATIO AND A CLOSED - FORM SOLUTION
To solve the problem (12), we can pretrain r , the log-ratio of d E ( s, a ) and d U ( s, a ) .
To this end, we train a discriminator c : S ×
A → [0 , 1] using the following maximization objective:
[Equation]
whose maximizer is c ∗ ( s, a )
=
[Equation]
By using c ∗ , r can be also obtained as
[Equation]
Since the optimization (8) is a convex optimization and the strong duality holds, the optimization (8) is equal to
[Equation]
The closed-form solution of inner max optimization of (15) turns out to be w ∗
ν ( s, a ) =
exp (cid:0) A ν
( s,a )
+ α − 1 (cid:1) for all ( s, a ) , where A ν ( s, a ) :
= r ( s, a )+ γ ( T ν )( s, a ) − ν ( s ) .
By using w ∗ ν ( s, a ) , optimization (15) can be reduced to a simple convex minimization problem as follows:
[Equation]
Although L ( w ∗ ν , ν ; r ) in Equation 16 is convex on ν , we observe that L ( w ∗ ν , ν ; r ) in Equation 16 is numerically unstable in practice.
Especially, the exponential term in L ( w ∗ ν , ν ; r ) is prone to explosion and so is the gradient ∇ ν L ( w ∗ ν , ν ; r ) .
3. A LTERNATIVE CONVEX OPTIMIZATION
In order to derive a numerically-stable alternative objective, we describe the following theoretical result:
Proposition 1.
Deﬁne the objective (cid:101) L ( ν ; r ) as
[Equation]
Then, the following equality holds:
[Equation]
(The proof can be found in Appendix A.)
The objective L ( w ∗ ν , ν ; r ) in Equation 16 has the instability issue since the gradient with respect to ν involves an unbounded easily-exploding function exp( · ) .
In contrast, the alternative objective (cid:101) L ( ν ; r ) in Equation 17 does not suffer from the same stability issue because in this case, the gradient forms a soft-max and is bounded by 1 .
Furthermore, we observe that the surrogate objective is still convex:
Proposition 2.
The objective (cid:101) L ( ν ; r ) is convex with respect to ν .
(The proof is in Appendix C.)
To summarize, we have presented a training objective (cid:101) L ( ν ; r ) which signiﬁcantly stabilizes the objective L ( w ∗ ν , ν ; r ) while having the same optimal value.
We converted the minimax optimization (15) to a single minimization, which is much more stable.
Proposition 1 is, however, limited in that it is not explain how to sample actions from learned ˜ ν .
Thus, we need a method to sample action from learned ˜ ν
and the next section presents such a method.
3. P OLICY EXTRACTION
In this section, we present a method for extracting a policy from the optimal ˜ ν ∗ = arg min ν (cid:101)
L ( ν ; r ) .
Our method is based on the following optimization, which amounts to the variant of weighted BC:
[Equation] [Equation]
where w ∗ ( s, a ) =
d π ∗ ( s,a ) d U ( s,a ) .
It can be estimated by ν ∗ ∈ arg min ν L ( w ∗ ν , ν ; r ) , but we use (cid:101) L ( ν ; r ) in Equation 17 to train ν .
Here, from the Proposition 1, we introduce theoretical results to explain the relation between ν ∗ ∈ arg min ν L ( w ∗ ν , ν ; r ) and ˜ ν ∗ ∈ arg min ˜
ν
L (˜ ν ; r ) :
Corollary 1.
Let (cid:101)
V = arg min ν (cid:101)
L ( ν ; r ) and V = arg min ν L ( w ∗ ν ,
ν ; r ) be the sets of ν .
Then the following equation holds: (We provide the proof in Appendix B.)
[Equation]
We observe that for any ˜ ν ∗ ∈ (cid:101) V , there are ν ∗ ∈ V and c ∈ R such that ˜ ν ∗ =
ν ∗ + c .
Based on this relationship, we can easily derive the following proportion: (see Appendix D for more detailed derivation)
[Equation]
In other words, using ˜ ν ∗ ∈ (cid:101) V , we can only obtain an unnormalized stationary distribution ratio instead of exact ratio.
Therefore, we estimate Equation 18 using self-normalized importance sampling (Owen, 2013) as shown below:
[Equation]
Based on the weighted BC, we simply extract the policy from ˜ ν ∗ ∈ (cid:101) V .
Our policy extraction is highly related to variants of weighted BC (Wang et al., 2018; 2020; Siegel et al., 2020) in ofﬂine RL.
The common idea of these methods is (1) training an action-value function Q ( s, a ) and (2) estimating the corresponding advantage
A ( s, a ) .
Using an increasing, non-negative function f ( · ) ( e.g., exp( · ) ), they perform weighted BC on precollected dataset with deﬁning weight f
( A ( s, a ) , s, a ) as: arg min π
E ( s,a ) ∼ d U [ − f
( A ( s, a ) , s, a ) log π ( a | s )] .
Finally, if we regard the advantage of optimal ν ∗ in Equation 16 as an advantage function, our policy extraction matches the ofﬂine RL based on weighted BC.
R ELATED W ORKS
Learning from imperfect demonstrations Among several recent works (Wu et al., 2019; Brown et al., 2019; 2020; Brantley et al., 2020; Tangkaratt et al., 2020; Sasaki & Yamashina, 2021; Wang et al., 2021) that attempted to leverage imperfect demonstrations in IL, two-step importance weighting imitation learning (2IWIL), generative adversarial imitation learning with imperfect demonstration and conﬁdence (IC-GAIL) (Wu et al., 2019), weighted GAIL (WGAIL) (Wang et al., 2021), and
Behavioral Cloning from Noisy Demonstrations (BCND) (Sasaki & Yamashina, 2021) are closely related to our work.
2IWIL, IC-GAIL and WGAIL suppose imperfect demonstrations which are composed of expert and non-expert trajectories and this deﬁnition is the same as that of DemoDICE.
Assuming that some of trajectories in the imperfect demonstrations are provided with labels of conﬁdence of optimalities, 2IWIL trains a semi-supervised classiﬁer to predict conﬁdence scores for unlabeled imperfect demonstrations and train the policy using the score.
Instead of such two-step learning, IC-GAIL trains the policy in an end-to-end fashion.
Inspired from Wu et al.
(2019), Wang et al.
(2021) suggest a general weighted GAIL objective function.
Unlike previous work, WGAIL predicts the weight for imperfect demonstrations without information about their quality.
However, 2IWIL, IC-GAIL, and WGAIL are online on-policy imitation learning method, whereas our algorithm, DemoDICE, is an ofﬂine imitation learning method.
BCND assumes that the expert demonstrations are sampled from a noisy expert policy.
BCND learns ensemble policies with a weighted BC objective, where the weight is policy learned by weighted BC in the previous internal iteration.
As the learning progresses, the mode of the learned policy moves towards that of the noisy expert policy.
Thus, the learned policy tends to converge to a non-expert policy when the non-expert trajectories comprise the majority of demonstrations.
Stationary distribution corrections In RL and IL, some prior works have used distribution corrections.
In AlgaeDICE (Nachum et al., 2019b), regularization in the space of stationary distribution is augmented to a policy optimization objective to solve off-policy RL problems.
In addition, by using dual formulation of f-divergence and change of variables (Nachum et al., 2019a), an off-policy learning objective is derived in AlgaeDICE.
ValueDICE (Kostrikov et al., 2020) minimizes the KL divergence between the agent and the expert stationary distributions to solve IL problems.
Similar to the AlgaeDICE, ValueDICE obtains an off-policy learning objective for distribution matching by using the dual formulation of KL divergence and change of variables.
Both AlgaeDICE and ValueDICE objectives are optimized by nested optimization, which may suffer from numerical instability.
In contrast, OptiDICE (Lee et al., 2021) reduces the same optimization used in AlgaeDICE to unconstrained convex optimization.
Although it shows promising performance in ofﬂine RL tasks, directly applying it to ofﬂine IL from expert and imperfect demonstrations is not trivial as we discussed in
Section 3.
Off policy learning from observations (OPOLO) (Zhu et al., 2020) addresses off-policy IL using expert demonstrations composed only of states.
It has similarities to DemoDICE in two aspects.
Firstly, OPOLO adopts a discriminator to estimate a state-transition distribution ratio.
Furthermore,
DICE-like technique is applied to the derivation of the objective of OPOLO, which is similar to ours.
Thus, based on the relation between DemoDICE and OPOLO, ofﬂine IL using expert demonstrations containing only states can be considered as a promising direction for future work.
E XPERIMENTS
In this section, we present the empirical performance of DemoDICE and baseline methods on MuJoCo continuous control environments (Todorov et al., 2012) using the OpenAI Gym (Brockman et al., 2016) framework.
We provide experimental results for 4 MuJoCo environments: Hopper,
Walker2d, HalfCheetah, and Ant.
We utilize D4RL datasets (Fu et al., 2020) to construct expert and imperfect demonstrations for our experiments.
We construct datasets, baselines, and evaluation protocol according to the following procedures:
Datasets For each of MuJoCo environments, we utilize three types of D4RL datasets (Fu et al., 2020), whose name end with “-expert-v2”, “-full replay-v2”, or “-random-v2”.
In the remainder of this section, we refer them by using corresponding sufﬁxes.
We regard the trajectories in expert-v2 as expert trajectories.
The set of expert demonstrations D E consists of the ﬁrst trajectory in expert-v2 .
Baselines We compare our method with three strong baseline methods, BC, BCND (Sasaki &
Yamashina, 2021), and ValueDICE (Kostrikov et al., 2020).
To consider the potential beneﬁt of utilizing D I , we carefully tuned BC with 5 different values of β ∈ { 1 , 0 . 75 , 0 . 5 , 0 .
25 , 0 } , which controls the balance between minimizing negative log-likelihood of D E and minimizing that of
D U
= D E ∪ D
I as follows:
[Equation]
We denote those 5 different settings by BC( β =1), BC( β =0.75), BC( β =0.5), BC( β =0.25), BC( β =0).
For BCND, we use all demonstrations as noisy expert demonstrations and report statistics of the best performance among 5 internal iterations over 5 seeds.
We provide the details for BCND in
Appenedix E.1.
Lastly, we made ValueDICE to utilize imperfect datasets by plugging in all the demonstrations to replay buffer.
Evaluation metric
The normalized scores for each environment are measured by normalized score = 100 ×
[Equation]
where expert and random scores are average returns of trajectories in expert-v2 and random-v2 , respectively.
We compute the average normalized score and the standard error over ﬁve random seeds.
In the following subsections, we provide experimental results on two types of imperfect datasets.
[Figure]
5. M IXED DATASET
DemoDICE mainly aims to overcome distributional drift caused by the lack of expert demonstrations.
When DemoDICE successfully learns the optimal unnormalized stationary distribution ratio, we can expect that DemoDICE distinguishes expert trajectories from non-expert ones to update its own policy.
Based on this intuition, we hypothesize that if the same sufﬁciently-many expert trajectories are included in imperfect demonstrations, DemoDICE achieves the optimal performance invariant to the number of sub-expert trajectories in imperfect demonstrations.
To see if it is the case, we use mixed datasets which have the same expert trajectories but have different numbers of non-expert trajectories in imperfect demonstrations.
Experimental setup Across all environments, we consider 3 tasks, each of which is called one of M1, M2, or M3 and is provided with expert and imperfect demonstrations.
While sharing the expert demonstrations composed of the same single expert trajectory, imperfect demonstrations in each task are composed of expert and random trajectories with different ratios such as 1:0.25 (M1), 1:1 (M2), and 1:4 (M3).
Speciﬁcally, imperfect demonstrations in M1, M2, and M3 are composed of 400 expert trajectories
sampled from expert-v2 and 100, 400, and 1600 random trajectories sampled from random-v2, respectively
Result We report the result of DemoDICE and comparing methods in Figure 1.
For simplicity,
BC ( β = 0 . 25) and BC ( β = 0 . 75) are in Appendix F. Note that imperfect demonstrations in the tasks
M1, M2, and M3 share the same expert trajectories, while having the different number of random trajectories.
Ideally, if an algorithm uses only expert trajectories from imperfect demonstrations, the algorithm should have the same performance in the tasks M1, M2, and M3, as discussed at the beginning of this subsection, except convergence speed.
While ValueDICE, BCND, and all the variants of BC catastrophically fail to recover expert policies in task M3 on all the environments except Walker2d, DemoDICE reaches the expert performance regardless of environments and tasks.
[Figure]
The result strongly implies that DemoDICE succeeds to learn appropriate stationary distribution correction, which is possible only when the algorithm effectively leverages expert trajectories in imperfect demonstrations ignoring the rest.
Furthermore, it is remarkable that DemoDICE reaches the expert performance in HalfCheetah in the end, which can be found in Appendix F.1.
To summarize, we have empirically shown that DemoDICE is robust to any choices of the environments and tasks and signiﬁcantly outperforms existing methods on ofﬂine IL.
5. R EPLAY BUFFER DATASET
When we use mixed datasets as imperfect demonstration sets, ValueDICE poorly performs in all environments except Walker2d as we observed in Section 5.1.
Since full replay-v2 consists of the replay buffer’s data during off-policy training, we believe it is one of the most practicallyrelevant datasets to ﬁll ValueDICE’s replay buffer.
Therefore, we conduct additional experiments using this dataset as the imperfect demonstrations.
Experimental setup Applying the same expert demonstrations used in Section 5.1, we employ another task named RB, which uses full replay-v2 directly as its imperfect dataset.
Result Figure 2 summarizes the result of DemoDICE and baseline methods.
We observe that DemoDICE performs on par with ValueDICE in Walker2d.
However, on the other environments, DemoDICE strictly outperforms ValueDICE.
We also observe that DemoDICE performs on par with
BCND in Ant, HalfCheetah environments, and signiﬁcantly outperforms in Hopper and Walker2d environments.
While showing the performance competitive to the best performing baseline methods in Walker2d, Ant, and HalfCheetah, DemoDICE outperforms all the methods in Hopper by a signiﬁcant margin.
Since the replay buffer dataset is collected by the policy which evolves (in policy training phase), it can be regarded as a set of imperfect demonstration from a wide range of sources.
We emphasize that
DemoDICE performs well not only when bad trajectories make up majority of the given imperfect demonstrations but also when they are generated from multiple sources.
C ONCLUSION
We have presented DemoDICE, an algorithm for ofﬂine IL from expert and imperfect demonstrations that achieves state-of-the-art performance on various ofﬂine IL tasks.
We ﬁrst introduced a regularized ofﬂine IL objective and reformulated the objective so as to make it natural to compute closed-form solution.
We then tackled the instability coming from the naive application of closedform solution by the alternative objective which yields not only the same optimal value but also a stable convex optimization.
Furthermore, we presented the method to extract an optimal policy with simple weighted BC.
Lastly, our extensive empirical evaluations showed that DemoDICE achieves remarkable performance close to the optimal by exploiting imperfect demonstrations effectively.
A CKNOWLEDGMENTS
This work was supported by the National Research Foundation (NRF) of Korea (NRF2019R1A2C1087634, NRF-2021M3I1A1097938) and Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
(No.20190-00075, No.2020-0-00940, No.2021-0-02068) HY was supported by the Engineering Research
Center Program through the National Research Foundation of Korea (NRF) funded by the Korean
Government MSIT (NRF-2018R1A5A1059921) and also by the Institute for Basic Science (IBSR029-C1).
A P ROOF OF P ROPOSITION
We provide the detailed proofs based on the previous works (Nachum & Dai, 2020; Lee et al., 2021).
Therefore, we highly recommend to read the works for a better understanding.
We ﬁrst provide a lemma to prove the proposition:
Lemma 1 (Fenchel duality) .
Consider a primal problem given by
[Equation]
where g, h : Ω → R are convex, lower-semi-continuous
and A is a linear operator.
Then, the dual of this problem is given by
[Equation]
where A ∗ is a (Hermitian) adjoint operator of A , i.e., (cid:104) y, Ax (cid:105)
= (cid:104)
A ∗ y, x (cid:105) for all x, y .
Proof.
We borrow the proof steps from Nachum & Dai (2020).
Remark that the deﬁnition of conjugate function of a function f :
[Equation]
where (cid:104)· , ·(cid:105) is inner product.
Based on the deﬁnition, we can easily derive the following equations:
[Equation]
Now, we prove the proposition:
Proposition 1.
Deﬁne the objective (cid:101) L ( ν ; r ) as
[Equation]
Then, the following equality holds:
[Equation]
(The proof can be found in Appendix A.)
Proof.
We ﬁrst deﬁne two functions:
[Equation]
where
[Equation]
∞ otherwise .
Then, the corresponding conjugate functions are given by
[Equation]
Now, we rewrite the optimization problem (5-7) as follows:
[Equation]
By Lemma
[Equation]
we reformulate the optimization problem as:
[Equation]
[Equation]
It means that min ν L ( w ∗ ν , ν ; r ) and min ν (cid:101) L ( ν ; r ) are the same optimization problems.
We obtain the conjugate function g ∗ using the fact that (cid:104) a, y (cid:105) is the conjugate of δ a ( x ) .
We also compute h ∗ using the two facts: b · f ∗ ( y − ab ) is the conjugate of (cid:104) a, x (cid:105) + b · f ( x ) and log E p [exp( y )] is the conjugate of D KL ( x (cid:107) p ) .
B P ROOF OF C OROLLARY
We remind the objectives in Equation 16 and Equation 17:
[Equation]
where
[Equation]
Note that for any constant C , the advantage for ν + C is:
[Equation]
Lemma 2.
For arbitrary function ν and constant C , the equality
[Equation]
holds.
Proof.
From the deﬁnition of (cid:101) L ( ν ; r ) , we derive the following equation:
[Equation]
= (1 − γ )
E s ∼ p
[ ν ( s ) + C ] + (1 + α ) log E ( s,a ) ∼ d U (cid:20) exp (cid:18)
A ( ν + C ) ( s, a ) 1 + α
(cid:19)(cid:21) =
(1 − γ ) E s ∼ p
[ ν ( s ) + C ] + (1 + α ) log E ( s,a ) ∼ d U (cid:20) exp (cid:18)
A ν ( s, a ) − (1 − γ ) C 1 + α (cid:19)(cid:21) =
(1 − γ ) E s ∼ p
[ ν ( s ) + C ] + (1 + α ) log (cid:20) exp (cid:18) − (1 − γ )
C
+ α (cid:19) E ( s,a ) ∼ d U (cid:20) exp (cid:18)
A ν ( s, a )
+ α (cid:19)(cid:21)(cid:21) =
(1 − γ ) E s ∼ p
[ ν ( s ) + C ]
+ (1 + α ) (cid:20) log E ( s,a ) ∼ d U (cid:20) exp (cid:18)
A ν ( s, a ) 1 + α (cid:19)(cid:21) − (1 − γ ) C
+ α (cid:21) =
(1 − γ ) E s ∼ p
[ ν ( s )]
+ (1 + α ) log E ( s,a ) ∼ d U (cid:20) exp (cid:18)
A ν ( s, a ) 1 + α (cid:19)(cid:21) =
(cid:101) L ( ν ; r )
Lemma 3.
For any function ν , the following inequality always holds:
[Equation]
Equality holds if and only if
[Equation]
Proof.
For any y ≥ 0 , y − 1 ≥ log y, and equality holds if and only if y = 1 .
From this inequality, we can derive the inequality directly:
[Equation]
and equality holds if and only if
[Equation]
Finally
[Equation]
we obtain the following results:
[Equation]
The equality holds if and only if
[Equation]
Lemma 4.
For any optimal solution ˜
ν ∗ = arg min ν (cid:101)
L ( ν ; r ) , there is a constant C such that ˜
ν ∗ + C is an optimal solution of min ν L ( w ∗ ν , ν ; r ) .
Proof.
Let ˜
ν
∗ be an optimal solution of arg min ν (cid:101)
L ( ν ; r ) and
[Equation]
Then, ˆ ν := ˜
ν ∗ + C ∗ satisﬁes
E ( s,a ) ∼ d U (cid:20) exp
[Equation] [Equation]
, a ) 1 +
[Equation]
(cid:19)(cid:21) =
E ( s,
[Equation] [Equation]
U (cid:20)
[Equation] [Equation]
=
E ( s,a ) ∼ d U (cid:20)
[Equation]
a )
[Equation] [Equation]
[Equation] [Equation]
=
[Equation]
( s,
[Equation] [Equation]
U
[Equation]
=
E ( s,a ) ∼ d U (cid:20) exp (cid:18)
A ˜ ν ∗ ( s, a ) 1 + α −(cid:19)(cid:21) exp (cid:18) − 1 − γ 1 + αC ∗ (cid:19)
=
E ( s,a ) ∼ d U (cid:20) exp (cid:18)
A ˜ ν ∗ ( s, a ) 1 + α −(cid:19)(cid:21) (cid:26) E ( s,a ) ∼ d U (cid:20) exp (cid:18) A ˜ ν ∗ ( s, a ) 1 + α −(cid:19)(cid:21)(cid:27) −= 1 .
Furthermore, ˆ ν is also an optimal solution of min ν (cid:101)
L ( ν ; r ) by Lemma 2.
Then, by equality condition in Lemma 3,
[Equation]
It means, ˆ ν is an optimal solution of min ν L ( w ∗ ν , ν ; r )
Lemma 5.
An optimal solution ν ∗ =
arg min ν L ( w ∗ ν , ν ; r ) is also an optimal solution of min ν (cid:101)
L ( ν ; r )
Proof.
From the Lemma 3,
[Equation]
Lemma 4 show that min ν L ( w ∗ ν , ν ; r ) and min ν (cid:101) L ( ν ; r ) have the same minimum value, (cid:101) L ( ν ∗ ; r ) ≤ L ( w ∗ ν ∗ , ν ∗ ; r ) =
min ν L ( w ∗ ν , ν ; r ) =
min ν (cid:101)
L ( ν ; r ) holds.
It means that (cid:101) L ( ν ∗ ; r ) is the minimum value of (cid:101) L and therefore, ν ∗ is an optimal solution of min ν (cid:101)
L ( ν ; r ) .
Finally, we ready to prove the corollary:
Corollary 1.
Let (cid:101)
V = arg min ν (cid:101)
L ( ν ; r ) and V = arg min ν L ( w ∗ ν ,
ν ; r ) be the sets of ν .
Then the following equation holds: (We provide the proof in Appendix B.)
[Equation]
Proof.
Lemma 4 and Lemma 5 shows that the last equation holds.
C P ROOF OF P ROPOSITION
Proposition 2.
The objective (cid:101) L ( ν ; r ) is convex with respect to ν .
(The proof is in Appendix C.)
Proof.
For any functions ν , ν (cid:48) and constant λ ∈ [0 , 1] , we can derive the following equality:
[Equation]
It means
[Equation]
A ν is the linear function with respect to ν .
Furthermore, using the convexity of log-sumexp,
[Equation]
Therefore, (cid:101) L ( ν ; r ) is convex function with respect to ν .
D P OLICY EXTRACTION
By the deﬁnition of A ν ,
[Equation]
Therefore,
[Equation]
E E XPERIMENTAL D ETAILS
Implementation detail For fair comparison, we use the same learning rate to train actors of BC and DemoDICE.
In DemoDICE, we applied the same regularization coefﬁcient α = 0 .
05 in Equation 17 across all the tasks and dataset conﬁgurations.
We implement our network architectures for
BC and DemoDICE based on the implementation of OptiDICE 2 .
For ValueDICE, we use its ofﬁcial implementation 3 without any modiﬁcation to network architectures or hyperparameters.
Based on DAC (Kostrikov et al., 2019), we treat terminal states as absorbing states in ValueDICE and
DemoDICE.
For stable discriminator learning, we use gradient penalty regularization on the r ( s, a ) function, which was proposed in Gulrajani et al.
(2017) to enforce 1-Lipschitz constraint.
To stabilize critic training, we add gradient L2-norm to the critic loss for the regularization.
Detailed hyperparameter conﬁgurations used for our main experiments are summarized in Table 1.
[Table]
E. E XPERIMENTAL D ETAILS FOR BCND
We implement the BCND based on the BCND paper (Sasaki & Yamashina, 2021).
We shufﬂe all demonstrations and divide them into K disjoint sets { D k } Kk =1 .
Then, for each iteration, we perform stochastic gradient ascent to update each policy π θ k using D K .
Finally, we update reward ˆ R ( s, a ) and η using
[Equation]
Here, we use following hyperparameters as reported in paper (Sasaki & Yamashina, 2021):
We use M = 5 (number of iteration), K = 5 (number of policy / disjoint dataset), N = 128 (batch size), L = 500 × | D k | /N for each D k ( k ∈ { 1 , . .
, K } , in our tasks, L ∈ [3 × 10 5 , 1 . 1 × 10 6 ] ), π θ ( a | s )
= 1 K (cid:80) K k
=1 π θ k
( a | s ) , ζ = 10 − 4 (shared learning rate), Adam optimizer with learning rate ζ × η for gradient ascent, and Gaussian policy π θ k = N ( µ θ k ( s ) , σ 2 θ k ) .
For Guassian policy, we use neural network with size [100 , 100] for µ θ k , and a trainable independent vector for σ θ k , as reported in the paper.
We measure the performance of algorithm for each internal iteration.
In Figure 1 and Figure 2, we report the best performance and corresponding standard error of the algorithm among the M internal iterations (we compute the average performance for each iteration and pick the maximum average performance as the best performance).
Require: Noisy expert demonstrations D , policy parameters { θ k } Kk =1 , learning rate ζ
Ensure: Ensemble policy π θ = 1 K (cid:80)
Kk
=1 π θ k
Set
ˆ R ( s, a ) = 1 for ∀
( s, a ) ∈ D .
Split D into K disjoint sets { D 1 , D 2 , . . .
, D for iteration = 1 ,
, M do for do
Initialize do end for end for
Update end for
F A DDITIONAL E XPERIMENTAL R ESULTS
In this section
[Figure]
we report additional experimental results for detailed empirical analysis of our approach.
We focus three experiments as follows: (1) asymptotic performance of our approach on
HalfCheetah environment for the mixed dataset we considered in Section 5.1, (2) behavior cloning with more diverse β selections which are omitted in Figure 1, (3) ablation study on hyperparameter α for DemoDICE, which is appeared in Equation 17.
F. A SYMPTOTIC P ERFORMANCE OF D EMO DICE ON H ALF C HEETAH
To inspect asymptotic behaviors of DemoDICE on our mixed dataset tasks in HalfCheetah environment
[Figure]
we extend the number of training iterations into 2 million with ﬁxing the other hyperparameter settings.
As Figure 3 shows, DemoDICE converges to the expert performance for all tasks we considered in Section 5.
[Figure]
F. B EHAVIOR C LONING P ERFORMANCE
In Figure
[Figure]
we omitted performance of BC with β = 0 .
25 and β = 0 .
75 for simplicity.
Figureshows performance of entire BC conﬁgurations, i.e., β ∈ { 0 , 0 .
[Figure]
F. D IFFERENT Q UALITY OF I MPERFECT D EMONSTRATIONS
In this section
[Figure]
we construct new tasks where random trajectories in mixed dataset (see 5.1) are replaced by medium trajectories, named I1,
I
[Figure]
and I3.
Here, all tasks share the expert demonstrations composed of the single expert trajectory.
Imperfect demonstrations in each task are composed ofexpert trajectories and medium trajectories
[Figure]
and the number of medium trajectories is 100 for the task
I1, 400 for I2, and 1600 for I3.
In Figure
[Figure]
we observed that DemoDICE performs well in these new tasks
[Figure]
F. D IFFERENT C OVERAGE OF E XPERT D EMONSTRATIONS
In this section
[Figure]
we introduce additional experiments with different coverages of expert demonstrations.
Concretely
[Figure]
we constructed expert demonstrations using an increasing number of expert trajectories, namely 1, 2, 5, up to 10.
Because DemoDICE already performs well in M1 and M2 tasks, we compare M3 and RB tasks for this ablation study.
In this experiments, DemoDICE is not significantly better than other baselines when the expert demonstrations cover enough space.
We remind the readers that when the coverage is not high, DemoDICE signiﬁcantly outperforms the others as shown in Figure 1 and 2 in the main text
[Figure]
Ant-M3(400:1600)
10 Number of Expert Trajectories 0.
Ant-RB
Expert
BC ( =0)
BC ( =0.25) BC ( =0.5)
BC ( =0.75)
BC ( =1)
ValueDICE
DemoDICE (ours)
F. A BLATION T EST ON H YPERPARAMETER α
As we can notice in Equation
[Figure]
DemoDICE introduces a hyperparameter α , which is a regularization coefﬁcient.
In this section
[Figure]
we aim to study how does the choice of α effect to training processes and performance of our algorithm.
We select α ∈ { 10 , 1 , 0 . 5 , 0 . 1 , 0 .
05 , 0 } and train on M3 datasets and replay buffer datasets by preserving the other settings same as before.
Figure 7 describes the results of ablation study.
We point out that DemoDICE shows almost the same performance in when α ∈ [0 , 0 . 1]
[Figure]
F. A BLATION T EST ON P URE O FFLINE I MITATION L EARNING
In this section, we compare DemoDICE and BC using only expert demonstrations.
In Figure 8, we observe that DemoDICE and BC show similar performance.
In pure ofﬂine IL, DemoDICE essentially tries to ﬁnd a stationary distribution that minimizes D KL ( d || d E ) on the MLE MDP constructed by D E .
Let ˆ d E be the empirical stationary distribution of d E .
Then, the optimal solution is trivially given by d ∗ = ˆ d E with D KL ( d || ˆ d E ) = 0 since ˆ d E itself is a valid stationary distribution on the MLE MDP.
Finally, extracting a policy from d E reduces to a simple behavior cloning: min π − E ( s,a ) ∼ d E [log π ( a | s )] , which concludes that DemoDICE would not be better than BC in the pure ofﬂine IL setting.
[Figure]
li z
[Figure]
d R
[Figure]
t u r
[Figure]
G D ETAILED A NALYSIS OF E MPIRICAL R ESULTS
In this section, we provide additional discussions about empirical results.
First, M1 is an easy task where most of the trajectories are from the expert, so some baselines also succeed to achieve good performance.
[Table]
To discuss about RB tasks, we show the normalized score statistics of trajectories in the replay buffer in the Table 2.
For each environment, we counted the number of trajectories that have a normalized score greater than R as > R , so the numbers above are cumulative counts.
Thus, this table suggests natural upper bound of performances of imitation learning algorithms, e.g. it would be very hard to score around 100 for HalfCheetah task.
In the Ant-RB task, we have enough good trajectories, especially > 100 , so both DemoDICE and
BC( β = 0 ) recover expert policy.
In the HalfCheetah-RB task, there is no trajectory with > 80 , so DemoDICE eagerly distinguishes the non-expert trajectories from expert ones, throwing away non-expert demonstrations.
Thus it overall behaves like BC( β = 0 ).
In the Walker2d-RB task, both ValueDICE and DemoDICE show similar performances at aroundat the end.
Looking at the statistics above, since the best trajectory in RB has its normalized score less than 90, we believe that ValueDICE and DemoDICE both achieved reasonable performance levels.