VILD: V ARIATIONAL I MITATION L EARNING WITH D IVERSE - QUALITY D EMONSTRATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations.
However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs.
IL in such situations can be challenging, especially when the level of demonstrators’ expertise is unknown.
We propose a new IL paradigm called
Variational Imitation Learning with Diverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators’ expertise with a probabilistic graphical model and estimate it along with a reward function.
We show that a naive estimation approach is not suitable to large state and action spaces, and ﬁx this issue by using a variational approach that can be easily implemented using existing reinforcement-learning methods.
Experiments on continuous-control benchmarks and real-world crowdsourced demonstrations denote that VILD outperforms stateof-the-art methods.
Our work enables scalable and data-efﬁcient IL under more realistic settings than before.
I NTRODUCTION
The goal of sequential decision making is to learn a policy that makes good decisions (Puterman, 1994).
As an important branch of sequential decision making, imitation learning (IL) (Schaal, 1999) aims to learn such a policy from demonstrations (i.e., sequences of decisions) collected from experts.
However, high-quality demonstrations can be difﬁcult to obtain in reality, since such experts may not always be available and sometimes are too costly (Osa et al., 2018).
This is especially true when the quality of decisions depends on speciﬁc domain-knowledge not typically available to amateurs; e.g., in applications such as robot control (Osa et al., 2018) and autonomous driving (Silver et al., 2012).
In practice, demonstrations are often diverse in quality, since it is cheaper to collect demonstrations from mixed demonstrators, containing both experts and amateurs (Audiffren et al., 2015).
Unfortunately, IL in such settings tends to perform poorly, since low-quality demonstrations often negatively affect the performance of IL methods (Shiarlis et al., 2016).
For example, amateurs’ demonstrations for robotics can be cheaply collected via a robot simulation (Mandlekar et al., 2018), but such demonstrations may cause damages to the robot which is catastrophic in the real-world (Shiarlis et al., 2016).
Similarly, demonstrations for autonomous driving can be collected from drivers in public roads (Fridman et al., 2017), which may contain trafﬁc-accident demonstrations.
Learning a self-driving car from these low-quality demonstrations may cause trafﬁc accidents.
When the level of demonstrators’ expertise is known, multi-modal IL (MM-IL) can be used to learn a good policy with diverse-quality demonstrations (Li et al., 2017; Hausman et al., 2017; Wang et al., 2017).
Speciﬁcally, MM-IL aims to learn a multi-modal policy, where each mode of the policy represents the decision making of each demonstrator.
When knowing the level of demonstrators’ expertise, good policies can be obtained by selecting modes that correspond to the decision making of high-expertise demonstrators.
However, in practice, it is difﬁcult to truly determine the level of demonstrators’ expertise beforehand.
Without knowing the level of expertise, it is difﬁcult to distinguish the decision making of experts and amateurs, and learning a good policy is challenging.
To overcome the issue of MM-IL, pioneer works have proposed to estimate the quality of each demonstration using auxiliary information from experts (Audiffren et al., 2015; Wu et al., 2019;
Brown et al., 2019).
Speciﬁcally, Audiffren et al.
(2015) inferred the demonstration quality using similarities between diverse-quality demonstrations and high-quality demonstrations, where the latter are collected in a small number from experts.
In contrast, Wu et al.
(2019) proposed to estimate the demonstration quality using a small number of demonstrations with conﬁdence scores.
Namely, the score value given by an expert is proportion to the demonstration quality.
Similarly, the demonstration quality can be estimated by ranked demonstrations, where ranking from an expert is evaluated due to the relative quality (Brown et al., 2019).
To sum up, these methods rely on auxiliary information from experts, namely high-quality demonstrations, conﬁdence scores, and ranking.
In practice, these pieces of information can be scarce or noisy, which leads to a poor performance of these methods.
In this paper, we consider a novel but realistic setting of IL where only diverse-quality demonstrations are available.
Meanwhile, the level of demonstrators’ expertise and auxiliary information from experts are fully absent.
To tackle this challenging setting, we propose a new learning paradigm called variational imitation learning with diverse-quality demonstrations (VILD).
The central idea of VILD is to model the level of demonstrators’ expertise via a probabilistic graphical model, and learn it along with a reward function that represents an intention of expert’s decision making.
To scale up our model for large state and action spaces, we leverage the variational approach (Jordan et al., 1999), which can be implemented using reinforcement learning (RL) for ﬂexibility (Sutton & Barto, 1998).
To further improve data-efﬁciency of VILD when learning the reward function, we utilize importance sampling (IS) to re-weight a sampling distribution according to the estimated level of demonstrators’ expertise.
Experiments on continuous-control benchmarks and real-world crowdsourced demonstrations (Mandlekar et al., 2018) denote that: 1) VILD is robust against diverse-quality demonstrations and outperforms existing methods signiﬁcantly.
2) VILD with IS is data-efﬁcient, since it learns the policy using a less number of transition samples.
IL FROM D IVERSE - QUALITY D EMONSTRATIONS AND ITS C HALLENGE
Before delving into our main contribution, we ﬁrst give the minimum background about RL and IL.
Then, we formulate a new setting in IL called diverse-quality demonstrations , discuss its challenge, and reveal the deﬁciency of existing methods.
Reinforcement learning.
Reinforcement learning (RL) (Sutton & Barto, 1998) aims to learn an optimal policy of a sequential decision making problem, which is often mathematically formulated as a Markov decision process (MDP) (Puterman, 1994).
We consider a ﬁnite-horizon MDP with continuous state and action spaces deﬁned by a tuple M “ p S , A , p p s 1 | s , a q , p 1 p s 1 q , r p s , a qq with a state s t P S Ď R
d
s , an action a t P A Ď R d a , an initial state density p 1 p s 1 q , a transition probability density p p s t ` 1 | s t , a t q , and a reward function r : S ˆ A ÞÑ R , where the subscript t P t 1 , . .
, T u denotes the time step.
A sequence of states and actions, p s 1: T , a 1: T q , is called a trajectory.
A decision making of an agent is determined by a policy π p a t | s t q , which is a conditional probability density of action given state.
RL seeks for an optimal policy π ‹ p a t | s t q which maximizes the expected cumulative reward: E p π p s 1: T , a 1: T q r Σ
Tt “ 1 r p s t , a t qs , where p π p s 1: T , a 1: T q “ p 1 p s 1 q Π
Tt “ 1 p p s t ` 1 | s t , a t q π p a t | s t q is a trajectory probability density induced by π . RL has shown great successes recently, especially when combined with deep neural networks (Silver et al., 2017).
However, a major limitation of RL is that it relies on the reward function which may be unavailable in practice (Schaal, 1999).
Imitation learning.
To address the above limitation of RL, imitation learning (IL) was proposed (Schaal, 1999).
Without using the reward function, IL aims to learn the optimal policy from demonstrations that encode information about the optimal policy.
A common assumption in most IL methods is that, demonstrations are collected by K ě 1 demonstrators who execute actions a t drawn from π ‹ p a t
| s t q for every states s t .
A graphical model describing this data collection process is depicted in Figure 1(a), where a random variable k P t 1 , . . .
, K u denotes each demonstrator’s identiﬁcation number and p p k q denotes the probability of collecting a demonstration from the k -th demonstrator.
Under this assumption, demonstrations tp s 1: T , a 1: T , k q n u Nn “ 1 (i.e., observed random variables in Figure 1(a)) are called expert demonstrations and are regarded to be drawn independently from a probability density p ‹ p s 1: T , a 1: T q p p k q “ p p k q p 1 p s 1 q Π
Tt “ 1 p p s t ` 1 | s t , a t q π ‹ p a t | s t q .
We note that k does not affect the trajectory density p ‹ p s 1: T , a 1: T q and can be omitted.
We assume a common assumption that p 1 p s 1 q and p p s t ` 1 | s t , a t q are unknown but we can sample states from them.
IL has shown great successes in benchmark settings (Ho & Ermon, 2016;
Fu et al., 2018; Peng et al., 2019).
However, practical applications of IL in the real-world is relatively few (Schroecker et al., 2019).
One of the main reasons is that most IL methods aim to learn with expert demonstrations.
In practice, such demonstrations are often too costly to obtain due to a limited number of experts, and
[Figure]
even when we obtain them, the number of demonstrations is often too few to accurately learn the optimal policy (Audiffren et al., 2015; Wu et al., 2019; Brown et al., 2019).
New setting in IL: Diverse-quality demonstrations.
To improve practicality of IL, we consider a new learning paradigm called IL with diverse-quality demonstrations , where demonstrations are collected from demonstrators with different level of expertise.
Compared to expert demonstrations, diverse-quality demonstrations can be collected more cheaply, e.g., via crowdsourcing (Mandlekar et al., 2018).
The graphical model in Figure 1(b) depicts the process of collecting such demonstrations from K ą 1 demonstrators.
Formally, we select the k -th demonstrator according to a distribution p p k q .
After selecting k , for each time step t , the k -th demonstrator observes state s t and samples action a t using π ‹ p a t |
s t q .
However, the demonstrator may not execute a t in the MDP if this demonstrator is not expertised.
Instead, he/she may sample an action u t P A with another probability density p p u t | s t , a t , k q and execute it.
Then, the next state s t ` 1 is observed with a probability density p
p s t ` 1 | s t , u t q , and the demonstrator continues making decision until time step T .
We repeat this process for N times to collect diverse-quality demonstrations D d “ tp s 1: T , u 1: T , k q n u Nn “ 1 .
These demonstrations are regarded to be drawn independently from a probability density
[Equation]
We refer to p p u t | s t , a t , k q as a noisy policy of the k -th demonstrator, since it is used to execute a noisy action u t .
Our goal is to learn the optimal policy π ‹ using diverse-quality demonstrations D d . Note that Eq.
(1) can be described equivalently by using a marginal density π p u t | s t , k q “ ş
A π ‹ p a t | s t q p p u t | s t , a t , k q d a t and removing a t from the graphical model.
However, we explicitly write a t as above to emphasize the dependency between π ‹ p a t | s t q and p p u t | s t , a t , k q .
This emphasis will be made more clear in Section 3.1 when we describe our choice of model.
The deﬁciency of existing methods.
We conjecture that existing IL methods are not suitable to learn with diverse-quality demonstrations according to p d .
Speciﬁcally, these methods always treat observed demonstrations as if they were drawn from p ‹ .
By comparing p ‹ and p d , we can see that existing methods would learn π p u t | s t q such that π p u t | s t q « Σ Kk “ 1 p p k q
[Equation]
In other words, they learn a policy that averages over decisions of all demonstrators.
This would be problematic when amateurs are present, as averaged decisions of all demonstrators would be highly different from those of all experts.
Worse yet, state distributions of amateurs and experts tend to be highly different, which often leads to the unstable learning: The learned policy oscillated between well-performed policy and poorly-performed policy.
For these reasons, we believe that existing methods tend to learn a policy that achieves average performances, and are not suitable for handling the setting of diverse-quality demonstrations.
VILD: A R OBUST M ETHOD FOR D IVERSE - QUALITY D EMONSTRATIONS
This section presents VILD, namely a robust method for tackling the challenge from diverse-quality demonstrations.
Speciﬁcally, we build a probabilistic model that explicitly describes the level of demonstrators’ expertise and a reward function (Section 3.1), and estimate its parameters by a variational approach (Section 3.2), which can be implemented easily by RL (Section 3.3).
We also improve data-efﬁciency by using importance sampling (Section 3.4).
Mathematical derivations are provided in Appendix A.
3. M ODELING DIVERSE - QUALITY DEMONSTRATIONS
This section describes a model which enables estimating the level of demonstrators’ expertise.
We ﬁrst describe a naive model, whose parameters can be estimated trivially via supervised learning, but suffers from the issue of compounding error.
Then, we describe our proposed model, which avoids the issue of the naive model by learning a reward function.
Naive model.
Based on p d , one of the simplest models to handle diverse-quality demonstrations is p θ , ω p s 1: T , u 1: T , k q “ p p k q p p s 1 q Π
Tt “ 1 p p s t ` 1 | s t , u t q ş
A π θ p a t | s t q p ω p u t | s t , a t , k q d a t , where π θ and p ω are learned to estimate the optimal policy and the noisy policy, respectively.
The parameters θ and ω can be learned by minimizing the Kullback-Leibler (KL) divergence from the data distribution to the model.
This naive model can be regarded as an extension of a model proposed by Raykar et al.
(2010) for handling diverse-quality data in supervised learning.
The main advantage of this naive model is that its parameters can be estimated trivially via supervised learning.
However, this native model suffers from the issue of compounding error (Ross & Bagnell, 2010) and tends to perform poorly.
Speciﬁcally, supervised-learning methods assume that data distributions during training and testing are identical.
However, data distributions during training and testing are different in IL, since data distributions depend on policies (Puterman, 1994).
A discrepancy of data distributions causes compounding errors during testing, where prediction errors increase further in future predictions.
Due to this issue, supervised-learning methods often perform poorly in IL (Ross & Bagnell, 2010).
The issue becomes even worse with diverse-quality demonstrations, since data distributions of different demonstrators tend to be highly different.
For these reasons, this naive model is not suitable for our setting.
Proposed model.
To avoid the issue of compounding error, our method utilizes the inverse RL (IRL) approach (Ng & Russell, 2000), where we aim to learn a reward function from diverse-quality demonstrations 1 .
IL problems can be solved by a combination of IRL and RL, where we learn a reward function by IRL and then learn a policy from the reward function by RL.
This combination avoids the issue of compounding error, since the policy is learned by RL which generalizes to states not presented in demonstrations (Ho & Ermon, 2016).
Speciﬁcally, our proposed model is based on a model of maximum entropy IRL (MaxEntIRL) (Ziebart et al., 2010).
Brieﬂy speaking, MaxEnt-IRL learns a reward function from expert demonstrations by using a model p
φ p s 1: T , a 1: T q 9 p p s 1 q Π
Tt “ 1 p 1 p s t ` 1 | s t , a t q exp p r φ p s t , a t qq .
Based on this model, we propose to learn the reward function and the level of expertise by a model p φ , ω p
[Equation]
: T , u
[Equation]
[Equation]
[Equation]
) where φ and ω are parameters.
We denote a normalization term of this model by Z φ , ω .
By comparing the proposed model p φ , ω to the data distribution p d , the reward parameter φ should be learned so that the cumulative reward is proportion to a joint probability density of actions given by the optimal policy, i.e., exp p Σ
Tt “ 1 r φ p s t , a t qq 9 Π
Tt “ 1 π ‹ p a t | s t q .
In other words, the cumulative reward is large for trajectories induced by the optimal policy.
Therefore, the optimal policy can be learned by maximizing the cumulative reward.
Meanwhile, the density p ω p u t | s t , a t , k q is learned to estimate the noisy policy p p u t | s t , a t , k q .
In the remainder, we refer to ω as an expertise parameter.
To learn parameters of this model, we propose to minimize the KL divergence from the data distribution to the model:
[Equation]
By rearranging terms and ignoring constant terms, minimizing this KL divergence is equivalent to solving an optimization problem max φ , ω f p φ , ω q ´ g p φ , ω q , where f p φ , ω q “ E p d p s 1: T , u 1: T | k q p p k q r Σ
Tt “ 1 log p ş
A exp p r φ p s t , a t qq
p ω p u t | s t , a t , k q d a t qs and g p φ , ω q “ log Z φ , ω .
To solve this optimization, we need to compute the integrals over both state space S and action space A .
Computing these integrals is feasible for small state and action spaces, but is infeasible for large state and action spaces.
To scale up our model to MDPs with large state and action spaces, we leverage a variational approach in the followings.
3. V ARIATIONAL APPROACH FOR PARAMETER ESTIMATION
The central idea of the variational approach is to lower-bound an integral by the Jensen inequality and a variational distribution (Jordan et al., 1999).
The main beneﬁt of the variational approach is that the integral can be indirectly computed via the lower-bound, given an optimal variational distribution.
However, ﬁnding the optimal distribution often requires solving a sub-optimization problem.
Before we proceed, notice that f p φ , ω q ´ g p φ , ω q is not a joint concave function of the integrals, and this prohibits using the Jensen inequality.
However, we can separately lower-bound f and g by the Jensen inequality, since they are concave functions of their corresponding integrals.
Speciﬁcally, let l φ , ω p s t , a t , u t , k q “ r φ p s t , a t q ` log p ω p u t | s t , a t , k q .
By using a variational distribution q ψ p a t | s t , u t , k q with parameter ψ , we obtain an inequality f p φ , ω q ě F p φ , ω , ψ q , where
[Equation]
and H t p q ψ q “ ´ E q ψ p a t | s t , u t ,k q r log q ψ p a t | s t , u t , k qs .
It is trivial to verify that the equality f p φ , ω q “ max ψ F p φ , ω , ψ q holds (Jordan et al., 1999), where the maximizer ψ ‹ of the lowerbound yields q ψ ‹ p a t | s t , u t , k q 9 exp p l φ , ω p s t , a t , u t , k qq .
Therefore, the function f p φ , ω q can be substituted by max ψ F p φ , ω , ψ q .
Meanwhile, by using a variational distribution q θ p a t , u t | s t , k q with parameter θ , we obtain an inequality g p φ , ω q ě G p φ , ω , θ q , where
[Equation]
and r q θ p s 1: T , u 1: T , a 1: T , k q “ p p k q p 1 p s
q Π Tt “ 1 p p s t ` 1 | s t , u t q q θ p a t , u t | s t , k q .
The lower-bound G resembles an objective function of maximum entropy RL (MaxEnt-RL) (Ziebart et al., 2010).
By using the optimality results of MaxEnt-RL (Haarnoja et al., 2018), we have an equality g p φ , ω q “ max θ G p φ , ω , θ q .
Therefore, the function g p φ , ω q can be substituted by max θ G p φ , ω , θ q .
By using these lower-bounds, we have that max φ , ω f p φ , ω
q
´
g p φ , ω q “ max φ , ω , ψ F p φ , ω , ψ q ´ max θ G p φ , ω , θ q “ max φ , ω , ψ min θ F p φ , ω , ψ q ´ G p φ , ω , θ q .
Solving the max-min problem is often feasible even for large state and action spaces, since F p φ , ω , ψ q and G p φ , ω , θ q are deﬁned as an expectation and can be optimized straightforwardly.
Nevertheless, in practice, we represent the variational distributions by parameterized functions, and iteratively solve the sub-optimization (w.r.t.
ψ and θ ) by stochastic optimization methods.
However, in this scenario, the equalities f p φ , ω q “ max ψ F p φ , ω , ψ q and g p φ , ω q “ max θ G p φ , ω , θ q may not hold for two reasons.
First, the optimal variational distributions may not be in the space of our parameterized functions.
Second, stochastic optimization methods may yield local solutions.
Nonetheless, when the variational distributions are represented by deep neural networks, the obtained variational distributions are often reasonably accurate and the equalities approximately hold (Ranganath et al., 2014).
3. M ODEL SPECIFICATION
In practice, we are required to specify models for q θ and p ω .
We propose to use q θ p a t , u t | s t , k q “ q θ p a t | s t q N p u t | a t , Σ q and p ω p u t | s t , a t , k q “ N p u t | a t , C ω p k qq .
As shown below, the choice for q θ p a t , u t | s t , k q enables us to solve the sub-optimization w.r.t.
θ by using RL with reward function r φ .
Meanwhile, the choice for p ω p u t | s t , a t , k q incorporates our prior assumption that the noisy policy tends to Gaussian, which is a reasonable assumption for actual human motor behavior (van Beers et al., 2004).
Under these model speciﬁcations, solving max φ , ω , ψ min θ F p φ , ω , ψ q ´ G p φ , ω , θ q is equivalent to solving max φ , ω , ψ min θ H p φ , ω , ψ , θ q , where
H p φ , ω , ψ , θ
[Equation]
Here, r q θ p s 1: T , a 1: T q “ p 1 p s 1 q Π
Tt “ş
A p p s t ` 1 | s t , u t q N p u t | a t , Σ
q d u t q θ p a t | s t q is a noisy trajectory density induced by a policy q θ p a t | s t q , where N p u t | a t , Σ q can be regarded as an approximation of the noisy policy in Figure 1(b).
Minimizing H w.r.t.
θ resembles solving a MaxEnt-RL problem with a reward function r φ p s t , a t q , except that trajectories are collected according to the noisy trajectory density.
In other words, this minimization problem can be solved using RL, and q θ p a t | s t q can be regarded as an approximation of the optimal policy.
The hyper-parameter Σ determines the quality of this approximation: smaller value of Σ gives a better approximation.
Therefore, by choosing a reasonably small value of Σ , solving the max-min problem in Eq.
(5) yields a reward function r φ p s t , a t q and a policy q θ p a t | s t q .
This policy imitates the optimal policy, which is the goal of IL.
The model speciﬁcation for p ω incorporates our prior assumption about the noisy policy.
Namely, p ω p u t | s t , a t , k q “ N p u t | a t , C ω p k qq assumes that the noisy policy tends to Gaussian, where C ω p k q gives an estimated expertise of the k -th demonstrator: High-expertise demonstrators have small
C ω p k q and vice-versa for low-expertise demonstrators.
Note that VILD is not restricted to this choice.
Different choices of p ω incorporate different prior assumptions.
For example, a Laplace distribution incorporates a prior assumption about demonstrators who tend to execute outlier actions (Murphy, 2013).
In such a case, the squared error in H is replaced by the absolute error (see Appendix A.3).
It should be mentioned that q ψ p a t | s t , u t , k q maximizes the immediate reward and minimizes the weighted squared error between u t and a t .
The trade-off between the reward and squared-error is determined by C ω p k q .
Speciﬁcally, for demonstrators with a small C ω p k q (i.e., high-expertise demonstrators), the squared error has a large magnitude and q ψ tends to minimize the squared error.
Meanwhile, for demonstrators with a large value of C ω p k q (i.e., low-expertise demonstrators), the squared error has a small magnitude and q ψ tends to maximize the immediate reward.
We implement VILD with deep neural networks where we iteratively update φ , ω , and ψ by stochastic gradient methods, and update θ by policy gradient methods.
A pseudo-code of VILD and implementation details are given in Appendix B.
In our implementation, we include a regularization term
L p ω q “ T E p p k
q r log | C ´ 1 ω p k q|s{ 2 , to penalize large value of C ω p k q .
Without this regularization,
C ω p k q can be overly large which makes learning degenerate.
We note that H already includes such a penalty via the trace term: E p p k q r Tr p C ´ 1 ω p k q Σ qs .
However, the strength of this penalty tends to be too small, since we choose Σ to be small.
VILD requires variable k to be given along with demonstrations.
However, There is no need for this variable to be provided by experts.
When k is not given, a simple strategy is to set k “ n and K “ N .
In other words, this strategy assumes that there is a one-to-one mapping between demonstration and demonstrator.
We apply this strategy in our experiments with real-world demonstrations.
3. I MPORTANCE SAMPLING FOR REWARD LEARNING
To improve the convergence rate of VILD when updating φ , we use importance sampling (IS).
Speciﬁcally, by analyzing the gradient ∇ φ H “ ∇ φ t E [Equation]
, u [Equation] [Equation] ´ E r q
[Equation]
[Equation] , a 1: T q r Σ Tt “ 1 r φ p s t , a t qsu , we can see that the reward function is updated to maximize the expected cumulative reward obtained by demonstrators and q ψ , while minimizing the expected cumulative reward obtained by q θ .
However, low-quality demonstrations often yield low reward values.
For this reason, stochastic gradients estimated by these demonstrations tend to be uninformative, which leads to slow convergence and poor data-efﬁciency.
To avoid estimating such uninformative gradients, we use IS to estimate gradients using high-quality demonstrations which are sampled with high probability.
Brieﬂy, IS is a technique for estimating an expectation over a distribution by using samples from a different distribution (Robert &
Casella, 2005).
For VILD, we propose to sample k from a distribution ˜ p
p k q 9 } vec p C ´ 1 ω p k qq} 1 .
This distribution assigns high probabilities to demonstrators with high estimated level of expertise (i.e., demonstrators with a small C ω p k q ).
With this distribution, the estimated gradients tend to be more informative which leads to a faster convergence.
To reduce a sampling bias, we use a truncated importance weight: w p k q “ min p p p k q{ ˜ p p k q , 1 q (Ionides, 2008), which leads to an IS gradient: ∇ φ H IS “ ∇ φ t E p
d p s 1: T , u 1: T | k q ˜ p p
k q r w p k q Σ Tt “ 1 E q ψ p a t | s t , u t ,k q r r φ p s t , a t
qss ´ E r q θ p s 1: T , a 1: T q r Σ Tt “ 1 r φ p s t , a t qsu .
Computing w p k q requires p p k q , which can be estimated accurately since k is a discrete random variable.
For simplicity, we assume that p p k q is a uniform distribution.
R ELATED W ORK
In this section, we will discuss a related area of supervised learning with diverse-quality data.
Besides, we will discuss existing IL methods that use the variational approach.
Supervised learning with diverse-quality data.
In supervised learning, diverse-quality data has been extensively studied, e.g. learning with noisy labels (Angluin & Laird, 1988).
This task assumes that human labelers may assign incorrect labels to training inputs.
With such labelers, the obtained dataset consists of high-quality data with correct labels and low-quality data with incorrect labels.
To handle this setting, many methods were proposed (Natarajan et al., 2013; Han et al., 2018).
The most related methods are probabilistic models, which aim to infer correct labels and the level of labelers’ expertise (Raykar et al., 2010; Khetan et al., 2018).
Speciﬁcally, Raykar et al.
(2010) proposed a method based on a two-coin model which enables estimating the correct labels and level of expertise.
Recently, Khetan et al.
(2018) proposed a method based on weighted loss functions, where the weight is determined by the estimated labels and level of expertise.
Methods for supervised learning with diverse-quality data can be leveraged to learn a policy in our setting.
However, they tend to perform poorly due to the issue of compounding error, as discussed previously in Section 3.1.
Variational approach in IL.
The variational approach has been previously utilized in IL to perform
MM-IL and reduce over-ﬁtting.
Speciﬁcally, MM-IL aims to learn a multi-modal policy from diverse demonstrations collected by many experts (Li et al., 2017), where each mode of the policy represents decision making of each expert 2 .
A multi-modal policy is commonly represented by a contextdependent policy, where each context represents each mode of the policy.
The variational approach has been used to learn such contexts, i.e., by learning a variational auto-encoder (Wang et al., 2017) and maximizing a variational lower-bound of mutual information (Li et al., 2017; Hausman et al., 2017).
Meanwhile, variational information bottleneck (VIB) (Alemi et al., 2017) has been used to reduce over-ﬁtting in IL (Peng et al., 2019).
Speciﬁcally, VIB aims to compress information ﬂow by minimizing a variational bound of mutual information.
This compression ﬁlters irrelevant signals, which leads to less over-ﬁtting.
Unlike these existing works, we utilize the variational approach to aid computing integrals in large state-action spaces, but not for learning a variational auto-encoder or optimizing a variational bound of mutual information.
E XPERIMENTS
In this section, we experimentally evaluate the performance of VILD (with and without IS) in continuous-control benchmarks and real-world crowdsourced demonstrations. For benchmarks, we use four continuous-control tasks from OpenAI gym (Brockman et al., 2016) with demonstrations from a pre-trained RL agent.
For real-world demonstrations, we use a robosuite reaching task (Fan et al., 2018) with demonstrations from real-world crowdsourcing platform (Mandlekar et al., 2018).
Performance is evaluated using a cumulative ground-truth reward along trajectories (i.e., higher is better)
(Ho & Ermon, 2016), and this cumulative reward is computed using test trajectories generated by learned policies (i.e., q θ p a t | s t q ).
We use 10 test trajectories for the benchmark tasks, and usetest trajectories for the robosuite reaching task.
Note that we use a larger number of test trajectories due to high variability of initial states in the robosuite reaching task.
We repeat experiments fortrials with different random seeds and report the mean and standard error.
Baseline.
We compare VILD against GAIL (Ho & Ermon, 2016), AIRL (Fu et al., 2018),
VAIL (Peng et al., 2019), MaxEnt-IRL (Ziebart et al., 2010), and InfoGAIL (Li et al., 2017).
These are online IL methods which collect transition samples to learn policies.
We use trust region policy optimization (TRPO)
(Schulman et al., 2015) to update policies, except for the Humanoid task where we use soft actor-critic (SAC) (Haarnoja et al., 2018).
For InfoGAIL, we report the performance averaged over uniformly sampled contexts, as well as the performance with the best context chosen during testing.
Data generation. To generate demonstrations from π ‹ (pre-trained by TRPO) according to Figure 1(b), we use two types of noisy policy p p u t | a t , s t , k q : Gaussian noisy policy: N p u t | a t , σ 2 k I q and time-signal-dependent (TSD) noisy policy: N p u t | a t , diag p b k p t q ˆ } a t } 1 { d a qq , where b k p t q is sampled from a noise process.
We use K “ 10 demonstrators with different σ k and noise processes for b k p t q .
Each demonstrator generates trajectories with approximately T “ 1000 time steps.
The number of state-action pairs in each dataset is approximately 10000.
Notice that for TSD, the noise variance depends on time and magnitude of actions.
This characteristic of TSD has been observed in human motor control (van Beers et al., 2004).
More details of data generation are given in Appendix C.
Results against online IL methods.
Figure 2 shows learning curves of VILD and existing methods against the number of transition samples in HalfCheetah and Ant 3 , whereas Table 1 reports the performance achieved in the last 100 iterations.
Clearly, VILD with IS overall outperforms existing methods in terms of both data-efﬁciency and ﬁnal performance, i.e., VILD with IS learns better policies using less numbers of transition samples.
VILD without IS tends to outperform existing methods in terms of the ﬁnal performance.
However, it is less data-efﬁcient when compared to VILD with IS, except on Humanoid with the Gaussian noisy policy, where VILD without IS tends to perform better than VILD with IS.
We conjecture that this is because IS slightly biases gradient estimation, which may have a negative effect on the performance.
Nonetheless, the overall good performance of
VILD with IS suggests that it is an effective method to handle diverse-quality demonstrations.
On the contrary, existing methods perform poorly as expected, except on the Humanoid task.
For the Humanoid task, VILD tends to perform the best in terms of the mean performance.
Nonetheless, all methods except GAIL achieve statistically comparable performance according to t-test.
This is perhaps because amateurs in this task perform relatively well compared to amateurs in other tasks, as seen from demonstrators’ performance given in Table 2 and 3 (Appendix C).
Since amateurs perform relatively well, demonstrations from these amateurs do not severely affect the performance of IL methods in this task when compared to the other tasks.
We found that InfoGAIL, which learns a context-dependent policy, may achieve good performance when the policy is conditioned on speciﬁc contexts.
For instance, InfoGAIL (best context) performs quite well in the Walker2d task with the TSD noisy policy (the learning curves are provided in
Figure 7(b)).
However, as shown in Figure 10, its performance varies across contexts and is quite poor on average when using contexts from a uniform distribution.
These results support our conjecture that MM-IL methods are not suitable for our setting where the level of demonstrators’ expertise is absent.
It can be seen that VILD without IS performs better for the Gaussian noisy policy when compared to the TSD noisy policy.
This is because the model of VILD is correctly speciﬁed for the Gaussian noisy policy, but the model is incorrectly speciﬁed for the TSD noisy policy; misspeciﬁed model indeed leads to the reduction in performance.
Nonetheless, VILD with IS still performs well for both types of noisy policy.
This is perhaps because negative effects of a misspeciﬁed model are not too severe for learning expertise parameters, which are required to compute r p p k q .
We also conduct the following evaluations.
Due to space limitation, ﬁgures are given in Appendix D.
Results against ofﬂine IL methods.
We compare VILD against ofﬂine IL methods based on supervised learning, namely behavior cloning (BC) (Pomerleau, 1988), Co-Teaching which is based on a method for learning with noisy labels (Han et al., 2018), and BC from diverse-quality demonstrations (BC-D) which optimizes the naive model described in Section 3.1.
Results in Figure 8 show that these methods perform worse than VILD overall; BC performs the worst since it severely suffers from both the compounding error and low-quality demonstrations.
Compared to BC, BC-D and Co-teaching are quite robust against low-quality demonstrations, but they still perform worse than VILD with IS.
Accuracy of estimated expertise parameter.
To evaluate accuracy of estimated expertise parameter, we compare the ground-truth value of σ k under the Gaussian noisy policy against the learned covariance C ω p k q .
Figure 9 shows that VILD learns an accurate ranking of demonstrators’ expertise.
The values of these parameters are also quite accurate compared to the ground-truth, except for demonstrators with low-level of expertise.
A reason for this phenomena is that low-quality demonstrations are highly dissimilar, which makes learning the expertise more challenging. VILD (with IS) VILD (without IS) AIRL
GAIL VAIL MaxEnt-IRL InfoGAIL
InfoGAIL (best context)
5 Transition samples 1e
C u m u l a t
i v e R e w a r d s 1e3 HalfCheetah (TRPO)
5 Transition samples 1e
C u m u l a t
i v e R e w a r d s 1e
Ant (TRPO)
[Figure]
5 Transition samples 1e
C u m u l a t
i v e r e w a r d s 1e3 HalfCheetah (TRPO)     5 Transition samples 1e
C u m u l a t
i v e r e w a r d s 1e
Ant (TRPO)
[Figure]
[Figure]
VILD (IS)
VILD (w/o IS
[Table]
5. R OBOSUITE REACHING TASK
In this experiment
[Table]
we evaluate the robustness of VILD against real-world demonstrations.
Speciﬁcally
[Table]
we conduct an experiment using real-world demonstrations collected by a robotic crowdsourcing platform (Mandlekar et al., 2018).
The public datasets were collected in the robosuite environment for object-manipulation tasks such as assembly tasks (Fan et al., 2018).
In our experiment
[Table]
we consider a reaching task, where demonstrations come from clipped assembly tasks when the robot’s end-effector contacts the target object.
We uses N “ 10 demonstrations whose length are approximately T “and set K “ 10 .
The number of state-action pairs in a demonstration dataset is approximately 5000.
For VILD, we apply the log-sigmoid function to the reward function, which improves the performance in this task.
More details of the experimental setting are provided in Appendix C.2.
Figure 3 shows the performance of all methods, except VILD without IS and VAIL, since IS improves the performance and VAIL is comparable to GAIL.
It can be seen that VILD with IS performs better than GAIL, AIRL, and MaxEnt-IRL.
VILD also performs better than InfoGAIL in terms of the ﬁnal performance; InfoGAIL learns faster in the early stage of learning, but its performance saturates and VILD eventually outperforms InfoGAIL.
These experimental results show that VILD is more robust against real-world demonstrations with diverse-quality when compared existing state-of-the-art methods.
An example of trajectory generated by VILD’s policy is shown in Figure 5.
Figure 4 shows the performance of InfoGAIL with different context variables z
(Li et al., 2017).
We can see that InfoGAIL performs well when the policy is conditioned on speciﬁc contexts, e.g., z “ 7 .
Indeed, the best context during testing can improve the performance of InfoGAIL.
The effectiveness of such an approach is demonstrated in Figure 3, where InfoGAIL (best context) performs very well.
However, InfoGAIL (best context) is less practical than VILD, since choosing the best context requires an expert to evaluate the performance of all contexts.
In contrast, the performance of VILD does not depend on contexts, since VILD does not learn a context-dependent policy.
Moreover, the
[Figure]
[Figure]
performance of InfoGAIL (best context) is quite unstable, and it is still outperformed by VILD in terms of the ﬁnal performance.
C ONCLUSION AND F UTURE W ORK
In this paper [Figure] we explored a practical setting in IL where demonstrations have diverse-quality.
We showed the deﬁciency of existing methods, and proposed a robust method called VILD, which learns both the reward function and the level of demonstrators’ expertise by using the variational approach.
Empirical results demonstrated that our work enables scalable and data-efﬁcient IL under this practical setting.
In future [Figure] we will explore other approaches to efﬁciently estimate parameters of the proposed model except the variational approach.
We will also explore approaches to handle model misspeciﬁcation, i e., scenarios where the noisy policy differs from the model p ω .
Speciﬁcally, we will explore more ﬂexible models of p ω such as neural networks, as well as using the tempered posterior approach (Grünwald & van Ommen, 2017) to improve robustness of our model
A D ERIVATIONS
This section derives the lower-bounds of f p φ ,
ω q and g p φ , ω q presented in the paper.
We also derive the objective function H p φ , ω , ψ , θ q of VILD.
A. L OWER - BOUND OF f
Let l φ , ω p s t , a t , u t , k q “ r φ p s t , a t q ` log p ω p u t | s t , a t , k q , we have that f p φ , ω q “
E p d p s 1: T , u 1: [Equation] | k q p p k q ”ř
Tt “ 1 f t p φ , ω q ı , where f t p φ , ω q “ log ş A exp p l φ , ω p s t , a t , u t , k qq d a t .
By using a variational distribution q ψ p a t | s t , u t , k q with parameter ψ , we can bound f t p φ , ω q from below by using the Jensen inequality as follows:
[Equation]
Then, by using the linearity of expectation, we obtain the lower-bound of f p φ , ω q as follows:
[Equation]
To verify that f p φ , ω q “ max ψ F p φ , ω , ψ q , we maximize F t p φ , ω , ψ q w.r.t.
q ψ under the constraint that q ψ is a valid probability density, i.e., q ψ p a t | s t , u t , k q ą 0 and ş A q ψ p a t | s t , u t , k q d a t “ 1 .
By setting the derivative of F t p φ , ω , ψ q w.r.t.
q ψ to zero, we obtain
[Equation]
where the last line follows from the constraint ş A q ψ p a t | s t , u t , k
q d a t “ 1 .
To show that this is indeed the maximizer, we substitute q ψ ‹ p a t | s t , u t , k q “ [Equation] [Equation]
[Equation] ,k qq ş [Equation] s
[Equation]
[Equation] , u
[Equation]
,k qq d a t into F t p φ , ω
[Equation]
q :
[Equation] [Equation]
ˆż
[Equation]
˙ .
This equality veriﬁes that f t p φ , ω q “ max ψ F t p φ , ω , ψ q .
Finally, by using the linearity of expectation, we have that f p φ , ω q “ max ψ F p φ , ω , ψ q .
A. L OWER - BOUND OF g
Next, we derive the lower-bound of g p φ , ω q presented in the paper.
We ﬁrst derive a trivial lower-bound using a general variational distribution over trajectories and reveal its issues.
Then, we derive a lower-bound presented in the paper by using a structured variational distribution.
Recall that the function g p φ , ω q “ log Z φ , ω is g p φ , ω q “ log ¨
[Equation]
˛ ‹‚
[Equation]
Lower-bound via a variational distribution A lower-bound of g can be obtained by using a variational distribution s q β p s 1: T , u 1: T , a 1: T , k q with parameter β .
We note that this variational distribution allows any dependency between the random variables s 1: T , u 1: T , a 1: T , and k .
By using this distribution, we have a [Table]
[Equation]
The main issue of using this lower-bound is that, s G p φ , ω , β q can be computed or approximated only when we have an access to the transition probability p p s t ` 1 | s t , u t q .
In many practical tasks, the transition probability is unknown and needs to be estimated.
However, estimating the transition probability for large state and action spaces is known to be highly challenging (Sutton & Barto, 1998).
For these reasons, this lower-bound is not suitable for our method.
Lower-bound via a structured variational distribution To avoid the above issue, we use the structure variational approach (Hoffman & Blei, 2015), where the key idea is to pre-deﬁne conditional dependency to ease computation.
Speciﬁcally, we use a variational distribution q θ p a t , u t | s t , k q with parameter θ and deﬁne dependencies between states according to the transition probability of an MDP.
With this variational distribution, we lower-bound g as follows:
[Equation]
where r q θ p s 1: T , u 1: T , a 1: T , k q “ p p k q p 1 p s 1 q Π
Tt “ 1 p p s t ` 1 | s t , u t q q θ p a t , u t | s t , k q .
The optimal variational distri- bution q θ ‹ p a t , u t | s t , k q can be founded by maximizing G p φ , ω , θ q w.r.t.
q θ .
Solving this maximization problem is identical to solving a maximum entropy RL (MaxEnt-RL) problem (Ziebart et al., 2010) for an MDP deﬁned by a tuple M “ p S ˆ N ` , A ˆ A , p p s 1 , | s , u q I k “ k 1 , p 1 p s 1 q p p k 1 q , l φ , ω p s , a , u , k qq .
Speciﬁcally, this MDP is deﬁned with a state variable p s t , k t q P S ˆ N , an action variable p a t , u t q P
A ˆ A , a transition probability density p p s t ` 1 , | s t , u t q I k t “ k t ` 1 , an initial state density p 1 p s 1 q p p k 1 q , and a reward function l φ , ω p s t , a t , u t , k q .
Here, I a “ b is the indicator function which equals to 1 if a “ b and 0 otherwise.
By adopting the optimality results of MaxEnt-RL (Ziebart et al., 2010; Haarnoja et al., 2018), we have g p φ , ω q “ max θ G p φ , ω , θ q , where the optimal variational distribution is
[Equation]
The functions Q and V are soft-value functions deﬁned as
[Equation]
A. O BJECTIVE FUNCTION H OF VILD
This section derives the objective function H p φ , ω , ψ , θ q from F p φ , ω , ψ q ´ G p φ , ω , θ q .
Speciﬁcally, we substitute the models p ω p u t | s t , a t , k q “ N p u t | a t , C ω p k qq and q θ p a t , u t | s t , k q “ q θ p a t | s t q N p u t | a t , Σ q .
We also give an example when using a Laplace distribution for p ω p u t | s t , a t , k q instead of the Gaussian distribution.
First, we substitute q θ p a t , u t | s t , k q “ q θ p a t | s t q N p u t | a t , Σ q into G : G p φ , ω , θ q “ E r q θ p s 1: T , u 1: T , a 1: T ,k q
[Equation]
where c 1 is a constant corresponding to the log-normalization term of the Gaussian distribution.
Next, by using the re-parameterization trick, we rewrite r q θ p s 1: T , u 1: T , a 1: T , k q as r q θ
[Equation]
s 1: T , u 1: T , a 1: T , [Equation] [Equation] , where we use u t “ a t ` Σ 1 { 2 (cid:15) t with (cid:15) t „ N p (cid:15) t | 0 , I q .
With this, the expectation of Σ Tt “ 1 } u t ´ a t } 2 Σ ´ 1 over r q θ p s 1: T , u 1: T , a 1: T , k q can be written as [Equation]
[Equation]
which is a constant.
Then, the quantity G can be expressed as
[Equation]
By ignoring the constant, the optimization problem max φ , ω , ψ min θ F p φ , ω , ψ q ´ G p φ , ω , θ q is equivalent to
[Equation]
Our next step is to substitute
p ω p u t | s t , a t , k q by our choice of model.
First, let us consider a Gaussian distribution p
[Equation]
qq , where the covariance depends on state.
With this model, the second term in Eq.
(13) is given by
[Equation]
where c 2 “ ´ d a 2 log 2 π is a constant.
By using the reparameterization trick, we write the expectation of Σ Tt “ 1 } u t ´ a t } 2 C ´ 1 ω p s t ,k q as follows:
[Equation]
Using this equality, the second term in Eq.
(13) is given by
[Equation]
Maximizing this quantity w.r.t.
θ has an implication as follows:
q θ p a t | s t q maximizes the expected cu- mulative reward while avoiding states that are difﬁcult for demonstrators.
Speciﬁcally, a large value of
E p p k
q r log | C ω p s t , k q|s indicates that demonstrators have a low level of expertise for state s t on average, given by our estimated covariance.
In other words, this state is difﬁcult to accurately execute optimal actions for all demonstrators on averages.
Since the policy q θ p a t | s t q should minimize E p p k q r log | C ω p s t , k q|s , the policy should avoid states that are difﬁcult for demonstrators.
We expect that this property may improve exploration-exploitation trade-off in IL.
Nonetheless, we leave an investigation of this property for future work, since this is not in the scope of the paper.
In this paper, we specify that the covariance does not depend on state: C ω p s t , k q “ C ω p k q .
This model speciﬁcation enables us to simplify Eq.
(14) as follows:
[Equation]
where r q θ p s 1: T , a 1: T q “ p 1 p s
[Equation]
ś Tt “
[Equation]
ş A p p s t
[Equation]
1 | s t , u t q N p u t | a t , Σ q d u t q θ p a t | s t q . The last line follows from the quadratic form identity: E N p (cid:15) t | 0 , I q ”
[Equation]
Σ 1 { 2 (cid:15) [Equation] } 2 C
[Equation]
1 ω p
[Equation]
ı “ Tr p C ´ 1 ω p k q Σ q .
Next, we substitute
p ω
p u t | s t , a t , k q “ N p u t | a t , C ω p k qq into the ﬁrst term of Eq.
[Equation]
Lastly, by ignoring constants, Eq.
(13) is equivalent to max φ , ω , ψ min θ H p φ , ω , ψ , θ q , where H p φ , ω , ψ , θ q “
[Equation]
[Equation]
q
[Equation]
This concludes the derivation of VILD
[Equation] As mentioned, other distributions beside the Gaussian distribution can be used for p ω .
For instance
[Equation]
let us consider a multivariate-independent Laplace distribution: p ω p u t
[Equation]
s t ,
[Equation]
a
[Equation]
1 q
[Equation]
where a division of vector by vector denotes element-wise division.
The Laplace distribution has heavier tails when compared to the Gaussian distribution, which makes the Laplace distribution more suitable for modeling demonstrators who tend to execute outlier actions.
By using the Laplace distribution for p ω p u t | s t , a t , k q , we obtain an objective
[Equation]
We can see that differences between H Lap and H are the absolute error and scaling of the trace term.
B I MPLEMENTATION DETAILS
We implement VILD using the PyTorch deep learning framework.
For all function approximators, we use neural networks with 2 hidden-layers of 100 tanh units, except for the Humanoid task and the robosuite reaching task Algorithm 1: Input: Diverse-quality demonstrations D d “ tp s 1: T , u 1: T , k q n u Nn “ 1 and a replay buffer B “ ∅ .
2: while Not converge do 3: while | B | ă B with batch size B do Ź Collect samples from r q θ p s 1: T , a 1: T q 4:
Sample a t „ q θ p a t | s t q and (cid:15) t „ N p (cid:15) t | 0 , Σ q .
Execute a t ` (cid:15) t , observe s 1 t „ p p s 1 t | s t , a t ` (cid:15) t q , and include p s t , a t , s 1 t q into B 6: Update q ψ by an estimate of ∇ ψ H p φ , ω , ψ , θ q . Update p ω by an estimate of ∇ ω
H p φ , ω , ψ , θ q ` ∇ ω L p ω q .
Update r φ by an estimate of ∇ φ H IS p φ , ω , ψ , θ q .
Update q θ by an RL method (e.g., TRPO or SAC) with reward function r φ .
where we use neural networks with 2 hidden-layers of 100 relu units.
We optimize parameters φ , ω , and ψ by Adam with step-size 3 ˆ 10 ´ 4 , β 1 “ 0 . 9 , β 2 “ 0 . 999 and mini-batch size 256.
To optimize the policy parameter θ , we use trust region policy optimization (TRPO)
(Schulman et al., 2015) with batch size 1000, except on the Humanoid task where we use soft actor-critic (SAC) (Haarnoja et al., 2018) with mini-batch size 256.
Note that TRPO is an on-policy RL method that uses only trajectories collected by the current policy, while SAC is an off-policy RL method that use trajectories collected by previous policies.
On-policy methods are generally more stable than off-policy methods, while off-policy methods are generally more data-efﬁcient (Gu et al., 2017).
We use SAC for Humanoid mainly due to its high data-efﬁciency.
When SAC is used, we also use trajectories collected by previous policies to approximate the expectation over the trajectory density ˜ q θ p s 1: T , a 1: T q .
For the distribution p ω p u t | s t , a t , k q “ N p u t | a t , C ω p k qq , we use diagonal covariances C ω p k q “ diag p c k q , where ω “ t c k u Kk “ 1 and c k P R d a ` are parameter vectors to be learned.
For the distribution q ψ p a t | s t , u t , k q , we use a Gaussian distribution with diagonal covariance, where the mean and logarithm of the standard deviation are the outputs of neural networks.
Since k is a discrete variable, we represent q ψ p a t | s t , u t , k q by neural networks that have K output heads and take input vectors p s t , u t q ; The k -th output head corresponds to (the mean and log-standard-deviation of) q ψ p a t | s t , u t , k q .
We also pre-train the mean function of q ψ p a t | s t , u t , k q , by performing least-squares regression for 1000 gradient steps with target value u t .
This pre-training is done to obtain reasonable initial predictions.
For the policy q θ p a t | s t q , we use a Gaussian policy with diagonal covariance, where the mean and logarithm of the standard deviation are outputs of neural networks.
We use Σ “ 10 ´ 8 I in experiments.
To control exploration-exploitation trade-off, we use an entropy coefﬁcient α “ 0 . 0001 in TRPO.
In SAC, the value of α is optimized so that the policy has a certain value of entropy, as described by Haarnoja et al.
Note that including α in VILD is equivalent to rescaling quantities in the model by α , i.e., exp p
r φ p s t , a t q{ α q and p p ω p u t | s t , a t , k qq 1 α .
A discount factor 0 ă γ ă 1 may be included similarly, and we use γ “ 0 . 99 in experiments.
For all methods, we regularize the reward/discriminator function by the gradient penalty (Gulrajani et al., 2017) with coefﬁcient 10 , since it was previously shown to improve performance of generative adversarial learning methods.
For methods that learn a reward function, namely VILD, AIRL, and MaxEnt-IRL, we apply a sigmoid function to the output of a reward network to bound reward values.
We found that without the bounds, reward values of the agent can be highly negative in the early stage of learning, which makes RL methods prematurely converge to poor policies.
An explanation of this phenomenon is that, in MDPs with large state and action spaces, distribution of demonstrations and distribution of agent’s trajectories are not overlapped in the early stage of learning.
In such a scenario, it is trivial to learn a reward function which tends to positive-inﬁnity values for demonstrations and negative-inﬁnity values for agent’s trajectories.
While the gradient penalty regularizer slightly remedies this issue, we found that the regularizer alone is insufﬁcient to prevent this scenario.
Moreover, for VILD, it is beneﬁcial to bound the reward function to control a trade-off between the immediate reward and the squared error when optimizing ψ .
A pseudo-code of VILD with IS is given in Algorithm 1, where the reward parameter is updated by IS gradient in line 8.
For VILD without IS, the reward parameter is instead updated by an estimate of ∇ φ H p φ , ω , ψ , θ q .
The regularizer L p ω q “ T E p p k
q r log | C ´ 1 ω p k q|s{ 2 penalizes large value of C ω p k q .
A source-code of our implementation will be publicly available.
C E XPERIMENT D ETAILS
In this section, we describe experimental settings and data generation.
We also give brief reviews of methods compared against VILD in the experiments.
[Table]
[Table]
C. E XPERIMENTAL SETTING AND DATA GENERATION FOR BENCHMARK TASKS
For the benchmark experiment in Section 5.1, we evaluate VILD on four continuous-control benchmark tasks from OpenAI gym platform (Brockman et al., 2016) with the Mujoco physics simulator: HalfCheetah, Ant, Walker2d, and Humanoid.
To obtain the optimal policy for generating demonstrations, we use the ground-truth reward function of each task to pre-train π ‹ with TRPO.
We generate diverse-quality demonstrations by using K “ 10 demonstrators according to the graphical model in Figure 1(b).
We consider two types of the noisy policy p p u t | s t , a t , k q : a Gaussian noisy policy and a time-signal-dependent (TSD) noisy policy.
Gaussian noisy policy.
We use a Gaussian noisy policy N p u t | a t , σ 2 k I q with a constant covariance.
The value of σ k for each of the 10 demonstrators is 0 .
9 and 1 . 0 , respectively.
Note that our model assumption on p ω corresponds to this Gaussian noisy policy.
Table 2 shows the performance of demonstrators (in terms of cumulative ground-truth rewards) with this Gaussian noisy policy.
A random policy π 0 is an initial policy neural network for learning; The network weights are initialized such that the magnitude of actions is small.
Note that this initialization scheme is a common practice in deep RL (Gu et al., 2017).
[Figure]
TSD noisy policy.
To make learning more challenging, we generate demonstrations according to a noise characteristic of human motor control, where a magnitude of noises is proportion to a magnitude of actions and increases with execution time (van Beers et al., 2004).
Speciﬁcally, we generate demonstrations using a Gaussian distribution
N p u t | a t , diag p b k p t q ˆ } a t } 1 { d a qq , where the covariance is propor- tion to the magnitude of action and depends on time steps and ˆ denotes an element-wise product.
We call this policy time-signal- dependent (TSD) noisy policy.
Here, b k p t q is a sample of a noise process whose noise variance increases over time, as shown in Figure 6.
We obtain this noise process for the k -th demonstrator by reversing
Ornstein–Uhlenbeck (OU) processes with parameters θ “ 0 . 15 and σ “ σ k (Uhlenbeck & Ornstein, 1930) 4 .
The value of σ k for each demonstrator is 0 . 01 , 0 .
9 , and 1 . 0 , respectively.
Table 3 shows the performance of demonstrators with this TSD noisy policy.
Learning from demonstrations generated by
TSD is challenging; The Gaussian model of p ω cannot perfectly model the TSD noisy policy, since the ground-truth variance is a function of actions and time steps. C. E XPERIMENTAL SETTING FOR ROBOSUITE REACHING TASK
For the real-world data experiment in Section 5.2, we use a robot control task from the robosuite environment Fan et al.
(2018) and a crowdsourced demonstration dataset from Mandlekar et al.
These demonstrations are collected for object-manipulation tasks such as assembly tasks.
These object-manipulation tasks require generate the the the time We use the publicly available dataset html the agent to perform three subtasks: reaching, picking, and placing. In our preliminary experiments, none of IL methods successfully learns object-manipulation policies, since the agent often fails at picking the object. We expect that a hierarchical policy is necessary to perform these manipulation tasks, due to the hierarchical structure (i.e., subtasks) of these tasks.
Since hierarchical IL is not in the scope of this paper, we consider the subtask of reaching where non-hierarchical policy sufﬁces, and leave an extension of VILD to hierarchical policy for future work.
In this experiment, we consider the subtask of reaching, which is still challenging for IL due to diverse quality of crowdsourced demonstrations.
To obtain reaching demonstrations from the original object-manipulation demonstrations (we use the SawyerNutAssemblyRound dataset), we terminate demonstrations after the robot’s end-effector contacts the target object.
After applying such a termination procedure, the dataset used in this experiment consists of 10 randomly chosen demonstrations ( N “ 10 ) whose length T is approximately 500 time steps.
The number of state-action pairs in this demonstration dataset is approximately 5000.
Since we do not know the actual number of demonstrators that collected these N “ 10 demonstrations, we use the strategy described in Section 3.3; we set K “ N and k “ n .
We use true states of the robot and do not use visual observations.
Since the reaching task does not require picking the object, we disable the gripper control command of the robot.
The state space of this task is S P R 44 , and the action space of this task is A P R 7 .
The performance of learned policies are evaluated using a reward function whose values are inverse proportion to the distance between the object and the end-effector (i.e., small distance yields high reward).
We repeat the experiment for 5 trials using the same dataset and report the average performance (undiscounted cumulative rewards).
For each trial, we generate 100 test trajectories for evaluating the performance.
Note that the number of test trajectories in this experiment is larger than that in the benchmark experiments.
This is because the initial states of this reaching task is much more varied than those in benchmark tasks.
We do not evaluate VILD without
IS and VAIL, since in benchmarks VILD with IS performs better than VILD without IS and VAIL is comparable to GAIL.
For all methods, we use neural networks with 2 hidden-layers of 100 relu units.
We update policy parameters by TRPO with the same hyper-parameters as the benchmark experiments.
We pre-train the mean of Gaussian policies for all methods by behavior cloning (i.e., we apply 1000 gradient descent steps of least-squares regression).
To pre-train InfoGAIL which learns a context-dependent policy, we use the variable k as context for pre-training.
For VILD, we apply the log-sigmoid function to the reward function.
Speciﬁcally, we parameterize the reward function as r φ p s , a q “ log D φ p s , a q where D φ p s , a q “ exp p d φ p s , a qq exp p
d
φ p s , a qq` 1 and d φ : S ˆ A Ñ R .
We also apply a substitution ´ log D φ p s , a q Ñ log p 1 ´ D
φ p s , a qq , which is a common practice in GAN literature (Fedus et al., 2018).
By doing so, we obtain an objective of VILD that closely resembles the objective of GAIL:
[Equation]
We use this variant of VILD in this experiment since it performs better than VILD with the standard reward function.
Although we omit the IS distribution in this equation for clarity, we use IS in this experiment.
C. C OMPARISON METHODS
Here, we brieﬂy review methods compared against VILD in our experiments.
We ﬁrstly review online IL methods, which learn a policy by RL and require additional transition samples from MDPs.
MaxEnt-IRL.
Maximum (causal) entropy IRL (MaxEnt-IRL)
(Ziebart et al., 2010) is a well-known
IRL method.
The original derivation of the method is based on the maximum entropy principle (Jaynes, 1957) but for causal interactions, and uses a linear-in-parameter reward function: r φ p s t , a t q “ φ J b p s t , a t q with a basis function b .
Here, we consider an alternative derivation which is applicable to non- linear reward function (Finn et al., 2016).
Brieﬂy speaking, MaxEnt-IRL learns a reward parameter by minimizing a KL divergence from a data distribution p ‹ p s 1: T , a 1: T q to a model p φ p s 1: T , a 1: T q “
Z
[Equation]
p
[Equation]
q Π Tt “
[Equation]
p p s [Equation]
| s t ,
[Equation]
t q exp p r φ p s t , a t q{ α q , where Z φ is the normalization term.
Minimizing this KL divergence is equivalent to solving max φ E p ‹ p s 1: T , a 1: T q “
[Equation] log Z φ . To compute log Z φ , we can use the importance sampling approach (Finn et al., 2016) or the variational approache as done in VILD.
The latter leads to a max-min problem max
[Equation]
where q θ p s 1: T , a 1: T q “ p 1 p s 1 q Π
Tt “ 1 p p s t ` 1 | s t , a t q q θ p a t | s t q .
The policy q
As we mentioned, the proposed model in VILD is based on the model of MaxEnt-IRL.
By comparing the max-min problem of MaxEnt-IRL and the max-min problem of VILD, we can see that the main difference are the variational distribution q ψ and the noisy policy model p ω .
If we assume that q ψ and p ω are Dirac delta functions: q ψ p a t | s t , u t , k q “ δ a t “ u t and p ω p u t | a t , s t , k q “ δ u t “ a t , then the max-min problem of VILD reduces to the max-min problem of MaxEnt-IRL.
In other words, if we assume that all demonstrators execute the optimal policy and have an equal level of expertise, then VILD reduces to MaxEnt-IRL.
GAIL.
Generative adversarial IL (GAIL) (Ho & Ermon, 2016) performs occupancy measure matching via gen- erative adversarial networks (GAN) to learn the optimal policy from expert demonstrations.
Speciﬁcally, GAIL ﬁnds a parameterized policy π θ such that the occupancy measure ρ π θ p s , a q of π θ is similar to the occupancy measure ρ π ‹ p s , a q of π ‹ .
Here, ρ π p s , a q “ E p π p s 1: T , a 1: T q r Σ Tt “ 0 δ p s t ´ s , a t ´ a qs is the state-action occupancy measure of π and satisﬁes the equality E p
π p s 1: T , a 1: T q r Σ Tt “ 1 r p s t , a t qs “ ť
S ˆ
A ρ π p s , a q r p s , a q d s d a “ E π r r p s , a qs .
To measure the similarity, GAIL uses the Jensen-Shannon divergence, which is estimated and minimized by the following generative-adversarial training objective:
[Equation]
| s t qs , where D φ p s , a q “ exp p d φ p s , a qq exp p d φ p s , a qq` 1 is called a discriminator.
The minimization problem w.r.t.
θ is achieved using RL with a reward function ´ log p [-
-]1 ´ D
φ p s , a qq .
AIRL.
Adversarial IRL (AIRL) (Fu et al., 2018) was proposed to overcome a limitation of GAIL regarding reward function: GAIL does not learn the expert reward function, since GAIL has D φ p s , a q “ 0 . 5 at the saddle point for every states and actions.
To overcome this limitation while taking advantage of generative-adversarial training, AIRL learns a reward function by solving
[Equation]
where D φ p s , a q “ exp p r φ p s , a qq exp p
r φ p s , a qq` q θ p a | s q .
The policy q θ p a t | s t q is learned by RL with a reward function r φ p s t , a t q .
Fu et al.
(2018) showed that the gradient of this objective w.r.t.
φ is equivalent to the gradi- ent of MaxEnt-IRL w.r.t.
φ .
The authors also proposed an approach to disentangle reward function, which leads to a better performance in transfer learning settings.
Nonetheless, this disentangle approach is general and can be applied to other IRL methods, including MaxEnt-IRL and VILD.
We do not evaluate AIRL with disentangle reward function.
We note that, based on the relation between MaxEnt-IRL and VILD, we can extend VILD to use a training procedure of AIRL.
Speciﬁcally, by applying the same derivation from MaxEnt-IRL to AIRL by Fu et al.
(2018), we can derive a variant of VILD which learns a reward parameter by solving max
[Equation]
We do not evaluate this variant of VILD in our experiment.
VAIL.
Variational adversarial IL (VAIL) (Peng et al., 2019) improves upon GAIL by using variational information bottleneck (VIB)
(Alemi et al., 2017).
VIB aims to compress information ﬂow by minimizing a variational bound of mutual information.
This compression ﬁlters irrelevant signals, which leads to less over-ﬁtting.
To achieve this in GAIL, VAIL learns the discriminator D φ by an optimization problem
[Equation]
where z is an encode vector,
E p
z
| s , a q is an encoder, p p
z q is a prior distribution of z , I c is the target value of mutual information, and β ą 0 is a Lagrange multiplier.
With this discriminator, the policy π θ p a t | s t q is learned by RL with a reward function ´ log p
´ D φ p E E p
z | s , a q r z sqq .
It might be expected that the compression may make VAIL robust against diverse-quality demonstrations, since irrelevant signals in low-quality demonstrations are ﬁltered out via the encoder.
However, we ﬁnd that this is not the case, and VAIL does not improve much upon GAIL in our experiments.
This is perhaps because VAIL compress information from both demonstrators and agent’s trajectories.
Meanwhile in our setting, irrelevant signals are generated only by demonstrators.
Therefore, the information bottleneck may also ﬁlter out relevant signals in agent’s trajectories, which lead to poor performances.
InfoGAIL.
Information maximizing GAIL (InfoGAIL)
(Li et al., 2017) is an extension of GAIL for learning a multi-modal policy in MM-IL.
The key idea of InfoGAIL is to introduce a context variable z to the GAIL formulation and learn a context-dependent policy π θ p a | s , z q , where each context represents each mode of the multi-modal policy.
To ensure that the context is not ignored during learning, InfoGAIL regularizes GAIL’s objective so that a mutual information between contexts and state-action variables is maximized.
This mutual information is indirectly maximized via maximizing a variational lower-bound of mutual information.
By doing so, InfoGAIL solves a min-max problem
[Equation]
where L p π θ , Q q “ E p p z q π θ p a | s ,z q r log Q p z | s , a q ´ log p
p
z qs is a lower-bound of mutual information, Q p z | s , a q is an encoder neural network, and p p
z q is a prior distribution of contexts.
In our experiment, the number of context z is set to be the number of demonstrators K .
As discussed in Section 1, when knowing the level of demonstrators’ expertise, we may choose contexts that correspond to high-expertise demonstrator.
In other words, we may hand-craft the prior distribution p p
z q so that a probability of contexts is proportion to the level of demonstrators’ expertise.
Nonetheless, for fair comparison, we do not use the oracle knowledge about the level of demonstrators’ expertise, and set p p z q to be a uniform distribution.
For the Humanoid task in our experiment, we use the Wasserstein-distance variant of InfoGAIL (Li et al., 2017), since the Jensen-Shannon-divergence variant does not perform well in this task.
Next, we review ofﬂine IL methods.
These methods learn a policy based on supervised learning and do not require additional transition samples from MDPs.
BC.
Behavior cloning (BC) (Pomerleau, 1988) is perhaps the simplest IL method.
BC treats an IL problem as a standard supervised learning problem and ignores dependency between states distributions and policy.
For continuous action space, BC solves a least-square regression problem to learn a parameter θ of a deterministic policy π
[Equation]
BC-D. BC with Diverse-quality demonstrations (BC-D) is a simple extension of BC for handling diversequality demonstrations.
This method is based on the naive model in Section 3.1, and we consider it mainly for evaluation purpose.
BC-D uses supervised learning to learn a policy parameter θ and expertise parameter ω of a model p θ , ω p s 1: T , u 1: T , k q “ p p k
q p p s 1 q Σ Tt “ 1 p p s t ` 1 | s t , u t q ş A π θ p a t | s t q p ω p u t | s t , a t , k q d a t .
To learn the parameters, we minimize the KL divergence from data distribution to the model.
By using the variational approach to handle integration over the action space, BC-D solves an optimization problem [Equation] ıı
[Equation]
where q ν p a t | s t , u t , k q is a variational distribution with parameters ν .
We note that the model p θ , ω p s 1: T , u 1: T , k q of BC-D can be regarded as a regression-extension of the two-coin model proposed by Raykar et al.
(2010) for classiﬁcation with noisy labels.
Co-teaching.
Co-teaching (Han et al., 2018) is the state-of-the-art method to perform classiﬁcation with noisy labels.
This method trains two neural networks such that mini-batch samples are exchanged under a small loss criteria.
We extend this method to learn a policy by least-square regression.
Speciﬁcally, let π θ 1 p s t q and π θ 2 p
s t q be two neural networks representing policies, and ∇ θ L p θ , B q “ ∇ θ Σ p s , a qP B } a
´ π θ p s q} 22 be gradients of a least-square loss estimated by using a mini-batch B .
The parameters θ 1 and θ 2 are updated by iterates: [Equation]
The mini-batch B θ 2 for updating θ 1 is obtained such that B θ 2 incurs small loss when using prediction from π θ 2 , i.e., B θ 2 “ argmin B 1 L p θ 2 , B 1 q .
Similarly, the mini-batch B θ 1 for updating θ 2 is obtained such that B θ 1 incurs small loss when using prediction from π θ 1 .
For evaluating the performance, we use the policy network π θ 1 .
D A DDITIONAL EXPERIMENTAL RESULTS
Results against online IL methods.
Figure 7 shows the learning curves of VILD and existing online IL methods against the number of transition samples.
It can be seen that for both types of noisy policy, VILD with and without IS outperform existing methods overall, except on the Humanoid tasks where most methods achieve comparable performance.
Results against ofﬂine IL methods.
Figure 8 shows learning curves of ofﬂine IL methods, namely BC,
BC-D, and Co-teaching.
For comparison, the ﬁgure also shows the ﬁnal performance of VILD with and without IS, according to Table 1.
We can see that these ofﬂine methods do not perform well, especially on the highdimensional Humanoid task.
The poor performance of these methods is due to the issues of compounding error and low-quality demonstrations.
Speciﬁcally, BC performs the worst, since it suffers from both issues.
Still, BC may learn well in the early stage of learning, but its performance sharply degrades, as seen in Ant and Walker2d.
This phenomena can be explained as an empirical effect of memorization in deep neural networks (Arpit et al., 2017).
Namely, deep neural networks learn to remember samples with simple patterns ﬁrst (i.e., high-quality demonstrations from experts), but as learning progresses the networks overﬁt to samples with difﬁcult patterns (i.e., low-quality demonstrations from amateurs).
Co-teaching is the-state-of-the-art method to avoid this effect, and we can see that it performs signiﬁcantly better than BC.
Meanwhile, BC-D, which learns the policy and level of demonstrators’ expertise, also performs better than BC and is comparable to Co-teaching.
Nonetheless, the performance of Co-teaching and BC-D is still much worse than VILD with IS.
Accuracy of estimated expertise parameter.
Figure 9 shows the estimated parameters ω “ t c k u Kk “ 1 of
N p u t | a t , diag p c k qq and the ground-truth variance t σ 2 k u Kk “ 1 of the Gaussian noisy policy N p u t | a t , σ 2 k I q .
The results show that VILD learns an accurate ranking of the variance compared to the ground-truth.
The values of these parameters are also quite accurate compared to the ground truth, except for demonstrators with low-levels of expertise.
A possible reason for this phenomena is that low-quality demonstrations are highly dissimilar, which makes learning the expertise more challenging.
We can also see that the difference between expertise parameters of VILD with IS and VILD without IS is small and negligible.
InfoGAIL with different values of context. Figure 10 shows the learning curves of InfoGAIL across different values of context z . We can see that the performance of InfoGAIL depends on the context, i.e., there is a discrepancy between the best and worst performances of InfoGAIL. The discrepancy is clearer in the Walker2d task with the TSD noisy policy and in the robosuite reaching task (Figure 4).
[Figure]
Humanoid
[Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]