U NDERSTANDING THE R ELATION B ETWEEN M AXIMUM -E NTROPY I NVERSE R EINFORCEMENT L EARNING AND B EHAVIOUR C LONING
A BSTRACT
In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations.
The most common approaches under this framework are Behaviour Cloning (BC), and Inverse
Reinforcement Learning (IRL).
Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods often fail.
Unfortunately, directly comparing the algorithms for these methods does not provide adequate intuition for understanding this difference in performance.
This is the motivating factor for our work.
We begin by presenting f -MAX, a generalization of AIRL (Fu et al., 2017), a state-of-the-art IRL method.
f -MAX provides grounds for more directly comparing the objectives for LfD objectives.
We empirically evaluate the factors of difference between these objectives in the continuous control domain.
I NTRODUCTION
Modern advances in reinforcement learning aim to alleviate the need for hand-engineered decisionmaking and control algorithms by designing general purpose methods that learn to optimize provided reward functions.
In many cases however, it is either too challenging to optimize a given reward (e.g. due to sparsity of signal), or it is simply impossible to design a reward function that captures the intricate details of desired outcomes.
One approach to overcoming such hurdles is Learning from Demonstrations (LfD), where algorithms are provided with expert demonstrations of how to accomplish desired tasks.
The most common approaches in the LfD framework are Behaviour Cloning (BC) and Inverse Reinforcement Learning (IRL) (Russell, 1998;
Ng et al., 2000).
In standard BC, learning from demonstrations is treated as a supervised learning problem and policies are trained to regress expert actions from a dataset of expert demonstrations.
Other forms of Behaviour Cloning, such as DAgger (Ross et al., 2011), consider how to make use of an expert in a more optimal fashion.
On the other hand, in
IRL the aim is to infer the reward function of the expert, and subsequently train a policy to optimize this reward.
The motivation for IRL stems from the intuition that the reward function is the most concise and portable representation of a task (Ng et al., 2000; Abbeel & Ng, 2004).
Unfortunately, the standard IRL formulation (Ng et al., 2000) faces degeneracy issues 1 .
A successful framework for overcoming such challenges is the Maximum-Entropy (Max-Ent) IRL method (Ziebart et al., 2008; Ziebart, 2010).
A line of research stemming from the Max-Ent IRL framework has lead to recent “adversarial” methods (Ho & Ermon, 2016; Finn et al., 2016a;
Fu et al., 2017).
These approaches aim to directly recover the expert policy ensuing from the Max-Ent IRL process without explicitly modelling the reward function, and have shown tremendous success.
In benchmarks for continuous control (Brockman et al., 2016), these methods outperform Behaviour
Cloning by a wide margin, particularly in the low data regime where a very limited number of expert trajectories are available.
However, it is not immediately clear why adversarial Max-Ent IRL methods would outperform BC, since at optimality both methods exactly recover the expert policy.
This questions motivates the work presented here.
Drawing upon the literature on f -divergences (Lin, 1991; Nowozin et al., 2016), we begin by presenting f -MAX, an algorithm for Max-Ent IRL.
We demonstrate how f -MAX generalizes AIRL (Fu et al., 2017), and provides new intuition for what this algorithm accomplishes.
Speciﬁcally we demonstrate that the objective in AIRL is equivalent to minimizing the reverse KL divergence between the joint state-action marginal distribution of the policy and that of the expert.
From these ﬁndings, we generate hypotheses for why direct Max-Ent IRL methods outperform BC, and empirically evaluate them in continuous control benchmarks.
To tease apart the differences between standard BC and AIRL, we also address the degeneracy of f -MAX in a special case, and provide a one-line modiﬁcation of AIRL, named FAIRL, which minimizes the forward KL divergence between the joint state-action marginal of the expert and the policy.
We discuss how our ﬁndings may relate to common observations (Bishop, 2006) regarding the mode-covering/modeseeking behaviour of the different KL divergence directions.
B ACKGROUND 2. M AXIMUM E NTROPY I NVERSE R EINFORCEMENT L EARNING
Consider a Markov Decision Process (MDP) represented as a tuple ( S , A , P , r, ρ 0 , γ ) with statespace S , action-space A , dynamics P : S ×
A × S → [0 , 1] , reward function r ( s, a ) , initial state distribution ρ 0 , and discount factor γ ∈ (0 , 1) .
In Maximum Entropy (Max-Ent) reinforcement learning (Todorov, 2008; Toussaint, 2009; Rawlik et al., 2013; Fox et al., 2015; Haarnoja et al., 2017; 2018), the goal is to ﬁnd a policy π such that trajectories sampled using this policy follow the distribution
[Equation]
where τ = ( s 0 , a 0 , s 1 , a 1 , ... ) denotes a trajectory, and R ( τ )
= (cid:80) t r ( s t , a t ) and Z is the partition function.
Hence, trajectories that accumulate more reward are exponentially more likely to be sampled.
Converse to the standard RL setting, in Max-Ent IRL (Ziebart et al., 2008; Ziebart, 2010) we are instead presented with an optimal policy π exp , or more realistically, sample trajectories from such a policy, and we seek to ﬁnd a reward function r that maximizes the likelihood of the trajectories sampled from π exp .
Formally, our objective is:
[Equation]
Being an energy-based modelling objective, the difﬁculty in performing this optimization arises from estimating the partition function Z .
Initial methods addressed this problem using dynamic programming (Ziebart et al., 2008; Ziebart, 2010), and recent approaches present methods aimed at intractable domains with unknown dynamics (Finn et al., 2016b; Ho & Ermon, 2016; Finn et al., 2016a; Fu et al., 2017; Kostrikov et al., 2018).
Instead of recovering the expert’s reward function and policy, recent successful methods in Max-Ent
IRL aim to directly recover the policy that would result from the full process.
Since such methods only recover the policy, it would be more accurate to refer to them as Imitation Learning algorithms.
However, to avoid confusion with Behaviour Cloning methods, in this work we will refer to them as direct methods for Max-Ent IRL.
GAIL: Generative Adversarial Imitation Learning Before describing the work of Ho & Ermon (2016), we establish the deﬁnition of causal entropy H causal ( π ) : = E ρ π ( s ) [ log π ( a | s )] (Ziebart, 2010; Bloem & Bambos, 2014).
Intuitively, causal entropy can be thought of as the “amount of options” the policy has in each state, in expectation.
Let C denote a class of cost functions (negative reward functions).
Furthermore, let ρ exp ( s, a ) , ρ π ( s, a ) denote the state-action marginal distributions of the expert and student policy respectively.
Ho & Ermon (2016) begin with a regularized Max-Ent IRL objective,
[Equation]
where ψ : C → R is a convex regularization function on the space of cost functions, and IRL ψ ( π exp ) returns the optimal cost function given the expert and choice of regularization. Also, while not immediately clear, note that min π −H causal ( π ) + E π [ c ( s, a )] is the Max-Ent RL objective given cost function c ( s, a ) . Let RL ( c ) :
=
arg min π −H causal ( π ) + E π
[ c ( s, a )] , be a function that returns the optimal Max-Ent policy given cost c ( s, a ) .
Ho & Ermon (2016) show that
[Equation]
where ψ ∗ denotes the convex conjugate of ψ .
This tells us that if we were to ﬁnd the cost function c ( s, a ) using the regularized Max-Ent IRL objective 3, and subsequently ﬁnd the optimal Max-Ent policy for this cost, we would arrive at the same policy had we directly optimized objective 4 by searching for the policy.
Directly optimizing 4 is challenging for many choices of ψ .
Interestingly however, Ho & Ermon (2016) show that for any symmetric f -divergences (Lin, 1991), there exists a choice of ψ such that equation 4 is equivalent to RL ◦ IRL ψ ( π exp ) = arg min π H causal ( π ) + D f ( ρ π ( s, a ) || ρ exp ( s, a )) .
In such settings, due to a close connection between binary classiﬁers and symmetric f -divergences (Nguyen et al., 2009), efﬁcient algorithms can be formed. The special case for Jensen-Shannon divergence leads to the successful method dubbed Generative Adversarial Imitation Learning (GAIL). As before, let ρ exp ( s, a ) , ρ π ( s, a ) denote the state-action marginal distributions of the expert and student policy respectively.
Let D ( s, a ) : S ×A → [0 , 1] be a binary classiﬁer - often referred to as the discriminator - for identifying positive samples (sampled from ρ exp ( s, a ) ) from negative samples (sampled from ρ π ( s, a ) ).
Using RL, the student policy is trained to maximize E τ ∼ π
[ (cid:80) t log D ( s t , a t )] − λ H causal ( π ) , where λ is a hyperparameter. The training procedure alternates between optimizing the discriminator and updating the policy.
As noted, it is shown that this training procedure minimizes the Jensen-Shannon divergence between ρ exp ( s, a ) and ρ π ( s, a ) (Ho & Ermon, 2016).
AIRL: Adversarial Inverse Reinforcement Learning
Subsequent to the advent of GAIL (Ho & Ermon, 2016), Finn et al.
(2016a)
present a theoretical discussion relating Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), IRL, and energy-based models. They demonstrate how an adversarial training approach could recover the Max-Ent reward function and simultaneously train the Max-Ent policy corresponding to that reward.
Building on this discussion, Fu et al.
(2017) present a practical implementation of this method, named Adversarial Inverse Reinforcement Learning (AIRL).
As before, let ρ exp ( s, a ) , ρ π ( s, a ) denote the state-action marginal distributions of the expert and student policy respectively and let D ( s, a ) : S
×
A → [0 , 1] be the discriminator.
In AIRL, the discriminator is parameterized as,
[Equation]
where f ( s, a ) : S ×
A → R , and π ( a | s ) denotes the likelihood of the action under the policy.
AIRL deﬁnes the reward function, r ( s, a ) :
= log D ( s, a ) − log (1 − D ( s, a )) , and sets the objective for the student policy to be the RL objective, max π E τ ∼ π
[ (cid:80) t r ( s t , a t )] .
As in GAIL, this leads to an iterative optimization process alternating between optimizing the discriminator and the policy. At convergence, the advantage function of the expert is recovered.
Given this observation, important considerations are made regarding how to extract the true reward function from f ( s, a ) .
When the objective is only to perform Imitation Learning, and we do not care to recover the reward function, the discriminator does not use the special parameterization discussed above and is instead direclty represented as a function D ( s, a ) : S
×
A → [0 , 1] , as done in GAIL (Ho & Ermon, 2016).
Performance With Respect to BC Methods such as GAIL and AIRL have demonstrated significant performance gains compared to Behaviour Cloning.
In particular, in standard Mujoco benchmarks (Todorov et al., 2012; Brockman et al., 2016), adversarial methods for Max-Ent IRL achieve strong performance using a very limited amount of demonstrations from an expert policy, an important failure scenario for standard Behaviour Cloning.
2. f -D IVERGENCES
Ho & Ermon (2016) demonstrate that Max-Ent IRL is the dual problem of matching ρ π ( s, a ) to ρ exp ( s, a ) ; indeed as noted above, GAIL (Ho & Ermon, 2016) optimizes the Jensen-Shannon divergence between the two distributions.
In section 3 we present f -MAX, a method for matching ρ π ( s, a ) to ρ exp ( s, a ) using arbitrary f -divergences (Lin, 1991).
Hence, in this section we recall this class of statistical divergences as well as methods for using them for training generative models.
Let P, Q be two distributions with density functions p, q .
For any convex, lower-semicontinuous function f : R + → R a statistical divergence can be deﬁned as: D f ( P || Q ) = (cid:82) χ q ( x ) f (cid:16) p ( x ) q ( x ) (cid:17) .
Divergences derived in this manner are called f-divergences and amongst many interesting divergences, include the forward and reverse KL.
Nguyen et al.
(2010) present a variational estimation method for f -divergences between arbitrary distributions P, Q. Using the notation of Nowozin et al.
we can write,
[Equation]
where T is an arbitrary class of functions T ω : X → R , and f ∗ is the convex conjugate of f .
Under mild conditions (Nguyen et al., 2010) equality holds between the two sides.
Motivated by this variational approximation as well as Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), Nowozin et al.
(2016) present an iterative optimization scheme for matching an implicit distribution 2 Q to a ﬁxed distribution P using any f -divergence.
For a given f -divergence, the corresponding minimax optimization is,
[Equation]
Nowozin et al.
(2016) discuss practical parameterizations of T ω , but to avoid notational clutter we will use the form above.
f -MAX: f -D IVERGENCE M AX -E NT IRL
We begin by presenting f -MAX, a generalization of AIRL (Fu et al., 2017) which provides a more intuitive interpretation of what similar algorithms accomplish.
Imagine, for some f , we aim to train a policy by optimizing the f -divergence
D f ( ρ exp ( s, a ) ||
ρ π ( s, a )) .
To do so, we propose the following iterative optimization procedure,
[Equation]
) generators where f ∗ and T ω are as deﬁned in section 2.2.
Equation 8 is the same as the inner maximization of the f -GAN objective in equation 7; this objective optimizes T ω so that equation 8 best approximates
D f ( ρ exp ( s, a ) ||
ρ π ( s, a )) .
On the other hand, for the policy objective, using the identities in appendix A we have,
[Equation]
which implies that the policy objective is equivalent to minimizing equation 8 with respect to π . With an identical proof as in Goodfellow et al. (2014, Proposition 2), if in each iteration the optimal
T ω is found, the described optimization procedure converges to the global optimum where the policy’s state-action marginal distribution matches that of the expert’s.
This is equivalent to iteratively computing D
f
( ρ exp ( s, a ) ||
ρ π ( s, a )) and optimiizing the policy to minimize it.
3. C OROLLARY : A S IMPLE D ERIVATION AND I NTUITION FOR AIRL
Choosing f ( u ) :
= − log u leads to D f ( ρ exp ( s, a ) ||
ρ π ( s, a ))
=
KL ( ρ π ( s, a ) ||
ρ exp ( s, a )) .
This divergence is commonly referred to as the “reverse” KL divergence.
In this setting we have, f ∗ ( t ) = − 1 − log ( − t ) , and T πω
( s, a )
= − ρ π ( s,a ) ρ exp ( s,a )
(Nowozin et al., 2016).
Hence, given T πω , the policy objective in equation 9 takes the form,
[Equation]
On the other hand, plugging the optimal discriminator D π ( s, a ) =
[Equation]
Goodfellow et al., 2014) into the AIRL (Fu et al., 2017) policy objective, we get,
[Equation]
As can be seen, the right hand side of equation 12 matches that of equation 11 up to a constant 3 , meaning that AIRL is solving the Max-Ent IRL problem by minimizing the reverse KL divergence,
[Equation]
3. R ELATION TO GAIL
As discussed above, Ho & Ermon (2016) present a class of methods for Max-Ent IRL that directly retrieve the expert policy without explicitly ﬁnding the reward function of the expert (sec. 2.1).
Using an interesting connection between surrogate cost functions for binary classiﬁcation and fdivergences (Nguyen et al., 2009), Ho & Ermon (2016) derive a special case of their method for minimizing any symmetric 4 f -divergence between ρ exp ( s, a ) and ρ π ( s, a ) .
Choosing the symmetric f -divergence to be the Jensen-Shannon divergence leads to the successful special case, GAIL (sec 2.1). Surprisingly, we now show that f -MAX is a subset of the cost-regularized Max-Ent IRL framework laid out in Ho & Ermon (2016)!
Recall the following equations from this framework, [Equation] We call an f -divergence symmetric if for any P,Q we have D f ( P || Q ) = D f ( Q, P )
[Table]
expert, and ρ agg 1: n ( s ) = 1 n (cid:80) ni
=1 ρ π ( i ) ( s ) .
where ψ ( c ) :
C → R was a closed, proper, and convex regularization function on the space of cost function, and ψ ∗ its convex conjugate.
For our proof we will operate in the ﬁnite state-action space, as in the original work (Ho & Ermon, 2016).
In this setting, cost functions can be represented as vectors in R S×A , and joint state-action distributions can be represented as vectors in [0 , 1] S×A .
Let f be the function deﬁning some fdivergence.
Given the expert for the task, we can deﬁne the following cost function regularizer,
[Equation]
[Equation]
where f ∗ is the convex conjugate of f .
Given this choice, with simple algebraic manipulation done in appendix B we have,
[Equation]
Typically, the causal entropy term is considered a policy regularizer, and is weighted by 0 ≤ λ ≤ 1 .
Therefore, modulo the term H causal ( π ) , our derivations show that f -MAX, and by inheritance AIRL (Fu et al., 2017), all fall under the cost-regularized Max-Ent IRL framework of Ho & Ermon (2016)!
U NDERSTANDING THE RELATIONS AMONG LEARNING - FROM - DEMONSTRATION ALGORITHMS
Given results derived in the prior section, we can now begin to populate table 1, writing various Imitation Learning algorithms in a common form, as the minimization of some statistical divergence between ρ exp ( s, a ) and ρ π ( s, a ) .
In Behaviour Cloning we minimize E ρ exp ( s ) [ KL ( ρ exp ( a | s ) ||
ρ π ( a | s ))] 5 .
On the other hand, the corollary in section 3.1 demonstrates that AIRL (Fu et al., 2017) minimizes KL ( ρ π ( s, a ) || ρ exp ( s, a )) , while GAIL (Ho & Ermon, 2016) optimizes D JS ( ρ exp ( s, a ) ||
ρ π ( s, a ))
− λ H causal ( π ) .
Hence, there are two ways in which the direct
IRL methods differ from BC.
First, in standard BC the policy is optimized to match the conditional distribution ρ exp ( a | s ) , whereas in the other two the policy is explicitly encouraged to match the marginal state distributions as well.
Second, in BC we make use of the forward KL divergence, whereas AIRL and GAIL use divergences that exhibit more mode-seeking behaviour.
These observations allow us to generate the following two hypotheses about why direct IRL methods outperform
BC, particularly in the low-data regime,
Hypothesis 1 In common MDPs of interest, the reward function depends more on the state than the action.
Hence it is plausible that matching state marginals is more useful than matching action conditional marginals.
Hypothesis
It is known that optimization using the forward KL divergence results in distributions with a mode-covering behaviour, whereas using the reverse KL results in modeseeking behaviour (Bishop, 2006).
Therefore, since in Reinforcement Learning we care about the “quality of trajectories”, being mode-seeking is more beneﬁcial than mode-covering, particularly in the low-data regime.
In what follows, we seek to experimentally evaluate our hypotheses.
To tease apart the differences between the direct Max-Ent IRL methods and BC, we present an algorithm that optimizes
KL ( ρ exp ( s, a ) ||
ρ π ( s, a )) .
We then compare its performance to Behaviour Cloning and the standard
AIRL algorithm using varying amounts of expert demonstrations.
D ERIVING F ORWARD KL
While f -MAX is a general algorithm, useful for most choices of f , it unfortunately cannot be used for the special case of forward KL, i.e. KL ( ρ exp ( s, a ) ||
ρ π ( s, a )) .
In the following sections we identify the problem and present a separate direct Max-Ent IRL method that optimizes this divergence.
5. D ERIVING F ORWARD KL FROM f -MAX
Let T πω denote the maximizer of equation 8 for a given policy π . For the case of forward KL, drawing upon equations from Nowozin et al.
we have,
[Equation]
Given this, the objective for the policy (equation 9) under the optimal T πω becomes,
[Equation]
Hence, there is no signal to train the policy!
5. A N A LTERNATIVE M ETHOD FOR F ORWARD KL
In this section we derive an algorithm for optimizing KL ( ρ exp ( s, a ) ||
ρ π ( s, a )) .
Similar to AIRL (Fu et al., 2017), let us have a discriminator, D
( s, a ) whose objective is to discriminate between expert and policy state-action pairs, [Figure]
[Equation]
We now deﬁne the objective for the policy to be,
[Equation]
In appendix C we show,
[Equation]
This is a refreshing result since it demonstrates that we can convert the AIRL algorithm (Fu et al., 2017) into its forward KL counterpart by simply modifying the reward function used; in AIRL (reverse KL) the reward is deﬁned as r ( s, a ) :
= log D ( s, a ) − log (1 − D ( s, a )) , whereas for forward KL it is deﬁned as r ( s, a ) :
= D ( s,a ) 1 − D ( s,a ) · log 1 − D ( s,a ) D ( s,a ) .
We refer to this forward KL version of AIRL as FAIRL.
If we parameterize the discriminator as D ( s, a ) := σ ( (cid:96) ( s, a )) , where σ represents the sigmoid activation function, the logit of the discriminator, (cid:96) ( s, a ) , is equal to log D ( s, a ) − log (1 − D ( s, a )) .
Hence, for an optimal discriminator, D π , we have (cid:96) π ( s, a ) = log ρ exp ( s,a ) ρ π ( s,a ) .
It is instructive to plot the reward functions under the two different settings as a function of (cid:96) π
( s, a ) ; ﬁgure 1 presents these plots.
As can be seen, in the forward KL version of AIRL, if for a state-action pair the expert puts more probability mass than the policy, the policy is severely punished.
However, if for some stateaction pairs the policy places a lot more mass than the expert, it almost does not matter.
As a result, the policy spreads its mass.
On the other hand, in the original AIRL formulation (reverse KL), the policy is always encouraged to put less mass than the expert.
These observations are in line with standard intuitions about the mode-covering/mode-seeking behaviours of the two KL divergences (Bishop, 2006).
E XPERIMENTS
In this section we provide empirical comparisons between AIRL, FAIRL, and standard BC in the
Ant and Halfcheetah environments found in Open-AI Gym (Brockman et al., 2016). [Figure]
6. S ETUP
Expert Policy To simulate access to expert demonstrations we train an expert policy using SoftActor-Critic (SAC) (Haarnoja et al., 2018), a state-of-the-art reinforcement learning algorithm for continuous control.
The expert policy consists of a 2-layer MLP with 256-dim layers, ReLU activations, and two output streams for the mean and the diagonal covariance of a Tanh(Normal( µ, σ )) distribution 7 .
We use the default hyperparameter settings for training the expert.
Evaluation Setup Using a trained expert policy, we generated 4 sets of expert demonstrations of that contain { 4 , 8 , 16 , 32 } trajectories.
Starting from a random offset, each trajectory is subsampled by a factor of 20.
This is standard protocol employed in prior direct methods for Max-Ent IRL (Ho & Ermon, 2016; Fu et al., 2017).
Also note that when generating demonstrations we sample from the expert’s action distribution rather than taking the mode.
This way, since the expert was trained using Soft-Actor-Critic, the expert should correspond to the Max-Ent optimal policy for the reward function 1 τ r g ( s, a ) , where τ is the SAC temperature used and r g ( s, a ) is the groundtruth reward function.
To compare the various learning-from-demonstration algorithms we train each method at each amount of expert demonstrations using 2 random seeds.
For each seed, we checkpoint the model at its best validation loss 8 throughout training.
At the end of training, the resulting checkpoints are evaluated on 50 test episodes.
Details for AIRL & FAIRL For AIRL and FAIRL, the student policy has an identical architecture to that of the expert, and the discriminator is a 2-layer MLP with 256-dim layers and Tanh activations.
We normalize the observations from the environment by computing the mean and standard deviations of the expert demonstrations.
The RL algorithm used for the student policies is SAC (Haarnoja et al., 2018), and the temperature parameter is tuned separately for AIRL & FAIRL.
Details for BC For BC, we use an identical architecture as the expert.
The model was ﬁt using
Maximum Likelihood Estimation 9 .
As before, the observations from the environment are normalized using the mean and standard deviation of the expert demonstrations.
6. R ESULTS & I NTERPRETATIONS
To match state-action marginals, the optimal student policy must sample actions from the stateconditional distribution, π ( a | s ) .
On the other hand, when we deploy a trained policy it is reasonable to instead choose the mode of this distribution, which we call the deterministic setting.
Here, we present evaluation results under the latter, and defer the results for the stochastic setting to the appendix.
[Figure] Figure 4 demonstrates that both AIRL and FAIRL outperform BC by a large margin, especially in the low data regime.
Speciﬁcally, the fact that FAIRL outperforms BC supports hypothesis 1 that the performance gain of Max-Ent IRL is not necessarily due to the direction of KL divergence used, but is the result of explicitly encouraging the policy to match the marginal state distribution of the expert in addition to the matching of conditional action distribution.
To compare AIRL and FAIRL, in ﬁgure 5 we plot the validation curves throughout training.
While
FAIRL has some gains over AIRL in the Ant environment, it does worse in Halfcheetah.
However in the stochastic evaluation setting AIRL seems to outperform FAIRL (appendix D).
Another trend that we do consistently observe is that, as we increase the amount of demonstrations, the performance of
FAIRL slightly diminishes.
Such observations provide initial positive support for hypothesis 2; as more expert demonstrations are provided, the policy trained with FAIRL broadens its distribution to cover the data-distribution, resulting in trajectories accumulating less reward in expectation.
We note however that more detailed experiments are necessary for adequately comparing the two methods.
C ONCLUSION & F UTURE W ORK
The motivation for this work stemmed from the superior performance of recent direct Max-Ent
IRL methods (Ho & Ermon, 2016; Fu et al., 2017) compared to BC in the low-data regime, and the desire to understand the relation between various approaches for Learning from Demonstrations.
We ﬁrst presented f -MAX, a generalization of AIRL (Fu et al., 2017), which allowed us to interpret AIRL as optimizing for KL ( ρ π ( s, a ) || ρ exp ( s, a )) .
Comparing to the standard BC objective, E ρ exp ( s ) [ KL ( ρ exp ( a | s ) ||
ρ π ( a | s ))] , we hypothesized two reasons for the superior performance of
AIRL: 1) the additional terms in the objective encouraging the matching of marginal state distributions, and 2) the direction of the KL divergence being optimized.
Setting out to empirically evaluate these claims we presented FAIRL, a one-line modiﬁcation of the AIRL algorithm that optimizes
KL ( ρ exp ( s, a ) ||
ρ π ( s, a )) .
FAIRL outperformed BC in a similar fashion to AIRL, which allowed us to conclude the key factor being the matching of state marginals.
Additional comparisons between
FAIRL and AIRL provided initial understanding about the role of the direction of the KL being optimized.
In future work we aim to produce results on a more diverse set of more challenging environments.
Additionally, evaluating other choices of f -divergence beyond forward and reverse KL may present interesting avenues for improvement (Wang et al., 2018).
Lastly, but importantly, we would like to understand whether the mode-covering behaviour of FAIRL could result in more robust policies ( et al., 2017).
A S OME U SEFUL I DENTITIES
Let h : S ×
A → R be an arbitrary function.
If all episodes have the same length T [Equation] we have,
[Equation]
In a somewhat similar fashion
[Equation]
in the inﬁnite horizon case with ﬁxed probability γ ∈ (
[Equation]
1) of transitioning to a terminal state
[Equation]
for the discounted sum below we have,
[Equation]
where Γ := 11 − γ is the normalizer of the sum (cid:80) t γ t .
Since the integral of an inﬁnite series is not always equal to the inﬁnite series of integrals [Equation] some analytic considerations must be made to go from equation 34 to 35.
But, one simple case in which it holds is when the ranges of h and all ρ π ( s t , a t ) are bounded [Equation]
B S IMPLE A LGEBRAIC M ANIPULATION
[Equation]
To go from 41 to 42 we simply changed notation T ω ( s, a ) :
= c ( s, a ) , and we can go from 42 to 43 because it is the exact same form as the variational characterization of f -divergences shown in equation ?? .
Note that equation 42 suggests the same training procedure as described for f -MAX.
C D ERIVATION FOR FAIRL
Below we present the derivation for equation 27.
Recalling deﬁnitions,
[Equation] [Equation]
45) and assuming the discriminator is optimal 10 , we have,
[Equation]
D S TOCHASTIC E VALUATION
[Figure] [Figure]