A DVERSARIAL I MITATION VIA V ARIATIONAL I NVERSE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider a problem of learning a reward and policy from expert examples under unknown dynamics in high-dimensional scenarios.
Our proposed method builds on the framework of generative adversarial networks and exploits reward shaping to learn near-optimal rewards and policies.
Potential-based reward shaping functions are known to guide the learning agent whereas in this paper we bring forward their beneﬁts in learning near-optimal rewards.
Our method simultaneously learns a potential-based reward shaping function through variational information maximization along with the reward and policy under the adversarial learning formulation.
We evaluate our method on various high-dimensional complex control tasks.
We also evaluate our learned rewards in transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure.
Our experimentation shows that our proposed method not only learns near-optimal rewards and policies matching expert behavior, but also performs signiﬁcantly better than state-of-the-art inverse reinforcement learning algorithms.
I NTRODUCTION
Reinforcement learning (RL) has emerged as a promising tool for solving complex decision-making and control tasks from predeﬁned high-level reward functions (Silver et al., 2016; Qureshi et al., 2017).
However, deﬁning an optimizable reward function that inculcates the desired behavior can be challenging for many robotic applications, which include learning social-interaction skills (Qureshi et al., 2018), dexterous manipulation (Finn et al., 2016b), autonomous driving (Kuderer et al., 2015), and robotic surgery (Yip & Das, 2017).
Inverse reinforcement learning (IRL) (Ng et al., 2000) addresses the problem of learning reward functions from expert demonstrations, and it is often considered as a branch of imitation learning (Argall et al., 2009).
The prior work in IRL includes maximum-margin (Abbeel & Ng, 2004; Ratliff et al., 2006) and maximum-entropy (Ziebart et al., 2008) formulations.
Currently, maximum entropy (MaxEnt) IRL is a widely used approach towards IRL, and has been extended to use non-linear function approximators such as neural networks in scenarios with unknown dynamics by leveraging sampling-based techniques (Boularias et al., 2011; Finn et al., 2016b; Kalakrishnan et al., 2013).
However, designing the IRL algorithm is usually complicated as it requires, to some extent, hand engineering such as deciding domain-speciﬁc regularizers (Finn et al., 2016b).
Rather than learning reward functions and solving the IRL problem, the imitation learning (IL) methods were proposed that learn a policy directly from expert demonstrations.
Prior work addressed the
IL problem through behavior cloning (BC) which learns a policy from expert trajectories using supervised learning (Pomerleau, 1991).
Although BC methods are simple solutions to IL, these methods require a large amount of data because of compounding errors induced by covariate shift (Ross et al., 2011).
To overcome BC limitations a generative adversarial imitation learning (GAIL) algorithm (Ho & Ermon, 2016) was proposed.
GAIL uses the formulation of Generative Adversarial Networks (GANs)(Goodfellow et al., 2014), i.e., a generator-discriminator framework, where generator learns to generate expert-like trajectories and discriminator learns to distinguish between generated and expert trajectories.
Although GAIL is highly effective and efﬁcient framework, it does not recover transferable/portable reward functions along with the policies.
Reward function learning is ultimately preferable, if possible, over direct imitation learning as rewards are portable functions that represent the most basic and complete representation of agent intention, and can be re-optimized in new environments and new agents.
Reward learning is challenging as there can be many optimal policies explaining a set of demonstrations and many reward functions inducing an optimal policy (Ng et al., 1999).
Recently, an adversarial inverse reinforcement learning (AIRL) framework (Fu et al., 2017), an extension of GAIL, was proposed that offers a solution to the former issue by exploiting the maximum entropy IRL method (Ziebart et al., 2008) whereas the latter issue is addressed through learning disentangled reward functions, i.e., the reward is a function of state only instead of both state and action.
The disentangled reward prevents actions-driven reward shaping (Fu et al., 2017) and is able to recover transferable reward functions, but has two main disadvantages.
First, AIRL fails to recover the ground truth reward when the ground truth reward is a function of both state and action.
For example, the reward function in any locomotion or ambulation tasks contains a penalty term that discourages actions with large magnitudes.
This need for action regularization is well known in optimal control literature and limits the use cases of a state-only reward function in most practical real-life applications.
Second, reward shaping plays a vital role in quickly recovering invariant policies (Ng et al., 1999) and thus for AIRL, it is usually not possible to simultaneously recover optimal/near-optimal policies when learning disentangled rewards.
In this paper, we propose the empowerment-based adversarial inverse reinforcement learning (EAIRL) algorithm 1 .
Empowerment (Salge et al., 2014) is a mutual information-based theoretic measure, like state- or action-value functions, that assigns a value to a given state to quantify the extent to which an agent can inﬂuence its environment.
Our method uses variational information maximization (Mohamed & Rezende, 2015) to learn empowerment in parallel to learning the reward and policy from expert data.
The empowerment acts as a potential function for shaping rewards.
Our experimentation shows that the proposed method recovers not only near-optimal policies but also recovers robust, near-optimal, transferable, non-disentangled (state-action) reward functions.
The results on reward learning show that EAIRL outperforms several state-of-the-art methods by recovering ground-truth reward functions.
On policy learning, results demonstrate that policies learned through EAIRL perform comparably to GAIL and AIRL with non-disentangled (state-action) reward function but signiﬁcantly outperform policies learned through AIRL with disentangled reward and GAN interpretation of Guided Cost Learning (GAN-GCL)
(Finn et al., 2016a).
B ACKGROUND
We consider a Markov decision process (MDP) represented as a tuple ( S , A , P , R , ρ 0 , γ ) where S denotes the state-space, A denotes the action-space, P represents the transition probability distribution, i.e., P : S ×
A × S → [0 , 1] , R ( s, a ) corresponds to the reward function, ρ 0 is the initial state distribution ρ 0 : S → R , and γ ∈ (0 , 1) is the discount factor.
Let q ( a | s, s (cid:48) ) be an inverse model that maps current state s ∈ S and next state s (cid:48)
∈ S to a distribution over actions A , i.e., q : S × S ×
A → [0 , 1] .
Let π be a stochastic policy that takes a state and outputs a distribution over actions such that π : S ×
A → [0 , 1] .
Let τ and τ E denote a set of trajectories, a sequence of state-action pairs ( s 0 , a 0 , · · · s T , a T ) , generated by a policy π and an expert policy π E , respectively, where T denotes the terminal time.
Finally, let Φ( s ) be a potential function that quantiﬁes a utility of a given state s ∈ S , i.e., Φ : S → R .
In our proposed work, we use an empowerment-based potential function Φ( s ) for reward shaping to adversarially learn both reward function and policy.
Therefore, the following sections provide a brief background on potential-based reward shaping functions and their beneﬁts to imitation learning, adversarial reward and policy learning, and variational information-maximization approach to learn the empowerment.
2. S HAPING R EWARDS
In this section, we brieﬂy describe a formal framework of reward-shaping and its importance to policy and reward learning (for details see (Ng et al., 1999)).
We consider a general form of reward function R : S × A × S
→ R , i.e., the reward R ( s, a, s (cid:48) ) is a function of current state s ∈ S , action a ∈ A , and next state s (cid:48) ∈ S .
Let F : S ×
A × S → R be a reward shaping function and R (cid:48) be a transformed reward function denoted as R (cid:48)
=
R + F .
Ng et al.
(1999) proved that an optimal behavior of a policy remains unchanged if the reward undergoes transformation through a shaping function F of form γ Φ( s (cid:48) )
− Φ( s ) , i.e.,
Theorem 1 (Ng et al., 1999)
We say F : S ×
A × S → R is a potential-based shaping function if there exist a real-valued function Φ : S → R and F = γ Φ( s (cid:48) ) − Φ( s ) .
Then a potential-based shaping function F is a necessary and sufﬁcient condition to guarantee that an optimal policy π learned in MDP M (cid:48) =
( S , A , P , R (cid:48)
=
R + F, ρ 0 , γ ) is also optimal in the MDP M =
( S , A , P , R , ρ 0 , γ ) , i.e., a policy π is invariant to reward transformations.
Reward shaping plays a vital role in learning both rewards and policies from expert demonstrations (Ng et al., 1999).
In the former case, reward shaping determines the extent to which a true reward function can be recovered whereas in the latter case, reward shaping speeds up the learning process by supplementing an actual reward function to guide the learning process.
Despite several advantages of shaping rewards, a potential-based shaping function F = γ Φ( s (cid:48) ) − Φ( s ) which is a sufﬁcient and necessary condition for preserving policy behavior (Ng et al., 1999) is usually not available.
There exist several methods (Asmuth et al., 2008; Grzes & Kudenko, 2009) to learn potential-based reward shaping functions but they assume the availability of transition model P , and are furthermore demonstrated in small-scale maze-solving problems.
In this paper, we show we are able to learn the potential-based reward shaping functions without a transition model, as well as one that also scales to higher dimensional problems by modeling a function Φ as Empowerment (Salge et al., 2014) which we learn efﬁciently online through variational information-maximization (Mohamed & Rezende, 2015).
2. A DVERSARIAL I NVERSE R EINFORCEMENT L EARNING
This section brieﬂy describes Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2017) algorithm which forms a baseline of our proposed method.
AIRL is state-of-the-art IRL method that builds on GAIL (Ho & Ermon, 2016), maximum entropy IRL framework (Ziebart et al., 2008) and
GAN-GCL, a GAN interpretation of Guided Cost Learning (Finn et al., 2016b;a).
GAIL is a model-free adversarial learning framework, inspired from GANs (Goodfellow et al., 2014), where the policy π learns to imitate the expert policy behavior π E by minimizing the JensenShannon divergence between the state-action distributions generated by π and the expert state-action distribution by π E through following objective
[Equation]
where D is the discriminator that performs the binary classiﬁcation to distinguish between samples generated by π and π E , λ is a hyper-parameter, and H ( π ) is an entropy regularization term E π [log π ] .
Note that GAIL does not recover reward; however, Finn et al.
(2016a) shows that the discriminator can be modeled as a reward function.
Thus AIRL (Fu et al., 2017) presents a formal implementation of (Finn et al., 2016a) and extends GAIL to recover reward along with the policy by imposing a following structure on the discriminator:
[Equation]
where f ξ,ϕ ( s, a, s (cid:48) )
=
r ξ ( s ) + γh ϕ ( s (cid:48) ) − h ϕ ( s ) comprises disentangled reward term r ξ ( s ) with training parameters ξ , and shaping term F
= γh ϕ ( s (cid:48) ) − h ϕ ( s ) with training parameters ϕ .
The entire D ξ,ϕ ( s, a, s (cid:48) ) is trained as a binary classiﬁer to distinguish between expert demonstrations τ E and policy generated demonstrations τ .
The policy is trained to maximize the discriminative reward ˆ r ( s, a, s (cid:48) )
=
log( D ( s, a, s (cid:48) )
− log(1 − D ( s, a, s (cid:48) ))) .
Note that the function F = γh ϕ ( s (cid:48) ) − h ϕ ( s ) consists of free-parameters as no structure is imposed on h ϕ ( · ) , and as mentioned in (Fu et al., 2017), the reward function r ξ ( · ) and function F are tied upto a constant ( γ − 1) c , where c ∈ R , thus the impact of F , the shaping term, on the recovered reward r is quite limited and therefore, the beneﬁts of reward shaping are not fully realized.
2. E MPOWERMENT AS M AXIMAL M UTUAL I NFORMATION
Mutual information (MI), an information-theoretic measure, quantiﬁes the dependency between two random variables.
In intrinsically-motivated reinforcement learning, a maximal of mutual information between a sequence of K actions a and the ﬁnal state s (cid:48) reached after the execution of a , conditioned on current state s is often used as a measure of internal reward (Mohamed & Rezende, 2015), known as Empowerment Φ( s ) , i.e.,
[Equation]
where p ( s (cid:48) | a , s ) is a K -step transition probability, w ( a | s ) is a distribution over a , and p ( a , s (cid:48) | s ) is a joint-distribution of K actions a and ﬁnal state s (cid:48) 2 .
Intuitively, the empowerment Φ( s ) of a state s quantiﬁes an extent to which an agent can inﬂuence its future.
Empowerment, like value functions, is a potential function that has been previously used in reinforcement learning but its applications were limited to small-scale cases due to computational intractability of MI maximization in higher-dimensional problems.
However, recently a scalable method (Mohamed & Rezende, 2015) was proposed that learns the empowerment through the more-efﬁcient maximization of variational lower bound, which has been shown to be equivalent to maximizing MI (Agakov, 2004).
The lower bound was derived (for complete derivation see Appendix A.1) by representing MI in term of the difference in conditional entropies H ( · ) and utilizing the non-negativity property of KL-divergence, i.e.,
[Equation]
where H ( a | s )
= − E w ( a | s ) [log w ( a | s )] , H ( a | s (cid:48) , s )
= − E p ( s (cid:48) | a,s ) w ( a | s ) [log p ( a | s (cid:48) , s )] , q φ ( · ) is a variational distribution with parameters φ and w θ ( · ) is a distribution over actions with parameters θ .
Finally, the lower bound in Eqn. 4 is maximized under the constraint H ( a | s ) <
η (to avoid divergence, see (Mohamed & Rezende, 2015)) to compute empowerment as follow:
[Equation]
where β is η dependent temperature term.
Mohamed & Rezende (2015) also applied the principles of Expectation-Maximization (EM) (Agakov, 2004) to learn empowerment, i.e., alternatively maximizing Eqn.
5 with respect to w θ ( a | s ) and q φ ( a | s (cid:48) , s ) .
Given a set of training trajectories τ , the maximization of Eqn.
5 w.r.t q φ ( · ) is shown to be a supervised maximum log-likelihood problem whereas the maximization w.r.t w θ ( · ) is determined through the functional derivative ∂I/∂w =
under the constraint (cid:80) a w ( a | s )
= 1 .
The optimal w ∗ that maximizes Eqn.
5 turns out to be
[Equation]
[log q
[Equation]
( a | s
[Equation]
s
[Equation]
)]) , where Z ( s ) is a normalization term.
By substituting w ∗ in
Eqn. 5 showed that the empowerment Φ( s ) = 1 β log Z ( s ) (for full derivation, see Appendix A.2).
Since w ∗ ( a | s ) is an unnormalized distribution, Mohamed & Rezende (2015) introduced an approximation w ∗ ( a | s )
≈ log π ( a | s ) + Φ( s ) where π ( a | s ) is a normalized distribution which leaves the scalar function Φ( s ) to account for the normalization term log Z ( s ) .
Finally, the parameters of policy π and scalar function Φ are optimized by minimizing the discrepancy between the two approximations (log π ( a | s ) + Φ( s )) and β log q φ ( a | s (cid:48) , s )) through the squared error as follow:
[Equation]
E MPOWERED A DVERSARIAL I NVERSE R EINFORCEMENT L EARNING
We present an inverse reinforcement learning algorithm that simultaneously and adversarially learns a robust, transferable reward function and policy from expert demonstrations.
Our proposed method comprises (i) an inverse model q φ ( a | s (cid:48) , s ) that takes the current state s and the next state s (cid:48) to output a distribution over actions
A that resulted in s to s (cid:48) transition, (ii) a reward r ξ ( s, a ) , with parameters ξ , that is a function of both state and action, (iii) an empowerment-based potential function Φ ϕ ( · )
[Equation]
with parameters ϕ
that determines the reward-shaping function F = γ Φ ϕ ( s (cid:48) )
− Φ ϕ ( s ) , and (iv) a policy model π θ ( a | s ) outputs a distribution over actions given the current state s .
All these models are trained simultaneously based on the objective functions described in the following sections.
3. I NVERSE MODEL q φ ( a | s, s (cid:48) ) OPTIMIZATION
As mentioned in Section 2.3, learning the inverse model q φ ( a | s, s (cid:48) ) is a maximum log-likelihood supervised learning problem.
Therefore, given a set of trajectories τ ∼ π , where a single trajectory is a sequence states and actions, i.e., τ i = { s 0 , a 0 , · · · , s T , a T } i , the inverse model q φ ( a | s (cid:48) , s ) is trained to minimize the mean-square error between its predicted action q ( a | s (cid:48) , s ) and the action a taken according to the generated trajectory τ , i.e.,
[Equation]
3. E MPOWERMENT Φ ϕ ( s ) OPTIMIZATION
Empowerment will be expressed in terms of normalization function Z ( s ) of optimal w ∗ ( a | s ) , i.e., Φ ϕ ( s ) = 1 β log Z ( s ) .
Therefore, the estimation of empowerment Φ ϕ ( s ) is approximated by minimizing the loss function l I ( s, a, s (cid:48) ) , presented in Eqn. 6, w.r.t parameters ϕ , and the inputs ( s, a, s (cid:48) ) are sampled from the policy-generated trajectories τ .
3. R EWARD FUNCTION r ξ ( s, a )
To train the reward function, we ﬁrst compute the discriminator as follow:
[Equation]
where r ξ ( s, a ) is the reward function to be learned with parameters ξ .
We also maintain the target ϕ (cid:48) and learning ϕ parameters of the empowerment-based potential function.
The target parameters ϕ (cid:48) are a replica of ϕ except that the target parameters ϕ (cid:48) are updated to learning parameters ϕ after every n training epochs.
Note that keeping a stationary target Φ ϕ (cid:48) stabilizes the learning as also highlighted in (Mnih et al., 2015).
Finally, the discriminator/reward function parameters ξ are trained via binary logistic regression to discriminate between expert τ E and generated τ trajectories, i.e.,
[Equation]
3. P OLICY OPTIMIZATION POLICY π θ ( a | s )
We train our policy π θ ( a | s ) to maximize the discriminative reward ˆ r ( s, a, s (cid:48) )
=
log( D ( s, a, s (cid:48) )
− log(1 − D ( s, a, s (cid:48) ))) and to minimize the loss function l I ( s, a, s (cid:48) )
= (cid:0) β log q φ ( a | s, s (cid:48) ) − (log π θ ( a | s ) + Φ ϕ ( s ))
(cid:1) 2 to learn the empowerment.
Hence, the overall policy training objective is:
[Equation]
(10) where policy parameters θ are updated by taking KL-constrained natural gradient step using any policy optimization method such as TRPO (Schulman et al., 2015) or an approximated step such as
PPO (Schulman et al., 2017).
Algorithm 1 outlines the overall training procedure to train all function approximators simultaneously.
Note that the expert samples τ E are seen by the discriminator only, whereas all other models are trained using the policy generated samples τ .
Furthermore, as highlighted in (Fu et al., 2017), the discriminating reward ˆ r ( s, a, s (cid:48) ) boils down to the following expression
[Equation]
11) where f ( s, a, s (cid:48) ) =
r ξ ( s, a ) + γ Φ ϕ (cid:48)
( s (cid:48) )
− Φ ϕ ( s ) .
Hence, our policy training objective maximizes the learned shaped-reward function f
( s, a, s (cid:48) ) and minimizes the discrepancy between (log π ( a | s )+ Φ( s )) and β log q φ ( a | s (cid:48) , s )) , with the term log π ( a | s ) acting as a regularizer.
As a result, our method trains the policy under reward transformations which leads to learning an invariant and robust policy from expert demonstrations.
Initialize parameters of policy π θ , and inverse model q φ
Initialize parameters of target Φ ϕ (cid:48) and training Φ ϕ empowerment, and reward r ξ functions
Obtain expert demonstrations τ E by running expert policy π E for i ← 0 to N do
Collect trajectories τ by executing π θ
Update φ i to φ i +1 with the gradient
E τ [ (cid:53) φ i l q ( s, a, s (cid:48) )]
Update ϕ
i to ϕ
i +1 with the gradient E τ
[ (cid:53) ϕ
i l I ( s, a, s (cid:48) )]
Update ξ i to ξ i +1 with the gradient:
E τ [ (cid:53) ξ i log D ξ i ( s, a, s (cid:48) )]
+ E τ E [ (cid:53)
ξ i (1 − log D ξ i ( s, a, s (cid:48) ))]
Update θ i to θ
i +1 using TRPO/PPO update rule with the following objective:
E τ (cid:2) (cid:53) θ
i log π θ i ( a | s )
ˆ r ξ i +1 ( s, a, s (cid:48) )
(cid:3)
+ λ I E τ (cid:2) (cid:53) θ
i l I ( s, a, s (cid:48) )
(cid:3)
After every n epochs sync ϕ (cid:48) with ϕ
[Figure]
R ESULTS
Our proposed method, EAIRL, simultaneously learns reward and policy from expert demonstrations.
We evaluate our method against state-of-the-art policy and reward learning techniques on several control tasks in OpenAI Gym.
In case of policy learning, we compare our method against
GAIL, GAN-GCL, AIRL with state-only reward, denoted as AIRL( s ) , and AIRL with state-action reward, denoted as AIRL( s, a ) .
In reward learning, we only compare our method against AIRL( s ) and AIRL( s, a ) as GAIL does not recover reward, and GAN-GCL is shown to exhibit inferior performance than AIRL (Fu et al., 2017).
Furthermore, in the comparisons, we also include the expert performances which represents a policy learned by optimizing a ground-truth reward using TRPO.
The performance of different methods are evaluated in term of average total reward accumulated (denoted as score) by an agent during the trial, and for each experiment, we run ﬁve trials.
4. R EWARD LEARNING PERFORMANCE (T RANSFER LEARNING EXPERIMENTS )
To evaluate the learned rewards, we consider a transfer learning problem in which the testing environments are made to be different from the training environments.
More precisely, the rewards learned via IRL in the training environments are used to re-optimize a new policy in the testing envi- [Figure] [Table] ronment.
We consider two test cases, shown in the Fig.
1 and Fig.
2, in which the agent’s dynamics and physical environment is modiﬁed, respectively.
In the ﬁrst test case, as shown in Fig. 1(a), we modify the agent itself during testing.
We trained a reward function to make a standard quadrupled ant to run forward.
During testing, we disabled the front two legs (indicated in red) of the ant (crippled-ant), and the learned reward is used to reoptimize the policy to make a crippled-ant move forward.
Note that the crippled-ant cannot move sideways (Appendix B.1).
Therefore, the agent has to change the gait to run forward.
In the second test case, shown in Fig 2(a), the agent learns to navigate a 2D point-mass to the goal region in a simple maze.
We re-position the maze central-wall during testing so that the agent has to take a different path, compared to the training environment, to reach the target (Appendix B.2).
Fig. 1(b) and Fig.
2(b) compare the policy performance scores over ﬁve different trials of EAIRL,
AIRL( s ) and AIRL( s, a ) in the aforementioned transfer learning tasks.
The expert score is shown as a horizontal line to indicate the standard set by an expert policy.
Table 1 summarizes the mean score of ﬁve trials with a standard deviation in above-mentioned transfer learning experiments.
It can be seen that our method recovers near-optimal reward functions as the policy scores almost reach the expert scores in all ﬁve trials.
Furthermore, our method performs signiﬁcantly better than both
AIRL( s ) and AIRL( s, a ) in matching an expert’s performance.
4. P OLICY LEARNING PERFORMANCE (I MITATION LEARNING )
Table 2 presents the means and standard deviations of policy learning performance scores, over the ﬁve different trials, in various control tasks.
For each algorithm, we provided 20 expert demonstrations for imitation, generated by optimizing a policy on a ground-truth reward using TRPO.
The tasks, shown in Fig.
3, include (i) making a 2D cheetah robot to run forward, (ii) making a 3D quadrupled robot (ant) to move forward, (iii) making a 2D robot to swim (swimmer), and (iv) keeping a friction less pendulum to stand vertically up. It can be seen that EAIRL, AIRL( s, a ) and GAIL
[Figure]
demonstrate similar performance and successfully learn to imitate expert policy whereas AIRL( s ) and GAN-GCL fails to recover a policy.
[Table]
D ISCUSSION
This section highlights the importance of state-action rewards and potential-based reward shaping functions on learning policies and rewards, respectively, from expert demonstrations.
Ng et al.
(1999) theoretically discussed the importance of potential-based reward shaping in a structural prediction of the MDP but no prior work has reported a practical approach to learn potentialbased reward shaping function and its implications to IRL to the best of our knowledge.
Note that our method, EARIL [Table] and AIRL with state-action reward function, i.e., AIRL( s, a ) , shares the same discriminator formulation except that AIRL( s, a ) does not impose any structure on the reward-shaping function, while our method models the reward-shaping function through empowerment.
The numerical results of reward learning, reported in the previous section, indicate that AIRL( s, a ) fails to learn rewards whereas EAIRL recovers the near-optimal reward functions.
This highlights the positive impact of using a potential-based reward-shaping function on reward learning.
Thus our experimentation validates the theoretical propositions of (Ng et al., 1999) that the reward shaping function determines an extent to which the true reward function can be recovered from the expert demonstrations.
Our experimentation highlights the importance of modeling discriminator/reward functions as a function of both state and action in reward and policy learning under GANs framework.
The reward learning results show that AIRL with state-only rewards (AIRL(s)) does not recover the action dependent terms of the ground-truth reward function that penalizes high torques. Therefore, the agent shows aggressive behavior and ﬂips over after few steps (see the accompanying video), which is also the reason that crippled-ant trained with AIRL’s disentangled reward function reaches only the half-way to expert scores as shown in the Table 1. Therefore, the formulation of rewards as a function of both states and actions is crucial in many real-life robotics applications for action regularization in any locomotion or ambulation tasks that discourage actions with large magnitudes. The policy learning results further validate the importance of state-action reward formulation.
Table 2 shows that AIRL with state-only rewards, AIRL(s), fails to learn a policy whereas EAIRL, GAIL, and AIRL that include state-action reward/discriminator formulation successfully recover the expert-like behavior policies.
Hence, our empirical results show that it is crucial to model re- ward/discriminator as a function of state-action as otherwise, adversarial imitation learning fails to learn ground-truth rewards and expert-like policies from expert data.
C ONCLUSIONS AND F UTURE W ORK
We present an approach to adversarial reward and policy learning from expert demonstrations by efﬁciently and effectively utilizing reward shaping for inverse reinforcement learning.
We learn a potential-based reward shaping function in parallel to learning the reward and policy.
Our method transforms the learning reward through shaping function that leads to acquiring a rewardtransformations preserving invariant policy.
The invariant policy in turn guides the reward-learning process to recover near-optimal reward.
We show that our method successfully learns near-optimal rewards, policies, and performs signiﬁcantly better than state-of-the-art IRL methods in both imitation learning and transfer learning.
The learned rewards are shown to be transferable to environments that are dynamically or structurally different from training environments.
In our future work, we plan to extend our method to learn rewards and policies from diverse human/expert demonstrations as the proposed method assumes that a single expert generates the training data.
Another exciting direction is to learn from sub-optimal demonstrations that also contains failures in addition to optimal behaviors.
A PPENDICES A V ARIATIONAL E MPOWERMENT
For completeness, we present a derivation of presenting mutual information (MI) as variational lower bound and maximization of lower bound to learn empowerment.
A. V ARIATIONAL I NFORMATION L OWER B OUND
As mentioned in section 2.
[Equation]
the variational lower bound representation of MI is computed by deﬁning MI as a difference in conditional entropies, and the derivation is formalized as follow.
[Equation]
A. V ARIATIONAL I NFORMATION M AXIMIZATION
The empowerment is a maximal of MI and it can be formalized as follow by exploiting the variational lower bound formulation (for details see (Mohamed & Rezende, 2015)).
[Equation]
As mentioned in section 2.3, given a training trajectories, the maximization of Eqn.
12 w.r.t inverse model q ( a | s (cid:48) , s ) is a supervised maximum log-likelihood problem.
The maximization of
Eqn. 12 w.r.t w ( a | s ) is derived through a functional derivative ∂I w,q /∂w =
under the constraint (cid:80) a w ( a | s ) = 1 .
For simplicity, we consider discrete state and action spaces, and the derivation is as follow:
[Equation]
By using the constraint (cid:80) a w ( a | s ) = 1 , it can be shown that the optimal solution w ∗ ( a | s ) =
Z ( s ) exp( u ( s, a )) , where u ( s, a ) =
β E p ( s (cid:48) | a,s )
[log q ( a | s (cid:48) , s )] and Z ( s ) =
(cid:80) a u ( s, a ) .
This solution maximizes the lower bound since ∂ 2 I w ( s ) /∂w
= − (cid:80) a w (
[Equation]
B T RANSFER LEARNING PROBLEMS B. A NT ENVIRONMENT
The following ﬁgures show the difference between the path proﬁles of standard and crippled Ant.
It can be seen that the standard Ant can move sideways whereas the crippled ant has to rotate in order to move forward.
B. M AZE ENVIRONMENT
The following ﬁgures show the path proﬁles of a 2D point-mass agent to reach the target in training and testing environment.
It can be seen that in the testing environment the agent has to take the opposite route compared to the training environment to reach the target.
C I MPLEMENTATION D ETAILS C. N ETWORK A RCHITECTURES
We use two-layer ReLU network with 32 units in each layer for the potential function h ϕ ( · ) and Φ ϕ ( · ) , reward function r ξ ( · ) , discriminators of GAIL and GAN-GCL.
Furthermore, policy π θ ( · ) of all presented models and the inverse model q φ ( · ) of EAIRL are presented by two-layer RELU network with 32 units in each layer, where the network’s output parametrizes the Gaussian distribution, i.e., we assume a Gaussian policy.
C. H YPERPARAMETERS
For all experiments, we use the temperature term β = 1 .
We set entropy regularization weight to 0.1 and 0.001 for reward and policy learning, respectively.
The hyperparameter λ
I was set to 1 for reward learning and 0.001 for policy learning.
The target parameters of the empowerment-based potential function Φ ϕ (cid:48) ( · ) were updated every 5 and 2 epochs during reward and policy learning respectively.
Although reward learning parameters are also applicable to policy learning, we decrease the magnitude of entropy and information regularizers during policy learning to speed up the policy convergence to optimal values.
Furthermore, we set the batch size to 2000- and 20000-steps per
TRPO update for the pendulum and remaining environments, respectively.
For the methods (Fu et al., 2017; Ho & Ermon, 2016) presented for comparison, we use their suggested hyperparameters.
We also use policy samples from previous 20 iterations as negative data to train the discriminator of all IRL methods presented in this paper to prevent the parametrized reward functions from overﬁtting the current policy samples.