Off-Policy Adversarial Inverse Reinforcement Learning
Abstract
Adversarial Imitation Learning (AIL) is a class of algorithms in Reinforcement learning (RL), which tries to imitate an expert without taking any reward from the environment and does not provide expert behavior directly to the policy training.
Rather, an agent learns a policy distribution that minimizes the difference from expert behavior in adversarial setting.
Adversarial Inverse Reinforcement Learning (AIRL) leverages the idea of
AIL, integrates a reward function approximation along with learning the policy and shows the utility of IRL in the transfer learning setting.
But the reward function approximator that enables transfer learning does not perform well in imitation tasks.
We propose an Off-Policy Adversarial Inverse Reinforcement Learning (Off-policy-AIRL) algorithm which is sample efﬁcient as well as gives good imitation performance compared to the state-of-the-art AIL algorithm in the continuous control tasks.
For the same reward function approximator, we show the utility of learning our algorithm over AIL by using the learned reward function to retrain the policy over a task under signiﬁcant variation where expert demonstrations are absent.
1. Introduction
Reinforcement learning (RL) is a very useful framework for learning complex behavior in control, where an agent interacts with an environment and tries to learn the expected behavior by optimizing an objective function that ensures the highest cumulative reward over speciﬁc time steps.
A reward is often human-engineered function that tries to ensure higher rewards for expected behavior within the environment.
Designing a reward function for a complex environment can be tricky.
Poorly designed reward function may lead to a fatal mistake when applied in real-world applications.
Generative Adversarial Imitation Learning (GAIL) (Ho &
Ermon, 2016) introduces AIL where a generator (policy) is used to compute trajectory and a binary classiﬁer called discriminator classiﬁes generated behavior from the expert.
As imitation learning algorithms only learn policy, they miserably fail when there is a change in environmental dynamics.
Inverse reinforcement learning (IRL) (Ng et al., 1999a;
Abbeel & Ng, 2004) addresses the importance of learning reward function and thus learn reward function approximator along with policy.
Adversarial Inverse Reinforcement
Learning (AIRL) (Fu et al., 2018) proposes an IRL algorithm in adversarial learning, which shows a promising result when there is considerable variability in the environment from the demonstration setting.
But compared to
AIL algorithms, AIRL does not perform well in imitation performance (Kostrikov et al., 2019; Fu et al., 2018).
It is also important to note the best performing reward function approximator conﬁguration for imitation and transfer learning tasks in AIRL algorithm are different.
In our work we perform transfer learning using the same conﬁguration that works best in imitation performance.
The paper makes the following contributions: (i)
We propose an Off-Policy Adversarial Inverse Reinforcement
Learning algorithm (off-policy-AIRL) which is sample efﬁcient and improves imitation performance of prior IRL algorithms through comparing imitation performance with state-of-the-art AIL algorithm on continuous control tasks.
(ii) We show utility of learning IRL algorithm over imitation by reusing the learned reward function to train policy under certain changes in dynamics where expert demonstrations are absent.
We use Soft Actor-Critic (Haarnoja et al., 2018) (SAC), which makes our algorithm more sample efﬁcient. (iii)
We also show using Multiplicative Compositional
Policies (Peng et al., 2019) (MCP) allows more ﬂexibility in retraining policy in transfer task and at the same time improves imitation performance.
2. Background
We deﬁne a Markov decision process (MDP) as a tuple ( S, A, T, r, γ, ρ 0 ) , where S , A are the state and action space respectively, γ ∈ [0 , 1) is the discount factor.
We consider the dynamic or transition distribution T ( s (cid:48) | s, a ) , initial state distribution ρ 0 ( s ) and reward function r ( s, a ) are unknown in IRL setup and can be learned through interaction with the
MDP.
AIL evolved from maximum casual entropy IRL framework (Ziebart et al., 2010), which considers an entropy regularized MDP with the goal to ﬁnd the optimal policy π ∗ that maximizes the expected entropy-regularized discounted reward under π , T and ρ 0 :
[Equation]
Here τ =
( s 0 , a 0 ...s T , a T ) deﬁnes the sequence of states and actions induced by policy and H ( π ( .
| s t )) is the discounted causal entropy of policy π .
In imitation task, expert demonstrations D = τ 1 , τ 2 ...
τ n are given and an agent is expected to produce trajectory which are similar to the expert and infer reward function.
Under certain constrain of feature matching (Ziebart et al., 2008) derived following objective function:
[Equation]
where z = (cid:82) τ e R ( τ ) is the partition function and can be expressed as probability of trajectories with equal reward/cost are equal likely and probability of trajectory with higher rewards are exponentially more likely.
Equation (2) can be interpreted as solving a maximum likelihood problem:
[Equation]
But the assumptions that are made in (Ziebart et al., 2010) are model / transition function T ( s (cid:48) | s, a ) is known and applied on small state-action space.
GAIL (Ho & Ermon, 2016), a model-free algorithm, is the ﬁrst framework to draw connection between generative adversarial network (GAN) and imitation learning which works in continuous state-action space.
GAIL uses a generative model or generator, G that acts as the policy.
Purpose of the generator is to learn policy π θ such that the stateaction visitation frequency ρ π θ is similar to expert’s ρ π E without directly being provided with the expert demonstrations.
A discriminator D is a binary classiﬁer which tries to distinguish the data distribution of the generator ρ π from the expert’s ρ π E .
Objective for a generator is to minimize the difference with expert visitation frequency, while a discriminator wants to maximize the difference and thus derived the entropy regularized objective as following:
[Equation]
where entropy term H is policy regularizer controlled by λ ≥ 0 .
GAIL uses two separate neural-networks to represent generator and discriminator.
For any given state generator tries to take expert like action.
Discriminator takes state-action pairs as input and computes the likelihood of the input coming from an expert.
Generator uses a reward function R ( s, a ) =
− log D ψ ( s, a ) in order to train it’s network parameters.
GAIL is sample efﬁcient in terms of number of expert demonstrations required but not quite sample efﬁcient in terms of required interaction with the environment to learn the policy.
Thus GAIL is not suitable for many real-world applications.
Discriminator Actor critic (DAC)
(Kostrikov et al., 2019) uses Twin-delayed deep deterministic actorcritic (TD3) (Fujimoto et al., 2018), an off-policy algorithm as generator to improve upon the sample efﬁciency of existing methods, and it extends the learning environment with absorbing states.
DAC criticises AIL algorithms (i.e
GAIL) for inducing reward biases by using strict positive or negative reward function and proposes a reward function r ( s, a )
= log( D θ ( s, a ))
− log(1 − D
θ ( s, a ))
in order to achieve unbiased rewards.
Combined, these changes remove the survival bias and solves sample inefﬁciency problem of
GAIL.
For being an imitation learning algorithm, instead of learning cost/reward function, both GAIL and DAC only recover expert policy and thus will fail miserably in dynamic environments.
A discriminator function, that learns policy as well as the cost/reward function, is proposed by (Finn et al., 2016) by optimizing following objective for discriminator network:
[Equation]
thus
[Equation] [Equation]
) where p ( τ ) is the data distribution and q ( τ ) is the policy distribution and (a) consider derived objective function (2) from (Ziebart et al., 2010).
But (Finn et al., 2016) contains no experimental demonstration of imitation performance.
Later on AIRL (Fu et al., 2018) shows, when we consider a whole trajectory to compute discriminator update it suffers from high variance issue and that leads to a poor performing generator.
Instead AIRL computes discriminator update using (s,a,s’)
tuple:
[Equation]
where f ψ,ω ( s, a, s (cid:48) )
=
r ψ ( s, a ) + γ Φ ω ( s (cid:48) ) − Φ ω ( s ) is a potential-based reward function, r ψ ( s, a ) is a reward function approximator and Φ ω is reward shaping term controlled by the dynamics.
In (Fu et al., 2018)
f ψ,ω ( s, a, s (cid:48) ) is referred as disentangled reward function (Ng et al., 1999b) due to r ψ ( s ) being indifferent to the changes in dynamics
T .
3. Off-policy Adversarial Inverse Reinforcement Learning
Off-policy-AIRL algorithm is inspired from DAC (Kostrikov et al., 2019), which is an adversarial imitation learning algorithm.
DAC addresses two existing problem with AIL algorithms, which are reward function bias and sample inefﬁciency.
Reward functions used in these algorithms induce an implicit bias by using strict positive or negative reward functions, which may work for some environments but in many cases become a reason for sub-optimal behavior.
Furthermore, (Kostrikov et al., 2019) shows adversarial methods improperly handle terminal states.
This introduces implicit reward priors that can affect the policy‘s performance.
In particular, many imitation learning implementations (Kim & Park, 2018; Ho & Ermon, 2016; Fu et al., 2018) and MDPs omit absorbing states s a .
Thus they implicitly assign 0 reward to terminal/absorbing states and adds a bias to the reward learning process.
Thus we use the same approach to learn the reward for absorbing states.
Return of the terminal state is deﬁned as
R T = r
( s T , a T ) + (cid:80) ∞
t = T +1 γ t − T r ( s a , . ) , where r ( s a , 0) is learned reward, instead of just R T = r
( s T , a T ) .
To make the implementation more stable, the terminal reward is analytically derived using the following equation:
[Equation]
Like any other imitation learning algorithm DAC only learns the policy.
We want to use the best of both worlds and thus replace the binary discriminator of DAC and use formulated discriminator function from AIRL (Fu et al., 2018) to learn a reward function approximator along with the policy.
From (Fu et al., 2018) it is evident that AIRL provides a poor imitation performance compared to GAIL.
It is important to note that the best performing policy in (Fu et al., 2018) for imitation task is trained using state-action dependent reward function r ψ ( s, a ) , which fails to re-train policy in the transfer learning task.
For state dependent reward function r ψ ( s ) (Fu et al., 2018) successfully completes transfer learning task but it comes with a cost of poor imitation performance.
In this work, we perform transfer learning using the same conﬁguration that works best in imitation performance.
We improve upon the prior IRL algorithm performance and while still being sample efﬁcient by using Soft-Actor-Critic (SAC) (Haarnoja et al., 2018) as the generator.
We store our experience in a replay buffer R to utilize them during the off-policy update of the generator.
After each time the environment reaches a terminal condition and resets, we update our discriminator and generator for the same episodic timesteps.
We use an expert buffer R E to store the expert trajectory and sample the ( s E , a E , s E (cid:48) ) pair in batch while update our discriminator.
3.1. Discriminator update rule
During discriminator update we randomly sample minibatch of state-action-next-state pairs ( s, a, .
, s (cid:48) )
b from both expert buffer R E and replay buffer R .
We use following equation derived in (Fu et al., 2018) to compute the discriminator output,
[Equation]
where
[Equation]
Here reward approximator r ψ can be a function of state, state-action or state-action-next-state and Φ can be any function that gives a measure of being at any state s .
Similar to (Fu et al., 2018)
we use value function V ω as reward shaping terms.
We ﬁnd the best imitation performance using state dependent reward function.
Output of the discriminator will predict likelihood of being an expert and thus objective of the discriminator is to minimize following binary cross entropy loss:
[Equation]
With in few iterations of the discriminator update, it easily can classify the expert from the generator data.
Thus to make the generator learning more stable we use gradient penalty (Gulrajani et al., 2017; Kostrikov et al., 2019).
3.2. Generator update rule
In IRL setting, we consider the actual reward from the environment is non-existent and rather formulate a reward approximator to train policy.
AIRL (Fu et al., 2018) algorithm uses a on-policy policy-gradient algorithm as its generator and proposes following reward approximator ˆ r ( s, a, s (cid:48) ) to update the policy:
[Equation]
Using equation (12) it is also referred as entropy regularized reward function:
[Equation]
For improving imitation performance and sample efﬁciency we use SAC as our generator.
SAC being an off-policy algorithm, samples ( s, a, s (cid:48) , done ) tuple from buffer when trains the network.
For our algorithm we experiment different reward approximator which is discussed in following section and use trained reward approximator to update critic or Q φ function in SAC with entropy term through following gradient step:
[Equation]
Similar to (Fujimoto et al., 2018;
Haarnoja et al., 2018)
we update actor or policy π θ for every second update of our Q
φ function using following gradient step:
[Equation]
3.3. Reward function selection
Implementation of AIRL discriminator (Fu et al., 2018) does not work off the shelf as now we have to train an off-policy generator.
We experiment on different way to compute the reward function for the generator update.
In AIRL, the author uses the value function of the states
V ω ( s ) to measure additional shaping terms.
We experiment on different variations of reward approximator ˆ r ψ,ω to update policy and disentangled reward function f ψ,ω to compute discriminator output so that we ﬁnd the combination of these two approximators that gives the best performance in off-policy-AIRL.
Implementation 1 :
• Policy update using :
[Equation]
• Disentangled reward function :
[Equation]
Implementation
• Policy update using :
[Equation]
• Disentangled reward function :
[Equation] + γV ω ( s (cid:48) )
− V ω ( s ) Implementation• Policy update using : ˆ r ψ,ω
( s, a, s (cid:48) )
=
r ψ ( s, a ) • Disentangled reward function : f ψ,ω =
r ψ ( s, a ) + γV ω ( s (cid:48) )
− V ω ( s ) Implementation• Policy update using : ˆ r ψ,ω
( s, a, s (cid:48) )
=
r ψ ( s ) • Disentangled reward function : f ψ,ω
= r ψ ( s ) + γV ω ( s (cid:48) )
− V ω ( s )
For implementation (1) and (2), we use the exact policy update rule which is used in AIRL (Fu et al., 2018).
In implementation (1) we consider a state dependent reward function r ψ ( s ) and for implementation (2) we consider reward to be a function of state-action pair r ψ ( s, a ) .
As we see from equation (16), reward computed to update policy is equivalent to entropy regularized reward function.
We evaluate the disentangled reward function for both state dependent r ψ ( s ) and state-action dependent r ψ ( s, a ) .
For implementation (3) and (4) we follow similar comparative study where policy update discard entropy regularization and use direct output from our reward approximator r ψ .
As discussed in DAC (Kostrikov et al., 2019), strict negative or positive reward approximator induces bias.
Similar to
DAC, our reward approximator r ψ gives both positive and negative rewards thus does not suffer reward biasing.
4. Transfer learning task
Advantage of IRL over imitation learning is that we can leverage the learned reward function to retrain a new policy.
If the reward function captures the underlying objective of an agent, then it is possible to use the reward function under robust changes of the dynamics.
Imitation learning successfully learn policy in the training domain but it is not possible to use the policy when there is a signiﬁcant domain shift (Fu et al., 2018).
We use the learned reward function from off-policy-AIRL algorithm to re-train our policy in transfer learning setting.
To the best of our knowledge AIRL (Fu et al., 2018) is the only IRL framework that shows utility of reward function in transfer learning for continuous control tasks and thus we demonstrate our experiments using same environments from (Fu et al., 2018)
(see Figure 1) and we also consider expert demonstrations to be absent for the transfer tasks.
For transfer learning experiments we consider dynamic changes in following two criterion, where either (1) dynamics of the agent or (2) dynamics of the environment changes while the goal/objective of the agent and action dimension remain the same.
Through theses experiments we show the learned reward function r ψ indeed can help the policy to behave under dynamic changes even when expert behavior is completely absent.
For transfer learning experiments we use the same hyper-parameter and update rules described in prior.
4.1. Criterion:
By changing dynamics of an agent, we refer to a signiﬁcant structural change of an agent.
To evaluate transfer learning performance under this criterion we use Customized Ant (Fu et al., 2018) environment.
In both imitation and transfer task the quadrupedal Ant requires to run forward, while dynamics of the agent during test session is changed by disabling and shrinking two legs.
In MuJoCo setting, (Fu et al., 2018) sets the gear ratios to 1, whereas the other joints were set to 150.
The gear ratio is a parameter which translates actions to torques on the joints, so the disabled joints are ∼ 150 x more difﬁcult to move.
This signiﬁcantly changes its gait.
In a MDP this puts a restriction in action space A , which results in a new transition dynamics T (cid:48) ( s | s, a ) .
Thus optimal policy π ∗ is changed even though the objective and action-dimension remains the same.
4.2. Criterion:
By changing dynamics of the environment, we consider notable variation in the trained environment such that it directly results in a different transition dynamics T (cid:48)
( s | s, a ) without putting restriction in action-space A .
Objective of
Shifting Maze (Fu et al., 2018) task is to reach to a speciﬁc goal position, where during imitation learning, agent is trained to go around the wall on the left side, but during transfer task environment setup is changed and must go around the wall on the right.
Thus optimal policy π ∗ is changed even though the objective and the dynamics of the agent remains same.
5. Multiplicative Compositional Policies
Policy often gets overﬁtted over one task and thus become harder to re-train during transfer learning task.
Different techniques such as early stopping or regularization can be helpful in this case.
An alternative solution is provided by (Peng et al., 2018) through using variational bottleneck technique to control over the information that is fed to the discriminator using mutual information theory, which allows more controlled update in the generator.
But this technique is highly dependent on hyper-parameter of the bottleneck itself, thus performance can be varied vastly if not tuned properly for each task.
We want to learn a policy that gives a good performance during transfer learning without being dependent on exact hyper-parameter tuning.
We propose using Multiplicative Compositional Policy (MCP) (Peng et al., 2019) to improve policy performance in transfer learning task.
MCP was introduced to combine multiple primitive skills in RL setting and these primitve skills are controlled using a gating function.
We propose using multiple primitive networks to learn single skill in IRL setting.
Using a weighted multiplicative composition of Gaussian primitives improves imitation performance and allows more ﬂexibility in composing into new policy in transfer learning.
MCP (Peng et al., 2019) tries to learn different skills with different primitive policies and then reuse those learned skills by combining them to do a more sophisticated task.
In hierarchical learning (Faloutsos et al., 2001;
Sutton et al., 1999; Jordan & Jacobs, 1992; Peng et al., 2016)
it is common to learn premitives
and then ﬁnd a composite policy π ( a | s ) by weighted sum of distribution from premitives π i ( a | s ) .
A gating function computes the weight w i , that determines the probability of activating each premitive for a given state, s .
Composite policy can be written as:
[Equation]
Here k denotes the number of primitives and this can be referred as additive model .
Standard hierarchical learning models can sequentially select particular skill over time.
In other words, can activate single primitive at each timestep.
MCP (Peng et al., 2019) proposes an multiplicative model , where multiple primitives are activated at a given time-step.
This allows an agent to learn a complex task requiring to perform more than one sub-task at a time.
MCP (Peng et al., 2019) decomposes agent’s policy into premitives, train those primitive policies on different subtasks and then obtain a composite-policy by multiplicative composition of these premitives.
Here each primitive is considered as distribution over actions and the composite policy is a multiplicative composition of these distribution.
Thus,
[Equation]
MCP enables an agent to activate multiple primitives simultaneously with each primitive specializing in different behaviors that can be composed to produce a continuous spectrum of skills, whereas, standard hierarchical algorithm only activate single primitive at a time.
6. MCP in off-policy-AIRL
MCP learns premitive policies
π i by training them into different sub-tasks and then try to learn more complex task leveraging the learned skills.
But in transfer learning experiments, we use MCP to learn single task.
The underlying idea is to observe whether it can relearn the composition of different action parameters when we put them in dynamic scenarios.
For example, during transfer learning experiments (Fu et al., 2018) disables two legs and thus it is important for the agent to re-optimise it’s policy to walk in this transfer learning setup.
Our initial objective is to see if MCP can decompose motor skills for a single task and re-optimise the learned policy by re-training the gating function.
We refer this new policy as SAC-MCP.
In MCP (Peng et al., 2019), experiments were conducted using on-policy algorithms in RL setting and the policy parameters were considered to be state and goal dependent.
We explore this concept to decompose motor skill over single task in IRL setting using off-policy algorithm.
[Figure]
7. Experimental Setup
Similar to (Ho & Ermon, 2016; Kostrikov et al., 2019) discriminator is 2 layer MLP of 100 hidden units with tanh activation.
Our generator consists of separate Actor and
Critic neural network
and follows the architecture used in (Kostrikov et al., 2019;
Fujimoto et al., 2018), where both of these networks have 2 layer MLP of 400 and 300 hidden units with ReLU activation.
To implement SAC-MCP, we modify Actor network (see Figure 2) by adding multiple premitive networks with same conﬁguration described in (Peng et al., 2019).
We have trained all networks with the Adam optimizer (Kingma & Ba, 2014) from PyTorch , which uses default learning rate of 1 e − 3 .
For our experiment we trained our algorithms on MuJoCo (Todorov et al., 2012) continuous control task environments.
For transfer learning experiments we use Custom-Ant and Shifting Maze environments from (Fu et al., 2018).
Performance curve is obtained using the mean over 10 experiments for 0-9 seeds and evaluated after each 5000 interaction with the environment.
During each evaluation we have stored the average performance ofruns.
8. Results 8.1. Reward function selection
We conduct experiment on CustomAnt-v0 (Fu et al., 2018) to select reward function for policy update.
We train SAC policy and collected 50 trajectory from the expert.
We keep our seed ﬁxed to 0 .
Figure 3 ( a ) gives the performance curve using actual reward from the environment and it is evident that reward signal directly from the reward function approximator (Implementation 4) gives better policy performance in imitation task.
We also see the cumulative reward achieved from reward approximator r θ (see Figure 3 ( b ) ) replicates the actual performance curve.
As our experiment shows drastic difference in performance we have not done comparative study for multiple seeds or other environments.
[Figure]
As demonstrated in Figure 4 using implementation-4 offpolicy AIRL gives a better imitation performance than
DAC in Hopper-v2 and Walker2d-v2 environments within 1 e 6 iterations [Figure] while provides comparable performance for
HalfCheetah-v2 and Ant-v2 environments after 2 e 6 iterations.
[Figure]
8.2. Performance in transfer learning
We test trained agent in transfer learning task under criterion (1) and (2).
We demonstrate the utility of learning reward function which gives us an advantage of learning IRL over imitation learning.
In IRL setting, Off-policy AIRL learns expert like behavior in Custom Ant environment but suffers from variance in Shifting maze task.
We do not tune separate hyper-parameter for individual environment but doing so may reduce performance variance for Shifting maze environment.
As illustrated in Figure 1 during transfer learning under criterion-
[Figure]
agent’s gait changes signiﬁcantly and as can be
[Figure]
seen in Figure 4 off-policy-AIRL fails to re-learn the policy using reward function.
But we do not conclude that it is not possible to relearn policy under this criterion using offpolicy-AIRL.
It is demonstrated in (Ha, 2018) that changing a better structure of agent’s body not only is better suited for the task but also facilitates policy learning.
The environment that we use in criterion-1 was introduced in (Fu et al., 2018), where the structure of the body of the quadrupedal
Ant was changed (Fu et al., 2018) for transfer learning by shrinking two legs and making other two larger.
But the speciﬁc increment or decrement of the legs may have favored for (Fu et al., 2018) to re-learn policy when two legs are paralyzed in transfer learning.
A hyper-parameter tuning of the structural change may work for our proposed algorithm as well.
Experiments with more structural ﬂexibility are required to come to a solid conclusion under this criterion.
On the other hand for Shifting maze environment (see Figure 4) it successfully completes the task but suffer from high variance in performance.
8.3. Performance comparison for using MCP
Performance of our MCP implementation in IRL setting is shown in Figure 6.
Using SAC-MCP improves the imitation performance in multiple (HalfCheetah-v2 and Ant-v2)
MuJoCo control taks.
We again conduct our transfer learning experiments on
Shifting-Maze tasks.
Here we do not load prior gating function.
We compare performance with and without re-training the actor.
Our initial hypothesis is that having multiplicative composition of Gaussian primitives should allow us with diverse behavior by only training the gating function during transfer task.
But Figure 7 shows we are not able to learn shifting maze task without retraining the actor networks.
We compare performance of SAC-MCP for k = { 4 , 8 } premitives (see Figure 7).
For K = 8 it reduces the performance
[Figure]
over continuous control task.
variance suffered by SAC.
[Figure]
8.4. Random State Initialization
We initialize the environment from a random state.
As we retrain already learned policy over one task, it does not give much exploratory action.
Thus initializing environment at different state will help to explore the new task.
We use expert agent to run for 0-100 steps and then stop at random, thus enabling the learning agent to start at random state every time the environment resets.
It allows an agent to explore from a state that is expected to be visited by an expert.
At the same time, using learned policy (from prior imitation task) to take actions will result into better quality of sequential observations.
We see in Figure 8 random initialization improves performance and also reduces performance variance for both SAC and SAC-MCP on transfer learning task for shifting maze environment.
9. Discussion and Conclusion
We propose an off-policy IRL algorithm in adversarial learning which learns policy using its approximated reward function.
We use SAC as generator to improve imitation performance while still being sample efﬁcient.
We compare different way to compute reward functions for off-policy generator update and experimentally ﬁnd simple state depen-
[Figure]
dent reward network without entropy regularization works best for our algorithm.
We compare our imitation performance with state-of-the-art imitation learning algorithm to show our persistent expert like performance over continuous control environments
[Figure]
We experimentally demonstrate the utility of learned reward function using off-policy-AIRL when the expert demonstrations are non-existent in transfer learning.
We compare performance over two transfer learning criterion, where either (1) the dynamics of the agent or (2) the dynamics of the environment changes while the goal/objective and action dimension remain the same.
Furthermore, we propose using MCP model to learn single primitive to achieve more ﬂexibility in retraining policy in transfer learning.
We observe an improved performance for using SAC-MCP on both imitation performance and transfer learning.
We also conduct experiments under random state initialization, which helps the agent to explore from an state that is expected to be visited in the new environment and reduces performance variance over prior experiments.
But unfortunately, our algorithm (using both SAC and SACMCP as generator) fails when the (criterion 1) dynamics of the agent is changed in transfer learning.
The quadrupedal
Ant has a very speciﬁc increment and decrement of the legs, which may have favored for (Fu et al., 2018) to re-learn policy when two legs are paralyzed in transfer learning.
As speciﬁc body structure of an agent can facilitate learning policy over a task (Ha, 2018), experiments using environments with more structural ﬂexibility is required to come to a conclusion under this criterion.