D EEP I NVERSE R EINFORCEMENT L EARNING VIA A DVERSARIAL O NE -C LASS C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Traditional inverse reinforcement learning (IRL) methods require a loop to ﬁnd the optimal policy for each reward update (called an inner loop), resulting in very time-consuming reward estimation.
In contrast, classiﬁcation-based IRL methods, which have been studied recently, do not require an inner loop and estimate rewards quickly, although it is difﬁcult to prepare an appropriate baseline corresponding to the expert trajectory.
In this study, we introduced adversarial oneclass classiﬁcation into the classiﬁcation-based IRL framework, and consequently developed a novel IRL method that requires only expert trajectories.
We experimentally veriﬁed that the developed method can achieve the same performance as existing methods.
I NTRODUCTION
Inverse reinforcement learning (IRL) (Russell, 1998) refers to the problem of estimating rewards for reinforcement learning (RL) agents to acquire policies that can reproduce expert behavior.
An RL algorithm learns a policy that maximizes the cumulative discounted reward under a given reward function.
An IRL algorithm does the opposite; it estimates the reward from the given policies or trajectories to satisfy the condition under the assumption that the expert is maximizing the reward.
IRL has been applied in two main areas (Ramachandran & Amir, 2007).
The ﬁrst is apprenticeship learning, which enables the learning of complex policies for which it is difﬁcult to design a reward function.
Compared to behavioral cloning, IRL is robust to the covariate shift problem (Ross et al., 2011) and achieves superior performance even when the amount of data is small.
The second is reward learning, where IRL is used to estimate rewards from the trajectory data of human and animal action sequences and to analyze the intention of the subject.
In previous studies, IRL methods have been used to analyze human walking paths (Kitani et al., 2012) and the behavior of nematodes (Yamaguchi et al., 2018).
In traditional IRL methods, the IRL loop has an inner loop that computes the optimal policy for the reward being estimated until convergence.
This inner loop presents a difﬁculty in applying IRL to tasks with a large state-action space because it is computation-intensive.
As a solution to this, classiﬁcation-based IRL methods transform the IRL problem into a problem of classifying the expert’s trajectory and the trajectory to be compared.
Notable methods include AIRL (Fu et al., 2017), LogReg-IRL (Uchibe, 2018), and T-REX (Brown et al., 2019).
These methods differ in the ways they are formulated, but they result in similar learning methods.
Online methods, such as AIRL, collect the trajectories to be compared from the environment.
Contrastingly, ofﬂine methods, such as LogReg-IRL and T-REX, collect the trajectories to be compared in advance, which enables them to further speed up and stabilize learning by not requiring access to the environment during training.
However, the learning performance of current ofﬂine methods depends heavily on the properties of the trajectories to be compared or the ranking of the trajectories, which is difﬁcult to collect.
In this study, we exploited the fact that the learning process of LogReg-IRL by binary classiﬁcation is equivalent to that of a discriminator in adversarial learning, such as with generative adversarial networks (GANs) (Goodfellow et al., 2014).
Speciﬁcally, we developed an innovative deep IRL method, called state-only learned one-class classiﬁer for IRL (SOLO-IRL), in which binary classiﬁcation is replaced with adversarial one-class classiﬁcation.
Figure 1 compares the traditional and proposed IRL methods.
The proposed method does not require an inner loop and is an ofﬂine method; thus, it can be trained extremely fast.
In addition, it does not require that trajectories be compared.
With these advantages, the proposed method greatly advances the application of IRL methods to real-world problems.
P RELIMINARIES 2. M ARKOV DECISION PROCESS (MDP)
RL is a learning problem based on the Markov decision process (MDP).
The MDP consists of a tuple M = (cid:104) S, A, P, R, γ (cid:105) , where S is the state space,
A is the action space, P is the state-transition probability, R is the reward function, and γ is the discount factor indicating the degree of importance for future rewards.
In the MDP, the state-value function for state s t at time t is represented by the
Bellman equation, as follows:
[Equation]
where R ( s t , a t ) is the reward for taking action a t in state s t and p ( s t +1 | s t , a t ) is the probability of transitioning to the next state s t +1 when taking action a t in state s t .
2. L INEARLY SOLVABLE MDP (LMDP)
The linearly solvable MDP (LMDP) is an extension of the MDP in which the agent directly determines the transition probability u
( s t +1 | s t ) from the current state s t to the next state s t +1 as the control, instead of the action a t in the MDP.
Then, the Bellman equation is linearized under two assumptions.
First, the state-transition probability p ( s t +1 | s t , u ) is assumed to be expressed as the product of the uncontrolled transition probability ¯ p ( s t +1 | s t ) and u as follows:
[Equation]
The uncontrolled transition probability ¯ p ( s t +1 | s t ) indicates a transitional relationship between the states in the environment.
When a transition is impossible, i.e., ¯ p = 0 , then p = 0 .
The second assumption is that the reward R ( s t , u ) is composed of a state-dependent reward r ( s t ) and penalty term D KL ( u || ¯ p ) for control u over the divergence from the uncontrolled transition probability ¯ p .
This assumption can be formulated as follows:
[Equation]
where D KL ( P x || P y ) represents the Kullback–Leibler (KL) divergence of P x and P y .
By rearranging
Eq.
(3) according to the deﬁnition of the KL divergence, the following equation is obtained:
[Equation]
Substituting Eq.
(4) into the Bellman equation in Eq.
(1) gives the following:
[Equation]
Eq. (2) is then substituted into Eq.
(5) and the Lagrange multiplier applied with (cid:80) s (cid:48) p ( s (cid:48) | s t , u ) =as a constraint.
Finally, the max operator is removed, resulting in the linear Bellman equation as follows:
[Equation]
The optimal control u ∗ in the LMDP is given by
[Equation]
2. L OGISTIC R EGRESSION -B ASED IRL (L OG R EG -IRL)
LogReg-IRL (Uchibe, 2018) is a deep IRL method in the LMDP.
The following is an overview of the IRL framework in LogReg-IRL.
By rearranging the linear Bellman equation in Eq.
(6), the following is obtained:
[Equation]
Then, substituting Eq.
(8) into Eq.
(7) and rearranging the result gives
[Equation]
Applying Bayes’ theorem to Eq.
(9) we obtain
[Equation]
The left-hand side and the ﬁrst term on the right-hand side of Eq.
(10) are the density-ratios.
The density-ratio p a /p b can be estimated by assigning the label η = 1 to the samples from the probability distribution p a , assigning η = − 1 to the samples from p b , and training a classiﬁer using logistic regression (Qin, 1998; Cheng et al., {+
 ;+} Bickel et al., 2007).
First, by Bayes’ theorem, the following is obtained:
[Equation]
Next, the ﬁrst discriminator D 1 ( x ) is deﬁned by the sigmoid function σ ( x ) = 1 / { 1 + exp( − x ) } and a neural network f ( x ) :
[Equation]
where the second term on the right-hand side of Eq.
(11) can be approximated by calculating the sample number ratio N p a /N p b and taking its logarithm.
For the ﬁrst term, the following equation can be obtained from the deﬁnition of the discriminator in Eq.
[Equation]
From Eq.
(13), when N p a = N p b , the following holds:
[Equation]
[Equation]
Therefore, the density-ratio of the ﬁrst term in Eq.
(10) can be estimated by sampling the states s ∗ t ∼ τ ∗ and ¯ s t ∼ ¯ τ from the expert trajectory τ ∗ according to the optimal control u ∗ and the baseline trajectory ¯ τ according to the uncontrolled transition probability ¯ p , followed by training with the following cross-entropy loss: [Equation]
[Equation]
The density-ratio on the left-hand side of Eq.
(10) is deﬁned as follows using the trained f ( x ) , reward-estimating neural network ˜ r ( x ) , and state-value-estimating neural network ˜ V ( x ) : [Equation]
[Equation]
The second discriminator D 2 for the state-transition pair is deﬁned as
[Equation]
As with D 1 , the discriminator D 2 is trained by cross-entropy loss L 2 , given as
[Equation]
In the original LogReg-IRL, an L2 regularization term is added to the loss function.
Following the process described above, LogReg-IRL estimates the reward and state-value by classifying the expert and baseline trajectories.
Unlike traditional IRL methods, LogReg-IRL does not require RL in the reward estimation process and, thus, it can be trained very quickly.
2. D IFFICULTY COLLECTING BASELINE TRAJECTORIES
LogReg-IRL showed that IRL in LMDP can be formulated by learning two discriminators.
However, LogReg-IRL has a problem in that its learning performance is greatly affected by the baseline trajectory.
For the baseline trajectory, it is desirable to collect data that follow uncontrolled transition probability ¯ p , such as trajectories obtained under a random policy, for a wide range of states.
However, for some tasks, the number of states that can be transitioned by a random policy may be limited.
For example, in a game task, such as an Atari game, the game does not progress, and in a driving simulator task such as TORCS (Wymann et al., 2000), the car crashes into a wall immediately.
In such cases, data according to a random policy cannot cover a wide range of states.
Several methods have been proposed to collect the baseline trajectory in LogReg-IRL.
For Atari games, a method using state-transition pairs from random policy in any state in the expert trajectory was proposed (Uchibe, 2018).
For TORCS, a method using the trajectory recorded by driving with noise added to the action output of the expert agent was proposed (Kishikawa & Arai, 2021).
However, those proposed methods are task-speciﬁc, and there is no well-established generalized method for collecting baseline trajectories.
An inappropriate baseline trajectory leads to inappropriate reward estimation.
The density-ratios diverge with respect to the states that the agent can reach and those for which there is no baseline trajectory, and high rewards are estimated where there are no experts.
Therefore, an agent that learns according to the estimated reward may acquire a different action from the expert as the optimal policy.
[Figure]
S TATE -O NLY L EARNED O NE - CLASS C LASSIFIER FOR IRL (SOLO-IRL)
We propose the novel IRL method SOLO-IRL, which estimates the reward given only the expert.
SOLO-IRL is a combination of an IRL framework based on LMDP, a transition generator based on adversarial one-class classiﬁcation, and least-squares loss.
Each of these is explained below.
3. S OLUTION BY ADVERSARIAL ONE - CLASS CLASSIFICATION
Classiﬁcation-based IRL methods are equivalent to learning a discriminator in adversarial learning frameworks, such as GANs (Goodfellow et al., 2014), which are binary classiﬁcation problems.
This means that the IRL problem can be solved by learning a discriminator that classiﬁes the trajectory as expert or not.
Broadening our perspective to other ﬁelds, we ﬁnd that the anomaly detection problem also requires a binary classiﬁer to distinguish between normal and abnormal samples.
However, in real problems, we often encounter situations in which we can obtain many normal samples but few abnormal samples.
Therefore, one-class classiﬁcation is a method for obtaining a binary classiﬁer using only normal samples.
Recently, the adversarially learned one-class classiﬁer (ALOCC)
(Sabokrou et al., 2018) was proposed as a one-class classiﬁcation method.
In the ALOCC, the discriminator is adversarially trained with a denoising autoencoder that generates fake normal samples.
Consequently, the discriminator is trained as a binary classiﬁer that identiﬁes normal and abnormal samples using only normal samples.
SOLO-IRL combines a classiﬁcation-based IRL method with the ALOCC.
The structure of SOLOIRL is illustrated in Figure 2, and its algorithm is given as Algorithm 1.
In the following, we describe the learning process and features of SOLO-IRL as well as the objective function used for learning SOLO-IRL.
3. L EARNING PROCESS OF SOLO-IRL
The training of SOLO-IRL consists of two stages.
In the ﬁrst stage, we learn a discriminator Dthat classiﬁes whether a state is sampled from expert data or not.
Generator G 1 is composed of an encoder–decoder network R 1 and generates a fake current state ¯ s t from the true current state s ∗ t plus noise ν 1 , as shown in Eq.
Then, the generator learns such that the generated ¯ s t and s ∗ t are close, and D 1 judges ¯ s t to be the true current state. [Equation]
[Equation]
Meanwhile, discriminator D 1 is deﬁned by Eq.
(12) and outputs the probability that a given state is sampled from the expert.
D 1 learns to distinguish between states sampled from a true expert and a fake expert from a generator.
Here, the density-ratio representing the expertness of each state is obtained in D 1 .
In the second stage, generator G 2 learns to generate a state-transition pair that is close to the sampled expert data and that discriminator D 2 judges as expert.
Generator G 2 uses two encoder–decoder networks R 2 and R 3 to generate a fake state-transition pair (¯ s t , ¯ s t +1 ) from the expert’s current state s ∗ t plus noise ν 2 , as given by Eqs.
(20) and (21).
Then, the generator learns such that the generated (¯ s t , ¯ s t +1 ) and ( s ∗ t , s ∗ t +1 ) are close to each other, and discriminator D 2 judges the generated data to be a true state-transition pair.
[Equation]
The training of D 2 is the same as that of D 1 in the ﬁrst stage.
Here, discriminator D 2 contains a reward network ˜ r and state-value network ˜ V , as shown in Eq.
Finally, it works as an IRL algorithm to estimate the reward and state-value.
By introducing a generator and training the discriminator in an adversarial manner, the decision boundary around the expert is reﬁned, and the appropriate reward is estimated.
This makes preparation of the baseline trajectory by trial and error unnecessary.
In addition, because SOLO-IRL is an ofﬂine method, it learns quickly without executing RL or interacting with the environment.
To the best of our knowledge, SOLO-IRL is the only method that estimates the reward and state-value exclusively from expert trajectories.
3. L EAST - SQUARES LOSS AS ADVERSARIAL OBJECTIVE
To train the generator and discriminator, we propose using least-squares loss as an adversarial objective.
The least-squares loss is represented by the following equations:
[Equation] [Equation]
23) where d ∗ and ¯ d denote the true and fake states or state-transition pairs, respectively.
The leastsquares loss was proposed alongside LSGAN (Mao et al., 2017), which solved the learning instability and mode collapse problems of previous GANs.
In SOLO-IRL, this least-squares loss is used instead of the cross-entropy loss to address the problems of learning stability and mode collapse.
3. R ECONSTRUCTION OBJECTIVE
Meanwhile, the generators G 1 and G 2 must be trained to be close to the true expert data in addition to training by the adversarial objective.
The reconstruction objective is trained by the least-squares loss given by the following equations, as in the original ALOCC:
[Equation]
Finally, the objective of discriminators D 1 and D 2 becomes L adv ( D ) , and the objective of generators G 1 and G 2 becomes
L adv ( G ) + L rec ( G ) .
Using these objectives, SOLO-IRL estimates the reward and state-value by training each neural network using the gradient descent method.
E XPERIMENTAL RESULTS AND DISCUSSION
We validated the performance of the proposed method in the OpenAI Gym environment (Brockman et al., 2016).
For the expert trajectory, we used a trajectory generated by an agent that had learned the optimal action for the true reward as an expert.
As RL algorithms, we used PPO (Schulman et al., 2017) for any task with a discrete action space and TD3 (Fujimoto et al., 2018) for any task with a continuous action space.
In SOLO-IRL, a three-layer multilayer perceptron was used as both the encoder and decoder in the generator, and a three-layer multilayer perceptron as the discriminator.
Adam (Kingma & Ba, 2014) was used to optimize the neural network.
Require:
Expert trajectories τ ∗ , discount factor γ , noise ν
and ν 2 ,
numbers of iterations n 1 and n
Ensure: Reward network ˜ r ( x ) , state-value network ˜ V
( x ) 1: Initialize neural network f ( x ) , R 1 ( x ) 2: Deﬁne discriminator D 1 ( s t ) =
σ ( f ( s t )) 3: for i = 0 , · · · , n 1 do 4:
Sample expert state s ∗ t from τ ∗ 5: ¯ s t ← R 1 ( s ∗ t + ν 1 ) 6:
Train D 1 according to the loss L adv ( D ) in Eq.
Train R 1 with loss L
adv ( G ) + L rec ( G 1 ) in Eqs.
(23) and (24) 8: end for 9: Initialize neural network ˜ r
( x ) , ˜ V ( x ) , R 2 ( x ) , R 3 ( x ) 10: Deﬁne discriminator D 2 ( s t , s t +1 )
=
σ ( f ( s t ) + ˜ r
( s t )
+
γ ˜ V
( s t +1 )
− ˜ V ( s t )) 11: Disable f updates 12: for i = 0 , · · · , n 2 do 13:
Sample expert current state s ∗ t and next state s ∗ t +1 from τ ∗ 14: ¯ s t ← R 2 ( s ∗ t + ν 2 ) 15: ¯ s t +1 ← R 3 ( s ∗ t + ν 2 ) 16:
Train D 2 according to the loss L adv ( D ) in Eq.
Train R 2 and R 3 according to the loss L adv ( G )
+ L rec ( G ) in Eqs.
and ( ) 18: end for CartPole
BipedalWalker [Table] We compared SOLO-IRL to LogReg-IRL (Uchibe, 2018).
The baseline trajectory in LogReg-IRL was collected by running a random policy based on a uniform distribution in the environment. The LogReg-IRL implementation was created based on the SOLO-IRL implementation with the following changes: removal of the generator, changed cross-entropy loss, and addition of a weight decay term (coefﬁcient: 1e-3) to the loss.
CartPole.
CartPole (Gym, b) is a basic RL task in which the agent must keep a pendulum connected to a cart upright by controlling the cart.
A survival reward is given to the pendulum in CartPole as long as it remains upright.
BipedalWalker.
BipedalWalker
(Gym, a) is a task in which a robot with four joints learns a bipedal walking task.
The agent is required to output the velocities of its two hips and knees as actions, taking the state given by virtual sensor inputs as the input.
The true reward consists of a bonus for speed and penalty for magnitude of the action or a fall.
Analysis of results.
First, we evaluated the expert agent’s policy, the random policy, and the policies of the agents trained according to the rewards estimated by LogReg-IRL and SOLO-IRL, respectively, in each environment; the results are shown in Table 1.
There are two metrics for evaluating the performance of IRL: expected value difference (EVD) (Levine et al., 2011) and expected cumulative reward. To calculate the EVD, we use the discount rate.
However, since the appropriate discount rate varies depending on the environment, we cannot be sure of the value we use. Therefore, we decided to evaluate the performance using the expected cumulative reward, which is actually used for learning.
[Figure] [Figure]
[Figure]
[Figure]
The results for CartPole show that SOLO-IRL was able to achieve performance on par with that of LogReg-IRL. Furthermore, in the results for BipedalWalker, we can see that LogReg-IRL failed to learn
[Figure]
whereas SOLO-IRL obtained a score. This can be attributed to the fact that BipedalWalker has a vast state space, whereas CartPole has a relatively small state space. As a result, the random policy could only collect data near the starting point, and could not provide an appropriate baseline trajectory for the expert trajectory that progressed to the goal. Relationship between trajectory distribution and estimated reward in the BipedalWalker task.
Among the 24 dimensions in the BipedalWalker state, Figure 3 visualized the distribution of the expert and baseline trajectories for “velocity in the X-axis direction” and “angle of hip 1,” and Figures 4 and 5 visualized the estimated reward for the same two dimensions at the starting point.
It can be seen that the baseline trajectory does not cover part of the distribution of the expert trajectory. Owing to the lack of a baseline trajectory, LogReg-IRL’s reward estimation failed, yielding a meaningless reward. In contrast, SOLO-IRL’s reward estimation, which was trained by generating samples near the expert, resulted in an appropriate reward that drove the agent forward. Behavior of adversarial and reconstruction objectives.
Figures 6 and 7 show the behavior of adversarial loss and reconstruction loss during training in SOLO-IRL. With adversarial learning, the probabilities of truthfulness of the true and fake samples converge to an equilibrium point near 0.5.
The discriminator narrows the decision boundary towards the expert neighborhood through learning, and the generator eventually succeeds in producing samples that are close to true. The success of the generator learning is also evident from the convergence of the reconstruction loss. Summary of experimental results.
There is room for improving the performance of LogReg-IRL by using complex rule-based policies or combining multiple policies to collect baseline trajectories, however this is expected to be much more difﬁcult than adjusting the noise in SOLO-IRL.
Therefore, we consider that SOLO-IRL can achieve the same or better performance than LogReg-IRL more easily.
C ONCLUSION
In this study, we exploited the fact that the classiﬁcation-based IRL framework is equivalent to training a discriminator in adversarial learning and developed SOLO-IRL, in which a generator is incorporated to generate fake expert data.
SOLO-IRL can be trained quickly without the need for an inner loop and easily estimates rewards with high performance exclusively from expert trajectories, with no need for baseline trajectories.
However, although it is simpler than collecting baseline trajectories, the noise used for reconstruction by the generator still requires adjustment.
In future work, we will develop a method that does not require noise adjustment.
We will also consider application to more advanced image-based control tasks and the incorporation of recent advances in GANs. R EPRODUCIBILITY S TATEMENT
In Appendices A and B we have described the details of the hyperparameters and noise needed to reproduce our experiments.
We also attached the source code used in our experiments
A H YPERPARAMETERS OF RL
This section describes the main hyperparameters used in the experiments.
For RL, we used the same settings for both the training of the expert agent and the training based on IRL results.
A. H YPERPARAMETERS OF TD3 AGENT
The implementation of TD3 is based on https://github.com/chainer/chainerrl/ blob/master/examples/mujoco/reproduction/td3/train_td3.py , which is identical except for the environment name and the values listed in Table 2.
[Table]
A. H YPERPARAMETERS OF PPO AGENT
The implementation of PPO is based on https://github.com/chainer/chainerrl/ blob/master/examples/mujoco/train_ppo_gym.py , which is identical except for the environment name and the values listed in Table 3.
[Table]
B IRL HYPERPARAMETERS
The hyperparameters employed in training SOLO-IRL and LogReg-IRL are shown in Table 4.
[Table]
B. A DDITIVE NOISE
For CartPole, we added noise that follows a normal distribution N (0 , 0 . 001) to the true state.
For each dimension of BipedalWalker, we added either noise based on the normal distribution or noise for the labels.
The tuning results for the experiment are shown in Table 5.
The “Types of noise” column in the table lists the noise addition operations described below, and the “Parameters” column lists the noise parameters.
• “Normal” · · · s add ← s ∗ + ν , ν ∼ N ( µ, σ 2 ) • “Label” · · · s add ← min(max( s ∗ + ν, 0) , 1) , ν is randomly sampled from [ − 1 , 0 , 1] with probability [ p a , p b , p c ]
[Table]