D EEP I NVERSE R EINFORCEMENT L EARNING VIA A DVERSARIAL O NE -C LASS C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Traditional inverse reinforcement learning (IRL) methods require a loop to ﬁnd the optimal policy for each reward update (called an inner loop), resulting in very time-consuming reward estimation.
In contrast, classiﬁcation-based IRL methods, which have been studied recently, do not require an inner loop and estimate rewards quickly, although it is difﬁcult to prepare an appropriate baseline corresponding to the expert trajectory.
In this study, we introduced adversarial oneclass classiﬁcation into the classiﬁcation-based IRL framework, and consequently developed a novel IRL method that requires only expert trajectories.
We experimentally veriﬁed that the developed method can achieve the same performance as existing methods.
I NTRODUCTION
Inverse reinforcement learning (IRL) (Russell, 1998) refers to the problem of estimating rewards for reinforcement learning (RL) agents to acquire policies that can reproduce expert behavior.
An RL algorithm learns a policy that maximizes the cumulative discounted reward under a given reward function.
An IRL algorithm does the opposite; it estimates the reward from the given policies or trajectories to satisfy the condition under the assumption that the expert is maximizing the reward.
IRL has been applied in two main areas (Ramachandran & Amir, 2007).
The ﬁrst is apprenticeship learning, which enables the learning of complex policies for which it is difﬁcult to design a reward function.
Compared to behavioral cloning, IRL is robust to the covariate shift problem (Ross et al., 2011) and achieves superior performance even when the amount of data is small.
The second is reward learning, where IRL is used to estimate rewards from the trajectory data of human and animal action sequences and to analyze the intention of the subject.
In previous studies, IRL methods have been used to analyze human walking paths (Kitani et al., 2012) and the behavior of nematodes (Yamaguchi et al., 2018).
In traditional IRL methods, the IRL loop has an inner loop that computes the optimal policy for the reward being estimated until convergence.
This inner loop presents a difﬁculty in applying
IRL to tasks with a large state-action space because it is computation-intensive.
As a solution to this, classiﬁcation-based IRL methods transform the IRL problem into a problem of classifying the expert’s trajectory and the trajectory to be compared.
Notable methods include AIRL (Fu et al., 2017), LogReg-IRL (Uchibe, 2018), and T-REX (Brown et al., 2019).
These methods differ in the ways they are formulated, but they result in similar learning methods.
Online methods, such as AIRL, collect the trajectories to be compared from the environment.
Contrastingly, ofﬂine methods, such as LogReg-IRL and T-REX, collect the trajectories to be compared in advance, which enables them to further speed up and stabilize learning by not requiring access to the environment during training.
However, the learning performance of current ofﬂine methods depends heavily on the properties of the trajectories to be compared or the ranking of the trajectories, which is difﬁcult to collect.
In this study, we exploited the fact that the learning process of LogReg-IRL by binary classiﬁcation is equivalent to that of a discriminator in adversarial learning, such as with generative adversarial networks (GANs) (Goodfellow et al., 2014).
Speciﬁcally, we developed an innovative deep IRL method, called state-only learned one-class classiﬁer for IRL (SOLO-IRL), in which binary classiﬁcation is replaced with adversarial one-class classiﬁcation.
Figure 1 compares the traditional and proposed IRL methods.
The proposed method does not require an inner loop and is an ofﬂine method; thus, it can be trained extremely fast.
In addition, it does not require that trajectories be compared.
With these advantages, the proposed method greatly advances the application of IRL methods to real-world problems.
P RELIMINARIES 2. M ARKOV DECISION PROCESS (MDP)
RL is a learning problem based on the Markov decision process (MDP).
The MDP consists of a tuple M = (cid:104) S, A, P, R, γ (cid:105) , where S is the state space,
A is the action space, P is the state-transition probability, R is the reward function, and γ is the discount factor indicating the degree of importance for future rewards.
In the MDP, the state-value function for state s t at time t is represented by the
Bellman equation, as follows:
[Equation]
where R ( s t , a t ) is the reward for taking action a t in state s t and p ( s t +1 | s t , a t ) is the probability of transitioning to the next state s t +1 when taking action a t in state s t .
2. L INEARLY SOLVABLE MDP (LMDP)
The linearly solvable MDP (LMDP) is an extension of the MDP in which the agent directly determines the transition probability u
( s t +1 | s t ) from the current state s t to the next state s t +1 as the control, instead of the action a t in the MDP.
Then, the Bellman equation is linearized under two assumptions.
First, the state-transition probability p ( s t +1 | s t , u ) is assumed to be expressed as the product of the uncontrolled transition probability ¯ p ( s t +1 | s t ) and u as follows:
[Equation]
The uncontrolled transition probability ¯ p ( s t +1 | s t ) indicates a transitional relationship between the states in the environment.
When a transition is impossible, i.e., ¯ p = 0 , then p = 0 .
The second assumption is that the reward R ( s t , u ) is composed of a state-dependent reward r ( s t ) and penalty term D KL ( p || ¯ p ) for state-transition probability p over the divergence from the uncontrolled transition probability ¯ p .
This assumption can be formulated as follows:
[Equation]
where D KL ( P x || P y ) represents the Kullback–Leibler (KL) divergence of P x and P y .
By rearranging
Eq.
(3) according to the deﬁnition of the KL divergence, the following equation is obtained:
[Equation]
Substituting Eq.
(4) into the Bellman equation in Eq.
(1) gives the following:
[Equation]
Eq.
(2) is then substituted into Eq.
(5) and the Lagrange multiplier applied with (cid:80) s (cid:48) p ( s (cid:48) | s t , u ) =as a constraint.
Finally, the max operator is removed, resulting in the linear Bellman equation as follows:
[Equation]
The optimal control u ∗ in the LMDP is given by
[Equation]
2. L OGISTIC R EGRESSION -B ASED IRL (L OG R EG -IRL)
LogReg-IRL (Uchibe, 2018) is a deep IRL method in the LMDP.
The following is an overview of the IRL framework in LogReg-IRL.
By rearranging the linear Bellman equation in Eq.
(6), the following is obtained:
[Equation]
Then, substituting Eq.
(8) into Eq.
(7) and rearranging the result gives
[Equation]
Applying Bayes’ theorem to Eq.
(9) we obtain
[Equation]
The left-hand side and the ﬁrst term on the right-hand side of Eq.
(10) are the density-ratios.
The density-ratio p a /p b can be estimated by assigning the label η = 1 to the samples from the probability distribution p a , assigning η = − 1 to the samples from p b , and training a classiﬁer using logistic regression (Qin, 1998;
Cheng et al.,
; Bickel et al., 2007).
First, by Bayes’ theorem, the following is obtained:
[Equation]
Next, the ﬁrst discriminator D 1 ( x ) is deﬁned by the sigmoid function σ ( x ) = 1 / { 1 + exp( − x ) } and a neural network f ( x ) :
[Equation]
where the second term on the right-hand side of Eq.
(11) can be approximated by calculating the sample number ratio N p a /N p b and taking its logarithm.
For the ﬁrst term, the following equation can be obtained from the deﬁnition of the discriminator in Eq.
[Equation]
From Eq.
(13), when N p a = N p b , the following holds:
[Equation]
[Equation]
Therefore, the density-ratio of the ﬁrst term in Eq.
(10) can be estimated by sampling the states s ∗ t ∼ τ ∗ and ¯ s t ∼ ¯ τ from the expert trajectory τ ∗ according to the optimal control u ∗ and the baseline trajectory ¯ τ according to the uncontrolled transition probability ¯ p , followed by training with the following cross-entropy loss:
[Equation]
[Equation]
The density-ratio on the left-hand side of Eq.
(10) is deﬁned as follows using the trained f ( x ) , reward-estimating neural network ˜ r ( x ) , and state-value-estimating neural network ˜ V ( x ) :
[Equation]
[Equation]
The second discriminator D 2 for the state-transition pair is deﬁned as
[Equation]
As with D 1 , the discriminator D 2 is trained by cross-entropy loss L 2 , given as
[Equation]
In the original LogReg-IRL, an L2 regularization term is added to the loss function.
Following the process described above, LogReg-IRL estimates the reward and state-value by classifying the expert and baseline trajectories.
Unlike traditional IRL methods, LogReg-IRL does not require RL in the reward estimation process and, thus, it can be trained very quickly.
2. D IFFICULTY COLLECTING BASELINE TRAJECTORIES
LogReg-IRL showed that IRL in LMDP can be formulated by learning two discriminators.
However, LogReg-IRL has a problem in that its learning performance is greatly affected by the baseline trajectory.
For the baseline trajectory, it is desirable to collect data that follow uncontrolled transition probability ¯ p , such as trajectories obtained under a random policy, for a wide range of states.
However, for some tasks, the number of states that can be transitioned by a random policy may be limited.
For example, in a game task, such as an Atari game, the game does not progress, and in a driving simulator task such as TORCS (Wymann et al., 2000), the car crashes into a wall immediately.
In such cases, data according to a random policy cannot cover a wide range of states.
Several methods have been proposed to collect the baseline trajectory in LogReg-IRL.
For Atari games, a method using state-transition pairs from random policy in any state in the expert trajectory was proposed (Uchibe, 2018).
For TORCS, a method using the trajectory recorded by driving with noise added to the action output of the expert agent was proposed (Kishikawa & Arai, 2021).
However, those proposed methods are task-speciﬁc, and there is no well-established generalized method for collecting baseline trajectories.
An inappropriate baseline trajectory leads to inappropriate reward estimation.
The density-ratios diverge with respect to the states that the agent can reach and those for which there is no baseline trajectory, and high rewards are estimated where there are no experts.
Therefore, an agent that learns according to the estimated reward may acquire a different action from the expert as the optimal policy.
[Figure]
S TATE -O NLY L EARNED O NE - CLASS C LASSIFIER FOR IRL (SOLO-IRL)
We propose the novel IRL method SOLO-IRL, which estimates the reward given only the expert.
SOLO-IRL is a combination of an IRL framework based on LMDP, a transition generator based on adversarial one-class classiﬁcation, and least-squares loss.
Each of these is explained below.
3. S OLUTION BY ADVERSARIAL ONE - CLASS CLASSIFICATION
Classiﬁcation-based IRL methods are equivalent to learning a discriminator in adversarial learning frameworks, such as GANs (Goodfellow et al., 2014), which are binary classiﬁcation problems.
This means that the IRL problem can be solved by learning a discriminator that classiﬁes the trajectory as expert or not.
Broadening our perspective to other ﬁelds, we ﬁnd that the anomaly detection problem also requires a binary classiﬁer to distinguish between normal and abnormal samples.
However, in real problems, we often encounter situations in which we can obtain many normal samples but few abnormal samples.
Therefore, one-class classiﬁcation is a method for obtaining a binary classiﬁer using only normal samples.
Recently, the adversarially learned one-class classiﬁer (ALOCC)
(Sabokrou et al., 2018) was proposed as a one-class classiﬁcation method.
In the ALOCC, the discriminator is adversarially trained with a denoising autoencoder that generates fake normal samples.
Consequently, the discriminator is trained as a binary classiﬁer that identiﬁes normal and abnormal samples using only normal samples.
SOLO-IRL combines a classiﬁcation-based IRL method with the ALOCC.
The structure of SOLOIRL is illustrated in Figure 2, and its algorithm is given as Algorithm 1.
In the following, we describe the learning process and features of SOLO-IRL as well as the objective function used for learning
SOLO-IRL.
3. L EARNING PROCESS OF SOLO-IRL
The training of SOLO-IRL consists of two stages.
In the ﬁrst stage, we learn a discriminator Dthat classiﬁes whether a state is sampled from expert data or not.
Generator G 1 is composed of an encoder–decoder network R 1 and generates a fake current state ¯ s t from the true current state s ∗ t plus noise ν 1 , as shown in Eq.
Then, the generator learns such that the generated ¯ s t and s ∗ t are close, and D 1 judges ¯ s t to be the true current state.
[Equation]
[Equation]
Meanwhile, discriminator D 1 is deﬁned by Eq.
(12) and outputs the probability that a given state is sampled from the expert.
D 1 learns to distinguish between states sampled from a true expert and a fake expert from a generator.
Here, the density-ratio representing the expertness of each state is obtained in D 1 .
In the second stage, generator G 2 learns to generate a state-transition pair that is close to the sampled expert data and that discriminator D 2 judges as expert.
Generator G 2 uses two encoder–decoder networks R 2 and R 3 to generate a fake state-transition pair (¯ s t , ¯ s t +1 ) from the expert’s current state s ∗ t plus noise ν 2 , as given by Eqs.
(20) and (21).
Then, the generator learns such that the generated (¯ s t , ¯ s t +1 ) and ( s ∗ t , s ∗ t +1 ) are close to each other, and discriminator D 2 judges the generated data to be a true state-transition pair.
[Equation]
The training of D 2 is the same as that of D 1 in the ﬁrst stage.
Here, discriminator D 2 contains a reward network ˜ r and state-value network ˜ V , as shown in Eq.
Finally, it works as an IRL algorithm to estimate the reward and state-value.
By introducing a generator and training the discriminator in an adversarial manner, the decision boundary around the expert is reﬁned, and the appropriate reward is estimated.
This makes preparation of the baseline trajectory by trial and error unnecessary.
In addition, because SOLO-IRL is an ofﬂine method, it learns quickly without executing RL or interacting with the environment.
To the best of our knowledge, SOLO-IRL is the only method that estimates the reward and state-value exclusively from expert trajectories.
3. L EAST - SQUARES LOSS AS ADVERSARIAL OBJECTIVE
To train the generator and discriminator, we propose using least-squares loss as an adversarial objective.
The least-squares loss is represented by the following equations:
[Equation] [Equation]
23) where d ∗ and ¯ d denote the true and fake states or state-transition pairs, respectively.
The leastsquares loss was proposed alongside LSGAN (Mao et al., 2017), which solved the learning instability and mode collapse problems of previous GANs.
In SOLO-IRL, this least-squares loss is used instead of the cross-entropy loss to address the problems of learning stability and mode collapse.
3. R ECONSTRUCTION OBJECTIVE
Meanwhile, the generators G 1 and G 2 must be trained to be close to the true expert data in addition to training by the adversarial objective.
The reconstruction objective is trained by the least-squares loss given by the following equations, as in the original ALOCC:
[Equation]
Finally, the objective of discriminators D 1 and D 2 becomes L adv ( D ) , and the objective of generators G 1 and G 2 becomes
L adv ( G ) + L rec ( G ) .
Using these objectives, SOLO-IRL estimates the reward and state-value by training each neural network using the gradient descent method.
E XPERIMENTAL RESULTS AND DISCUSSION
We validated the performance of the proposed method in the OpenAI Gym environment (Brockman et al., 2016).
For the expert trajectory, we used a trajectory generated by an agent that had learned the optimal action for the true reward as an expert.
As RL algorithms, we used PPO (Schulman et al., 2017) for the tasks with a discrete action space and TD3 (Fujimoto et al., 2018) for the tasks with a continuous action space.
In SOLO-IRL, a three-layer multilayer perceptron was used as both the encoder and decoder in the generator, and a three-layer multilayer perceptron was used as the discriminator.
Adam (Kingma & Ba, 2014) was used to optimize the neural network.
Require:
Expert trajectories τ ∗ , discount factor γ , noise ν
and ν 2 ,
numbers of iterations n 1 and n
Ensure: Reward network ˜ r ( x ) , state-value network ˜ V
( x ) 1: Initialize neural network f ( x ) , R 1 ( x ) 2: Deﬁne discriminator D 1 ( s t ) =
σ ( f ( s t )) 3: for i = 0 , · · · , n 1 do 4:
Sample expert state s ∗ t from τ ∗ 5: ¯ s t ← R 1 ( s ∗ t + ν 1 ) 6:
Train D 1 according to the loss L adv ( D ) in Eq.
Train R 1 with loss L
adv ( G ) + L rec ( G 1 ) in Eqs.
(23) and (24) 8: end for 9: Initialize neural network ˜ r
( x ) , ˜ V ( x ) , R 2 ( x ) , R 3 ( x ) 10: Deﬁne discriminator D 2 ( s t , s t +1 )
=
σ ( f ( s t ) + ˜ r
( s t )
+
γ ˜ V
( s t +1 )
− ˜ V ( s t )) 11: Disable f updates 12: for i = 0 , · · · , n 2 do 13:
Sample expert current state s ∗ t and next state s ∗ t +1 from τ ∗ 14: ¯ s t ← R 2 ( s ∗ t + ν 2 ) 15: ¯ s t +1 ← R 3 ( s ∗ t + ν 2 ) 16:
Train D 2 according to the loss L adv ( D ) in Eq.
Train R 2 and R 3 according to the loss L adv ( G )
+ L rec ( G 2 ) in Eqs.
(23) and (
) 18: end for
CartPole
BipedalWalker
Hopper
[Table]
We compared SOLO-IRL to LogReg-IRL (Uchibe, 2018).
The baseline trajectory in LogReg-IRL was collected by running a random policy based on a uniform distribution in the environment.
The
LogReg-IRL implementation was created based on the SOLO-IRL implementation with the following changes: removal of the generator, changing of the cross-entropy loss, and addition of a weight decay term (coefﬁcient: 1e-3) to the loss. We used the following four tasks for validation.
CartPole.
CartPole (Gym, b) is a basic RL task in which the agent must keep a pendulum connected to a cart upright by controlling the cart.
A survival reward is given to the pendulum as long as it remains upright.
BipedalWalker.
BipedalWalker
(Gym, a) is a task in which a robot with four joints learns a bipedal walking task.
The agent is required to output the velocities of its two hips and knees as actions, taking the state given by the virtual sensor as the input.
The true reward comprises a bonus based on the speed and a penalty based on the magnitude of the action or fall.
MuJoCo. MuJoCo (Todorov et al., 2012) is an environment that collects tasks for the development of robotics. We selected Hopper and Walker2d as examples. Hopper is a task in which a one-legged robot moves forward, and Walker2d is one in which a two-legged robot walks.
These two tasks require more complex continuous control than BipedalWalker.
[Figure] [Figure] Analysis of results. First, in each environment, we evaluated the policy of the expert agent, the random policy, the policy of the agent trained according to the reward estimated by LogReg-IRL and SOLO-IRL, and the policy obtained via behavioral cloning.
We also tested the random expert distillation (RED) (Wang et al., 2019), which is a recent method of imitation learning in the MuJoCo environment 1 .
There are two metrics for evaluating the performance of IRL: expected value difference (EVD) (Levine et al., 2011) and expected cumulative reward.
To calculate the EVD, we used the discount rate.
However, since the appropriate discount rate varies depending on the environment, the value we use will also vary accordingly.
Therefore, we decided to evaluate the performance using the expected cumulative reward, which is a more common of the two.
The results are presented in Table 1.
The results for CartPole show that SOLO-IRL was able to achieve comparable with that of LogReg-IRL.
Furthermore, with regard to the results for BipedalWalker, we saw that LogReg-IRL failed to learn, whereas SOLO-IRL obtained a good score.
The aforementioned results can be attributed to the fact that BipedalWalker has a vast state space, whereas CartPole has a relatively small state space.
Consequently, the random policy could only collect data near the starting point and could not provide an appropriate baseline trajectory for the expert trajectory that progressed to the goal.
Behavioral cloning and RED generally perform better than the other methods that were used in the comparison.
However, in BipedalWalker, the proposed method showed better performance than behavioral cloning due to the changing environment.
In addition, RED requires more information than the proposed method because of the availability of RL.
Therefore, our proposed method can be said to be superior than the traditional methods in that it can learn purely using expert trajectories.
Relationship between trajectory distribution and estimated reward in the BipedalWalker task.
Among the 24 dimensions in the BipedalWalker state, Figure 3 visualizes the distribution of the expert and baseline trajectories for “velocity in the X-axis direction” and “angle of hip 1,” and Figures 4 and 5 visualize the estimated reward for these two dimensions at the starting point.
As can be seen, the baseline trajectory does not cover a part of the the expert trajectory.
Owing to the lack of a baseline trajectory, LogReg-IRL’s estimation failed, yielding a meaningless reward.
In contrast, SOLO-IRL’s estimation, which was trained via generating samples near the expert, resulted in an appropriate reward that drove the agent forward.
Behavior of adversarial and reconstruction objectives.
Figures 6 and 7 illustrate the behavior of adversarial loss and reconstruction loss during training in SOLO-IRL.
With adversarial learning, the probabilities of truthfulness of the true and fake samples converge to an equilibrium near 0.5.
The discriminator narrows the decision boundary toward the expert neighborhood through learning, and the generator eventually succeeds in producing samples that are close to the true.
The learning success of the generator is also evident from the convergence of the reconstruction loss.
Summary of the experimental results.
There is room for improving the performance of LogRegIRL using complex rule-based policies or by combining multiple policies to collect baseline tra- [Figure] [Figure] jectories; however, this is expected to be signiﬁcantly more difﬁcult than adjusting the noise in
SOLO-IRL. Therefore, it can be said that similar or better performance than LogReg-IRL can be achieved more easily using SOLO-IRL.  R ELATED WORKS IRL.
A method similar to the proposed method is the AIRL, which is an extension of maximum entropy IRL (Ziebart et al., 2008), guided cost learning (Finn et al., 2016), and generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016).
The theoretical background of the aforementioned methods is related to the proposed method in terms of entropy regularization (Chow et al., 2018). AIRL requires access to the environment, while the proposed method does not. In addition, DREX (Brown et al., 2020) is similar to the proposed method in that it creates the trajectory of the comparison target by adding noise. Since the proposed method uses adversarial learning, it can generate more appropriate samples for comparison and does not need to learn behavioral cloning.
Imitation learning.
There are several methods that have been recently proposed for imitating expert demonstrations, such as RED, disagreement-regularized imitation learning (Brantley et al., 2019), and O-NAIL
(Arenz & Neumann, 2020).
All these methods are extensions of GAIL and require information regarding the expert’s state-action pairs.
The proposed method can be used to estimate the reward from the trajectory comprising only states.
Behavioral cloning.
The simplest ofﬂine learning method to imitate expert trajectory data is behavioral cloning, and (Torabi et al., 2018) is such a method that can be applied to state-only trajectories.
However, compared to IRL methods, it is difﬁcult to deal with the issues of policy transferability and covariate shift using such a method.
C ONCLUSION
In this study, we exploited the fact that the classiﬁcation-based IRL framework is equivalent to training a discriminator in adversarial learning and developed SOLO-IRL, in which a generator is incorporated to generate fake expert data.
SOLO-IRL can be trained quickly without the need for an inner loop and easily estimates rewards with high performance exclusively from expert trajectories, with no need for baseline trajectories.
However, although it is simpler than collecting baseline trajectories, the noise used for reconstruction by the generator still requires adjustment.
In future work, we will develop a method that does not require noise adjustment.
We will also consider application to more advanced image-based control tasks and the incorporation of recent advances in GANs.
R EPRODUCIBILITY S TATEMENT
In Appendices we have described the details of the hyperparameters and noise needed to reproduce our experiments.
We also attached the source code used in our experiments.
A H YPERPARAMETERS OF RL
This section describes the main hyperparameters used in the experiments.
For RL, we used the same settings for both the training of the expert agent and the training based on IRL results.
A. H YPERPARAMETERS OF TD3 AGENT
The implementation of TD3 is based on https://github.com/chainer/chainerrl/ blob/master/examples/mujoco/reproduction/td3/train_td3.py , which is identical except for the environment name and the values listed in Table 2.
[Table]
A. H YPERPARAMETERS OF PPO AGENT
The implementation of PPO is based on https://github.com/chainer/chainerrl/ blob/master/examples/mujoco/train_ppo_gym.py , which is identical except for the environment name and the values listed in Table 3.
[Table]
B IRL HYPERPARAMETERS
The hyperparameters employed in training SOLO-IRL and LogReg-IRL are shown in Table 4.
[Table]
B. A DDITIVE NOISE
For CartPole, we added noise that follows a normal distribution N (0 , 0 . 001) to the true state.
For the MuJoCo task, we added a normal distribution N (0 , 0 . 00001)
noise to Hopper and a normal distribution N (0 , 0 . 1) noise to Walker2d.
For each dimension of BipedalWalker, we added either noise based on the normal distribution or noise for the labels.
The tuning results for the experiment are shown in Table 5.
The “Types of noise” column in the table lists the noise addition operations described below [Table] and the “Parameters” column lists the noise parameters.
• “Normal” · · · s add ← s ∗ + ν , ν ∼ N ( µ, σ 2 ) • “Label” · · · s add ← min(max( s ∗ + ν, 0) , 1) , ν is randomly sampled from − 1 , 0 , 1] with probability [ p a , p b , p c ] [Table] B. S ENSITIVITY TO NOISE SCALE
To analyze the sensitivity of the learning results to the noise scale, we conducted experiments in which we changed the standard deviation σ of the normal distribution N (0 , σ 2 ) used to generate the noise.
Walker2d in the MuJoCo environment was used for the experiments.
The results are shown in Figure 8.
It can be seen that the noise should have a certain level of magnitude and should not be too small or too large.
The appropriate noise level is considered to be task-dependent.
[Figure]
B. D ETAILS OF BEHAVIORAL CLONING
There are various implementation methods for behavioral cloning.
We used supervised learning as the simplest behavioral cloning method; speciﬁcally, recording the expert’s state and action sequences and using the state as input and the action as output.
More speciﬁcally, we trained a multilayer perceptron f ( x ; θ ) for tasks with a discrete action space using the softmax cross-entropy function shown in Eq.
(26) as the loss.
The number of inputs in the network was | S | , the number of outputs was equal to that of action options.
The other settings of the network were the same as those of IRL.
[Equation]
In the evaluation, the argmax policy in Eq.
(27) was used:
[Equation]
For tasks with a continuous action space, the multilayer perceptron was trained by using the squared loss shown in Eq.
The number of inputs in the network was | S | , and the number of outputs was | A | .
[Equation]
In the evaluation, the output of the network f ( s t ; θ ) was used.
Note that the output values were clipped in the deﬁned range of the action in the environment.
B. D ETAILS OF RED
To measure the RED scores, we used the author’s implementation https://github.com/
RuohanW/RED .
We used the same hyperparameters but changed the number of trajectories to 1000.