Original: A UTO -E NCODING
Revised: A UTO -E NCODING

Original: I NVERSE R EINFORCEMENT
Revised: I NVERSE R EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning (RL) provides a powerful framework for decisionmaking, but its application in practice often requires a carefully designed reward function.
Revised: Reinforcement learning (RL) provides a powerful framework for decisionmaking, but its application in practice often requires a carefully designed reward function.

Original: Inverse Reinforcement Learning (IRL) has shed light on automatic reward acquisition, but it is still difﬁcult to apply IRL to solve real-world tasks.
Revised: Inverse Reinforcement Learning (IRL) has shed light on automatic reward acquisition, but it is still difﬁcult to apply IRL to solve real-world tasks.

Original: In this work, we propose Auto-Encoding Inverse Reinforcement Learning (AEIRL), a robust and scalable IRL framework, which belongs to the adversarial imitation learning class.
Revised: In this work, we propose Auto-Encoding Inverse Reinforcement Learning (AEIRL), a robust and scalable IRL framework, which belongs to the adversarial imitation learning class.

Original: To recover reward functions from expert demonstrations, AEIRL utilizes the reconstruction error of an auto-encoder as the learning signal, which provides more information for optimizing policies, compared to the binary logistic loss.
Revised: To recover reward functions from expert demonstrations, AEIRL utilizes the reconstruction error of an auto-encoder as the learning signal, which provides more information for optimizing policies, compared to the binary logistic loss.

Original: Subsequently, we use the derived objective functions to train the reward function and the RL agent.
Revised: Subsequently, we use the derived objective functions to train the reward function and the RL agent.

Original: Experiments show that AEIRL performs superior in comparison with state-of-the-art methods in the MuJoCo environments.
Revised: Experiments show that AEIRL performs superior in comparison with state-of-the-art methods in the MuJoCo environments.

Original: More importantly, in more realistic settings, AEIRL shows much better robustness when the expert demonstrations are noisy.
Revised: More importantly, in more realistic settings, AEIRL shows much better robustness when the expert demonstrations are noisy.

Original: Speciﬁcally, our method achieves 16 .
Revised: Speciﬁcally, our method achieves 16 .

Original: 1% relative improvement compared to the best baseline FAIRL on clean expert data and 46 .
Revised: 1% relative improvement compared to the best baseline FAIRL on clean expert data and 46 .

Original: 5% relative improvement compared to the best baseline PWIL on noisy expert data both with the metric overall averaged scaled rewards.
Revised: 5% relative improvement compared to the best baseline PWIL on noisy expert data both with the metric overall averaged scaled rewards.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) provides a powerful framework for automating decision making.
Revised: Reinforcement learning (RL) provides a powerful framework for automating decision making.

Original: However, RL still requires signiﬁcantly engineered reward functions for good practical performance.
Revised: However, RL still requires signiﬁcantly engineered reward functions for good practical performance.

Original: To make RL more applicable in the real-world, it is important to learn a reward function from expert demonstrations.
Revised: To make RL more applicable in the real-world, it is important to learn a reward function from expert demonstrations.

Original: Imitation learning offers the instruments to learn policies directly from the data, without an explicit reward function.
Revised: Imitation learning offers the instruments to learn policies directly from the data, without an explicit reward function.

Original: Imitation learning enables the agents to learn to solve tasks from expert demonstrations, such as helicopter control (Abbeel et al., 2006; 2007; Ng et al., 2004;
Revised: Imitation learning enables the agents to learn to solve tasks from expert demonstrations, such as helicopter control (Abbeel et al., 2006; 2007; Ng et al., 2004;

Original: Coates et al., 2008; Abbeel et al., 2008a; 2010), robot navigation (Ratliff et al., 2006; Abbeel et al., 2008b; Ziebart et al., 2008; 2010), and building controls (Barrett & Linder, 2015).
Revised: Coates et al., 2008; Abbeel et al., 2008a; 2010), robot navigation (Ratliff et al., 2006; Abbeel et al., 2008b; Ziebart et al., 2008; 2010), and building controls (Barrett & Linder, 2015).

Original: The goal of imitation learning is to extract the expert policies from the expert demonstrations without the access to the reward signal from the environment.
Revised: The goal of imitation learning is to extract the expert policies from the expert demonstrations without the access to the reward signal from the environment.

Original: The algorithms in this ﬁeld can be divided into two broad categories: behavioral cloning (BC) and Inverse Reinforcement Learning (IRL).
Revised: The algorithms in this ﬁeld can be divided into two broad categories: behavioral cloning (BC) and Inverse Reinforcement Learning (IRL).

Original: BC formulates the learning task as a supervised learning problem, which learns a mapping from the states to the actions, using the expert trajectories.
Revised: BC formulates the learning task as a supervised learning problem, which learns a mapping from the states to the actions, using the expert trajectories.

Original: However, BC methods suffer from the problem of compounding errors, i.e., covariate shift, which only learn the actions of the expert but not reason about what the expert attempts to achieve.
Revised: However, BC methods suffer from the problem of compounding errors, i.e., covariate shift, which only learn the actions of the expert but not reason about what the expert attempts to achieve.

Original: In the contrast, IRL directly infers the underlying reward function from the data, and then teaches the RL agent to learn a policy to achieve the highest accumulated reward.
Revised: In the contrast, IRL directly infers the underlying reward function from the data, and then teaches the RL agent to learn a policy to achieve the highest accumulated reward.

Original: Empirical results show that IRL methods are more efﬁcient than BC methods in multi-step decision making tasks.
Revised: Empirical results show that IRL methods are more efﬁcient than BC methods in multi-step decision making tasks.

Original: However, how to make IRL stable and efﬁcient to use is still subject to research.
Revised: However, how to make IRL stable and efﬁcient to use is still subject to research.

Original: Current IRL methods, such as adversarial imitation learning approaches, model the reward function as a discriminator to learn the mapping from the state-action pair to a scalar value, i.e., the reward (Ho & Ermon, 2016; Fu et al., 2017; Ghasemipour et al., 2020).
Revised: Current IRL methods, such as adversarial imitation learning approaches, model the reward function as a discriminator to learn the mapping from the state-action pair to a scalar value, i.e., the reward (Ho & Ermon, 2016; Fu et al., 2017; Ghasemipour et al., 2020).

Original: However, this formulation may be easily over-conﬁdent to focus on the minor differences between the state-action features of the expert and generated samples.
Revised: However, this formulation may be easily over-conﬁdent to focus on the minor differences between the state-action features of the expert and generated samples.

Original: To the best of our knowledge, this is the ﬁrst instance of formulating the reward function as an auto-encoder, which has the ability to learn full scale differences between expert and alternative policies.
Revised: To the best of our knowledge, this is the ﬁrst instance of formulating the reward function as an auto-encoder, which has the ability to learn full scale differences between expert and alternative policies.

Original: The derived formulation ensures the reward signal is informative, which is efﬁcient for optimization.
Revised: The derived formulation ensures the reward signal is informative, which is efﬁcient for optimization.

Original: Additionally, the encoding-decoding process empowers the denoising capability and makes the agent robust to the noisy expert, in more realistic settings.
Revised: Additionally, the encoding-decoding process empowers the denoising capability and makes the agent robust to the noisy expert, in more realistic settings.

Original: In the experiments, we show that the proposed method AEIRL achieves the best overall performance on both clean and noisy expert demonstrations.
Revised: In the experiments, we show that the proposed method AEIRL achieves the best overall performance on both clean and noisy expert demonstrations.

Original: Our contributions are three-fold:
Revised: Our contributions are three-fold:

Original: • We propose the Auto-Encoding Inverse Reinforcement Learning (AEIRL) architecture, which models the reward function as a surrogate function using the reconstruction error of an auto-encoder.
Revised: • We propose the Auto-Encoding Inverse Reinforcement Learning (AEIRL) architecture, which models the reward function as a surrogate function using the reconstruction error of an auto-encoder.

Original: The reconstruction error provides more informative learning signals, compared to the binary logistic loss.
Revised: The reconstruction error provides more informative learning signals, compared to the binary logistic loss.

Original: • To show the contributing factors of our method, we conduct ablation studies based on different distribution divergences and alternative auto-encoders.
Revised: • To show the contributing factors of our method, we conduct ablation studies based on different distribution divergences and alternative auto-encoders.

Original: Experiments show that they achieve comparable results which indicates the encoding-decoding process is the major contributing factor for our method.
Revised: Experiments show that they achieve comparable results which indicates the encoding-decoding process is the major contributing factor for our method.

Original: • The experimental results on the MoJoCo tasks show that our method outperforms state-ofthe-art imitation learning methods on both clean and noisy expert demonstrations.
Revised: • The experimental results on the MoJoCo tasks show that our method outperforms state-ofthe-art imitation learning methods on both clean and noisy expert demonstrations.

Original: Empirical analysis show that our learned reward function can be more informative and robust.
Revised: Empirical analysis show that our learned reward function can be more informative and robust.

Original: Furthermore, the learning processes of our methods are also more stable in general.
Revised: Furthermore, the learning processes of our methods are also more stable in general.

Original:  R ELATED W ORK 2. I NVERSE R EINFORCEMENT L EARNING
Revised:  R ELATED W ORK 2. I NVERSE R EINFORCEMENT L EARNING

Original: Adversarial imitation learning such as GAIL, AIRL, DAC, f -GAIL, EAIRL, FAIRL (Ho & Ermon, 2016; Fu et al., 2017; Kostrikov et al., 2019; Zhang et al., 2020; Qureshi et al., 2019; Ghasemipour et al., 2020) formulates the learned reward function as a discriminator that learns to differentiate expert transitions from non-expert ones.
Revised: Adversarial imitation learning such as GAIL, AIRL, DAC, f -GAIL, EAIRL, FAIRL (Ho & Ermon, 2016; Fu et al., 2017; Kostrikov et al., 2019; Zhang et al., 2020; Qureshi et al., 2019; Ghasemipour et al., 2020) formulates the learned reward function as a discriminator that learns to differentiate expert transitions from non-expert ones.

Original: Among these methods, GAIL (Ho & Ermon, 2016) considers the Jensen-Shannon divergence, while AIRL (Fu et al., 2017) considers the Kullback-Leibler (KL) divergence.
Revised: Among these methods, GAIL (Ho & Ermon, 2016) considers the Jensen-Shannon divergence, while AIRL (Fu et al., 2017) considers the Kullback-Leibler (KL) divergence.

Original: DAC (Kostrikov et al., 2019) extends GAIL to the off-policy setting and signiﬁcantly improves the sample-efﬁciency of adversarial imitation learning.
Revised: DAC (Kostrikov et al., 2019) extends GAIL to the off-policy setting and signiﬁcantly improves the sample-efﬁciency of adversarial imitation learning.

Original: Furthermore, f -divergence is utilized in f -GAIL
Revised: Furthermore, f -divergence is utilized in f -GAIL

Original: (Zhang et al., 2020), which is considered more sample-efﬁcient.
Revised: (Zhang et al., 2020), which is considered more sample-efﬁcient.

Original: Recently,
Revised: Recently,

Original: FAIRL utilizes the forward KL divergence (Ghasemipour et al., 2020) and achieves better performance than AIRL (Fu et al., 2017) comprehensively, but it is still not robust enough.
Revised: FAIRL utilizes the forward KL divergence (Ghasemipour et al., 2020) and achieves better performance than AIRL (Fu et al., 2017) comprehensively, but it is still not robust enough.

Original: However, these methods rely heavily on a carefully tuned discriminator, which might easily overﬁt to the minor differences between the expert and the generated samples.
Revised: However, these methods rely heavily on a carefully tuned discriminator, which might easily overﬁt to the minor differences between the expert and the generated samples.

Original: In comparison, our auto-encoder based reward function helps to learn the full scale differences between the expert and generated samples, which provides more informative reward signals.
Revised: In comparison, our auto-encoder based reward function helps to learn the full scale differences between the expert and generated samples, which provides more informative reward signals.

Original: The robustness of adversarial imitation learning is also questionable with imperfection in observations (Stadie et al., 2017; Berseth & Pal, 2020), actions, transition models (Gangwani & Peng, 2020;
Revised: The robustness of adversarial imitation learning is also questionable with imperfection in observations (Stadie et al., 2017; Berseth & Pal, 2020), actions, transition models (Gangwani & Peng, 2020;

Original: Christiano et al., 2016), expert demonstrations (Brown et al., 2019; Shiarlis et al., 2016;
Revised: Christiano et al., 2016), expert demonstrations (Brown et al., 2019; Shiarlis et al., 2016;

Original: Jing et al., 2020) and their combinations (Kim et al., 2020).
Revised: Jing et al., 2020) and their combinations (Kim et al., 2020).

Original: Previous robust IRL methods require the demonstrations to be annotated with conﬁdence scores (Wu et al., 2019; Brown et al., 2019; Grollman & Billard, 2012), when the expert data is noisy.
Revised: Previous robust IRL methods require the demonstrations to be annotated with conﬁdence scores (Wu et al., 2019; Brown et al., 2019; Grollman & Billard, 2012), when the expert data is noisy.

Original: However, these annotations are rather expensive.
Revised: However, these annotations are rather expensive.

Original: Compared to this, our auto-encoder based reward function helps to denoise the expert data through the encoding-decoding process.
Revised: Compared to this, our auto-encoder based reward function helps to denoise the expert data through the encoding-decoding process.

Original: Our method AEIRL is relatively succinct and robust to noisy expert demonstrations and does not require any annotated data.
Revised: Our method AEIRL is relatively succinct and robust to noisy expert demonstrations and does not require any annotated data.

Original: Another category of IRL uses an ofﬂine similarity function to estimate the rewards (Boularias et al., 2011; Klein et al., 2013; Piot et al., 2016).
Revised: Another category of IRL uses an ofﬂine similarity function to estimate the rewards (Boularias et al., 2011; Klein et al., 2013; Piot et al., 2016).

Original: The idea of these methods is still inducing the expert policy by minimizing the distance between the state action distributions of the expert and sampled trajectories.
Revised: The idea of these methods is still inducing the expert policy by minimizing the distance between the state action distributions of the expert and sampled trajectories.

Original: To the best of our knowledge, the most powerful method in this category is Primal
Revised: To the best of our knowledge, the most powerful method in this category is Primal

Original: Wasserstein Imitation Learning (PWIL) (Dadashi et al., 2021), which utilizes the upper bound of its primal form as the optimization objective.
Revised: Wasserstein Imitation Learning (PWIL) (Dadashi et al., 2021), which utilizes the upper bound of its primal form as the optimization objective.

Original: The advantage of these methods is that they are relatively more robust compared to adversarial imitation learning methods, when the expert data is noisy.
Revised: The advantage of these methods is that they are relatively more robust compared to adversarial imitation learning methods, when the expert data is noisy.

Original: However, the performance of these methods heavily depends on the similarity measurement, and therefore, it varies greatly on different tasks.
Revised: However, the performance of these methods heavily depends on the similarity measurement, and therefore, it varies greatly on different tasks.

Original: Compared to PWIL, our method achieves superior performance.
Revised: Compared to PWIL, our method achieves superior performance.

Original: 2. A UTO -E NCODING BASED GAN S
Revised: 2. A UTO -E NCODING BASED GAN S

Original: Auto-encoders have been successfully applied to improve the training stability and modes capturing in GANs.
Revised: Auto-encoders have been successfully applied to improve the training stability and modes capturing in GANs.

Original: Auto-encoding based GANs can be classiﬁed into three categories: (1) utilizing an auto-encoder as the discriminator such as energy-based GANs (Zhao et al., 2016) and boundaryequilibrium GANs (Berthelot et al., 2017); (2) using a denoising auto-encoder to derive an auxiliary loss for the generator (Warde-Farley & Bengio, 2017); (3) combining variational auto-encoder and
Revised: Auto-encoding based GANs can be classiﬁed into three categories: (1) utilizing an auto-encoder as the discriminator such as energy-based GANs (Zhao et al., 2016) and boundaryequilibrium GANs (Berthelot et al., 2017); (2) using a denoising auto-encoder to derive an auxiliary loss for the generator (Warde-Farley & Bengio, 2017); (3) combining variational auto-encoder and

Original: GANs to generate both vivid and diverse samples by balancing the objective of reconstructing the training data and confusing the discriminator (Larsen et al., 2016).
Revised: GANs to generate both vivid and diverse samples by balancing the objective of reconstructing the training data and confusing the discriminator (Larsen et al., 2016).

Original: Our method AEIRL takes inspirations from EBGAN (Zhao et al., 2016) and utilizes an auto-encoder as the reward function and derives the efﬁcient reconstruction error based surrogate reward signal and its corresponding objective functions.
Revised: Our method AEIRL takes inspirations from EBGAN (Zhao et al., 2016) and utilizes an auto-encoder as the reward function and derives the efﬁcient reconstruction error based surrogate reward signal and its corresponding objective functions.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: A Markov decision process (MDP) is a tuple ( S, A, T, γ, P, r ) .
Revised: A Markov decision process (MDP) is a tuple ( S, A, T, γ, P, r ) .

Original: In this tuple, S is a state space; A is an action space; T is a probability matrix for state transitions; γ ∈ (0 , 1] is a discount factor; P is a initial-state transition distribution; and r : S ×
Revised: In this tuple, S is a state space; A is an action space; T is a probability matrix for state transitions; γ ∈ (0 , 1] is a discount factor; P is a initial-state transition distribution; and r : S ×

Original: A → R is a reward function.
Revised: A → R is a reward function.

Original: Additionally, we also deﬁne a stochastic policy π , which is a mapping from states to probability distributions over actions.
Revised: Additionally, we also deﬁne a stochastic policy π , which is a mapping from states to probability distributions over actions.

Original: IRL infers the reward function using the expert demonstrations, which are assumed to be the observations of optimal behaviors (Ng & Russell, 2000).
Revised: IRL infers the reward function using the expert demonstrations, which are assumed to be the observations of optimal behaviors (Ng & Russell, 2000).

Original: In general, IRL is formulated as a bi-level optimization process meaning that iteratively training the reward function and optimizing the policy.
Revised: In general, IRL is formulated as a bi-level optimization process meaning that iteratively training the reward function and optimizing the policy.

Original: Assume that we are given an expert policy π E , IRL (Ziebart et al., 2008; 2010) ﬁts a reward function from a family of functions R with the optimization problem:
Revised: Assume that we are given an expert policy π E , IRL (Ziebart et al., 2008; 2010) ﬁts a reward function from a family of functions R with the optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Moreover, the expert policy π E will only be provided by a set of expert demonstrations, so the expected reward of π E is estimated by these trajectories.
Revised: Moreover, the expert policy π E will only be provided by a set of expert demonstrations, so the expected reward of π E is estimated by these trajectories.

Original: IRL looks for a reward function r ( s, a ) that assigns high values to the expert policy and low values to other policies.
Revised: IRL looks for a reward function r ( s, a ) that assigns high values to the expert policy and low values to other policies.

Original: Therefore, it allows the expert policy to be found with a certain reinforcement learning procedure:
Revised: Therefore, it allows the expert policy to be found with a certain reinforcement learning procedure:

Original: [Equation]
Revised: [Equation]

Original: The RL process will induce the expert policy via maximizing the expected cumulative rewards.
Revised: The RL process will induce the expert policy via maximizing the expected cumulative rewards.

Original: Meanwhile the entropy term can be optionally added to the reinforcement learning objective to encourage the exploration in policy searching.
Revised: Meanwhile the entropy term can be optionally added to the reinforcement learning objective to encourage the exploration in policy searching.

Original: Typically, IRL models the reward function as a discriminator which can be easily overﬁt to the expert data.
Revised: Typically, IRL models the reward function as a discriminator which can be easily overﬁt to the expert data.

Original: Also its training stability is questionable once confronted with even a little noise in the expert data.
Revised: Also its training stability is questionable once confronted with even a little noise in the expert data.

Original: Therefore, we propose auto-encoding IRL to utilize an auto-encoder based reward function which achieve strong performance both on clean and noisy expert demonstrations.
Revised: Therefore, we propose auto-encoding IRL to utilize an auto-encoder based reward function which achieve strong performance both on clean and noisy expert demonstrations.

Original:  A UTO -E NCODING I NVERSE R EINFORCEMENT L EARNING 4. O VERVIEW
Revised:  A UTO -E NCODING I NVERSE R EINFORCEMENT L EARNING 4. O VERVIEW

Original: The reward function in GAIL is a discriminator, which attempts to assign high values to the regions near the expert demonstrations and assign low values to the other regions.
Revised: The reward function in GAIL is a discriminator, which attempts to assign high values to the regions near the expert demonstrations and assign low values to the other regions.

Original: However, this form of reward function could be easily overﬁtting to the expert data.
Revised: However, this form of reward function could be easily overﬁtting to the expert data.

Original: Consider the CartPole balancing task, the state of the environment is a feature consisting of position, angle, angle’s rate, and cart velocity, while the action is moving left or right.
Revised: Consider the CartPole balancing task, the state of the environment is a feature consisting of position, angle, angle’s rate, and cart velocity, while the action is moving left or right.

Original: Here, we assume that all the expert states’ velocity is 2 for example.
Revised: Here, we assume that all the expert states’ velocity is 2 for example.

Original: When the generated states’ velocity is 1 . 9 and other dimensions of state action pairs are
Revised: When the generated states’ velocity is 1 . 9 and other dimensions of state action pairs are

Original: [Figure]
Revised: [Figure]

Original: the same as the expert’s, the discriminator of GAIL would still give a low reward on these generated state action pairs.
Revised: the same as the expert’s, the discriminator of GAIL would still give a low reward on these generated state action pairs.

Original: However it may actually perform very well on the goal of mimicking the expert’s behaviors.
Revised: However it may actually perform very well on the goal of mimicking the expert’s behaviors.

Original: In other words, the reward function in GAIL on this example could easily overﬁt to the minor differences between the expert and the sampled data, while missing the underlying goal of the expert.
Revised: In other words, the reward function in GAIL on this example could easily overﬁt to the minor differences between the expert and the sampled data, while missing the underlying goal of the expert.

Original: In our paper, we propose an auto-encoder based reward function for inverse reinforcement learning.
Revised: In our paper, we propose an auto-encoder based reward function for inverse reinforcement learning.

Original: It utilizes the reconstruction error of the state action pairs to yield an informative reward signal.
Revised: It utilizes the reconstruction error of the state action pairs to yield an informative reward signal.

Original: The reconstruction error based reward signal signiﬁcantly retains the information of state action pairs, rather than focusing on the minor differences.
Revised: The reconstruction error based reward signal signiﬁcantly retains the information of state action pairs, rather than focusing on the minor differences.

Original: Such a reward signal wouldn’t lead to overconﬁdence in distinguishing the expert and the generated samples.
Revised: Such a reward signal wouldn’t lead to overconﬁdence in distinguishing the expert and the generated samples.

Original: Recall the CartPole balancing example, the mean square error between states’ velocity 1 . 9 and 2 is very small.
Revised: Recall the CartPole balancing example, the mean square error between states’ velocity 1 . 9 and 2 is very small.

Original: And it could still feed a good reward to the agent under this situation.
Revised: And it could still feed a good reward to the agent under this situation.

Original: Thus, the reconstruction error based reward signal focuses on the full scale differences between the expert and generated state action pairs rather than the minor parts.
Revised: Thus, the reconstruction error based reward signal focuses on the full scale differences between the expert and generated state action pairs rather than the minor parts.

Original: This yields a much more informative reward signal.
Revised: This yields a much more informative reward signal.

Original: Figure 5 (See in Section 5.4), shows a more informative reward signal recovered with our method.
Revised: Figure 5 (See in Section 5.4), shows a more informative reward signal recovered with our method.

Original: Furthermore, the training process of the policy becomes smoother in general, which is shown in Figure 11 (See in Appendix A.6).
Revised: Furthermore, the training process of the policy becomes smoother in general, which is shown in Figure 11 (See in Appendix A.6).

Original: N(cid:82)i(cid:86)(cid:92)
Revised: N(cid:82)i(cid:86)(cid:92)

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: The expert demonstrations usually contains noise bacause we sample these human trajectories with sensors and other devices in the real world.
Revised: The expert demonstrations usually contains noise bacause we sample these human trajectories with sensors and other devices in the real world.

Original: Therefore, adversarial imitation learning, such as GAIL, might be easily affected to learn the noisy expert behaviors, which are not the real intentions of the expert.
Revised: Therefore, adversarial imitation learning, such as GAIL, might be easily affected to learn the noisy expert behaviors, which are not the real intentions of the expert.

Original: The reward function under this situation could overﬁt to the noisy features.
Revised: The reward function under this situation could overﬁt to the noisy features.

Original: Recall the CartPole balancing example, we assume that the expert states’ velocity is 2 but it tends to be 

Revised: Recall the CartPole balancing example, we assume that the expert states’ velocity is 2 but it tends to be

Original: + δ due to the noisy sampling process.
Revised: + δ due to the noisy sampling process.

Original: When the learned policy is good enough as the sampled states’ velocity, the discriminator in GAIL still takes it as a bad policy, and therefore, it loses its efﬁcacy, when learning from noisy demonstrations.
Revised: When the learned policy is good enough as the sampled states’ velocity, the discriminator in GAIL still takes it as a bad policy, and therefore, it loses its efﬁcacy, when learning from noisy demonstrations.

Original: Figure 2 depicts the denoising process of an autoencoder.
Revised: Figure 2 depicts the denoising process of an autoencoder.

Original: When the auto-encoder is trained to minimize the averaged squared errors, the vector points approximately towards the nearest points on the manifold, since the auto-encoder estimates the center of mass of the clean points (Goodfellow et al., 2016).
Revised: When the auto-encoder is trained to minimize the averaged squared errors, the vector points approximately towards the nearest points on the manifold, since the auto-encoder estimates the center of mass of the clean points (Goodfellow et al., 2016).

Original: Therefore, the reconstruction error of the auto-encoder can eliminate some effects when learning from noisy expert demonstrations and make the reward signal more robust.
Revised: Therefore, the reconstruction error of the auto-encoder can eliminate some effects when learning from noisy expert demonstrations and make the reward signal more robust.

Original: Recall the CartPole example, the reconstruction error of the auto-encoder is δ 2 which doesn’t play an important role in the rewards for the agent.
Revised: Recall the CartPole example, the reconstruction error of the auto-encoder is δ 2 which doesn’t play an important role in the rewards for the agent.

Original: 4. M ETHOD
Revised: 4. M ETHOD

Original: Our approach is to minimize the distance between the state action distribution of the policy π θ and the expert demonstrations D E .
Revised: Our approach is to minimize the distance between the state action distribution of the policy π θ and the expert demonstrations D E .

Original: The distance we used in our method is the Wasserstein distance (Arjovsky et al., 2017):
Revised: The distance we used in our method is the Wasserstein distance (Arjovsky et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where the reward function network’s parameters are denoted as w and the policy network’s parameters are represented as θ .
Revised: where the reward function network’s parameters are denoted as w and the policy network’s parameters are represented as θ .

Original: We choose the metric functions in Wasserstein distance as a class of reward functions from neural network functions for computational convenience.
Revised: We choose the metric functions in Wasserstein distance as a class of reward functions from neural network functions for computational convenience.

Original: Minimizing this distance is actually inducing the expert policy from expert demonstrations.
Revised: Minimizing this distance is actually inducing the expert policy from expert demonstrations.

Original: Therefore, the optimization of the policy π θ and recovering period of the reward function r w ( s, a ) forms a bi-level optimization problem, which can be formally deﬁned as:
Revised: Therefore, the optimization of the policy π θ and recovering period of the reward function r w ( s, a ) forms a bi-level optimization problem, which can be formally deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: This leads to an adversarial formulation for inverse RL.
Revised: This leads to an adversarial formulation for inverse RL.

Original: The outer level minimization with respect to the policy leads to a learned policy which is close to the expert.
Revised: The outer level minimization with respect to the policy leads to a learned policy which is close to the expert.

Original: The inner level maximization recovers a reward function which attributes higher values to regions close to the expert data, and penalizes all other regions.
Revised: The inner level maximization recovers a reward function which attributes higher values to regions close to the expert data, and penalizes all other regions.

Original: In traditional adversarial imitation learning methods, such as WGAIL (Arjovsky et al., 2017;
Revised: In traditional adversarial imitation learning methods, such as WGAIL (Arjovsky et al., 2017;

Original: Ho &
Revised: Ho &

Original: Ermon, 2016) (Details See in Appendix A.2.3), the reward function is deﬁned as
Revised: Ermon, 2016) (Details See in Appendix A.2.3), the reward function is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where D w ( s, a ) is the output of the discriminator.
Revised: where D w ( s, a ) is the output of the discriminator.

Original: In our method, we use an auto-encoder based surrogate reward function, which is deﬁned as:
Revised: In our method, we use an auto-encoder based surrogate reward function, which is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where AE is the reconstruction error of an auto-encoder:
Revised: where AE is the reconstruction error of an auto-encoder:

Original: [Equation]
Revised: [Equation]

Original: Here, x represents the state-action pairs.
Revised: Here, x represents the state-action pairs.

Original: It is a mean square error between the sampled state action pairs and the reconstructed samples.
Revised: It is a mean square error between the sampled state action pairs and the reconstructed samples.

Original: This form of the reward signal uses the reconstruction error of an auto-encoder to score the state action pairs in trajectories.
Revised: This form of the reward signal uses the reconstruction error of an auto-encoder to score the state action pairs in trajectories.

Original: It is a monotonically decreasing function over the reconstruction error of the auto-encoder.
Revised: It is a monotonically decreasing function over the reconstruction error of the auto-encoder.

Original: Low reconstruction errors ensure high reward values for state action pairs and vice versa.
Revised: Low reconstruction errors ensure high reward values for state action pairs and vice versa.

Original: Section 4.1 tells that this form of reward signal focuses more on the full scale differences between the expert and generated samples and can help to denoise the expert demonstrations.
Revised: Section 4.1 tells that this form of reward signal focuses more on the full scale differences between the expert and generated samples and can help to denoise the expert demonstrations.

Original: Training the auto-encoder is an adversarial process considering the objective 4, which is minimizing the reconstruction error for the expert samples and meanwhile maximizing this error for generated samples.
Revised: Training the auto-encoder is an adversarial process considering the objective 4, which is minimizing the reconstruction error for the expert samples and meanwhile maximizing this error for generated samples.

Original: So the auto-encoder based reward function training objective is to minimize:
Revised: So the auto-encoder based reward function training objective is to minimize:

Original: [Equation]
Revised: [Equation]

Original: The auto-encoder learns to maximize the full scale differences between the expert and the generated samples with the adversarial objective.
Revised: The auto-encoder learns to maximize the full scale differences between the expert and the generated samples with the adversarial objective.

Original: It leads to a better feedback signal to the agent.
Revised: It leads to a better feedback signal to the agent.

Original: Furthermore, the auto-encoder also retains more information and helps to denoise the expert data via the encoding-decoding process.
Revised: Furthermore, the auto-encoder also retains more information and helps to denoise the expert data via the encoding-decoding process.

Original: This can further help us to match the distribution between the expert and generated behaviors in a latent space.
Revised: This can further help us to match the distribution between the expert and generated behaviors in a latent space.

Original: The variational auto-encoder (Kingma & Welling, 2014) is also an alternative form of reward function.
Revised: The variational auto-encoder (Kingma & Welling, 2014) is also an alternative form of reward function.

Original: When we use an variational auto-encoder based reward function, the objective function for training the variational auto-encoder in our method is:
Revised: When we use an variational auto-encoder based reward function, the objective function for training the variational auto-encoder in our method is:

Original: [Equation]
Revised: [Equation]

Original: where p model ( z )) is a ﬁxed Gaussian distribution.
Revised: where p model ( z )) is a ﬁxed Gaussian distribution.

Original: When minimizing this objective, the latent distribution for the expert samples is widely distributed to induce more diverse expert policies.
Revised: When minimizing this objective, the latent distribution for the expert samples is widely distributed to induce more diverse expert policies.

Original: Figure 1 depicts the architecture for AEIRL.
Revised: Figure 1 depicts the architecture for AEIRL.

Original: The auto-encoder based reward function takes either expert or the generated state action pairs, and estimates the reconstruction error based rewards accordingly.
Revised: The auto-encoder based reward function takes either expert or the generated state action pairs, and estimates the reconstruction error based rewards accordingly.

Original: The auto-encoder and the policy is iteratively optimized under this adversarial training paradigm.
Revised: The auto-encoder and the policy is iteratively optimized under this adversarial training paradigm.

Original: Algorithm 1 in Appendix A.1 depicts the pseudo code for training AEIRL.
Revised: Algorithm 1 in Appendix A.1 depicts the pseudo code for training AEIRL.

Original: Not limited to the utilized Wasserstein distance, other distribution divergences are also applicable to form the reward functions without loss of generality.
Revised: Not limited to the utilized Wasserstein distance, other distribution divergences are also applicable to form the reward functions without loss of generality.

Original: Appendix A.5 shows other two forms of objectives and reward functions following Jensen-Shannon divergence.
Revised: Appendix A.5 shows other two forms of objectives and reward functions following Jensen-Shannon divergence.

Original: Experiments show that different forms of objectives achieve comparable results on almost all locomotion tasks.
Revised: Experiments show that different forms of objectives achieve comparable results on almost all locomotion tasks.

Original: It indicates that an auto-encoder based reward function is the major contributing factor to improve the performance on mimicking the expert rather than the different distribution divergences and the corresponding forms of surrogate reward function.
Revised: It indicates that an auto-encoder based reward function is the major contributing factor to improve the performance on mimicking the expert rather than the different distribution divergences and the corresponding forms of surrogate reward function.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct the experiments on six locomotion tasks with varying dynamics and difﬁculty:
Revised: We conduct the experiments on six locomotion tasks with varying dynamics and difﬁculty:

Original: Swimmer-v2 (Schulman et al., 2015), Hopper-v2 (Levine & Koltun, 2013), Walker2d-v2 (Schulman et al., 2015), HalfCheetah-v2 (Heess et al., 2015), Ant-v2 (Schulman et al., 2016), and Humanoid-v (Tassa et al., 2012).
Revised: Swimmer-v2 (Schulman et al., 2015), Hopper-v2 (Levine & Koltun, 2013), Walker2d-v2 (Schulman et al., 2015), HalfCheetah-v2 (Heess et al., 2015), Ant-v2 (Schulman et al., 2016), and Humanoid-v (Tassa et al., 2012).

Original: The goal for all these tasks is to move forward as quickly as possible.
Revised: The goal for all these tasks is to move forward as quickly as possible.

Original: The baselines we choose for comparison includes: behavior cloning (BC) (Dhariwal et al., 2017), GAIL (Ho & Ermon, 2016), WGAIL (Arjovsky et al., 2017; Ho & Ermon, 2016), FAIRL (Ghasemipour et al., 2020), and PWIL (Dadashi et al., 2021).
Revised: The baselines we choose for comparison includes: behavior cloning (BC) (Dhariwal et al., 2017), GAIL (Ho & Ermon, 2016), WGAIL (Arjovsky et al., 2017; Ho & Ermon, 2016), FAIRL (Ghasemipour et al., 2020), and PWIL (Dadashi et al., 2021).

Original: For a fair comparison, we use TRPO (Schulman et al., 2015) as the policy search method for all the algorithms, which is implemented by
Revised: For a fair comparison, we use TRPO (Schulman et al., 2015) as the policy search method for all the algorithms, which is implemented by

Original: OpenAI Baselines (Dhariwal et al., 2017).
Revised: OpenAI Baselines (Dhariwal et al., 2017).

Original: Experimental details, including the implementation and more results, are shown in Appendix A.2 to A.7.
Revised: Experimental details, including the implementation and more results, are shown in Appendix A.2 to A.7.

Original: 5. L EARNING FROM E XPERT D EMONSTRATIONS
Revised: 5. L EARNING FROM E XPERT D EMONSTRATIONS

Original: [Table]
Revised: [Table]

Original: To evaluate the performance of our proposed methods and other baselines, we run the experiments on the clean expert demonstrations.
Revised: To evaluate the performance of our proposed methods and other baselines, we run the experiments on the clean expert demonstrations.

Original: The metric we used is the ground truth rewards for different learned policies.
Revised: The metric we used is the ground truth rewards for different learned policies.

Original: Higher rewards indicate better mimicking expert behaviors.
Revised: Higher rewards indicate better mimicking expert behaviors.

Original: Figure 3 depicts the training curves of ground truth rewards for different algorithms.
Revised: Figure 3 depicts the training curves of ground truth rewards for different algorithms.

Original: The training curve shows the imitation performance learning from non-noisy expert data.
Revised: The training curve shows the imitation performance learning from non-noisy expert data.

Original: Table 1 shows the ﬁnal learned policy performance on MuJoCo tasks with ground truth episode return.
Revised: Table 1 shows the ﬁnal learned policy performance on MuJoCo tasks with ground truth episode return.

Original: GAIL, WGAIL and FAIRL have comparable results, while WGAIL and FAIRL are little better than
Revised: GAIL, WGAIL and FAIRL have comparable results, while WGAIL and FAIRL are little better than

Original: GAIL comprehensively on these tasks but still not robust enough.
Revised: GAIL comprehensively on these tasks but still not robust enough.

Original: PWIL achieves higher performance compared with GAIL, WGAIL and FAIRL on Walker2d, Hopper and Humanoid tasks but performs worse than BC on Swimmer.
Revised: PWIL achieves higher performance compared with GAIL, WGAIL and FAIRL on Walker2d, Hopper and Humanoid tasks but performs worse than BC on Swimmer.

Original: Thus, the performance of PWIL varies greatly on different tasks.
Revised: Thus, the performance of PWIL varies greatly on different tasks.

Original: [Table]
Revised: [Table]

Original: The overall averaged scaled rewards for our method is about 0 .
Revised: The overall averaged scaled rewards for our method is about 0 .

Original: 907 where the best baseline is 0 .for
Revised: 907 where the best baseline is 0 .for

Original: FAIRL.
Revised: FAIRL.

Original: It is a more than 16 .
Revised: It is a more than 16 .

Original: 1% relative improvement.
Revised: 1% relative improvement.

Original: Our method outperforms state-of-the-art baselines on all locomotion tasks except for Hopper where our method doesn’t outperform PWIL.
Revised: Our method outperforms state-of-the-art baselines on all locomotion tasks except for Hopper where our method doesn’t outperform PWIL.

Original: Moreover, the variational auto-encoder variant of our method gets the best scores on Walker2d and Hopper.
Revised: Moreover, the variational auto-encoder variant of our method gets the best scores on Walker2d and Hopper.

Original: In summary, our methods, including vanilla auto-encoder based and variational autoencoder based variants, get better performance than other baselines comprehensively.
Revised: In summary, our methods, including vanilla auto-encoder based and variational autoencoder based variants, get better performance than other baselines comprehensively.

Original: 5. L EARNING FROM N OISY D EMONSTRATIONS
Revised: 5. L EARNING FROM N OISY D EMONSTRATIONS

Original: To show the robustness of our proposed AEIRL, we run our method and baselines on noisy expert demonstrations.
Revised: To show the robustness of our proposed AEIRL, we run our method and baselines on noisy expert demonstrations.

Original: The noise added to the expert data is a Gaussian noise (0 , 0 . 3) on every dimension of state action pairs on all tasks except for Ant and Humanoid.
Revised: The noise added to the expert data is a Gaussian noise (0 , 0 . 3) on every dimension of state action pairs on all tasks except for Ant and Humanoid.

Original: Since the dimension of state action pairs in Ant and Humanoid is much higher than other tasks, we choose (0 , 0 . 03) and (0 , 0 . 01)
Revised: Since the dimension of state action pairs in Ant and Humanoid is much higher than other tasks, we choose (0 , 0 . 03) and (0 , 0 . 01)

Original: Gaussian noises added to the dataset respectively in order to lower the effects of the noise component.
Revised: Gaussian noises added to the dataset respectively in order to lower the effects of the noise component.

Original: Figure 7 (See in Appendix A.3) depicts training curves for the ground truth rewards for different methods on noisy expert data while Table 2 shows the ﬁnal learned policy performance on MuJoCo tasks with ground truth episode return.
Revised: Figure 7 (See in Appendix A.3) depicts training curves for the ground truth rewards for different methods on noisy expert data while Table 2 shows the ﬁnal learned policy performance on MuJoCo tasks with ground truth episode return.

Original: These results show that our method outperforms other state-of-the-art algorithms on all tasks except for Hopper and Swimmer, on which PWIL and GAIL wins respectively.
Revised: These results show that our method outperforms other state-of-the-art algorithms on all tasks except for Hopper and Swimmer, on which PWIL and GAIL wins respectively.

Original: Table 1 and 2 show that the noise component affects the performance of mimicking the expert behaviors.
Revised: Table 1 and 2 show that the noise component affects the performance of mimicking the expert behaviors.

Original: But our method could help to denoise the expert demonstrations and provide more robust reward signals for inducing the expert policies.
Revised: But our method could help to denoise the expert demonstrations and provide more robust reward signals for inducing the expert policies.

Original: The overall scaled rewards for our AEIRL is 0 .
Revised: The overall scaled rewards for our AEIRL is 0 .

Original: 795 where the best baseline is 0 .for
Revised: 795 where the best baseline is 0 .for

Original: PWIL.
Revised: PWIL.

Original: Additionally, PWIL also has some capabilities to denoise the expert on part of the tasks.
Revised: Additionally, PWIL also has some capabilities to denoise the expert on part of the tasks.

Original: And other adversarial imitation learning methods, includes GAIL, WGAIL and FAIRL, are very sensitive to the noisy expert.
Revised: And other adversarial imitation learning methods, includes GAIL, WGAIL and FAIRL, are very sensitive to the noisy expert.

Original: 5. A NALYSIS ON R OBUSTNESS OF O UR M ETHOD
Revised: 5. A NALYSIS ON R OBUSTNESS OF O UR M ETHOD

Original: To compare the robustness of different reward signals of different methods, we use t-SNE visualization technique to visualize the latent space of the reward function network.
Revised: To compare the robustness of different reward signals of different methods, we use t-SNE visualization technique to visualize the latent space of the reward function network.

Original: For the discriminator based methods, we visualize the output of ﬁrst hidden (middle) layer for instance.
Revised: For the discriminator based methods, we visualize the output of ﬁrst hidden (middle) layer for instance.

Original: So the latent representations for both discriminator based and our auto-encoder based methods will show its robustness, especially to the corrupted expert data.
Revised: So the latent representations for both discriminator based and our auto-encoder based methods will show its robustness, especially to the corrupted expert data.

Original: Since PWIL (Dadashi et al., 2021) considers an ofﬂine reward function without neural networks, we only compare our methods with other three IRL baselines.
Revised: Since PWIL (Dadashi et al., 2021) considers an ofﬂine reward function without neural networks, we only compare our methods with other three IRL baselines.

Original: Figure 4 depicts that two sets of samples are more indistinguishable than other baselines both on clean and noisy expert data.
Revised: Figure 4 depicts that two sets of samples are more indistinguishable than other baselines both on clean and noisy expert data.

Original: It shows better robustness of our auto-encoder based reward function.
Revised: It shows better robustness of our auto-encoder based reward function.

Original: More details for using t-SNE and results with varying preplexity are shown in Appendix A.
Revised: More details for using t-SNE and results with varying preplexity are shown in Appendix A.

Original: A BLATION
Revised: A BLATION

Original: S TUDIES
Revised: S TUDIES

Original: Different Objectives To analyze the major contributing factors of our method, we conduct the ablation studies based on different distribution divergences and different formulation of reward functions.
Revised: Different Objectives To analyze the major contributing factors of our method, we conduct the ablation studies based on different distribution divergences and different formulation of reward functions.

Original: Comparable performances would indicate that the major contributing factor is that we utilize the reconstruction error of an auto-encoder rather than the formulation of reward function and its corresponding objective.
Revised: Comparable performances would indicate that the major contributing factor is that we utilize the reconstruction error of an auto-encoder rather than the formulation of reward function and its corresponding objective.

Original: We formulate some other forms of surrogate reward functions based on
Revised: We formulate some other forms of surrogate reward functions based on

Original: Jensen-Shannon divergence (Goodfellow et al., 2014) to justify this hypothesis.
Revised: Jensen-Shannon divergence (Goodfellow et al., 2014) to justify this hypothesis.

Original: Figure 9 (See in
Revised: Figure 9 (See in

Original: Appendix A.5) shows that three different formulations of surrogate reward signal achieve comparable results on almost all tasks which are all based on the vanilla auto-encoder.
Revised: Appendix A.5) shows that three different formulations of surrogate reward signal achieve comparable results on almost all tasks which are all based on the vanilla auto-encoder.

Original: This shows the major contributing factor in our method is the auto-encoder based reward function which provides more informative learning signals, compared to the discriminator based reward function.
Revised: This shows the major contributing factor in our method is the auto-encoder based reward function which provides more informative learning signals, compared to the discriminator based reward function.

Original: Alternative Auto-Encoders Our method is not limited to vanilla auto-encoders and we also take the variational auto-encoder as an alternative.
Revised: Alternative Auto-Encoders Our method is not limited to vanilla auto-encoders and we also take the variational auto-encoder as an alternative.

Original: We run variational auto-encoder based variant on MuJoCo tasks under the same setting with vanilla auto-encoder.
Revised: We run variational auto-encoder based variant on MuJoCo tasks under the same setting with vanilla auto-encoder.

Original: Figure 10 (See in Appendix A.5) shows the training curves for variational auto-encoder based and vanilla auto-encoder based variants.
Revised: Figure 10 (See in Appendix A.5) shows the training curves for variational auto-encoder based and vanilla auto-encoder based variants.

Original: Overall, the scaled rewards of variational auto-encoder based variant is 0 . 845 and 0 .
Revised: Overall, the scaled rewards of variational auto-encoder based variant is 0 . 845 and 0 .

Original: 727 on clean and noisy expert respectively, which is comparable to the vanilla auto-encoder based method.
Revised: 727 on clean and noisy expert respectively, which is comparable to the vanilla auto-encoder based method.

Original: Auto-Encoder based Reward Signal is Denser A denser reward signal is beneﬁcial for policy searching (Ng et al., 1999;
Revised: Auto-Encoder based Reward Signal is Denser A denser reward signal is beneﬁcial for policy searching (Ng et al., 1999;

Original: Hu et al., 2020).
Revised: Hu et al., 2020).

Original: Section 4.1 tells that our auto-encoder based reward function focuses on the full scale differences between the expert and generated samples.
Revised: Section 4.1 tells that our auto-encoder based reward function focuses on the full scale differences between the expert and generated samples.

Original: Thus, it provides more information to the agent compared to the discriminator based reward function.
Revised: Thus, it provides more information to the agent compared to the discriminator based reward function.

Original: To justify this hypothesis, we want to illustrate the generated rewards for different trajectories in the whole trajectory space.
Revised: To justify this hypothesis, we want to illustrate the generated rewards for different trajectories in the whole trajectory space.

Original: Meanwhile more distinct rewards for different trajectories means the reward signal is denser and more informative.
Revised: Meanwhile more distinct rewards for different trajectories means the reward signal is denser and more informative.

Original: So we use the ﬁnal learned reward function of our method and the baselines to score different trajectories for analysis.
Revised: So we use the ﬁnal learned reward function of our method and the baselines to score different trajectories for analysis.

Original: We collect 50 levels of trajectories with different performances from random to the expert performance.
Revised: We collect 50 levels of trajectories with different performances from random to the expert performance.

Original: This can be realized by saving trajectories every 100 iterations via a PPO.
Revised: This can be realized by saving trajectories every 100 iterations via a PPO.

Original: Since the trajectories with different performances have different lengths, we normalize the generated rewards with the length of trajectories.
Revised: Since the trajectories with different performances have different lengths, we normalize the generated rewards with the length of trajectories.

Original: We also scale the generated rewards considering the expert samples are 1 and random policy samples are 0 .
Revised: We also scale the generated rewards considering the expert samples are 1 and random policy samples are 0 .

Original: So the scaled normalized rewards for different levels of trajectories reﬂect the denseness of a reward signal.
Revised: So the scaled normalized rewards for different levels of trajectories reﬂect the denseness of a reward signal.

Original: And higher degree of distinction indicates it is a denser reward signal.
Revised: And higher degree of distinction indicates it is a denser reward signal.

Original: Figure 5 shows that the ﬁnal learned scaled and normalized rewards for different levels of the trajectories.
Revised: Figure 5 shows that the ﬁnal learned scaled and normalized rewards for different levels of the trajectories.

Original: The higher level in the abscissa means higher performance of its trajectories.
Revised: The higher level in the abscissa means higher performance of its trajectories.

Original: Note that learned rewards under our formulation are readily distinct at different levels when compared to learned rewards under WGAIL.
Revised: Note that learned rewards under our formulation are readily distinct at different levels when compared to learned rewards under WGAIL.

Original: Our much steeper curves indicates that the auto-encoder derived reward function provides a denser and more informative learning signal when compared to the discriminator based reward functions, which are all based on Wasserstein distance (Arjovsky et al., 2017).
Revised: Our much steeper curves indicates that the auto-encoder derived reward function provides a denser and more informative learning signal when compared to the discriminator based reward functions, which are all based on Wasserstein distance (Arjovsky et al., 2017).

Original: Stability of Our Method The variance of gradients is computed via different mini-batches of training examples.
Revised: Stability of Our Method The variance of gradients is computed via different mini-batches of training examples.

Original: Lower variance of gradients means smoother updating of the network (Faghri et al., 2020).
Revised: Lower variance of gradients means smoother updating of the network (Faghri et al., 2020).

Original: Now that our auto-encoder based reward function is more informative and denser than the discriminator based ones, how about the training stability of our method?
Revised: Now that our auto-encoder based reward function is more informative and denser than the discriminator based ones, how about the training stability of our method?

Original: The relative variance of policy gradients to the weights of policy network shows the training stability for different methods.
Revised: The relative variance of policy gradients to the weights of policy network shows the training stability for different methods.

Original: Figure 11 (See in Appendix A.6) shows that our method gets lower relative variance of gradients which means that the policy network training process can be much more stable.
Revised: Figure 11 (See in Appendix A.6) shows that our method gets lower relative variance of gradients which means that the policy network training process can be much more stable.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This paper presents a succinct and robust adversarial imitation learning method based on AutoEncoding (AEIRL).
Revised: This paper presents a succinct and robust adversarial imitation learning method based on AutoEncoding (AEIRL).

Original: We utilize the reconstruction error of an auto-encoder as the surrogate reward function for reinforcement learning.
Revised: We utilize the reconstruction error of an auto-encoder as the surrogate reward function for reinforcement learning.

Original: The advantages of our method can be summarized into two aspects: (1) the auto-encoder based reward function focuses on the full scale differences between the expert and generated samples, which won’t easily overﬁt to the expert data; (2) the auto-encoder denoises the expert data through the reconstruction process.
Revised: The advantages of our method can be summarized into two aspects: (1) the auto-encoder based reward function focuses on the full scale differences between the expert and generated samples, which won’t easily overﬁt to the expert data; (2) the auto-encoder denoises the expert data through the reconstruction process.

Original: Thus, the learned reward signal in our method is more robust and informative compared with the existing methods.
Revised: Thus, the learned reward signal in our method is more robust and informative compared with the existing methods.

Original: Experimental results show that our methods achieve strong performances on locomotion tasks on both clean and noisy expert demonstrations.
Revised: Experimental results show that our methods achieve strong performances on locomotion tasks on both clean and noisy expert demonstrations.

Original: In the future, we want to further investigate our method in more realistic scenarios, such as autonomous driving.
Revised: In the future, we want to further investigate our method in more realistic scenarios, such as autonomous driving.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Data and source code are attached in supplementary materials.
Revised: Data and source code are attached in supplementary materials.

Original: Experimental implementation details, including our methods and baselines, see in Appendix A.2.
Revised: Experimental implementation details, including our methods and baselines, see in Appendix A.2.

Original: A. I MPLEMENTATION
Revised: A. I MPLEMENTATION

Original: [Figure]
Revised: [Figure]

Original:  (e) ( [Figure]
Revised: ) (d) (e) (f)

Original: The goal for all these tasks is to move forward as quickly as possible.
Revised: The goal for all these tasks is to move forward as quickly as possible.

Original: These tasks are more challenging than the basic tasks due to high degrees of freedom.
Revised: These tasks are more challenging than the basic tasks due to high degrees of freedom.

Original: In addition, a great amount of exploration is needed to learn to move forward without getting stuck at local optima.
Revised: In addition, a great amount of exploration is needed to learn to move forward without getting stuck at local optima.

Original: Since we penalize for excessive controls as well as falling over, during the initial stage of learning, when the robot is not yet able to move forward for a sufﬁcient distance without falling, apparent local optima exist including staying at the origin or diving forward slowly.
Revised: Since we penalize for excessive controls as well as falling over, during the initial stage of learning, when the robot is not yet able to move forward for a sufﬁcient distance without falling, apparent local optima exist including staying at the origin or diving forward slowly.

Original: Figure 6 depicts locomotion tasks’ environments.
Revised: Figure 6 depicts locomotion tasks’ environments.

Original: A.2. B ASIC S ETTINGS
Revised: A.2. B ASIC S ETTINGS

Original: The expert trajectories are sampled with PPO on Walker2d, Hopper, and Swimmer, HalfCheetah and Ant with 25 trajectories.
Revised: The expert trajectories are sampled with PPO on Walker2d, Hopper, and Swimmer, HalfCheetah and Ant with 25 trajectories.

Original: And we sample 320 trajectories on Humanoid with SAC to get higher expert performance since PPO is not good enough on Humanoid.
Revised: And we sample 320 trajectories on Humanoid with SAC to get higher expert performance since PPO is not good enough on Humanoid.

Original: For each task, we normalize the state features and use the normalized features as the input for the auto-encoder.
Revised: For each task, we normalize the state features and use the normalized features as the input for the auto-encoder.

Original: But we use the raw data input to compute the mean square error.
Revised: But we use the raw data input to compute the mean square error.

Original: On Walker2d, Hopper, and Swimmer, we train these algorithms directly, while on HalfCheetah, Ant and Humanoid we use BC to pre-train the policy with 10k iterations.
Revised: On Walker2d, Hopper, and Swimmer, we train these algorithms directly, while on HalfCheetah, Ant and Humanoid we use BC to pre-train the policy with 10k iterations.

Original: We test all the algorithms on these six locomotion tasks with environment seed 0 , 1 , 2 , 3 , 4 , and using 10 rollouts to estimates the ground truth reward value to plot the training curves.
Revised: We test all the algorithms on these six locomotion tasks with environment seed 0 , 1 , 2 , 3 , 4 , and using 10 rollouts to estimates the ground truth reward value to plot the training curves.

Original: The maximum length for sampled trajectories is 1024.
Revised: The maximum length for sampled trajectories is 1024.

Original: The discounted factor is 0 .
Revised: The discounted factor is 0 .

Original: Each iteration we update 3 times of policies and 1 time of reward function (discriminator or auto-encoder).
Revised: Each iteration we update 3 times of policies and 1 time of reward function (discriminator or auto-encoder).

Original: While updating TRPO, we update the critic 5 times with learning rate 2e-4 every iteration.
Revised: While updating TRPO, we update the critic 5 times with learning rate 2e-4 every iteration.

Original: A.2. B ASELINES
Revised: A.2. B ASELINES

Original: We use open source code, OpenAI baselines (Dhariwal et al., 2017), to get the performance of BC and the training iterations is 100k for all tasks.
Revised: We use open source code, OpenAI baselines (Dhariwal et al., 2017), to get the performance of BC and the training iterations is 100k for all tasks.

Original: GAIL (Ho & Ermon, 2016) borrows the GAN (Goodfellow et al., 2014) architecture to realize the inverse reinforcement learning process and it is applicable in high dimensional dynamic environments.
Revised: GAIL (Ho & Ermon, 2016) borrows the GAN (Goodfellow et al., 2014) architecture to realize the inverse reinforcement learning process and it is applicable in high dimensional dynamic environments.

Original: We implement GAIL using open source code, OpenAI baselines (Dhariwal et al., 2017), with best tunned hyperparameters.
Revised: We implement GAIL using open source code, OpenAI baselines (Dhariwal et al., 2017), with best tunned hyperparameters.

Original: We choose TRPO as the reinforcement learning algorithm.
Revised: We choose TRPO as the reinforcement learning algorithm.

Original: WGAIL (Arjovsky et al., 2017; Ho & Ermon, 2016) utilizes the Wasserstein distance to train the policy and the reward function which can be more stable.
Revised: WGAIL (Arjovsky et al., 2017; Ho & Ermon, 2016) utilizes the Wasserstein distance to train the policy and the reward function which can be more stable.

Original: WGAIL is implemented based on GAIL with corresponding objective function and reward signals as in (Arjovsky et al., 2017).
Revised: WGAIL is implemented based on GAIL with corresponding objective function and reward signals as in (Arjovsky et al., 2017).

Original: We also choose TRPO as the reinforcement learning algorithm.
Revised: We also choose TRPO as the reinforcement learning algorithm.

Original: FAIRL (Ghasemipour et al., 2020) utilizes the forward KL divergence as the objective which shows competitive performances on MuJoCo tasks.
Revised: FAIRL (Ghasemipour et al., 2020) utilizes the forward KL divergence as the objective which shows competitive performances on MuJoCo tasks.

Original: FAIRL is implemented based on GAIL with corresponding objective function and reward signals as in (Ghasemipour et al., 2020).
Revised: FAIRL is implemented based on GAIL with corresponding objective function and reward signals as in (Ghasemipour et al., 2020).

Original: We also choose
Revised: We also choose

Original: TRPO as the reinforcement learning algorithm.
Revised: TRPO as the reinforcement learning algorithm.

Original: PWIL (Dadashi et al., 2021) is an adversarial imitation learning method based on an ofﬂine similarity reward function.
Revised: PWIL (Dadashi et al., 2021) is an adversarial imitation learning method based on an ofﬂine similarity reward function.

Original: It is quite robust and easy to ﬁne-tune the hyper-parameters of its reward function (only two hyper-parameters).
Revised: It is quite robust and easy to ﬁne-tune the hyper-parameters of its reward function (only two hyper-parameters).

Original: PWIL is implemented based on open source code (Dadashi et al., 2021) with TRPO as the reinforcement learning step for fair comparison (original PWIL code uses
Revised: PWIL is implemented based on open source code (Dadashi et al., 2021) with TRPO as the reinforcement learning step for fair comparison (original PWIL code uses

Original: D4PG as the RL algorithm).
Revised: D4PG as the RL algorithm).

Original: A.2. A RCHITECTURE OF O UR M ETHODS
Revised: A.2. A RCHITECTURE OF O UR M ETHODS

Original: Policy Net: the same with baselines, two hidden layers with 64 units, with tanh nonlinearities in between.
Revised: Policy Net: the same with baselines, two hidden layers with 64 units, with tanh nonlinearities in between.

Original: The learning rate to update the policy is 0.01.
Revised: The learning rate to update the policy is 0.01.

Original: Auto-Encoder Net: both vanilla auto-encoders and variational auto-encoder, the same number of layers with the architecture of discriminator in GAIL and WGAIL, 4 layers.
Revised: Auto-Encoder Net: both vanilla auto-encoders and variational auto-encoder, the same number of layers with the architecture of discriminator in GAIL and WGAIL, 4 layers.

Original: The two hideen layers are with 100 units, with tanh nonlinearities in between, the ﬁnal output layer is an identity layer.
Revised: The two hideen layers are with 100 units, with tanh nonlinearities in between, the ﬁnal output layer is an identity layer.

Original: And we normalize the state feature after the input layer and we use the raw state action input features to compute the mean square error.
Revised: And we normalize the state feature after the input layer and we use the raw state action input features to compute the mean square error.

Original: Learning rate for the auto-encoder is 3e-4.
Revised: Learning rate for the auto-encoder is 3e-4.

Original: A. O MITTED E XPERIMENT R ESULTS ON N OISY E XPERT D EMONSTRATIONS
Revised: A. O MITTED E XPERIMENT R ESULTS ON N OISY E XPERT D EMONSTRATIONS

Original: To show the robustness of our AEIRL method, we conduct the experiments on noisy expert demonstrations.
Revised: To show the robustness of our AEIRL method, we conduct the experiments on noisy expert demonstrations.

Original: The noise component is a Gaussian distribution noise which is (0 , 0 . 3) for Walker2d,
Revised: The noise component is a Gaussian distribution noise which is (0 , 0 . 3) for Walker2d,

Original: Hopper, Swimmer, HalfCheetah.
Revised: Hopper, Swimmer, HalfCheetah.

Original: Since the effects of the noise component performs more heavily on Ant and Humanoid, we choose (0 , 0 . 03) and (0 , 0 . 01)
Revised: Since the effects of the noise component performs more heavily on Ant and Humanoid, we choose (0 , 0 . 03) and (0 , 0 . 01)

Original: Gaussian distribution noise for Ant and Humanoid respectively.
Revised: Gaussian distribution noise for Ant and Humanoid respectively.

Original: So under this setting, higher ﬁnal ground truth rewards indicate better robustness of its corresponding algorithm.
Revised: So under this setting, higher ﬁnal ground truth rewards indicate better robustness of its corresponding algorithm.

Original: Figure 7 shows the training curves for different imitation learning algorithms on the noisy expert demonstrations.
Revised: Figure 7 shows the training curves for different imitation learning algorithms on the noisy expert demonstrations.

Original: On Walker2d, HalfCheetah, Ant, and Humanoid, our AEIRL clearly outperforms other baseline algorithms.
Revised: On Walker2d, HalfCheetah, Ant, and Humanoid, our AEIRL clearly outperforms other baseline algorithms.

Original: On the other hand, on Swimmer GAIL achieves the best performance while on Hopper, PWIL achieves the best performance.
Revised: On the other hand, on Swimmer GAIL achieves the best performance while on Hopper, PWIL achieves the best performance.

Original: We scale the ground truth rewards of ﬁnal policy performance for different methods which considers the expert trajectory’s rewards to be 1 and random policy trajectory’s rewards to be 0 .
Revised: We scale the ground truth rewards of ﬁnal policy performance for different methods which considers the expert trajectory’s rewards to be 1 and random policy trajectory’s rewards to be 0 .

Original: And the averaged scaled ground truth rewards is 0 .
Revised: And the averaged scaled ground truth rewards is 0 .

Original: 795 while the best baseline is 0 .
Revised: 795 while the best baseline is 0 .

Original: 542 for PWIL.
Revised: 542 for PWIL.

Original: So we achieve 46 .
Revised: So we achieve 46 .

Original: 5% relative improvement to PWIL.
Revised: 5% relative improvement to PWIL.

Original: In summary, our method is more robust than other baselines.
Revised: In summary, our method is more robust than other baselines.

Original: A. D ETAILS AND M ORE R ESULTS ON T -SNE V ISUALIZATION
Revised: A. D ETAILS AND M ORE R ESULTS ON T -SNE V ISUALIZATION

Original: The auto-encoder based reward function learns the mapping from state action space to a scalar value, i.e., the reward.
Revised: The auto-encoder based reward function learns the mapping from state action space to a scalar value, i.e., the reward.

Original: When the system of our method converges, the generated state action samples would be indistinguishable with the expert state action samples.
Revised: When the system of our method converges, the generated state action samples would be indistinguishable with the expert state action samples.

Original: For our auto-encoder based reward function, the latent representations between ﬁnal policy samples and expert samples would also be indistinguishable.
Revised: For our auto-encoder based reward function, the latent representations between ﬁnal policy samples and expert samples would also be indistinguishable.

Original: To compare the robustness of different reward signals of different methods, we use t-SNE visualization technique to visualize the latent space of the reward function network.
Revised: To compare the robustness of different reward signals of different methods, we use t-SNE visualization technique to visualize the latent space of the reward function network.

Original: For the discriminator based methods, we visualize the output of ﬁrst hidden (middle) layer for instance.
Revised: For the discriminator based methods, we visualize the output of ﬁrst hidden (middle) layer for instance.

Original: And for our autoencoder based reward function, we also visualize the output of ﬁrst hidden (middle) layer in the net.
Revised: And for our autoencoder based reward function, we also visualize the output of ﬁrst hidden (middle) layer in the net.

Original: So the latent representations for both discriminator based and our auto-encoder based methods will show its robustness, especially to the corrupted expert data.
Revised: So the latent representations for both discriminator based and our auto-encoder based methods will show its robustness, especially to the corrupted expert data.

Original: We plot the t-SNE embedding results with SciKit-Learn tools (i.e., sklearn.manifold.
Revised: We plot the t-SNE embedding results with SciKit-Learn tools (i.e., sklearn.manifold.

Original: TSNE function) with varying preplexity.
Revised: TSNE function) with varying preplexity.

Original: The hyper-parameters for t-SNE are (other hyper-parameters are default with scikit-learn): n
Revised: The hyper-parameters for t-SNE are (other hyper-parameters are default with scikit-learn): n

Original: component =
Revised: component =

Original: early
Revised: early

Original: exaggerationﬂoat = 12.
Revised: exaggerationﬂoat = 12.

Original: learning rate = 200.
Revised: learning rate = 200.

Original: n iterint =n iter without progressint =
Revised: n iterint =n iter without progressint =

Original: min
Revised: min

Original: grad normﬂoat =
Revised: grad normﬂoat =

Original: 1e-
Revised: 1e-

Original: metric
Revised: metric

Original: = ’euclidean’ init
Revised: = ’euclidean’ init

Original: =
Revised: =

Original: ’pca’
Revised: ’pca’

Original: Since PWIL (Dadashi et al., 2021) considers an ofﬂine reward function without neural networks, we only compare our methods with other three IRL baselines.
Revised: Since PWIL (Dadashi et al., 2021) considers an ofﬂine reward function without neural networks, we only compare our methods with other three IRL baselines.

Original: Figure 8 depicts that two sets of samples are more indistinguishable than other baselines both on clean and noisy expert data.
Revised: Figure 8 depicts that two sets of samples are more indistinguishable than other baselines both on clean and noisy expert data.

Original: The top row of
Revised: The top row of

Original: Figure 8 shows the embeddings with preplexity = 5 . 0 in the non-noisy setting.
Revised: Figure 8 shows the embeddings with preplexity = 5 . 0 in the non-noisy setting.

Original: The second row shows the embeddings with preplexity = 50 . 

Revised: The second row shows the embeddings with preplexity = 50 .

Original: in the non-noisy setting.
Revised: in the non-noisy setting.

Original: The third row of Figure 8 shows the embeddings with preplexity = 5 . 0 in the noisy setting.
Revised: The third row of Figure 8 shows the embeddings with preplexity = 5 . 0 in the noisy setting.

Original: The bottom row shows the embeddings with preplexity = 50 . 

Revised: The bottom row shows the embeddings with preplexity = 50 .

Original: in the noisy setting.
Revised: in the noisy setting.

Original: And the noise component is a Gaussian distribution (0 , 0 . 5) .
Revised: And the noise component is a Gaussian distribution (0 , 0 . 5) .

Original: And in the main text, Figure 4 shows the default preplexity = 30 . 0 results.
Revised: And in the main text, Figure 4 shows the default preplexity = 30 . 0 results.

Original: With varying hyper-parameter preplexity, we can see the overlapped areas for policy and expert samples are greater for our method.
Revised: With varying hyper-parameter preplexity, we can see the overlapped areas for policy and expert samples are greater for our method.

Original: In detail, we can observe that the policy samples for GAIL and
Revised: In detail, we can observe that the policy samples for GAIL and

Original: WGAIL hardly appear in the outer ring with preplexity = 30 . 0 .
Revised: WGAIL hardly appear in the outer ring with preplexity = 30 . 0 .

Original: And FAIRL is a little better than
Revised: And FAIRL is a little better than

Original: GAIL and WGAIL.
Revised: GAIL and WGAIL.

Original: Our AEIRL gets maximum overlap between two sets of samples compared to these baselines.
Revised: Our AEIRL gets maximum overlap between two sets of samples compared to these baselines.

Original: And in the noisy settings, the two sets of embeddings of GAIL and WGAIL are actually isolated with different ”preplexity” while they are gobally clustered (greatly overlapped) only with preplexity = 30 . 0 for FAIRL.
Revised: And in the noisy settings, the two sets of embeddings of GAIL and WGAIL are actually isolated with different ”preplexity” while they are gobally clustered (greatly overlapped) only with preplexity = 30 . 0 for FAIRL.

Original: And the embeddings are greatly overlapped with varing preplexity for our method.
Revised: And the embeddings are greatly overlapped with varing preplexity for our method.

Original: So it reﬂects that our auto-encoder based reward function is more robust to both the clean and noisy data.
Revised: So it reﬂects that our auto-encoder based reward function is more robust to both the clean and noisy data.

Original: A. A BLATION S TUDIES
Revised: A. A BLATION S TUDIES

Original: To illustrate the major contributing factors of our methods, we conduct the ablation studies with different forms of distribution divergences and other alternative auto-encoders.
Revised: To illustrate the major contributing factors of our methods, we conduct the ablation studies with different forms of distribution divergences and other alternative auto-encoders.

Original: [Table]
Revised: [Table]

Original: We conduct the experiments with another two forms of reward functions following the
Revised: We conduct the experiments with another two forms of reward functions following the

Original: Jensen–Shannon divergence (Goodfellow et al., 2014) for comparison.
Revised: Jensen–Shannon divergence (Goodfellow et al., 2014) for comparison.

Original: Typically, JS divergence is deﬁned as:
Revised: Typically, JS divergence is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where P r and P g represents the real data distribution and the generated data distribution.
Revised: where P r and P g represents the real data distribution and the generated data distribution.

Original: The adversarial training procedure of auto-encoding inverse reinforcement learning will match the state action distribution of expert and generated samples.
Revised: The adversarial training procedure of auto-encoding inverse reinforcement learning will match the state action distribution of expert and generated samples.

Original: In the ﬁrst form of alternative reward functions, we consider the exponential of negative reconstruction error as: P r ( x ) /
Revised: In the ﬁrst form of alternative reward functions, we consider the exponential of negative reconstruction error as: P r ( x ) /

Original: P r ( x ) + P g ( x ) .
Revised: P r ( x ) + P g ( x ) .

Original: The corresponding Jensen-Shannon divergence for updating the auto-encoder based reward function is:
Revised: The corresponding Jensen-Shannon divergence for updating the auto-encoder based reward function is:

Original: [Equation]
Revised: [Equation]

Original: Here, w represents the parameters of the auto-encoder network.
Revised: Here, w represents the parameters of the auto-encoder network.

Original: So its formulation of surrogate reward function is:
Revised: So its formulation of surrogate reward function is:

Original: [Equation]
Revised: [Equation]

Original: In this formulation, minimizing the Jensen-Shannon divergence between state action distribution of the expert and generated trajectories assigns low mean square errors to the expert samples and high mean square errors to other regions.
Revised: In this formulation, minimizing the Jensen-Shannon divergence between state action distribution of the expert and generated trajectories assigns low mean square errors to the expert samples and high mean square errors to other regions.

Original: In the second formulation of reward functions, we consider the sigmoid output of the negative reconstruction error as: P r ( x ) /
Revised: In the second formulation of reward functions, we consider the sigmoid output of the negative reconstruction error as: P r ( x ) /

Original: P r ( x )+ P g ( x ) .
Revised: P r ( x )+ P g ( x ) .

Original: So it corresponding objective function for updating the auto-encoder is:
Revised: So it corresponding objective function for updating the auto-encoder is:

Original: [Equation]
Revised: [Equation]

Original: where w also represents the parameters of the auto-encoder network.
Revised: where w also represents the parameters of the auto-encoder network.

Original: And its reward function formulation is:
Revised: And its reward function formulation is:

Original: [Equation]
Revised: [Equation]

Original: In experimental results, we note the ﬁrst formulation as ”ours-exponential” and the second formulation as ”ours-sigmoid”.
Revised: In experimental results, we note the ﬁrst formulation as ”ours-exponential” and the second formulation as ”ours-sigmoid”.

Original: Speciﬁcally, we don’t add the entropy term to the reinforcement objective which already achieve competitive results on MuJoCo tasks.
Revised: Speciﬁcally, we don’t add the entropy term to the reinforcement objective which already achieve competitive results on MuJoCo tasks.

Original: These three forms of reward signals achieve comparable performance on all tasks except for Humanoid shown in Figure 9.
Revised: These three forms of reward signals achieve comparable performance on all tasks except for Humanoid shown in Figure 9.

Original: This indicates that an auto-encoder based reward function is the major contributing factor to improve the performance on mimicking the expert rather than the different distribution divergences or forms of reward signal.
Revised: This indicates that an auto-encoder based reward function is the major contributing factor to improve the performance on mimicking the expert rather than the different distribution divergences or forms of reward signal.

Original: However, the performance on Humanoid is quite different between these different forms of reward signals.
Revised: However, the performance on Humanoid is quite different between these different forms of reward signals.

Original: It shows that the Wasserstein distance can be more robust to this task.
Revised: It shows that the Wasserstein distance can be more robust to this task.

Original: Our method is not limited to vanilla auto-encoders but also other alternative auto-encoders.
Revised: Our method is not limited to vanilla auto-encoders but also other alternative auto-encoders.

Original: To show the differences between our method and other alternative auto-encoders, we conduct the experiments with variational auto-encoder based reward function.
Revised: To show the differences between our method and other alternative auto-encoders, we conduct the experiments with variational auto-encoder based reward function.

Original: The variational auto-encoder (Kingma & Welling, 2014) is a directed model that uses learned approximate inference.
Revised: The variational auto-encoder (Kingma & Welling, 2014) is a directed model that uses learned approximate inference.

Original: The key insight behind training variational auto-encoder is to maximize the variational lower bound L ( q ) of the log-likelihood for the training examples (Goodfellow et al., 2016):
Revised: The key insight behind training variational auto-encoder is to maximize the variational lower bound L ( q ) of the log-likelihood for the training examples (Goodfellow et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: where z is the latent representation for data points x .
Revised: where z is the latent representation for data points x .

Original: And p model is chosen to be a ﬁxed Gaussian distribution N (0 , I ) .
Revised: And p model is chosen to be a ﬁxed Gaussian distribution N (0 , I ) .

Original: In Eq.
Revised: In Eq.

Original: 16, we recognize that the second entropy term encourages the variational posterior to place high probability mass on z values as many as possible, rather than collapse to a single most likely value point.
Revised: 16, we recognize that the second entropy term encourages the variational posterior to place high probability mass on z values as many as possible, rather than collapse to a single most likely value point.

Original: When we use an variational auto-encoder based reward function in our method, the latent representations of the expert samples can be widely distributed on a Gaussian distribution.
Revised: When we use an variational auto-encoder based reward function in our method, the latent representations of the expert samples can be widely distributed on a Gaussian distribution.

Original: Therefore, matching the expert and generated state action distributions in this latent space leads to more diverse behaviors inducing.
Revised: Therefore, matching the expert and generated state action distributions in this latent space leads to more diverse behaviors inducing.

Original: Thus, the reward signal would be more informative.
Revised: Thus, the reward signal would be more informative.

Original: Training the variational auto-encoder is still a adversarial process.
Revised: Training the variational auto-encoder is still a adversarial process.

Original: We notice that the ﬁrst term in
Revised: We notice that the ﬁrst term in

Original: Eq.
Revised: Eq.

Original: 17 is the reconstruction log-likelihood found in traditional auto-encoders.
Revised: 17 is the reconstruction log-likelihood found in traditional auto-encoders.

Original: So the objective for training the variational auto-encoder is:
Revised: So the objective for training the variational auto-encoder is:

Original: [Equation]
Revised: [Equation]

Original: where the choice for r w ( s, a ) is the same with Eq.
Revised: where the choice for r w ( s, a ) is the same with Eq.

Original: 6 in Section 4.2.
Revised: 6 in Section 4.2.

Original: Therefore, low mean square errors of the variational auto-encoder ensure high reward values for state action pairs and vice versa.
Revised: Therefore, low mean square errors of the variational auto-encoder ensure high reward values for state action pairs and vice versa.

Original: When minimizing this objective, the variational auto-encoder based reward function can still assign high values near the expert demonstrations and low values to other regions which is the same as in typical auto-encoders.
Revised: When minimizing this objective, the variational auto-encoder based reward function can still assign high values near the expert demonstrations and low values to other regions which is the same as in typical auto-encoders.

Original: On the other hand, optimizing this objective is restricting the latent distribution for the expert to a ﬁxed Gaussian distribution p model ( z )) and maximizing the KL distance between the generated samples and the same ﬁxed Gaussian distribution.
Revised: On the other hand, optimizing this objective is restricting the latent distribution for the expert to a ﬁxed Gaussian distribution p model ( z )) and maximizing the KL distance between the generated samples and the same ﬁxed Gaussian distribution.

Original: With the bi-level optimization process 4, we can induce the expert policy with more diverse behaviors from the expert demonstrations which addresses the problem of mode collapse in GAIL.
Revised: With the bi-level optimization process 4, we can induce the expert policy with more diverse behaviors from the expert demonstrations which addresses the problem of mode collapse in GAIL.

Original: And a more informative reward signal would be yield.
Revised: And a more informative reward signal would be yield.

Original: Figure 10 shows the training curves for vanilla auto-encoder based variant and varational autoencoder based variant on non-noisy expert demonstrations.
Revised: Figure 10 shows the training curves for vanilla auto-encoder based variant and varational autoencoder based variant on non-noisy expert demonstrations.

Original: The performances for these two variants are comparable on almost all tasks while vanilla auto-encoder based method is better than varational auto-encoder based variant on Humanoid.
Revised: The performances for these two variants are comparable on almost all tasks while vanilla auto-encoder based method is better than varational auto-encoder based variant on Humanoid.

Original: It indicates that the variational auto-encoder can be an alternative reward function formulation.
Revised: It indicates that the variational auto-encoder can be an alternative reward function formulation.

Original: A. A NALYSIS ON S TABILITY OF O UR M ETHODS
Revised: A. A NALYSIS ON S TABILITY OF O UR M ETHODS

Original: The variance of gradients is computed via different mini-batches of training examples.
Revised: The variance of gradients is computed via different mini-batches of training examples.

Original: Lower variance of gradients means smoother updating of the network (Faghri et al., 2020).
Revised: Lower variance of gradients means smoother updating of the network (Faghri et al., 2020).

Original: Now that our auto-encoder based reward function is more informative and denser than the discriminator based ones, how about the training stability of our method?
Revised: Now that our auto-encoder based reward function is more informative and denser than the discriminator based ones, how about the training stability of our method?

Original: To show the training stability of different methods, we record the norm of weights of the policy network and the variance of policy gradients.
Revised: To show the training stability of different methods, we record the norm of weights of the policy network and the variance of policy gradients.

Original: The relative variance of policy gradients shows the training stability for different methods.
Revised: The relative variance of policy gradients shows the training stability for different methods.

Original: While we utilize Wasserstein distance in our method, we compare our vanilla auto-encoder based variant and variational auto-encoder based variant with WGAIL.
Revised: While we utilize Wasserstein distance in our method, we compare our vanilla auto-encoder based variant and variational auto-encoder based variant with WGAIL.

Original: It could depict the training stability with our newly proposed auto-encoder based reward formulation.
Revised: It could depict the training stability with our newly proposed auto-encoder based reward formulation.

Original: Figure 11 shows the training stability of our method compared to the baseline WGAIL.
Revised: Figure 11 shows the training stability of our method compared to the baseline WGAIL.

Original: It depicts that our method can be more stable and the policy network updating is smoother.
Revised: It depicts that our method can be more stable and the policy network updating is smoother.

Original: A. A DDITIONAL E XPERIMENTS ON B ETTER S YNTHESIS E XPERTS
Revised: A. A DDITIONAL E XPERIMENTS ON B ETTER S YNTHESIS E XPERTS

Original: [Table]
Revised: [Table]

Original: To get more convincing results
Revised: To get more convincing results

Original: [Table]
Revised: [Table]

Original: we run the experiments on better synthesis experts.
Revised: we run the experiments on better synthesis experts.

Original: We collect the expert demonstration with 16 trajectories each task via D4PG (Barth-Maron et al., 2018).
Revised: We collect the expert demonstration with 16 trajectories each task via D4PG (Barth-Maron et al., 2018).

Original: Figure 12 depicts the training curves of ground truth rewards for different algorithms.
Revised: Figure 12 depicts the training curves of ground truth rewards for different algorithms.

Original: The training curve shows the imitation performance learning from non-noisy expert data.
Revised: The training curve shows the imitation performance learning from non-noisy expert data.

Original: Table 4 shows the ﬁnal learned policy performance on MuJoCo tasks with ground truth episode return.
Revised: Table 4 shows the ﬁnal learned policy performance on MuJoCo tasks with ground truth episode return.

Original: The overall results are similar with what we have done in the main text.
Revised: The overall results are similar with what we have done in the main text.

Original: On Walker2d, HalfCheetah and Ant, our method gets the best performance.
Revised: On Walker2d, HalfCheetah and Ant, our method gets the best performance.

Original: PWIL gets the best performance and our AEIRL is on par with PWIL on Hopper.
Revised: PWIL gets the best performance and our AEIRL is on par with PWIL on Hopper.

Original: We take the random and expert trajectory as 0 and 1 respectively to compute the scaled ﬁnal rewards for different methods.
Revised: We take the random and expert trajectory as 0 and 1 respectively to compute the scaled ﬁnal rewards for different methods.

Original: The overall averaged scaled rewards for our method is about 0 .
Revised: The overall averaged scaled rewards for our method is about 0 .

Original: 729 where the best baseline is 0 .
Revised: 729 where the best baseline is 0 .

Original: 546 for PWIL.
Revised: 546 for PWIL.

Original: It is a more than 33 .
Revised: It is a more than 33 .

Original: 7% relative improvement.
Revised: 7% relative improvement.

Original: So our method outperforms the other state-of-the-art baselines mostly.
Revised: So our method outperforms the other state-of-the-art baselines mostly.