From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: I have read the reviews, rebuttals, and discussion surrounding this paper proposing a novel adversarial imitation learning method. Even were we to discard the weakest score, the consensus remains in favour of rejection, and in the absence of a strong voice championing acceptance, I see no reason to overrule the paper. Furthermore, the reviewers giving borderline rejects gave detailed and actionable feedback which I think will help the authors make this paper better and the argument it present stronger, but would benefit from another round of reviews once the feedback is incorporated. I wish the authors the best of luck in doing so.
justification_for_why_not_higher_score: No strong argument for or champion in favour of acceptance, and detailed, actionable feedback for improving the paper which warrants another round of reviews.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Thanks for your reply
comment: Thanks for your reply. We address your concerns as follows.

**Scaling Issue for Entropy** Firstly, the entropy is a proxy measurement for denseness. The comparison should be under the same reward scale. Since entropy reflects the amounts of reward information for different rollout batches, entropy could be used to measure the "denseness". Secondly, scaling the rewards would increase entropy but doesn't change the optimal/sub-optimal policy in MDP [1]. Your question is now that scaling would increase entropy thus "denseness", then would the baselines be improved by scaling the rewards. Our answer is "no" since scaling rewards won't change the policy optimization period. However, our comparison for entropy is under the same scale. So it is meaningful and fair. 

**Why using TRPO?** Different people use different RL agents for imitation learning [2]. TRPO is still one of the best on-policy RL agent algorithms. In our paper, we use TRPO for fair comparison. In Appendix A.6, we also provide a thorough comparison with D4PG as RL agent. With both TRPO and D4PG, our AEAIL outperforms other baselines. 

[1] Policy invariance under reward transformations: Theory and application to reward shaping. A. Ng et al., ICML'1999.

[2] what matters for adversarial imitation learning. M. Orisini et al., NeurIPS 2021
 
Best

Authors

From: ICLR.cc/2023/Conference/Paper1734/Reviewer_9Nok
title: Thanks for the clarification
comment: I appreciate the efforts from the authors. My concern over the theoretical issue is resolved. and I have revised my score based on the new results. However, my other concerns are not fully addressed. I still disagree with the entropy metric and characterization of the reward being a random variable. The scaling issue is still present: can we just make other baselines better by tuning a scaling factor as a hyper-parameter to improve "denseness" of other rewards? For comparison with other methods, I disagree with using TRPO for all methods is fair. All methods should be allowed to use whatever the best RL algorithm it can use. Other subtle but important conditions that may change the results are number of expert demonstrations and the quality of the expert demonstrations. The current experiments haven't fully captured these conditions explored in the previous works.

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Thanks for your reply
comment: Thanks for your thoughtful reply. We address your concerns as follows.

**Q1: Note that the actual reward function being used for optimization is $1/(1+AE(x))$. Thus it may be necessary to further show that $1/(1+AE(x))$ is K-Lipschitz, in order to show that W-distance is being optimized.**

A1: Thanks for your suggestion. We will revise this part in the final version of our paper. Here, we prove that if $f(x)$ is K-Lipschitz, then $1/(1+f(x))$ is also K-Lipschitz.

Since $f(x)$ is K-Lipschitz, we can obtain that there exists $K$, for all real $x_1$, $x_2$, that

$$|f(x_1) - f(x_2)| \leq K |x_1-x_2|$$

We can obtain:

$$|1/(1+f(x_1)) - 1/(1+f(x_2))| = |(f(x_2)-f(x_1))/(f(x_1+f(x_2)+1+f(x_1)\cdot f(x_2)))|$$

Since $f(x)$ is the reconstruction error of an auto-encoder, $f(x) \geq 0$
So We can obtain:

$$|1/(1+f(x_1)) - 1/(1+f(x_2))| = |(f(x_2)-f(x_1))/(f(x_1+f(x_2)+1+f(x_1)\cdot f(x_2)))| \leq |f(x_2)-f(x_1)| \leq K |x_1-x_2|$$

So $1/(1+f(x))$ is also K-Lipschitz. 

**Q2: I'm not convinced that AE gives a denser reward.  It's still unclear to me why a reward is assigned a probability and what that probability represents. In addition, scaling still poses a significant issue for explaining the advantages of AEAIL, because any rewards can become "denser" by simple scaling. Overall, reward denseness is not well motivated and appears to be an insufficient justification for the performance of AEAIL.**

A2: (1) we view the rewards as a random variable since we randomly generates the rewards during training. Therefore, the probability is used to compute how the rewards are distributed. Thus, we can use these probabilities to compute entropy for reward information during optimization. Here, we define the denseness as the amount of reward information for the same batch from agent rollouts. Higher amounts of reward information for the same batch would be beneficial for policy training since it contains much more learning signals. And we define it as a denser reward signal. 

(2) Differential entropy is translation invariant but not invariant under arbitrary invertible maps [1]. In our paper, our comparison is based on the same reward scale (both AEAIL/FAIRL's rewards are in [0,1]). It satisfies the requirements for fair comparison. 

(3) entropy is a proxy for denseness justification in our paper. We use this experiment result to show that our AE-based rewards are much denser than the discriminator-based ones. 

[1] Wikipedia: https://en.wikipedia.org/wiki/Differential_entropy

**Q3: I'm not sure why absorbing state technique from DAC is added for this experiment, since it introduces additional components that may confound the final performance.**

A3: In this additional experiment, we want to show that our AEAIL is a general modification  to AIL methods. It could be built on any AIL methods for gaining better imitation performance. DAC is a great AIL variant which uses absorbing state techinique to solve the reward bias problem and uses off-policy RL agent to improve the sample efficiency. In this case, we prove that the performance gains of AEAIL are not due to survival reward bias. Meanwhile, we show that our AEAIL (built on DAC)  is better than PWIL+D4PG overall. It also indicates that it doesn't matter which RL agent we use in the experiments. Under both TRPO and D4PG, our AEAIL variants are better than  PWIL over all tasks. 

**Q4: My questions on lower-than-expected baseline performance and expert sample complexity (e.g., 11 trajectories in PWIL vs 320 in this work) are not addressed.**

A4: (1) *lower-than-expected baseline performance* Thanks for your comment. The experiment results are a little lower than PWIL paper since we use TRPO as the RL agent for all the baselines. In previous work, different people uses different RL agent. In our paper, we use TRPO as RL agent for a fair comparison. The experiment results are similar with other related papers such as in [2], which also uses TRPO as RL agent algorithm. In Appendix A.6, we also use D4PG as the RL agent for comparison. Under both D4PG and TRPO, our AEAIL is better than other baselines. 

(2) *sample complexity*  PWIL utilizes an offline similarity measurement for imitation learning. The advantage of PWIL is that they are relatively more robust and less demonstration dependent than AIL methods. So in PWIL paper, it only needs 11 trajectories for experiments. However, AIL methods need much more expert data to achieve good performance since it uses a generative adversarial training manner. In our experiments, we also use sufficient expert data for comparison (320 trajectories on Humanoid is the same as GAIL paper).  

[2] what matters for adversarial imitation learning. M Orisini et al., NeurIPS 2021


Best

Authors

From: ICLR.cc/2023/Conference/Paper1734/Reviewer_9Nok
title: Thanks for clarification
comment: I thanks the authors for the additional efforts and experiment results.

**K-Lipschitz and W-distance**
It is nice to see the derivation that auto-encoding functions are also K-Lipschitz. However, there appears to be a gap from Theorem 1 to Corollary 1. Note that the actual reward function being used for optimization is $1/(1+AE(x))$. Thus it may be necessary to further show that $1/(1+AE(x))$ is K-Lipschitz, in order to show that W-distance is being optimized.

**Denser reward** I'm not convinced that AE gives a denser reward. It's still unclear to me why a reward $r(s, a)$ is assigned a probability and what that probability represents. In addition, scaling still poses a significant issue for explaining the advantages of AEAIL, because any rewards can become "denser" by simple scaling. Overall, reward denseness is not well motivated and appears an insufficient justification for the performance of AEAIL.

**Additional experiments** I'm not sure why absorbing state technique from DAC is added for this experiment, since it introduces additional component that may confound the final performance. On a related note, does PWIL also implement absorbing state technique for fair comparison in the new experiments? For understanding the advantages of AEAIL, the original experiment setup appears sufficient where the only difference between two methods is the reward function. 

For Humanoid, the original GAIL paper has an expert performance of close to 10K and a similar expert is used in various follow-up papers (e.g. RED and PWIL). The broader question here that about whether the baseline methods are performing at their expected level, and if not, what the issues may have caused it. These details are important for understanding the advantages of the proposed method. My questions on lower-than-expected baseline performance and expert sample complexity (e.g., 11 trajectories in PWIL vs 320 in this work) are not addressed.

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Discussion Period Ending Soon
comment: Dear Reviewer 9Nok,

We kindly remind that the final stage of discussion is ending soon, and so please kindly let us know if our response has addressed your concerns.

Thanks again for your time and reviews, we will be happy to answer if there are additional issues or questions.

Authors

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Discussion Period Ending Soon
comment: Dear Reviewer ymvo,

We kindly remind that the final stage of discussion is ending soon, and so please kindly let us know if our response has addressed your concerns.

Thanks again for your time and reviews, we will be happy to answer if there are additional issues or questions.

Authors

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Discussion Period Ending Soon
comment: Dear Reviewer EB8r,

We kindly remind that the final stage of discussion is ending soon, and so please kindly let us know if our response has addressed your concerns.

Thanks again for your time and reviews, we will be happy to answer if there are additional issues or questions.

Authors

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Discussion Period Ending Soon
comment: Dear Reviewer vULj,

We kindly remind that the final stage of discussion is ending soon, and so please kindly let us know if our response has addressed your concerns.

Thanks again for your time and reviews, we will be happy to answer if there are additional issues or questions.

Authors

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Response to Reviewer 9Nok
comment: Thanks for your thoughtful and constructive review! We address your concerns as follows.

**Q1: K-Lipschitz should be upon $\Vert x -f(x)\Vert^2$ the reconstruction error rather the auto-encoder function. In addition, EBGAN optimizes the TV distance. How about AEAIL?**

A1: Thanks for your suggestion. We have revised this issue in the new version of our paper. In detail, when we clip the weights of the auto-encoder, we can prove that the $\Vert x -f(x)\Vert^2$ is $2m(K+1)$-Lipschtiz since $x$ and $f(x)$ is bounded while $f(x)$ is K-Lipschtiz. Therefore, our reward function can also be K-Lipschtiz.  We prove that the distance we optimized in AEAIL is W-distance in Appendix A.2 (reference: [1]). We revise this in the new version of our paper. 

[1] Wasserstein GAN. M. Arjovsky et al., ICML'2017.
 
**Q2: the denseness is unclear, how to compute differential entropy? The entropy measurement is also problematic for 2 reasons: 1) it appears that simply scaling the reward with a large constant would improve entropy for any reward function. 2) Entropy requires p(r(s,a)) to compute. It is unclear to me how the probability of a reward r(s,a) is obtained.**

A2: We define the denseness as the amount of reward information for the same batch from agent rollouts. Higher amounts of reward information for the same batch would be beneficial for policy training since it contains much more learning signals. And we define it as a denser reward signal. Specifically, we compute the differential entropy for each reward batch as:

$$H(x) = -\int_{\mathcal{X}}f(x)\log f(x)dx$$

where $f$ is the probability density function whose support is a set $\mathcal {X}$. 
(1) The scaling (like some transfomations) won't increase the "differential entropy" if we discrete the rewards into sections enough. Of course, scaling with a large constant would increase entropy but in our comparison, the rewards for FAIRL and our AEAIL are both at the same scale, i.e., [0, 1]. And we compute entropy with 100 sections, which is enough empirically. (2) For computation convenience, we discrete the rewards into 100 sections to compute the probabilities of rewards in each section. Therefore, we can get the entropy of each reward batch from agent rollouts.  
 
**Q3: only provide results with TRPO? Why not try a better RL agent algorithm. The performance of expert demonstrations for Humanoid is lower compared to PWIL paper.**

A3: Thanks for your suggestion. We have added the off-policy RL agent algorithm for comparison in Appendix A.6. Our AEAIL is a general modification of AIL methods with the auto-encoder as the reward function formulation. It doesn't matter which RL agent to choose. However, we added the comparison with original PWIL (D4PG) in Appendix A.6. It shows that our AEAIL is general and flexible for different RL agents. Our AEAIL is better than PWIL on all the tasks except for Swimmer. For Humanoid expert performance issue, we have tried several SoTA RL algorithms to get the expert demonstrations, about 5700 score is what we can get best in Humanoid task. And the report for Humanoid is around 5000~6000 in D4PG/TD3/SAC original paper. 

**Q4: the novelty is limited since the related techniques can be traced to existing works?**

**[1] Exploration by random network distillation, Burda et al. ICLR 2019.**

**[2] Random expert distillation: Imitation learning via expert policy support estimation, Wang et al. ICML 2019.**

**[3] Energy-based Generative Adversarial Networks, Zhao et al, ICLR 2017.**

A4: We compare our AEAIL with these works in section 2. RND [1] and EBGAN [3] are not about imitation learning. RND [1] is for better exploration of RL algorithms. EBGAN [3] is for generating vivid images and avoiding gradient vanishing in training GANs. RED [2] is using random network distillation for imitation learning. One of its variant is using AE in place of random network distillation. However, the authors of RED [2] acknowledge that the AE version's performance is worse than GAIL and thus worse than ours. On the other hand, the main technical difference between RED [2] and our AEAIL is that they used a fixed network, while we constantly update the AE with an adversarial loss.

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Response to Reviewer ymvo  
comment: Thanks for your thoughtful and constructive review! We address your concerns as follows.

**Q1:  Wasserstein divergence/WGAN are introduced quite lengthily in sec 3 and 4, but then not really used?  I recommend to also include [Xiao at al: Wasserstein adversarial imitation learning] as reference.**

A1: Thanks for your suggestion. The objective we use is actually the W-distance since we clip the weights of the auto-encoder to keep the K-Lipschitz constraint for the reward function. We have revised the writing of the paper to show the connection to the W-distance. 

**Q2: Is the Lipschitz constraint dropped in Eq. 3?**

A2: No, we clip the weights of the auto-encoder to keep the K-Lipschitz constraint for the auto-encoder function. Therefore, the reward function could also be K-Lipschitz (See Proof in Appendix A.2). We have revised this in the new version of our paper.  
 
**Q3: will the presentation of IL and IRL be improved?**

A3: Thanks for your suggestion. We have improved the writing in the new version of our paper. 
 
**Q4: Should the CartPole task be introduced more properly?**

A4: Thanks for your suggestion. We have introduced the details of CartPole task in section 4.1 in the new version of our paper.
 
**Q5: the auto-encoder learns a low-dimensional manifold. Thus, the elements lying on the same manifold would lead to the same error/reward. This may not be expressive enough as a reward function.**

A5: Our auto-encoder is very expressive since it learns the full-scale difference between the expert and generated samples. To justify this hypothesis, we illustrate the generated rewards for different samples in the whole trajectory space. Distinct rewards for different trajectories means the reward signal is expressive. In Appendix A.7, we added the results to show that the final learned scaled and normalized rewards for different samples with our AEAIL's reward function on Walker2d. Note that learned rewards under our formulation are readily distinct in different samples. Our steep curve indicates that the auto-encoder derived reward function is very expressive indeed.  
 
**Q6: the auto-encoder in this work - i.e., reconstruction error for expert states - is not robust to distributional changes etc. For example, the same state action could be expert state action while also agent generated pair. Therefore, GAIL or AEAIL would recognize this pair as positive sample but also negative sample. Is it not robust for GAIL and AEAIL to distributional changes?**

A6: Both GAIL and our AEAIL have this issue. However, this is orthogonal to our contribution. We appreciate the reviewer for raising this issue and we agree that it is worth future research. 

**Q7: the relation between an auto-encoder based approach to detect the expert versus non-expert, compared to the more classic distribution-divergence one.**

A7: To show the relation between AE and discriminator-based reward function, we visualize the latent representations of the discriminator (in AIL methods) and the auto-encoder (in our method) on both the expert and the policy-generated samples with t-SNE in Appendix A.8. Here, the policy-generated samples are sampled from the same expert policy. The t-SNE visualization shows that our AEAIL greatly overlaps the two sets of embeddings compared to other discriminator based formulations. It shows that the auto-encoder can learn full-scale difference between expert and generated samples rather than easily overfit to the minor difference. 
 
**Q8: the theoretical analysis seems incomplete?**

A8: Thanks for your suggestion. We have improved the theoretical part in the new version of our paper.

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Response to Reviewer EB8r
comment: Thanks for your thoughtful and constructive review! We address your concerns as follows.

**Q1: the idea of using reconstruction loss in RL already exists. Therefore, the novelty of the proposed approach is limited. -- "Loss is its own Reward: Self-Supervision for Reinforcement Learning" -- "Improving Sample Efficiency in Model-Free Reinforcement Learning from Images"**

A1: these two works are not about imitation learning. In "Loss is its own Reward: Self-Supervision for Reinforcement Learning ", this paper proposes a self-supervised architecture of training A3C. In "Improving Sample Efficiency in Model-Free Reinforcement Learning from Images", this paper proposes using regularized AE to augment SAC to achieve stable training from images in the off- policy regime. Both works are not about how to learn the reward function. In our work, we formulate the reward function in adversarial imitation learning as an auto-encoder which utilizes an adversarial training paradigm. It provides much more learning signals to the agent training. Therefore, the rewards are much denser than discriminator-based ones. 

**Q2: the experimental evaluation is limited as it is not quite convinced that the proposed approach results in significant improvement**

A2: we conduct experiments with six MuJoCo tasks which are thorough as lots of previous work did, such as in [1-3].   Our AEAIL gets best performance comprehensively compared to other state-of-the-art baselines, especially on the noisy expert demonstration setting. Empirically, our AEAIL achieves 11% and 50.7% relative improvement overall compared to the best baseline GAIL and PWIL on clean and noisy expert data respectively. 

**Q3: Theoretical justification to motivate the design choice is not convincing?**

A3: Thanks for your suggestion. We have added more illustrations for theoretical justification in the new version of our paper. In detail, we prove that clipping weights of the auto-encoder can help the reward function to be K-Lipschitz. Therefore, our AEAIL is actually optimizing the W-distance. And our AEAIL could converge globally under the corresponding assumptions as shown in the paper. 
 
**Q4: The statement "The higher entropy of learned rewards means the reward signal can offer more feedback for each state-action pair, thus providing a denser reward signal to the agent policy." is given without any justification or theoretical support.**

A4: Thanks for your suggestion. We have added more illustrations in the new version of our paper. We define the denseness as the amounts of reward information for the same batch from agent rollouts. Higher amounts of reward information for the same batch would be beneficial for policy training since it contains much more learning signals. And we define it as a denser reward signal. In this way, the higher entropy of reward batch during optimization indicates that our reward function is denser. 

[1] Generative Adversarial Imitation Learning, Ho et al., NeurIPS 2016.

[2] A Divergence Minimization Perspective on Imitation Learning Methods, Kamyar et al., CoRL 2020.

[3] Primal Wasserstein Imitation Learning, Dadashi et al., ICLR 2021

From: ICLR.cc/2023/Conference/Paper1734/Authors
title: Response to Reviewer vULj
comment: Thanks for your thoughtful and constructive review! We address your concerns as follows.

**Q1: the theoretical justification assumption 4.3(iii) should be stated more rigorously. What is "the distribution divergence" and how is it regularized? When is it "properly" regularized?**

A1: Thanks for your suggestion. We have revised this part in the new version of our paper. 
The distribution divergence

$$d_{\mathcal{F}(P_r, P_\theta)} = \sup_{r \in \mathcal{R}} E_{x \sim P_r}[r(x)] - E_{x\sim P_{\theta}}[r(x)] + \phi(r(x))$$

should be properly regularized to be strongly concave with respect to the reward function. Here, $\phi(r(x))$ is the regularizer. In addition, we prove that the reward function in our AEAIL is K-Lipschitz with clipping the weights of the auto-encoder. (Proof see in Appendix A.2). 
 
**Q2: How was the entropy for "Question 1" computed? Does the scaling increase the entropy?**

A2: To answer Question 1, we define the denseness as the amount of reward information for the same batch from agent rollouts. Higher amounts of reward information for the same batch would be beneficial for policy training since it contains much more learning signals. And we define it as a denser reward signal. Specifically, we compute the differential entropy for each reward batch as:

$$H(x)=-\int_{\mathcal{X}}f(x)\log f(x)dx$$
 
where $f$ is the probability density function whose support is a set $\mathcal{X}$. For computation convenience, we discrete the rewards into 100 sections to compute the probabilities. Therefore, we can get the entropy of each reward batch from agent rollouts.  The scaling (like some transfomations) won't increase the "differential entropy" if we discrete the rewards into sections enough. Of course, in our comparison, the rewards for FAIRL and our AEAIL are both at the same scale, i.e., [0, 1]. And we compute entropy with 100 sections.  
 
**Q3: Was exp(-AE(x)) in Eq. 8 restricted to the range [0,1]?**

A3: Yes. 

**Q4: How to solve the reward bias problem?**

A4: Thanks for your suggestion. Our AEAIL is a general modification for AIL methods. It views the reward function as an auto-encoder rather than the discriminator. So our AEAIL can be built on different variants of AIL methods. DAC is a good variant of GAIL as it uses absorbing states to solve the reward bias problem. In Appendix A.6, we provide more experiment results built upon DAC with absorbing states warpper. We show that our AEAIL (built on DAC) performs greatly on almost all tasks including the negative survival bias task -- Reacher. On the other hand, it also shows that our AEAIL is very general to different variants of AIL methods and can solve reward bias problem as the same as DAC.

From: ICLR.cc/2023/Conference/Paper1734/Reviewer_vULj
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a novel method for adversarial imitation learning that restricts the family of reward functions to a specific function that is monotonically decreasing in the reconstruction error of an auto-encoder (I think that "reconstruction-error" and "auto-encoder" are misnomers, as the encoder-decoder network is not trained to minimize the reconstruction error).
The method is compared to prior AIL methods (GAIL, DAC, PWIL, FAIRL) on Mujoco tasks, where achieves better performance, in particular when the expert data is perturbed by Gaussian noise. Furthermore, several ablations are conducted, for example to test a VAE-style and a GAIL-inspired objective for training the reward function.
strength_and_weaknesses: Strength:
- The paper is well written 
- The approach is novel

Weaknesses:
- The algorithm doesn't seem fully sound
- The experiments may suffer from reward bias 
clarity,_quality,_novelty_and_reproducibility: Clarity:
Overall, the paper is quite clear. However, some aspects need to be better described:
- The assumption 4.3 (iii) should be stated more rigorously. What is "the distribution divergence" and how is it regularized? When is it "properly" regularized?
- How was the entropy for "Question 1" computed? Does a scaling of the reward function increase the estimated entropy?
- Was $\exp(-\text{AE}(x))$ in Eq. 8 restricted to the range $[0,1]$?

Quality:
- The method seems purely empiric, as I can not find a convincing theoretical justification. What is the overall objective (e.g. which divergence) optimized by the method? Section 4.3 just lists a few assumption (that are partially unclear, see above) under which the method "converges globally" but it is unclear what solution it converges to.
- The experimental evaluation seems to suffer from reward bias (Kostrikov et al. 2019): The "reconstruction-error" and the resulting reward function are unable to express negative rewards. As the absorbing states (falling down in Mujoco) are assumed to have a reward of zero, the reward functions have a survival bias.

Novelty:
The approach is novel, but the contribution seems to be mainly restricted to proposing a specific model family for the reward function and using it for Wasserstein/JS-AIL.

Reproducibility:
The reproducibility seems good, as the authors provide code and details on the experiments in the appendix.
summary_of_the_review: The paper proposes a novel method for imitation learning that does not seem fully sound, but achieves good results in practice. I think the paper could be accepted in principle, however, currently the evaluation seems to suffer from reward bias, and thus, the value of the empirical results seems questionable. How would the algorithm perform on environments where the agent needs to reach an absorbing state (e.g. MountainCar?), or when learning the reward also for the absorbing states (as proposed by Kostrikov et al. 2019)?
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1734/Reviewer_EB8r
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes to use reconstruction loss as a reward function in adversarial imitation learning for solving Mujoco locomotion tasks. The idea is built on top of WGAIL/GAIL method, where an autoencoder replaces the discriminator. 

strength_and_weaknesses: **Strengths** 
Paper demonstrates that using reconstruction loss as a reward can lead to better performance than some prior methods. 

**weaknesses** 
- Overall, the idea of using reconstruction loss in RL already exists*. Therefore, the novelty of the proposed approach is limited.
-- *Loss is its own Reward: Self-Supervision for Reinforcement Learning
-- *Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
- The experimental evaluation is limited as it does not quite convince that the proposed approach results in significant improvement.
- Theoretical justification to motivate the design choice are not convincing either to support the idea of auto-encoder based surrogate reward function. The statement "The higher entropy of learned rewards means the reward signal can offer more feedback for each state-action pair, thus providing a denser reward signal to the agent policy." is given without any justification or theoretical support. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. The approach is simple and reproducible.  However, there are major concerns over the novelty/quality of the paper.
summary_of_the_review: The Paper demonstrates that reconstruction loss can be used as a discriminator in a GAIL-like framework and present performance in various Mujoco Tasks. However, the proposed approach lacks significant novelty and is also not well justified theoretically. Moreover, some of the statements are given without reference or mathematical support. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1734/Reviewer_ymvo
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: 
In this work, a version of the GAIL method is proposed where the reward in the form of the discriminator on the state-action distribution is replaced by a reward in the form of the reconstruction error of an autoencoder on state-actions. The method is evaluated in extensive simulated experiments.

strength_and_weaknesses: 
**Strengths:**

I see the motivation.
Clearly, GAIL has its weaknesses in terms of overall instabilities. And one specific problem can be the discriminator becoming "too good" (though maybe a bit more detail would be appropriate on what this means, given that, on some level of abstraction, the better the discriminator the better for the imitator; and, as far as I understand, this is more a problem of the particular training dynamics/diverging "speeds" of imitator and discriminator).

On a high level, I see that the idea of learning representations/autoencoders can help here. I see that an autoencoder can be a way to detect the "anomaly" of having a non-expert compared to an expert.

Extensive simulation experiments are conducted with mostly good performance of the proposed method.


**Weaknesses and improvement points:**


However, I think this work is too premature, in terms of writing as well as methodological contribution:


The structure, ordering and writing, in particular description of the method, need to be improved. Here are some more and some less problematic parts I see:
* I'm surprised that Wasserstein divergence/WGAN are introduced quite lengthily in sec 3 and 4, but then not really used? There is no further mentionin of it after eq (2)? That's confusing. I think if the Wasserstein GAN framework is actually not used, and the relation is merely that both have a bi-level/adversarial formulation, then Wasserstein does not have to be introduced so prominently. Btw., I recommend to also include [Xiao at al: Wasserstein adversarial imitation learning] as reference.
* Related to this: Is the Lipschitz-continuity constraint in the problem in eq. (3) dropped compared to (2)?
* I think the presentation of IL, IRL,etc., needs to be improved. I wouldn't subsume all those variants of GAIL under IRL! Because IRL is about recovering the expert's underlying reward function - assuming it "exists" - while GAIL etc. practically do not recover the expert reward (this can be seen from the fact that in equilibrium, the discriminator is always flat, and this is hardly a sensible expert reward). Overall, I would say, IL is the overarching task where demonstrator state-actions are given and have to be imitated. And then there are all these versions like BC, IRL, GAIL, … I think this is the proper conceptual background, while currently the structure and writing of intro and background (sec 3) leaves room for improvement.
* I feel that the whole intro, background and method sections jump back and forth between motivation and actual method. This is not fundamentally wrong, but a bit confusing in terms of structure.
* The CartPole balancing task is referred to without proper introduction. It needs to be introduced properly before discussing details like velocities (sec 4.1).





Method and its novelty/contribuion:
* I think an autoenconder just learns a lower-dimensional data *manifold* - and not a *distribution* (in contrast to a variational autoencoder, GAN, etc.). But, at least in some cases, this means that *all elements that in fact lie on the manifold will get the same error/reward*. But this may not be expressive enough as a reward function. In contrast, with distributional approaches, even within the manifold there is a further "weighting/ordering" between elements of the manifod.
* As a minor point, intuitively, I feel that an autoencoder should be something that learns a very general, robust representation. I feel the autoencoder/representation in this work - i.e., reconstruction error for expert states - is not robust to distributional changes etc. I think it often happens that in one instance (say, for one initialization), a state-action pair is good (taken by the expert), while in another instance it is bad (taken by non-expert). I.e., whether a state is good or bad can heavily depend on initial state, random fluctuations, etc. E.g., from one starting position in a maze, the expert may never want to visit a certain state, while from another one, the state would lie on the shortest path. It's hard for me to make this intuitive counterargument specific, and maybe GAIL suffers from similar problems, but it does seem to be a limitation.
* More positively speaking, I think it would be relevant to also conceptually better understand the relation between an autoencoder based appraoch to detect the expert versus non-expert, compared to the more classic distribution-divergence one.
* I feel sec 4.3's title makes quite a strong claim, but then there is little details on the precise conditions for the convergence guarantees. I don't say the full theory from the mentioned Guan et al is needed, but the treatment just feels a bit incomplete in terms of necessary assumptions.
* I'm wondering: if - as sec 4.3 seems to indicate - the proposed autoencoder-based method is just a special case of what has already been treated by Guan et al, this may indicate that the contribution of the present work is rather limited.
clarity,_quality,_novelty_and_reproducibility: (see above)
summary_of_the_review: The motivation in terms of unstable GAIL, as well as the high-level idea of the autoencoder-based version make sense. But I think this work is too premature, in terms of writing as well as methodological contribution.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1734/Reviewer_9Nok
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper introduces Auto-Encoding Adversarial Imitation Learning (AEAIL) for adversarial imitation learning (AIL). The framework uses the reconstruction error of an auto-encoder as the reward rather than classifier-based reward. The authors suggest that reconstruction errors provide denser rewards. Experimental results show that AEAIL outperforms baseline methods, especially when using noisy demonstrations.
strength_and_weaknesses: Strength:
1. The proposed method is clean and straightforward to implement.
2. The use of AE to overcome noisy demonstration appears robust and outperforms baselines empirically.

Weakness:
1. I can't agree with the theoretical claim that the proposed AE discriminator minimizes the Wasserstein distance.  The Wasserstein distance formulation from [1] and used in this paper requires the space of the reward function to be Lipschitz-K. However, this assumption may not hold true for AE in practical applications, as noted by the authors. For the practical attempt to clip the weights, note that [1] clips the weights of the discriminator $f$ directly. However, for AE, the function that needs to be Lipschitz-K is the actual reward $|||x - f(x)|^2$. It remains to be shown that weight clipping is effective for AE reward.

In addition, AE-based GAN is more closely associated with energy-based GAN (see the theoretical derivation in [1]), which shows that AE minimizes total variation distance, not Wasserstein distance.

2. The definition of reward denseness is unclear to me. I can't find any formal definition of it and the differential entropy in the experiment is only a "proxy" to the concept of denseness. The entropy measurement is also problematic for 2 reasons: 1) it appears that simply scaling the reward with a large constant would improve entropy for any reward function. 2) Entropy requires $p(r(s, a))$ to compute. It is unclear to me how the probability of a reward $r(s, a)$ is obtained.

3. I have several concerns over the empirical evaluation. a) Why TRPO is the only RL algorithm used for all methods, instead of more recent one like D4PG. In particular, PWIL originally used D4PG. It is well understood empirically that different RL algorithms have significant impacts on the method performance. Thus it would be preferable to select the best RL algorithm for each individual method. b) Some empirical results are hard to reconcile with previous works. For instance, the humanoid expert performance is quite low compared to previously used (5700 vs ~10000 in PWIL). PWIL was able to achieve a performance of 7000+ using 11 demonstrations in original PWIL, while this work only achieves ~3500 with 320 demonstrations. Could the authors please discuss how to reconcile the discrepancies?

[1] Wasserstein generative adversarial networks, Arjovsky et al., ICML 2017
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear and easy to follow. Some aspects of the claims are not well supported (e.g. AE yields Wassterstein distance; Reward denseness is not well defined). The method has limited novelty since most of the components may be traced to some existing works (e.g. the use of AE are similar to [1, 2, 3]). Source code is provided for reproducibility.

[1] Exploration by random network distillation, Burda et al. ICLR 2019.

[2] Random expert distillation: Imitation learning via expert policy support estimation, Wang et al. ICML 2019.

[3] Energy-based Generative Adversarial Networks, Zhao et al, ICLR 2017.
summary_of_the_review: The paper proposes an AE-based approach to AIL. While the method shows some promise for noisy demonstrations, it has limited novelty, issue with its theoretical interpretation and discrepancies with previous empirical results. Substantial revision may thus be needed for the current manuscript.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold