Paper: EvIL: Evolution Strategies for Generalisable Imitation Learning (ICLR-2024)

Reviewer comment (Novelty):
The literature review is insufficient. For example, the structure of EvIL is similar to the framework of learning and evolution. Elfwing et al. (2018) optimize hyperparameters of RL by a simple evolutionary algorithm in the outer loop while the policy is trained by RL. In Nielum et al. (2010), the reward function is optimized by genetic programming. The relation between EvIL and these classical studies should be discussed.

Author response:
The papers mentioned have been added to the background under the Evolution Strategies subsection (2.1). While Niekum et al. (2010) do utilize evolution strategies to train a reward function, their objective is to enhance RL performance. In our approach, this is not feasible due to the absence of a reward signal. Similarly, Elfwing et al. (2018) also optimize for a reward function, but our framework operates under the assumption that no reward function is available. Evolved Policy Gradient (Houthooft et al., 2018) employs a similar setup with an ES outer loop optimizing for rewards  in the inner loop. However, the optimization targets differ between the outer and inner loops in both approaches. To summarise, our approach is fundamentally different for all the papers above as we don’t assume access to a reward function signal we can optimise, but instead we optimise for a cross entropy loss against the expert’s trajectories.

Reviewer comment (Novelty):
A more detailed survey on meta-learning is needed. For example, Evolved Policy Gradient (Houthooft et al., 2018) optimizes a differentiable loss function, where a similar evolutionary strategy is used in the outer loop.

Author response:
The papers mentioned have been added to the background under the Evolution Strategies subsection (2.1). While Niekum et al. (2010) do utilize evolution strategies to train a reward function, their objective is to enhance RL performance. In our approach, this is not feasible due to the absence of a reward signal. Similarly, Elfwing et al. (2018) also optimize for a reward function, but our framework operates under the assumption that no reward function is available. Evolved Policy Gradient (Houthooft et al., 2018) employs a similar setup with an ES outer loop optimizing for rewards  in the inner loop. However, the optimization targets differ between the outer and inner loops in both approaches. To summarise, our approach is fundamentally different for all the papers above as we don’t assume access to a reward function signal we can optimise, but instead we optimise for a cross entropy loss against the expert’s trajectories.

Reviewer comment (Experiments):
The scalability of the algorithm is a concern. Since a naive evolutionary strategy is adopted to optimize the reward and transition functions, applying the proposed method to problems with a high-dimensional state space is problematic.

Author response:
While we acknowledge that our current experiments do not exhibit strong scalability, it is important to note that IRL is typically evaluated on MuJoCo environments, which have a smaller observation space compared to some of the environments we employed. Within these environments, we achieved performance parity with the AIRL baseline.

Reviewer comment (Experiments):
Figure 1 seems interesting, but the experimental conditions are unclear. The first paragraph of Section 4.1 mentions that the reward function optimized in the training environment is used to train policies in the test environment. However, Figure 1 shows that the reward and transition functions are optimized in the test environment. Clarification of the motivation for the experiments is needed.

Author response:
The right-side plots in Figure 1 illustrate how the reward function generalizes to the test environment during optimization on the training data. Notably, the reward function is never optimized on the test data or environment; rather, we solely evaluate its performance there.

Reviewer comment (Evaluation):
Figure 1 shows that the performance of EvIL is worse than that of AIRL in the training environment, but AIRL failed to improve its performance in the test environment. Does this imply that the initial reward is tuned for the training environment?

Author response:
The original reward function that both AIRL and EvIL strive to recover remains the same. However, the reward function recovered by EvIL exhibits superior generalization to the test environment.

Reviewer comment (Evaluation):
Discussions about Figure 2 are needed. There seem to be no significant differences among EvIL, EvIL with supervised loss, and EvIL with supervised model. In addition, the performance in the online setting is much better than those of other methods, implying that EvIL failed to find a good transition function even in the Poitmass environment. The quality of the optimized transition function should be discussed in detail.

Author response:
The plot demonstrates that EvIL, when combined with some knowledge of the reward function (as is often the case in real-world problems), can enhance performance in the offline setting. This is shown in Figure 2a. 
In Figure 2b, we show the performance of Offline Evil, here the transition function is learned only through Evolution Strategies and a Cross Entropy loss between the trained agent and the expert trajectories. This suggests our ability to recover transition dynamics from the data. 
In Figure 2d, we show the performance of EvIL where the dynamics model is learnt through supervised loss.
Figure 2f, where we show the performance of Online EvIL, is not directly comparable to the other plots, as the optimisation process has access to more information (i.e. the environment).
Offline EvIL did not match the performance of the expert (and Online EvIL) as it was only trained on a single expert trajectory. We have additional results on the Reacher environment (with more expert data) that intuitively demonstrate improved performance as the number of expert trajectories increases. The results have been added to the Appendix.

Reviewer comment (Reproducibility):
In Figure 3, it is unclear how features such as the obstacle position are retrieved from the reward function. The understanding is that the reward function is represented by a neural network, which maps a state to a scalar value. Clarification of the method used to retrieve the reward features is needed.

Author response:
In the experiment depicted in Figure 4, we employ a partially known transition function and aim to uncover the remaining unknown variables. Our results demonstrate the successful recovery of these unknown variables, including obstacle position and size. Therefore it is the transition function, rather than the reward function, that serves as the source of this information.

Reviewer comment (Writing):
Some acronomys are not defined, e.g. AIRL (adverserial inverse reinforcement learning). What are its training setup?

Author response:
We have addressed the reviewer's suggestions by including the BC hyperparameters in the appendix, defining the acronyms, and providing detailed descriptions of the environments. 
AIRL stands for Adversarial Inverse Reinforcement Learning, which is essentially equivalent to Generative Adversarial Inverse Reinforcement Learning (GAIL). In AIRL/GAIL, there are two players: a discriminator (used as reward function) and a player. The reward function player tries to pick out differences between the current learner policy and the expert demonstration, while the policy player attempts to maximise this reward function to move closer to expert behaviour. This setup is effectively a GAN. For a more theoretical explanation, please refer to [1].

Reviewer comment (Experiments):
Approach should be evaluated on more environments and potentially complexer ones. Is the approach generally better to BC or does it depend on the type of environment?

Author response:
We are currently in the process of expanding our method to MuJoCo tasks by changing environment parameters like link lengths and joint stiffness to generate a diverse range of dynamics. This approach generally surpasses BC due to its foundation in Inverse Reinforcement Learning (training the agent in the inner loop) instead of simply imitating the expert's actions. BC has been shown to suffer from compounding errors and poor generalization.

Reviewer comment (Experiments):
It would be useful to test how transferable the learned reward functions are to similar domains

Author response:
The transferability and generalizability of the recovered reward function is what we are testing for in the test version of our environment (e.g. different joint dampening in Reacher and a maze vs no-maze setting in the Gridworld)

Reviewer comment (Reproducibility):
Hyperparaemters for some of the experiment steps don’t seem to be mentioned (e.g. learning rates, etc.?)

Author response:
We have addressed the reviewer's suggestions by including the BC hyperparameters in the appendix, defining the acronyms, and providing detailed descriptions of the environments.

Reviewer comment (Experiments):
The environments are not described in the paper. Details should at least be added to the appendix.

Author response:
We have addressed the reviewer's suggestions by including the BC hyperparameters in the appendix, defining the acronyms, and providing detailed descriptions of the environments.

Reviewer comment (Reproducibility):
What are the parameters for the BC training?

Author response:
We have addressed the reviewer's suggestions by including the BC hyperparameters in the appendix, defining the acronyms, and providing detailed descriptions of the environments.

Reviewer comment (Experiments):
Since the paper only uses experiments to demonstrate the method's effectiveness, I believe that the experiments are too few and too simple. Usually, when someone talks about the generalization of RL regarding the transition function, I will think of more challenging and practical modifications like different gravity, mass, and friction.

Author response:
We are currently engaged in expanding our method's applicability to MuJoCo tasks, where we intend to introduce variations in environment parameters such as link lengths and joint stiffness to create a diverse range of dynamics.

Reviewer comment (Experiments):
The method is very slow since it involves multiple rounds of training.

Author response:
The iterative nature of our method, involving multiple rounds of training, aligns with standard practice for IRL methods. Both primal IRL approaches like GAIL/AIRL and dual IRL techniques like MaxEnt IRL are iterative procedures. Additionally, our method is not slower than standard IRL approaches, but is comparable due to the massive parallelization we are able to achieve via JAX. As an example, the PureJaxRL library archives a 1000x speed up over other standard PyTorch RL implementations.

Reviewer comment (Novelty):
The authors do not clarify why we should use their design since the method is not the most straightforward way to solve the problem of previous work. An easy way to focus on only one scenario first and talk about why we need the method in that scenario (e.g., online).

Author response:
As you rightly pointed out, our method's strength lies in its ability to optimize non-differentiable reward functions through a non-differentiable optimization process. Moreover, we emphasize behavioral prediction outside the training distribution and demonstrate our method's superior performance compared to standard AIRL. While AIRL performs satisfactorily within the training environment, our experiments highlight our ability to achieve parity with traditional methods while expanding the range of applicable reward model families. For instance, we can optimize for symbolic expressions of environment features, as demonstrated in the recent EUREKA paper [1]. Additionally, we show better performance in the test environments, indicating rewards recovered by EvIL generalize better.

Reviewer comment (Experiments):
It seems like in expert demonstrations you only use one kind of transition function is that correct?

Author response:
Yes, the expert trajectories are indeed sampled from a single environment.