Paper: Converging and Stabilizing Generative Adversarial Imitation Learning (ICLR-2024)

Reviewer comment (Novelty):
Many lemmas and theorems have appeared in existing work. Lemmas A.1 and A.2 follow from the original GAIL work and the concept of entropy-regularized MDPs. This connection should be clarified.

Author response:
Yes, lemma A.1 and A.2 follow the original GAIL work; we use it to derive the training dynamic of GAIL

Reviewer comment (Experiments):
Experiment results are weak. Many strong baselines, such as the mentioned DAC and IQ-learn [9], are not compared. Thus, the significance of improvement of GAIL is unclear.

Author response:
We finished the experiment on DAC on four expert trajectories with a faster convergence speed with our controller. We will include the result in our final submission

Reviewer comment (Writing):
The paper should indicate that the policy aims to minimize the cost. In particular, the problem setting is not well outlined—it should be said explicitly what assumptions we are making—Markov Decision Processes? Infinite horizon? Finite state-action spaces?

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
- We assume our work under infinite horizon and infinite state-action space

Reviewer comment (Theory):
Under the proof of theorem 4.2, what is the trace a function of, since \( z^* \) is already set a priori?

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
- The trace function is for the Jacobian of the dynamic system near the equilibrium point

Reviewer comment (Theory):
Under section 4.4, in practice the paper only considers the controller for discriminator. Then the theory appears to not really matter since the practical algorithm does not address the stability of \( \pi \)? But, we should still expect the discriminator to converge faster with the controller?

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
- In theory, we consider controlling both the discriminator and the generator, and in experiments, we only add the controller on the discriminator, and the entire training dynamic converges faster

Reviewer comment (Reproducibility):
The sensitivity analysis seems to be tested on a single seed—how confident are we in terms of replicating this result?

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
- The table we present is under three different seeds

Reviewer comment (Writing):
Definition 3.5: What is \( x_e \)?

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
- $x_e$ is a typo and it should be $\bar{x}$

Reviewer comment (Writing):
On page 1, introduction, line 5: "includes" instead of "include" and "behavioral" instead of "Behavioral".

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Section 2: "as a Dynamic" instead of "as Dynamic".

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Page 4, corollary 3.8: "eigenvalues" instead of "eigenvalue".

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Section 4, paragraph 1: "state-action" instead of "state action".

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Section 4.1, paragraph 2: Eq. 8 instead of Eq.8.

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Section 4.1, third paragraph: def. 3.1 instead of def.3.1.

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Page 5, line 1: "state \( s \) at step \( k \)".

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Page 5, paragraph 2, line 1: "described by Eq. 4 and Eq. 5".

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Writing):
Page 5, two lines after Eq. 17: Eq. 9 instead of Eq.9.

Author response:
Dear Reviewer 4PYc,

Thank you for your helpful thoughts. We respond to your questions below.
We also corrected those typos you have pointed out.

Reviewer comment (Experiments):
**Limited Empirical Evidence**: Although the paper mentions evaluating their approach on MuJoCo tasks, the evaluation is basically based on reward from the learned policy. Considering that the proposed solution tries to control and drag the learned policy to the expert's, it might be more informative to show a comparison between the ground-truth policy and the learned one. In addition, the authors might want to add more experiments with varying hyperparameters, number of expert demonstrations, etc to make the experiment complete.

Author response:
We conducted our experiments on GAIL-DAC, and our controller converges the training of GAIL-DAC faster. We will include our results in our final submission. We will also include more trials with different hyperparameters and various numbers of trajectories.

Reviewer comment (Theory):
**Potential Assumptions**: It is unclear why in eq.~(30), the objective omits the parts except for $V_2(D,\pi)$. This implementation is inconsistent with Assumption 4.1, where $\alpha$ is a real number, and $c$ represents the state distribution. The authors did not shed light on this implementation.

Author response:
In our theoretical analysis, we design our controller for both the policy generator and the discriminator. However, as we mentioned in section 4.4, we are unaware of the expert policy, so in the evaluation section, we only add the controller for the discriminator.

Reviewer comment (Writing):
**Writing & Notation**: The writing of this paper is not very clear. It would be better if the authors refine the notations and define them before using, for example, the $V^\prime$ in eq.~(29) and (30).

Author response:
Yes, we noticed that we have some typos and unclear notations in our submission. We will correct them in our final submission.

Reviewer comment (Experiments):
Will the performance change with different $\alpha$ values?

Author response:
Our experiments are conducted with the controller only on the discriminator, so the $\alpha$ value is irrelevant here

Reviewer comment (Evaluation):
Will adding the controller terms make the model more computationally costly to train?

Author response:
The controller term does not make the model more computationally costly to train. Notice that Figure 1 is plotted with respect to time.

Reviewer comment (Experiments):
Will the number of expert demonstrations change the performance?

Author response:
Our controller works with a different number of expert demonstrations. For example, the GAIL-DAC we include later is for only four expert demonstrations.

Reviewer comment (Novelty):
Ideas and derivations follow Luo 2023a. The proposed approach seems a straightforward extension of Luo 2023a to GAIL. I couldn't find out either theoretical or empirical challenges from the paper when Luo 2023a's idea (for GAN) was applied to GAIL.

Author response:
This work is inspired by Luo 2023a. However, GANs and GAIL have different training dynamics in terms of differential equations, so the BMC for GANs cannot be applied to GAIL. In our work, we design a new controller concerning the training dynamic of GAIL.

Reviewer comment (Experiments):
Empirical studies comparing Controlled GAIL and original GAIL should be improved. No hyperparameter ablation study was given, simply following the default setting of Gleave et al., 2022, which makes the empirical study in the paper unreliable.

Author response:
We also conduct our experiments on GAIL-DAC, and our controller converges the training of GAIL-DAC faster. We will include our results in our final submission.

Reviewer comment (Writing):
Readability and overall organization of writing need to be improved. Details will be mentioned below.

Author response:
We also thank you for pointing out our writing typos and ways to improve our readability; we will correct them in our final submission.

Reviewer comment (Theory):
We normally consider a discrete step $t$, but the training dynamics considers derivatives w.r.t. $t$, which implies that $t$ is continuous. What is the motivation behind, and what is the rationale to use the continuous $t$?

Author response:
The analysis is based on a continuous-time setting. However, in reality, the training dynamic should be discrete. We point out this difference in our discussion section

Reviewer comment (Writing):
Remark 3.2.,  if it does not exist an equilibrium --> if an equilibrium does not exist.

Author response:
We also thank you for pointing out our writing typos and ways to improve our readability; we will correct them in our final submission.

Reviewer comment (Writing):
Definition 3.5., $x_e$ --> $\bar{x}$.

Author response:
We also thank you for pointing out our writing typos and ways to improve our readability; we will correct them in our final submission.

Reviewer comment (Theory):
Where do we use Theorem 3.7 and Corollary 3.8?

Author response:
We use theorem 3.7 and corollary 3.8 to prove that our controlled dynamic system is asymptotically stable in theorem 4.2

Reviewer comment (Theory):
In Section 4.2., Assumption 4.1., what are the intuitive meanings for these assumptions? Also, are these assumption practical? For empirical studies, what are important factors? While $k$ was ablated in the experiments, how are other variables ($k$, $\alpha$) affecting the results?

Author response:
Assumption 4.1 guides the relationship between the hyperparameters we introduced. The assumption is practical since we tried to plot it in a graphic calculator with different values of $c$. In experiments, we omitted $k, \alpha$.

Reviewer comment (Experiments):
In Section 4.4., authors mentioned "In practice, we are unaware of the expert policy for the generator’s controller, so in the evaluation section, we only include the controller for the discriminator." Would you elaborate this explanation?

Author response:
Since the policy generator's equilibrium point is unknown, we cannot directly apply the controller for the generator to our experiment. Instead, we include the controller for the discriminator and achieve a faster rate of convergence in practice.

Reviewer comment (Experiments):
In Section 5. (Experiment), how many runs were considered? Would you share more details about Figure 1?

Author response:
In section 5, figure 1 is generated by taking the average of 3 runs with respect to the training time, and table 1 is generated by taking the average of 3 seeds recorded at the end of training.