Paper: PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward (ICLR-2024)

Reviewer comment (Theory):
On the 5th line from the bottom, please confirm whether the direction of the inequality sign is correct.

Author response:
The sign is correct, when $U_{r^-}(E)-max_\pi U_{r^-}(\pi) > U_{r^+}(E) - max_\pi U_{r^+}(\pi)$, reward function $r^-$ will be more 'optimal' than $r^+$. 
We use this inequality to show that we must consider the case where a sub-optimal reward function can align with the task while the optimal reward function is misaligned.

Reviewer comment (Experiments):
I'm curious if the antagonist policy can be a non-optimal policy. In Figure 2 (b), VAIL does not appear to have enough successful episodes to qualify as an optimal policy and serve as a proxy. Would it be acceptable for the policy to have non-zero returns instead of being fully optimal?

Author response:
In theory, the antagonist policy has to be optimal, while in practice, we have to train the antagonist policy together with the protagonist policy from scratch.
The fact that VAIL alone not being able to produce successful episodes does not directly relate to whether running VAIL in the PAGAR framework can produce better results.   
This is because the reward function in the PAGAR framework is optimized to increase the protagonist antagonist induced regret in Eq (2). This optimization incentivizes searching for a reward function that can makes the antagonist policy outperforms the protagonist policy.
Given that inverse RL can have multiple solutions -- multiple reward functions aligned with the demonstrations, PAGAR-based IL avoids those reward functions that are difficult for any policy to achieve a high utility, since this will result in both the antagonist and the protagonist policy having low performance.

Reviewer comment (Theory):
I'm not fully convinced of the motivation for the method. The paper argues that existing IL/IRL methods may fail in optimizing the underlying task because they learn a policy that optimizes a misaligned reward function. However, I do not think that the typically used problem formulation suffer from such problems. For example IL methods are often formulated as divergence minimization problems, and assuming that they succeed in bringing the divergence to zero, they would perform as well as the expert on any reward function. Of course, in practice we will typically not achieve zero divergence due function approximations, limited data and potentially unstable optimization, but I don't think that these challenges are specific to the problem formulation and they affect the proposed method in similar way. For some IRL methods, it was even shown that the optimal policy for the learned reward function may outperform the (suboptimal) expert, in the sense that it performs no worse but potentially better on every reward function within the hypothesis space. With that in mind, I also find Example 1 quite misleading. The example argues that IRL would lead to a bad policy in a simple toy problems, by arguing that a constant reward function would solve the IRL problem. However, already Ng & Russell [2000] noted, that such IRL formulation would be illposed, and additional constraints / objectives are required. For example, MaxEnt-IRL would maximize the likelihood for a MaxEnt-policy, where a constant reward function (leading to a uniform policy) would certainly not be a solution in the toy problem.

Author response:
Maximum entropy is a heuristic used to reduce reward ambiguity and not reward misalignment. 
In fact, MaxEnt cannot eliminate reward ambiguity even if we have infinite demonstrations ([Skalse. et al., 2022], which we cite in the first paragraph of Introduction).

Reward ambiguity and reward-task misalignment are two different concepts.  
Reward ambiguity is the problem of many different reward functions in the reward hypothesis space can explain the expert demonstrations equally well.
On the other hand, reward-task misalignment 
is the problem of policies that achieve a high utility under a reward function failing to finish the task (the well-known notion of reward hacking is a manifestation of this [Pan. et al., 2022], which we cite at the end of first paragraph).

Maximum entropy likelihood may lead to a policy that is highly probable in reproducing the demonstrated trajectories. 
It is equivalent to assuming that the underlying task is simply to generate the demonstrated trajectories with the highest probability.
Our work does not make such an assumption. As shown by our definition of $R_{E,\delta}$, we include not only the optimal but also the sub-optimal solutions of inverse RL as candidate reward functions. 

>  If the agent reproduces the behavior of the expert and we assume that the expert solves the task, the agent will also solve the task.

A policy being able to reproduce the demonstrated trajectories does not imply the policy is ready for the task. In a stochastic environment, any stochastic policy can reproduce the demonstrated trajectories ... We have reiterated that the outcomes of individual attempts or instances do not solely determine success. If by `reproduces the behavior` you mean to recover the expert policy, the problem is that IRL can fail to recover expert policy for so many reasons ... We have cited the literature in the paper and in our rebuttal.

> This claim is not substantiated and wrong ...

Our claim is substantiated and correct. We recommend revisiting `Ziebart, Brian D., et al. Maximum entropy inverse reinforcement learning` and searching for the keyword 'likelihood'. It is beyond doubt that max-ent IRL determines the reward weight by maximizing the likelihood of the max-ent policy generating the demonstrated trajectories. Max-Ent principle is not the only factor at play.

> The underlying task would always be to maximize the reward function.

This does not contradict our argument. The policy is trained with a reward function that is learned based on a maximum likelihood objective function.

> ... at least if we assume the expert policy to be in the hypothesis space

Our paper never made such an assumption at all.
It is common knowledge that IRL can be affected by various factors, including a limited number of expert demonstrations, expert policy not being in the hypothesis space, learning reward function in MDPs with different dynamics ... 
And our method is to mitigate the reward misalignment issue in challenging and varied conditions.
While Theorem 3 implicitly covers the ideal condition where IRL can reach Nash Equilibrium, all other theories do not account for the ideal condition.
Our experiments in MiniGrid aim to exhibit PAGAR's ability when those factors are present.

Reviewer comment (Theory):
The conditions used in Theorem 1 are not satisfiable if the hypothesize space of the reward function includes a reward function for which there is no gap between the best performing failing policy and a succeeding policy. For example, if a task is considered to succeed when the distance to a goal is below a certain threshold, and the reward uses a quadratic penalty on the distance to the threshold as cost, we can get arbitrary close in terms of reward to a succeeding policy while still failing. Potentially related to this, it not clear to me how the method optimizes with respect to a distribution $\mathcal{P}\_{\pi}(r)$ over reward functions. Where does this distribution appear in Algorithm 1?

Author response:
Firstly, in Definition 1, the intervals $S_r$ and $F_r$ are defined such that a policy's utility falling within $S_r$ or $F_r$ is deemed a sufficient but not necessary condition for determining success or failure. Put differently, if a policy $\pi$ achieves a utility in $S_r$, it is guaranteed to be successful, and if $\pi$ achieves a utility in $F_r$, it is guaranteed to be a failure. Note that $S_r$ and $F_r$ are not required to encompass all successful or failing policies' utilities. These intervals may also be very small for certain task. In the extreme scenario, $S_r$ could be a singleton, representing the maximum achievable policy utility under $r$, if the optimal policy under $r$ can successfully complete the task. Meanwhile, $F_r$ can be empty.
The impact is that if all reward functions in the set $R$ have very small $S_r$ and $F_r$ intervals, $MinimaxRegret$ may have to achieve a low regret to produce a successful policy.

Also, the conditions about the width of S and F intervals are sufficient conditions for $MinimaxRegret$ to learn a successful policy. Suppose all the reward functions in the candidate reward function set are task-aligned. Then, even if they all match the description 
 mentioned by the reviewer, i.e., `no gap between  $S_r$ and $F_r$`, $MinimaxRegret$ can still induce a successful policy in the task. 
> Proof
   * There must be a policy $\pi^+$ that makes $Regret(\pi^+, r)\leq|S_r|$ hold for all reward function $r$'s. Because its utility cannot be in the S interval under a reward function and in the F interval under another.
   * For any policy $\pi^-$ that fails the task, $Regret(\pi^+, r)>|S_r|$ holds for all reward function $r$'s. Because if there exists an $r$ under which $Regret(\pi^-, r) \leq  |S_r|$, $\pi^-$ is guaranteed to succeed in the task, contradicting the assumption. 
   * $MinimaxRegret$ will pick $\pi^+$ over $\pi^-$ as its solution because $Regret(\pi^+, r)<Regret(\pi^-, r)$ under all $r$'s

> ## ... Potentially related to this, it not clear to me how the 

Table 1 shows an equivalent objective function of $MinimaxRegret$
It only describes the analytical solution of PAGAR. 
Our approach to PAGAR is not based on Table 1.

Reviewer comment (Theory):
Section 4.2 defines a set $R\_{E,\delta} =  \\{ r \| U\_{r}(E) - \underset{\pi \in \Pi}{\max} U\_{r}(\pi) \ge \delta \\}$ where $\delta$ is chosen to be strictly positive. Wouldnt this set be empty if the expert policy is in $\Pi$?

Author response:
We mentioned after Corollary 2 that $\delta$ cannot be larger than the negative minimal loss of inverse RL. 
If the minimal loss of inverse RL is non-negative, i.e., the expert policy is in the policy set, $\delta$ will be non-positive.

Reviewer comment (Theory):
I am wondering how the success of a policy or its performance can be deterministic, given that both policy and environment are stochastic.

Author response:
The 'success' here indicates the ability of the policy to finish the task, not whether the policy finishes the task in a single run.
For instance, a success criterion can be "the probability of reaching the state must be higher than 'p'". 
If a reward function is well-designed, the success of this policy can be guaranteed if the policy's expected return exceeds a certain threshold.

Reviewer comment (Experiments):
The transfer experiment is not clear to me. Is the learned policy transferred to the new environment, or is it learned from scratch in the new environment, using demonstration from the source environment?

Author response:
The demonstrations are sampled in one environment, Algorithm 1 (reward learning, policy learning) is conducted in another environment.

Reviewer comment (Experiments):
The experimental results in the continuous domain are not convincing. For example, IQ-Learn should clearly outperform GAIL in these environments. The paper states that it should perform better if a bigger replay buffer was used, but then I wonder why no adequate hyperparameter tuning was performed for the comparisons.

Author response:
In our continuous control experiments, the x-axis represents policy update iterations, not policy roll-out steps. 
IQ-learn's implementation is based on soft-actor-critic which is an off-policy algorithm that conducts several policy updates between consecutive interactions with the environment.
The size of the replay buffer allocated for IQ-learn is consistent with the sample size used for PPO which is the RL algorithm used in our implementation.

Reviewer comment (Writing):
The paper does not answer the question of how the set R (under which reward optimization is performed) is chosen in the RL setting, instead redirecting to the IL setting. It is unclear whether PAGAR is meant to be applied in the RL setting or if the selection of R is prohibitively difficult and thus PAGAR as described is only practically applicable to IL. This should be made more explicit.

Author response:
We would like to clarify that 
Section 4.1, "RL with PAGAR", is about examining the conditions for $R$ under which $MinimaxRegret(R)$ can generate a policy to avoid task failure or guarantee task success. 
It does not deal with how to construct such an $R$ in an RL training scenario.
It is meant to support the main objective of the paper, which is mitigating the reward alignment problem in IRL-based IL, since IRL-based IL involves inferring a reward from the demonstrations and performing RL with the inferred reward.  

Applying PAGAR to IRL-based IL is equivalent to letting $R$ be the solution set of IRL. 
In this context, human demonstrations can be viewed as a supervision signal that help us identify $r$ that satisfies the conditions in Section 4.2 which are extensions of the conditions in Section 4.1 for a setting with expert demonstrations.

Reviewer comment (Reproducibility):
In the IL setting, it seems that generation of the R_E,δ set is difficult, but the paper does not discuss how to obtain such a set. Example 1 does not provide intuition for the basis functions chosen to parameterize R_E, making it difficult to understand. It is unclear whether the discussion at the bottom of page 5 is meant to be a construction of R_E or just a non-constructive description.

Author response:
We do not directly construct this set; rather, we utilize the aforementioned conditions to restrict the exploration of $r$.  
Our definition of $R_{E, \delta}$ is that for every $r\in R_{E,\delta}$, the difference between the optimal utility and the expert demonstration utility is within $\delta$. 
This difference equals the negative IRL loss under $r$. 
We can enforce selecting $r$ from $R_{E,\delta}$ by restricting the inverse RL loss of $r$ to be no greater than $-\delta$.

We mention in the paragraph above Example 1 that $\delta$ cannot be larger than the negative minimal inverse RL loss.
When $\delta$ reaches its maximum, $R_{E,\delta}$ equals the optimal solution set of inverse RL.
In this case, we simplify the notation as $R_E$.
We then use Example 1 to show that in some instances, inverse RL treats the entire reward space $R$ as its optimal solution set, in other words, $R_E=R$.
The basis function is picked to illustrate the misalignment problem in inverse RL, as the choice of the basis functions can be arbitrary.
However, by applying PAGAR, we can identify the ground-truth reward function.

Reviewer comment (Experiments):
Experiments are only conducted with a very limited number of demonstrations (10), which is not realistic for real-world deployment where alignment is important. It is unclear whether this limitation is because providing more demonstrations would reduce misalignment and thus the need for a complicated method like PAGAR.

Author response:
The effective utilization of a limited number of demonstrations is widely recognized as a crucial metric for evaluating Inverse Reinforcement Learning (IRL) and Imitation Learning (IL) algorithms in the literature. It is often used to asess the performance of these algorithms, as demonstrated in notable works such as the GAIL paper [Ho. et al., 2016], the AIRL paper [Fu. et al., 2018], and various other references cited in our study.
In GAIL [Ho. et al., 2016], which is a seminal work in imitation learning with generative adversarial networks, the algorithm demonstrates the ability to generate high-performance policies with just a single demonstration. 

As we mentioned in our introduction, a reward function aligning with the expert demonstrations does not imply aligning with the underlying task.
These two concepts are orthogonal.
Notice that the definition of Task-Reward Alignment (Definition 1) is independent of demonstrations.
More demonstrations may help reduce reward ambiguity (i.e. multiple demonstration-aligned rewards), which, however, cannot be eliminated even with an infinite number of data [Ng & Russell (2000); Cao et al. (2021);
Skalse et al. (2022a;b); Skalse & Abate (2022)].

Reviewer comment (Experiments):
Is PAGAR meant to be applied in the RL setting or is the selection of R prohibitively difficult and therefore PAGAR as described is only applicable in practice to IL?

Author response:
We would like to clarify that 
Section 4.1, "RL with PAGAR", is about examining the conditions for $R$ under which $MinimaxRegret(R)$ can generate a policy to avoid task failure or guarantee task success. 
It does not deal with how to construct such an $R$ in an RL training scenario.
It is meant to support the main objective of the paper, which is mitigating the reward alignment problem in IRL-based IL, since IRL-based IL involves inferring a reward from the demonstrations and performing RL with the inferred reward.  

Applying PAGAR to IRL-based IL is equivalent to letting $R$ be the solution set of IRL. 
In this context, human demonstrations can be viewed as a supervision signal that help us identify $r$ that satisfies the conditions in Section 4.2 which are extensions of the conditions in Section 4.1 for a setting with expert demonstrations.

Reviewer comment (Theory):
Is the discussion at the bottom of page 5 meant to be a construction of R_E or just a non-constructive description?

Author response:
This discussion is about the formula in Table 1, which is an alternative of $MinimaxRegret$. 
It is equivalent to $MinimaxRegret$ in the sense that it induces the same result as $MinimaxRegret$. 
It is meant to provide more insights about PAGAR.
It is not a constructive description.