Paper: Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR-2024)

Reviewer comment (Theory):
I have found a handful of math errors throughout the paper, especially in Appendix C, which critically degrades the quality of such a theory-driven paper. While many of them can be simple typos, some may actually require careful consideration.

Author response:
We thank the reviewer for the detailed and thoughtful review. We have uploaded a revision with all the suggested typos corrected. The changes are marked in red. Please find the clarification to the queries below:

Reviewer comment (Theory):
There is a typo in equation (35). The second term inside $f^*$ should be $\gamma \sum_{s', a'} p(s' \mid s, a) \pi(a' \mid s') Q(s', a')$, not $\gamma \sum_{s'} p(s' \mid s, a) \pi(a \mid s') Q(s', a')$.

Author response:
We made the changes in the equations above but missed the final equation. Thanks for pointing this out and is corrected now in the revised paper.

Reviewer comment (Presentation):
It is unclear what the left-most figure in Figure 1(a) represents and how it is related to the squared distribution gap plot right next to it.

Author response:
We have updated the caption in the revised paper as well as included a detailed explanation of the experiment in Appendix H.3. Please let us know if you need more clarification.

Reviewer comment (Theory):
Proposition 4: The authors write that CQL can be cast as a dual-Q problem with a right choice of f-divergence (Pearson). However, in the proof of Proposition 4 in Appendix D.1, Eq (70) actually corresponds to the objective of ATAC [Cheng et al. 2022], not CQL. See Eq (1) in [Cheng et al. 2022], which is exactly the same as Eq (70) here. To my understanding, the main difference between CQL and ATAC is that ATAC takes a maxmin formulation, whereas CQL doesn't, though they have a similar objective for learning Q. The dual-Q formulation here is a maxmin problem. The authors should update Proposition 4 to make the connection to ATAC, which would be more appropriate and precise.

Author response:
**Proposition 4: The authors write that CQL can be cast as a dual-Q problem with a right choice of f-divergence (Pearson). However, in the proof of Proposition 4 in Appendix D.1, Eq (70) actually corresponds to the objective of ATAC [Cheng et al. 2022], not CQL.**

We appreciate reviewer's suggestion of expanding the family of algorithms unified by dual-RL by including ATAC[1]. We have updated Proposition 4 and it's proof to point out the connection to ATAC. 

We believe Conservative Q-learning (CQL) and ATAC share some similarities in their objectives, which makes it hard to discard CQL from this family.  The reviewer mentions that CQL does not formulate their objective as a min-max game. However, we believe this is not completely correct. Directing the reviewer's attention to Equation 3 in CQL, a min-max game is proposed between an action-proposal distribution $\mu$ and the Q-function. The authors point out that $\mu$ can be taken as the last policy iterate (also used in their implementation for high-dimensional action spaces) making it particularly close to the objective coming out of dual-RL. A straightforward difference between CQL and ATAC is the reliance on Gradient-Descent Ascent vs the Stackelberg game strategy of updating the two-player game. We have included both ATAC and CQL in Proposition 4 in the revised paper.

Reviewer comment (Theory):
Bug in the proof of Theorem 1: In the block starting "Imitation from Arbitrary data (dualQ)" after Eq (108), max_d is moved inside min_Q, whereas max_d is outside min_Q in Eq (108). Thus Eq (109) is not just a summary, and the statement "we summarize the result of the derivation so far" can be misleading. This can be easily fixed by applying strong duality before moving forward. Additionally, it would be good to write out the proof of Eq (112) for completeness.

Author response:
**Eq (109) is not just a summary, and that "we summarize the result of the derivation so far" can be misleading. But this can be easily fixed though, by applying strong duality before moving forward.**

Thanks for pointing this out. We have made the correction.

Reviewer comment (Theory):
Validity of Eq (10): In the proof, the authors assume d_0 = d_E. This is a fairly strong assumption, as it implies that the initial state at test time would be drawn from the same distribution as the expert. This is rarely the case in practice and is not a common assumption in offline or off-policy RL. The authors should highlight this assumption in the main paper, or present Eq (140) as the main result. A minor bug in the proof is that the derivation misses a 1/4 factor in front of the square loss.

Author response:
**Validity Eq (10). In the proof, the author assume $d_0 = d_E$. I think that this is a fairly strong assumption, as it implies that the initial state at the test time would be drawn from the same distribution of the expert.**

We think there might be a misunderstanding here. We do not assume $d_0=d^E$ but rather assume $d_0=d^S$. We clarify that Eq.10 is a simplified objective and the full derivation under $\chi^2$ can be found in Appendix E.3. In the full derivation we downweigh the $Q$-function on states obtained from mixture distribution of offline data ($1-\gamma$) and expert data ($\gamma$), and the actions obtained from current policy. In practice, the simplified objective contains the offline data term only as our offline data is a superset of expert data and $\gamma$ is close to 1.

The offline distribution $d^S$ is a wider distribution encompassing the starting state $d_0$ distribution. This strategy allows us to learn a near-optimal policy from a wider initial state distribution and should not be detrimental to function representation with sufficient capacity. A similar strategy is common in (eg. [2], off-policy RL [3],[4] ). 

We have corrected Eq.10 with the missing constant of 1/4 and also updated the sentence before Eq.10 to say that we are presenting a simplified objective here.

Reviewer comment (Experiments):
The experiment design of offline IL can be confusing. It is stated that the agent is given 1 expert demonstration, but the suboptimal transition dataset actually contains 200 expert demonstrations, though they are not marked as expert demonstrations to the algorithm. A more convincing experiment would be to completely remove the expert demonstrations from the suboptimal data. Additionally, Table 2 lists 1000 expert transitions — is that the same as one expert trajectory?

Author response:
**The experiment design of offline IL can be a bit confusing. It says the agent is given 1 expert demonstration. But the suboptimal transition dataset actually contains 200 expert demonstrations, though I understand they're not marked as expert demonstrations to the algorithm. I think a more convincing experiment is to completely remove the expert demonstrations in the suboptimal data.**

The suboptimal dataset we consider either comes from random or medium (for locomotion)/ cloned-human (for manipulation), it might be unreasonable to expect any imitation learning method to achieve meaningful performance on these tasks with 1-expert trajectory as even RL with known dense reward function has struggled to obtain near-expert behavior from these datasets (eg. Table 1 in [3]). Second, the experimental design for imitation learning used here is not a contribution of our work rather is an evaluation benchmark established by prior works [5,6]. We note that in our work, we extended this benchmark to include manipulation tasks for which we collected our own data following a similar protocol outlined in [5,6].

Reviewer comment (Presentation):
The interpretation of Fig 1 is unclear. The authors should explain it more.

Author response:
**I don't know how to interpret Fig 1. Can you explain more?**

We have updated the caption in the revised paper as well as included a detailed explanation of the experiment in Appendix H.3. Figure 1 aims to test the ability of ReCOIL objective to learn distribution ratios. For a fixed policy, the inner optimization objective of Equation 9 captures the distribution ratio between mixture distributions of (current policy visitation+offline data visitation) and (expert visitation+offline data visitation). In Figure 1, we consider a 1-D MDP with simple dynamics. Given an expert visitation, replay visitation and a policy, we aim to test how accurately can we predict the given policy's visitation. Figure 1(a) also provides the ground truth visitation of the policy, and we plot MSE error with ground truth visitation in the right plot.

Reviewer comment (Writing):
Minor: The sentence "Unfortunately, one of the most successful classes of offline RL methods .. has evaded connections to regularized policy optimization. Proposition 2 shows, perhaps surprisingly, that XQL can be cast as a dual of regularized policy learning, concretely as a dual-V problem." is confusing to read.

Author response:
**This sentence is confusing to read. "Unfortunately, one of the most successful classes of offline RL methods..."**

We have rephrased this sentence. Thanks for pointing this out.

Reviewer comment (Writing):
Minor: The connection between the sentence "To prevent extrapolation error in the offline setting, we rely on an implicit maximizer [Garg et al., 2023] that estimates the maximum over the Q-function conservatively with in-distribution samples." and Eq (12) is unclear.

Author response:
**I don't understand the connection between the sentence "To prevent extrapolation error in the offline setting, we rely on an implicit maximizer [Garg et al., 2023] that estimates the maximum over the Q-function conservatively with in-distribution samples." and Eq (12)**

In the offline setting, optimizing Equation 10 results in value function overestimation. This phenomenon arises because policy maximization can choose actions that are not present in the dataset, whose values are incorrectly overestimated. Now, these overestimated values gets bootstrapped as a result of Bellman backup in the last term of Equation 10 resulting in value function blowup. To solve this issue, we subsitute the unconstrained policy maximization step with an in-sample maximizer -- only samples that are seen in datset are used to conservatively (under)estimate the maximum.

Reviewer comment (Presentation):
Figure 1 (a) is quite hard to follow. Please refine the caption to explain what the floats and the colors mean.

Author response:
We have elaborated on the experiment in Figure 1 (a) in the caption of the revised paper as well as added a detailed explanation of the experiment setup in Appendix H.3.

Reviewer comment (Evaluation):
The first paragraph of Section 7.1 lacks supportive evidence for this claim. Incorporating illustrative figures and evaluations would substantiate this statement.

Author response:
We want to request clarification on this question. It seems like Section 7.1 is the experimental setup section and perhaps the reviewer meant some other section.

Reviewer comment (Writing):
On page 36, typo in "We base the implementation of SAC off pytorch_sac ..."

Author response:
We have made the changes for all the suggested typos in the revision. The changes are highlighted in red. We appreciate the reviewer's attention for pointing these out. We provide clarification to reviewers queries below:

Reviewer comment (Experiments):
On page 37, I disagree with your claim that "Our experiments with the popular off-policy method SAC [Haarnoja et al., 2018] reveal its brittleness to off-policy data". SAC is not brittle to off-policy data. Filling the replay buffer arbitrarily with expert data at the beginning is not fair for an ADP-style algorithm. Please rewrite this paragraph.

Author response:
In this section of the appendix, our intention was to compare popular ADP-based off-policy methods vs dual-RL/DICE style off-policy methods. Any off-policy algorithm, by definition, should be able to make use of any kind of data obtained from the environment -- expert data is a special case. We have reworded this section to make our claims more accurate since as the reviewer points out considering one special case is not enough to say the the algorithm is brittle.

Reviewer comment (Presentation):
Some crucial experimental outcomes, currently in Appendix H, should be integrated into the main paper for better accessibility and coherence, especially those results referenced and analyzed within the main paper.

Author response:
We appreciate the reviewer's suggestion. We acknowledge that our appendix contains a number of interesting experiments. In this work, we were constrained by the page limit to strike a balance between facilitating understanding, demonstrating our contributions concisely, and providing empirical evidence to support the theoretical contributions. Currently, we take $\approx 0.6$ page for related works and $\approx 1.4$ page for preliminary. We believe introduction should not be considered a part of prior work and a similar format can be found in a number of influential ICLR works [1,2,3]. We would be happy to receive suggestions on what the reviewer thinks might be traded off in favor of more experiments.

Reviewer comment (Novelty):
Better not to claim "New" in the title directly. Just saying "A unified framework" is ok.

Author response:
We clarify that the use of the word 'New' was to signify that we leveraged the existing framework of duality in RL to propose new and improved algorithms for offline RL and IL and not to signify the novelty of the duality framework.

Reviewer comment (Evaluation):
While f-DVL outperforms XQL, its gains over other offline RL methods like IQL are marginal. The gains are not as significant as claimed over the full spectrum of baselines.

Author response:
The locomotion benchmark in D4RL for offline RL is simpler with performance to a certain extent saturated, compared to the Antmaze or Kitchen environments. Our proposed approach is competitive in these domains. A detailed discussion of the properties of these environments and datasets can be found in [1]. We notice that $f$-DVL outperforms in 6/9 tasks in antmaze and kitchen environments which pose a harder challenge of stitching suboptimal trajectories compared to recently proposed IQL[2], XQL[3], and DT[4].  We believe that these improvements over prior work are meaningful by the standards used in prior offline RL papers.
One of the contributions through $f$-DVL is also to provide another axis of improvement in a stable learning alternative to XQL. We would like to bring attention to the fact that XQL (ICLR 23) is a more recent method than IQL (ICLR 22) and our proposed method drives direct improvement over XQL.

Reviewer comment (Evaluation):
The presentation of empirical results could be improved by using standardized metrics, showing confidence intervals, and increasing clarity around performance highlights.

Author response:
We appreciate the reviewer's suggestions and note that the following changes have been made in our revised paper: a) In the results for imitation learning (ReCOIL) we highlight all methods that lie within the standard deviation of the best-performing method. b) In the offline RL (f-DVL) results we highlight methods that are within 1 performance point of the best-performing method. Additionally, we have also added standard deviations for our method in Table 10. Indeed, using statistical metrics would be more suitable but unfortunately prior works in offline RL like XQL have used mean as an estimate. We extend prior results in our work and computing metrics like Inter-Quartile mean (IQM) or std-dev would require rerunning all the baselines which would take more time and more compute budget than we have available.

Reviewer comment (Theory):
In Eq. 10, is the reward assumed to be zero in the Bellman consistency term?

Author response:
To clarify for Eq. 10, we are in the setting of imitation learning where we are not provided with task rewards but rather only expert demonstrations. Eq. 10 is a direct consequence of Theorem 1 which provides a principled objective for imitation learning signifying that bellman backup with zero rewards in conjunction with a contrastive loss on Q is sufficient and pricipled for imitation learning.

Reviewer comment (Reproducibility):
Can you expand on how ReCOIL estimates the policy visitation and reward for the results in Sec 7.1? Also clarify if Fig 1 and Fig 10 use OPOLO or SMODICE for comparison.

Author response:
We have added a detailed explanation for estimating policy visitation in Appendix H.3 in the revised paper and also edited the caption of Figure 1 to give more clarity. In brief, any dual-RL objective written in the $Q$-form allows us to extract density ratios by considering the inner optimization of Equation 9.

 For reward estimation, the paragraph below Theorem 1 explains how rewards can be inferred from learned Q-function and policy. In brief, given $Q^*$ and $\pi^*$, the rewards can be extracted as $r(s,a)= Q^*(s,a)-\mathcal{T}^{\pi^*}_0(Q^*(s,a))$
 
 3. **Also clarify if Fig 1 and Fig 10 use OPOLO or SMODICE for comparison.**
 
 We appreciate the reviewer pointing out this typo, we correctly mentioned in the figures that we used SMODICE but incorrectly in the text that OPOLO was used for comparison. We have made this correction in the text. SMODICE can be understood as an offline counterpart to OPOLO and in our work we focus on the offline setting. A more detailed discussion of this experiment and baselines can be found in Appendix H.3.

Reviewer comment (Writing):
What do the distributions D and d^O refer to in Eq. 12 and 13? Are they both d_{mix}^{E,S}?

Author response:
We appreciate the reviewer bringing attention to this typo. They are both indeed $d_{mix}^{E,S}$ and we have fixed this in the revised paper.