Paper: Score Models for Offline Goal-Conditioned Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
The target goal-transition distribution $q(s,a,g)$ defined in Section 3.1 might not be a valid discounted visitation distribution that fulfills the Bellman-flow constraint. Therefore, this suggests that the occupancy matching problem in Eq. 4 may not be appropriately formulated.

Author response:
We would like to clarify that irrespective of the goal-transitions distribution being an infeasible visitation distribution (which we discuss in the paper as well), the distribution matching objective in Eq.4 still maximizes a lower bound to the goal-conditioned RL objective. This is proved in Propositions 1 and 2 in the paper.

Reviewer comment (Experiments):
This paper optimizes surrogate objectives using IQL[2] rather than directly solving the objective derived from DICE reformulation, but does not provide explainations for this choice. Moreover, this paper does not conduct enough ablation studies on this aspect. It raises a question about the performance of GoFAR using this same technique. In my view, optimizing the surrogate objectives in Eq. 10-12 can mitigate the side-effect of residual learning in Eq. 9. Therefore, GoFAR may also obtain large performance gains using the IQL tricks.

Author response:
We would first like to point out a common misconception: GoFAR does not learn via residual learning even though it seems like it from Algorithm 2 in their paper. Their official code [linked here](https://github.com/JasonMa2016/GoFAR/blob/cfb2d4e36d69c5ec16edf6d127464eb263925e70/rl_modules/gofar_agent.py#L171) actually shows that semi-gradient learning is being used in GoFAR.

Our practical algorithm adapts the derived GCRL objective for the offline setting by using in-sample maximization to avoid overestimation. We appreciate the suggestion and have added this ablation in the Appendix [Table 13]. GoFAR with IQL is seen to have nearly the same performance as vanilla GoFAR.

We note that the in-sample maximization and the constrained policy optimization used in the practical algorithm is not a contribution of our work and is simply taken from [1,2,3] and it has been shown previously unconstrained optimization is detrimental in offline setting [4].

Reviewer comment (Novelty):
Some potential overclaims. Section 3.1 is essentially an extension of the conclusion of GoFAR[3] from state-occupancy matching to state-action-occupancy matching. The authors should provide more discussions on the relationships with GoFAR. Although they have made statements like "Proposition 1 extends the insights of formulating GCRL as an imitation learning problem from Ma et al. for goal-transition distributions when matching state-action-goal visitations", this similarity should be made clearer.

Author response:
We appreciate the suggestion and have clarified this in the updated paper. Our contribution lies mainly in proposing a mixture-distribution matching objective for GCRL as well as providing a connection to the original GCRL objective in Proposition 2.


 We do want to point out that while Proposition 1 bears similarity in the high-level intuition of distribution matching but differs in the fact that GoFAR matches state-goal transitions, which requires them to construct another lower bound (see Proposition 4.2 and Lemma B.2 in GoFAR[5]) in order to obtain a tractable optimization objective. In contrast, goal-transition distributions allow us to craft an objective that we can directly optimize. We believe that this is an important observation in itself. 

Please find our expanded discussion on the theoretical and algorithmic differences of our method compared to GoFAR in the general discussion.

Reviewer comment (Novelty):
The Problem Formulation section (Section 2) is largely similar to the one in GoFAR[3] paper, with only minor rewording. In my view, this section should be reorganized or rewording a lot to avoid potential plagiarism.

Author response:
We appreciate the reviewer for pointing this out. Although our problem setting is exactly the same we recognize the importance of setting up our own context for the work. We have updated the Problem Formulation to address reviewers' concerns. The suggestions have been incorporated into the revision.

Reviewer comment (Novelty):
The problem setting and Section 3.1 largely follow GoFAR[1] with minor changes. A key motivation from the authors is that methods like GOFAR need an unstable discriminator-based construction. However, this does not really hold. As in the GoFAR paper[1], their authors clearly mentioned in the paper that the discriminator can be bypassed by using the reward in the dataset.

Author response:
While GoFAR author suggests that discriminator can be bypassed by using rewards, their experiments are hardcoded to use discriminator (official code line [link](https://github.com/JasonMa2016/GoFAR/blob/cfb2d4e36d69c5ec16edf6d127464eb263925e70/train.py#L130). Please find our results using reward function instead of a discriminator with GoFAR in Table 13 of the revised paper and an extended discussion of differences with GoFAR in the general response comment. GoFAR without discriminator reduces to an RL with a sparse rewards algorithm and has poor performance.

Reviewer comment (Experiments):
Although using HER to generate augmented goal-transition samples could potentially improve distribution coverage and may contribute to certain level performance improvement. However, as discussed in GoFAR, using HER could also lead to sensitive hyperparameter tuning and suffer from hindsight bias.

Author response:
Figure 4 in GoFAR demonstrates that HER does not improve performance for GoFAR. Our work, as well as GoFAR, compares to baselines with access to HER data augmentation. A fair comparison directs that we hold our method to the same standards. Removing hindsight bias from HER is an orthogonal exploration (ex. [1]) and our method presents a goal-conditioned policy learning approach applicable for any offline data distribution. We have added a comparison of our method without HER in Table 12 where it can be seen that HER contributes to a small performance gain, even without which we outperform GoFAR.

Reviewer comment (Theory):
The biggest problem of this paper is the gap between theoretical derivation and the practical algorithm. Based on the augmented samples from HER, the proposed method constructs a goal-conditioned Q-DICE objective which needs to solve a max-$\pi$ and min-$S$ (analogous to Q function in typical Q-DICE algorithm like AlgaeDICE[5]). This actually caused some stability issues due to extracting $\pi$ through the max-min optimization problem. Hence the practical algorithm directly jumps to an in-sample learning framework which is similar to IQL[2], SQL/EQL[3], and XQL[4]. Although there are some connections between the DICE-based method and previous in-sample learning methods, there are also some distinctions. An apparent difference is that the DICE-based method requires minimizing $S$ (analogous to Q in other Q-DICE methods) in the first term of Eq.(9) using samples from initial states $d_0$, while in in-sample learning algorithms, this can be sampled from the whole dataset $\mathcal{D}$ (Eq.(10)). Second, DICE-based method only learn Q (similar to S in this paper) or V, while the previous in-sample learning algorithms learn both Q and V. In my opinion, the paper starts with a DICE formulation and goes a long way to turn it into a non-DICE algorithm. If the authors check the SQL/EQL[3] paper, a more straightforward approach will be starting with its implicit value regularization framework and designing a proper, goal-conditioned regularization function $f$, which will provide a neat and more principled algorithm.

Author response:
We respectfully disagree and demonstrate below how our practical algorithm closely follows our theoretical derivation and adapts to the offline setting we consider. Our derived objective maximizes over policies $\pi$ and minimizes over $S$. The objective contains terms that depend on next state making it susceptible to overestimation if $\pi$ chooses OOD actions when it is maximized at any given state. Hence, in our practical algorithm, we use:
  
$$
\text{in-sample-max}\_{\pi}[g(\pi)] ~~ \text{as a surrogate to} ~~ \max\_{\pi}[g(\pi)].
$$

In-sample maximization is a mathematical tool to avoid overestimation that has been applied to work outside offline RL methods (eg. [2,8]). 

Second, the reviewer mentions that DICE necessitates using samples from initial state distribution. This is incorrect. In a number of prior DICE methods, it has been found that setting the initial state distribution to the entire replay-buffer/offline-dataset distribution leads to more stable and performant optimization [3,4]. [3] connects the link between using the offline dataset as the initial state distribution directly to in-sample maximization. Finally, using a wider initial state distribution of offline dataset does not affect the policy performance at the environment's initial state distribution as long as the function-approximator has enough capacity (in our case they are neural networks) since the policy will only learn to achieve near-optimal behaviors from a wider variety of states. 

Third, we respectfully disagree that DICE methods cannot use both Q and V for learning. This has been done in the past in OptiDICE[9, ICML 21] as well as [3]. As opposed to using single sample targets, learning Q reduces variance in targets by averaging over stochasticity in dynamics.

We appreciate the reviewer's suggestion to make the connection to SQL/EQL, but point out that one key difference is that SQL/EQL are reward maximization algorithms with policy regularization whereas we aim to derive a distribution matching algorithm using policy visitations. For this purpose, we believe the duality in RL approach by Nachum et al[5] is more suitable. We have added a comparison with in-sample learning methods in Table 15 where we find SQL/EQL/XQL to be particularly more unstable in sparse reward settings.

Reviewer comment (Experiments):
The proposed algorithm has many hyperparameters, e.g. $\tau$ in Eq.(10), $\beta$ in Eq.(11), and $\alpha$ in Eq.(12). The paper conducts heavy tuning to obtain the best performance. First of all, for an offline RL algorithm, introducing too many hyperparameters and requiring heavy parameter tuning is an extremely bad practice. In practical offline RL applications, it is almost impossible to evaluate or tune model parameters given restricted access to the real environment. In practice, no one will use an offline RL algorithm if it needs careful hyperparameter tuning to achieve good performance.

Author response:
We believe there is a misunderstanding about the hyperparameter tuning done in this work. We clarify that $\alpha$ and $\beta$ were fixed to 3.0 and 0.5 throughout all the environments. The only parameter tuned per set of environments is $\beta$ which controls the conservativeness of the value function and is an MDP-dependent parameter. As we show in the general discussion comment, we found very minimal hyperparameter tuning was required and the final results use a fixed value of $\beta$ per class of environments. We have added this discussion in the appendix. Further, Table 9 demonstrates our method is robust across a range of $\beta$ values. This is significantly less hyperparameters tuned compared to popular algorithms like CQL[6] (pessimism, regularizer, SAC parameters), IQL[7] (expectile, policy temperature). 

**Question 1: Please report the hyperparameter values in your experiments.**
 
We have added this in the updated appendix.

**Question 2: How will the proposed method perform if not tuned individually for each task? Such as using 1~3 sets of hyperparameters.**

As we discussed above our method already uses one set of hyperparameters per environment class. We have clarified this in the appendix now.

**Question 3: All datasets in the experiments are mixed with some expert data. How will the proposed method perform if only uses sub-optimal data samples?**

We appreciate the reviewer's suggestion and have added this to Appendix (Table 14). In most environments, random data has no examples of methods reaching goal successfully and in turn, all methods are not expected to perform well.

Reviewer comment (Novelty):
Could you please elaborate more about the technical differences between SMORe and GoFAR? My understanding is that the main difference is whether the training involved a discriminator. Does any other difference in details improve the novelty of SMORe?

Author response:
Our novelty compared to GoFAR can be summarized in two parts: a) Theoretical: 1. We present a novel objective of mixture distribution matching for GCRL that provably optimizes the GCRL objective without requiring a discriminator. 2.  
GoFAR considers matching state-goal distributions which requires constructing a loose lower bound (Proposition B.1 and Lemma B.1 in [1]) that is not tight for any $f$-divergence. Using goal-transition distribution (state-action-goal) allows us to create a lower bound to the original GCRL objective that is tight for KL divergence. b) Algorithmic: 1. GoFAR requires a three-step learning process of first learning discriminator, then value, and finally extracting a policy. Our method presents a simpler two-step approach by directly learning the optimal score function and corresponding policy. Empirically our method achieves significant gains across the board compared to GoFAR. 

We refer the reviewer to the general discussion comment where we present a more thorough comparison with GoFAR.

Reviewer comment (Reproducibility):
Is the "0.25" in equation 9 a fixed number or a kind of coefficient that can be tuned?

Author response:
0.25 is a fixed number dictated by the use of $\chi^2$ divergence. It weighs the smoothness regularization vs the contrastive-objective ensuring the resulting objective we optimize has principled connections to the original GCRL objective we are interested in.

Reviewer comment (Experiments):
Any reason why you make the mixture of random/medium and expert data following 4.1 EXPERIMENTAL SETUP? I thought that there is already a pipeline for collecting random, medium, and expert sub-dataset?

Author response:
We do not collect our own dataset for Fetch or Sawyer environments. These datasets are part of the benchmark established by previous work [1,2]. We explain the dataset collection procedure used the by previous work in the appendix only for completeness.

Note that the locomotion tasks for GCRL as well their datasets are introduced by us for the first time.

Reviewer comment (Evaluation):
Can you explain why SMORe has an even higher discounted return in Figure 2 under 0.5 noise level compared with 0 noise? In addition, I am interested in the variance of Figure 2.

Author response:
We suspect that is a consequence of different quality of transitions in dataset collected with different noise levels. It can also be a result of noise with finite seeds. We appreciate the suggestion of showing variance and have included the standard deviation now in the updated paper.

Reviewer comment (Evaluation):
Is there any insight or analysis into why pick and place tasks in Figure 3 are relatively difficult for other baselines compared with the remaining tasks while the proposed method can have a significant improvement in pick and place tasks?

Author response:
The final pick and place task is more complicated in the sense that it requires precise manipulation. The gripper can collide with the drawer resulting in failure to complete the tasks. Our hypothesis is that, in other tasks, mistakes are more forgivable and allow other baselines to perform reasonably as well.