Paper: Distributional Distance Classifiers for Goal-Conditioned Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
I am not very convinced about the claim that "the number of bins does not affect the final performance". In the extreme case, where H=1, it becomes a quantity similar to the expected distance measure and it should perform similar to estimating expected distance measure in my opinion. I would like to see how the algorithm behaves when H is very small, such as H=1 or 2.

Author response:
In the case of H=1, Distributional NCE reduces to the Contrastive NCE baseline, which is worse than our method (see Fig. 4). However, we also observed that varying the number of bins between 11 and 101 did not significantly impact the performance in Appendix D.6 Fig. 12, which hints that increasing the number of bins beyond a point has diminishing improvements. We have revised the paper (Appendix D.6) to clarify that the performance our method does degrade for very small $H$.

Reviewer comment (Theory):
The problem formulation in this work is not exactly the stochastic shortest path problem as the discounting factor is < 1. Will the analysis still go through if the discounting factor is 1?

Author response:
The analysis in Sec. 4.2 and Appendix A shows that the (discounted) probability objective used in our work is closely related to the (undiscounted) stochastic shortest path problem. This analysis requires that (discounted) probability objective is a discount factor strictly less than one; we have revised Appendix A to note this. Please let us know if this doesn't make sense, or if we have misunderstood the question.

Reviewer comment (Novelty):
In related work, please also discuss the relationship between this work and the recent theoretical advancement in goal-conditioned RL, such as the followings: (1) Jean Tarbouriech, Evrard Garcelon, Michal Valko, Matteo Pirotta, and Alessandro Lazaric. No-regret exploration in goal-oriented reinforcement learning. In International Conference on Machine Learning, pages 9428–9437. PMLR, 2020 (2) Alon Cohen, Haim Kaplan, Yishay Mansour, and Aviv Rosenberg. Near-optimal regret bounds for stochastic shortest path. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 8210–8219. PMLR, 2020. (3) Liyu Chen, Mehdi Jafarnia-Jahromi, Rahul Jain, and Haipeng Luo. Implicit finite-horizon approximation and efficient optimal algorithms for stochastic shortest path. Advances in Neural Information Processing Systems, 2021a.

Author response:
We have added a paragraph to the related work section noting the similarities and differences with our work. We welcome additional suggestions for related work to discuss.

Reviewer comment (Novelty):
The proposed method has the same spirit as distributional RL. I am wondering whether quantile regression can also be extended to solving goal-oriented RL?

Author response:
This is a very exciting future direction that is beyond the scope of this paper. The contrastive RL framework offers flexibility to arbitrarily define the bins of the distributional NCE algorithm, as long as their union covers all the future timesteps [1, \infty). Instead of choosing bin boundaries as successive timesteps as we do in the paper (akin to C51), one can choose them to be the quantiles of a normalized distance classifier P(H | s, a, g) (akin to QR-DQN).

Reviewer comment (Theory):
I am not sure if propositions 1-3 are correct, the statements themselves are quite vague. The proofs, or lack thereof, are not convincing. For instance, what is the argument being made in the proof of proposition 2? That MC distances are not valid distances because they do not satisfy the triangle inequality? How is this being shown for any metric? What is the distance function \( d(s_1, s_2, a) \)? Is the proof for proposition 1 referring to the MDP in figure 1a)?

Author response:
> I am not sure if propositions 1-3 are correct, the statements themselves are quite vague

We believe that these propositions are correct, and have revised the statements to make the notation and results more clear. Please let us know if there is any notation is ambiguous or confusing, and we would be happy to further revise this section.

> what is the argument being made in the proof of proposition 2?

*Proposition 2* argues that the MC distance is generally not a valid metric. The proof is a proof by contradiction, so we construct an example where the MC distance is not a valid metric (i.e., it violates the triangle inequality). We have added a note at the start of the proof to highlight that it is a proof by contradiction.

> Is the proof for proposition 1 referring to the mdp in figure 1a?

Yes, the proof for *Proposition 1* is referring to the mdp in Fig. 1(a). We have revised the proof to clarify this point.

Reviewer comment (Theory):
The main weakness of the paper lies in the formalism of the mathematical writing. Certain mathematical expressions, such as the Monte Carlo Distance Function, are defined in words but not formally stated. This becomes an issue when writing proofs about these ill-defined terms.

Author response:
We thank the reviewer for the detailed review. It seems like the reviewer's main concern is about mathematical correctness and precision. We believe that all the theoretical results are correct, and have revised the paper to make the mathematical notion more clear (blue text). We answer specific questions below. **Do these revisions and answers fully address the reviewer's concerns?** We look forward to continuing the discussion.

Reviewer comment (Theory):
Why is Monte Carlo distance estimation equivalent in the limit to learning a normalized distance classifier?

Author response:
> Why is Monte-Carlo distance estimation equivalent in the limit to learning a normalized distance classifier?

We have revised Sec. 4.3 to clarify that these are not exactly equivalent. MC distance function can be estimated using a normalized distance classifier as follows:
- MC distance d(s,a,g) = \sum_{i=0}^\infty ( i * P(i | s, a, g) ),

where P(H | s, a, g) is the normalized distance classifier.

Reviewer comment (Presentation):
Appendix D.4 is empty.

Author response:
> Appendix D.4 is empty

We have added a sentence at the start of Appendix D.4 to clarify that this subsection contains just Fig. 10 and the text in its caption.

Reviewer comment (Novelty):
Another assumption is that the method of predicting the number of steps that elapse between one observation and another is common. A few works definitely explore this direction, but I think it is more common to simply use a shaping reward over the 0/1 reward function.

Author response:
We have revised Sec. 2 to clarify that, while there are a number of methods that predict distance, it is also a common practice to use shaping reward. Nonetheless, we believe our analysis is useful because we highlight the intricacies and shortcomings of using MC distance functions, and how these shortcomings can be fixed by taking a probabilistic perspective of distances.

Reviewer comment (Experiments):
In section 4.1, it is unclear in the toy examples when the episode ends. From the math it seems like the episodes end when the goal state is reached otherwise they continue indefinitely.

Author response:
The episode ends when the agent reaches the goal state. Our revised paper on OpenReview includes this clarification.

Reviewer comment (Experiments):
It would be interesting to see how this method compares with shaping rewards.

Author response:
Prior work has shown that reward shaping performs quite poorly on these Fetch environments, even when combined with HER (https://arxiv.org/pdf/1707.01495.pdf, Fig 5, success rates all <10%). We have revised the additional experiments section (Appendix D.1) to note this.

Reviewer comment (Theory):
In section 4.3, the estimated distance functions seem to be odd. Suppose, C(s, a, g)[H] = [0, 0, ...]. Will the estimated distance between s and g be 0? Also, if the classifer predicts normalized probabilities, then p(g|s, a2, H=1, 2, 3, ...) cannot be equal to [1, 1, 1, ...].

Author response:
The MC distance is ill-defined in this case. In practice, this corresponds to the setting where we've never seen any trajectories between s and g, so we cannot estimate the number of steps between them. We've revised Sec. 4.3 in the paper to include example.

Correct: a _normalized_classifier cannot equal [1, 1, 1, …]. p(g|s, a, H=i) refers to the probability of reaching the goal state g as opposed to some other state at i^th timesteps, and is thus a quantity in [0,1]. On the other hand, p(H=i |s, a, g) refers to the probability of reaching the goal g at i^th timestep, given that the policy reaches the desired goal at some future timestep. Thus,
- \sum_i p(g | s, a, H=i) \neq 1
- \sum_i p(H=i | s, a, g) = 1

Reviewer comment (Writing):
The algorithm is not completely clear. What is dt in the algorithm box?

Author response:
`dt` refers to the relative time index of the future state which is used to index the corresponding bin of the Distributional NCE classifier. We have updated the algorithm block to include this information.

Reviewer comment (Experiments):
A lot of research in goal-conditioned RL uses probability as criterion. The paper should include a comparison to compare the algorithm in the paper and previous algorithm.

Author response:
Our experiments in Fig. 4 do compare with prior methods that use probability as a criterion (Contrastive NCE (red), C-Learning (yellow)). We'd welcome suggestions for additional baselines to compare against.

Reviewer comment (Theory):
The temporal consistency objective does not make sense in certain reinforcement learning problems. Some goals can have a minimum number of steps needed before they can be reached. For example, in a gridworld where the shortest path to the goal is 10 steps away, P(reaching the goal in 9 steps) = 0, whereas P(reaching in 10) can be high for a good policy. It would be useful to address this aspect of temporal consistency or even highlight it in an experiment with such temporal constraints, or remove it as it does not naturally stem from the main hypothesis of the paper.

Author response:
Thanks for bringing up this great example; we believe the temporal consistency regularizer does make sense in this example. In this example, let’s imagine a start state s_0, a goal state g, and actions a_i ~ \pi(.|s_i) for all i \in [0, \infty). We further imagine that the goal state can only be reached after 10 steps:
- P(s_0, s_{i}=g,a_0) = 0 for all i < 10 && P(s_0, s_{0 + “10”}=g,a_0) > 0.
Nevertheless, the following still holds:
- P(s_0, s_{0 + “10”}=g,a_0) = E_{s_1 ~ p(. | s_0, a_0), a_1 ~ \pi(.|s_1)} [ P(s_1, s_{1 + “9”}=g,a_1) ]
Note that the term on the right-hand side refers to a future state that is 9 steps away relative to t=1, i.e. still the 10th timestep w.r.t the starting point t=0. We've revised the paper to include this discussion in Appendix C.

Reviewer comment (Writing):
While this work does acknowledge Contrastive NCE from the literature, the acronym is not explicitly spelled out anywhere. This would be useful to specify.

Author response:
Thanks for the suggestions to improve the paper writing! We have addressed these changes (in blue color) in the revised version of our paper on OpenReview:
- Added a formal definition of the Monte Carlo distance function in Sec. 4
- Revised Sec. 4.1 to better explain the pathological behaviors exhibited by MC distance functions.
- Clarified that NCE stands for Noise Contrastive Estimation. 
- Removed duplicate reference to Yecheng et al. 2022 (page 11)

Reviewer comment (Presentation):
It would be useful to specify the number of runs that were averaged over in the learning curves of Figures 3, 4, and those in the appendix. The work should also mention what the shaded area around each curve represents and how they curves were smoothed, if any such technique was applied.

Author response:
The curves show the mean performance with the shaded region indicatingthe 95% confidence interval computed across 5 random seeds. We do not use any smoothing or filters to post-process the results. These details were in the original submission (Appendix E), but we have revised the main text to point readers to this appendix.

Reviewer comment (Presentation):
On page 11, Yecheng et al. 2022 was included twice in the references.

Author response:
Thanks for the suggestions to improve the paper writing! We have addressed these changes (in blue color) in the revised version of our paper on OpenReview:
- Added a formal definition of the Monte Carlo distance function in Sec. 4
- Revised Sec. 4.1 to better explain the pathological behaviors exhibited by MC distance functions.
- Clarified that NCE stands for Noise Contrastive Estimation. 
- Removed duplicate reference to Yecheng et al. 2022 (page 11)