Paper: Contrastive Representations Make Planning Easy (ICLR-2024)

Reviewer comment (Experiments):
The by far weakest point of the paper is that the authors miss to demonstrate that their method works on a real problem. After all, this is still machine learning. No matter how beautiful the derivation is, a method that does not work in the real world has no value. I have severe doubts that it will ever work, say, for modeling an industrial process or a robot turning a knob to open a door. For my taste, the assumptions in section 3 are far too strong to be left unchecked on real data.

Author response:
As suggested, we have run additional experiments to validate our theory, including on a 39-dim robot door opening task:
* [Fig 3 (left)](https://i.imgur.com/BMAg3yU.png): We show how the analysis in Sec. 4 can be used to solve a maze. After learning representations, we used the closed form expression to infer the representations of intermediate states. This result shows how the representations effectively warp the state space so that linear motion in representations (i.e., $A \psi$) corresponds to fairly complex motion in the state space (i.e., navigating through a maze).
* [Fig 3 (right)](https://i.imgur.com/BMAg3yU.png): Using this same maze, we show that the inferred waypoints can be used for control (details below).
* [Fig 4](https://i.imgur.com/XgDrGOu.png): We apply the representations to a 39-dimensional robotic door opening task. Compared to a PCA baseline, linearly interpolated waypoints are a better fit for the true data waypoints.
* [Fig 6](https://i.imgur.com/gVnReJj.png): We apply the representations to a 46-dimensional robotic hammering task. Compared to a PCA baseline, linearly interpolated waypoints are a better fit for the true data waypoints.

Reviewer comment (Writing):
Use of terms: The term "planning" has a technical meaning in artificial intelligence. That meaning is very different from temporal inter- or extrapolation, which is what it refers to in the paper. Therefore, even the paper title is completely misleading. A similar consideration applies to the central term "representation", which is used as a map from observations to a latent space, and also for an element of that space, sometimes with different meaning within the same sentence.

Author response:
Thanks for pointing this out! We agree that there is no single clear meaning of this word and it is often misused. We have revised the introduction to explain that we will use ``planning’’ to refer to an inference problem, rather than an optimal control problem. We provide citations there to some prior work that takes a similar perspective (Botvinick 2012, Attias 2003), though we acknowledge that this may be a nonstandard usage within the AI community.

Thanks for raising this great point! It indeed seems like a potential source of confusion. To incorporate this feedback, we have significantly revised Sec. 4.1 to disambiguate “encoder” from “representation.” We have also reviewed every use of the word “representation” in the rest of the main text, making some minor changes in attempts to clarify this. **Are there remaining parts of the paper where this is unclear?** We would be happy to make additional changes to improve clarity and precision.

Reviewer comment (Writing):
Abbreviations: The abbreviations "CV" and "NLP" are of course generally understood in the community. It is nevertheless good style to state the complete terms at least once before using abbreviations. Other abbreviations are less commonly known, like NCE.

Author response:
We have written these terms out in full the first time they are used.

Reviewer comment (Presentation):
Figure 1 is pretty meaningless.

Author response:
As suggested, we have removed Figure 1.

Reviewer comment (Presentation):
The first paragraph of section 2 ends in the middle of a sentence.

Author response:
We have completed the sentence in the revision.

Reviewer comment (Presentation):
I am surprised to see that the title of section 4 coincides with the paper title.

Author response:
Our intention here was to make it easy for readers to identify where to find the main results. We also felt like it was an accurate description of the section. If the current title is unclear or imprecise, we would be happy to revise the section title.

Reviewer comment (Writing):
At the point figure 2 is presented (section 1), the symbols in the figure are not introduced yet.

Author response:
We have revised the caption to briefly explain the symbols

Reviewer comment (Presentation):
Why on earth should one gray out a proof?

Author response:
We have removed this. Our original intention was to indicate that readers could skip over this section if they were not interested in the proof, but we realize in hindsight it can cause confusion.

Reviewer comment (Experiments):
There is a lack of comprehensive experimentation in Section 5. I acknowledge the author's emphasis on the theoretical aspect of their work and I highly value theoretical contributions. From what I gather, the author asserts that their work simplifies the planning process by deriving the analytical Gaussian form. I agree with this statement. However, at present, it remains challenging to envision how this method can offer practical utility in moderately complex environments and tasks. It raises questions about its direct applicability and the potential necessity of heuristics. The inclusion of empirical evidence would significantly strengthen the case for acceptance. Currently, I find myself in a somewhat undecided position.

Author response:
Thank you for the detailed feedback. It seems like the main suggestion is to add more experiments, which we have done through new experiments on higher-dimensional tasks (up to 46-dimensional) and by demonstrating that the inferred representations are useful for control. **Together with the revisions (red text in new PDF) and clarifications discussed below, does this fully address the reviewer's concerns about the paper?** We look forward to continuing the discussion!

> The inclusion of empirical evidence would significantly strengthen the case for acceptance.

As suggested, we have added 4 new figures with empirical results to validate our theory:
* [Fig 3 (left)](https://i.imgur.com/BMAg3yU.png): We show how the analysis in Sec. 4 can be used to solve a maze. After learning representations, we used the closed form expression to infer the representations of intermediate states. This result shows how the representations effectively warp the state space so that linear motion in representations (i.e., $A \psi$) corresponds to fairly complex motion in the state space (i.e., navigating through a maze).
* [Fig 3 (right)](https://i.imgur.com/BMAg3yU.png): Using this same maze, we show that the inferred waypoints can be used for control (details below).
* [Fig 4](https://i.imgur.com/XgDrGOu.png): We apply the representations to a 39-dimensional robotic door opening task. Compared to a PCA baseline, linearly interpolated waypoints are a better fit for the true data waypoints.
* [Fig 6](https://i.imgur.com/gVnReJj.png): We apply the representations to a 46-dimensional robotic hammering task. Compared to a PCA baseline, linearly interpolated waypoints are a better fit for the true data waypoints.

> Evidence that planning by waypoint inference enables long-horizon control:

To show waypoint inference is useful for control, we have added an experiment in a 2d maze environment ([Fig 3](https://i.imgur.com/BMAg3yU.png)). We define a simple proportional controller for navigating between nearby states; as expected ([blue line in Fig 3](https://i.imgur.com/BMAg3yU.png)), this proportional controller fails to reach distant goals. Using the proposed inference over representations, we infer waypoint representations, retrieve waypoints states (nearest neighbor), and iteratively have the proportional controller navigate to the waypoint states. The resulting success rates ([orange line in Fig 3](https://i.imgur.com/BMAg3yU.png)) are up to 4.5x higher (19% -> 87%).

Reviewer comment (Theory):
Section 3: As a related suggestion, it would be advantageous to formalize the statements in this section within theorem/lemma/proposition frameworks, similar to the structure used in Lemma 4.1.

Author response:
> Section 3: it would be advantageous to formalize the statements in this section within theorem/lemma/proposition frameworks, similar to the structure used in Lemma 4.1.

We have reformatted this section to typeset the Assumptions in the same manner as the Lemmas in the paper. Our motivation for not phrasing Assumption 1 and Assumption 2 as "assumptions," rather than theorems, is that we want to be transparent about the assumptions behind our method. While it is true that both these assumptions are guaranteed to hold in certain settings, real world settings may violate these assumptions (e.g., because of sampling noise, function approximation error). We welcome other suggestions for making this section as precise as possible.

Reviewer comment (Novelty):
Is the primary assertion in Section 3, which states "learned representations follow isotropic Gaussian distributions," a novel contribution? The current phrasing does not distinctly convey what aspects are original. For instance, the reference to "supported by analysis based on Wang and Isola" is somewhat ambiguous. Does this imply that such analysis has been previously conducted in their work?

Author response:
> Is the primary assertion in Section 3, which states "learned representations follow isotropic Gaussian distributions," a novel contribution?

This assertion is a very minor extension of prior work, which is why we've placed it in the preliminaries section. We've added a note to explain this at the end of Section 3.

> For instance, the reference to "supported by analysis based on Wang and Isola" is somewhat ambiguous. Does this imply that such analysis has been previously conducted in their work?

We have revised this sentence to clarify the relationship with prior work. Prior work provides some theoretical intuition for why Assumption 2 should hold. For clarity, the Appendix explains why this intuition from prior work also holds for Gaussian distributions, rather than von Mises Fisher distributions.