Paper: Video Language Planning (ICLR-2024)

Reviewer comment (Experiments):
One of the key contributions of VLP is the use of text-to-video models as dynamics models. But authors’ experiments and baselines are not convincing about the value of using these models as dynamics models.

Author response:
**Text LLM for Model Dynamics.** To effectively use a LLM to model dynamics and plan on the precise manipulation settings we consider, the LLM must be able to accurately simulate changes in the visual state subject to each possible natural language action. This requires the first a text-based serialization of the objects in an image as well as resultant simulation based off this serialization. In appendix A.1 (Figure XV), we illustrate that existing LLMs, i.e. GPT-3.5, are unable to accurately simulate the fine dynamics changes of such a serialization.

Reviewer comment (Experiments):
Authors use relatively simple tasks to evaluate VLP. Why not use a text-based LLM or something like this: https://arxiv.org/abs/2106.00188 as a dynamics model instead of the text-to-video model given the simplicity of the evaluation tasks? It would have been great to see such a baseline to really understand the edge that video dynamic models provide.

Author response:
**Comparisons to Text-Based Dynamics Simulation.** We agree it would be better to compare with a text-based dynamics approach with access to training examples of serialized dynamics predictions. However, our existing training dataset only consists of sequences of images, as opposed to a text serialization, and there is no easy way to convert an image directly into a text representation to train a dynamics model. The most straightforward manner would be directly to tokenize images into multimodal tokens and predict dynamics in this manner and use this for planning (this is actually the first approach we experimented with -- see our response in generating multimodal tokens). If the reviewer believes this baseline is important, we are happy to add such a comparison in the final version of the paper.

Reviewer comment (Experiments):
It is also not clear which use-cases will really require VLP as compared to existing LLM-based planning models or more generalized agent architectures such as GATO (https://arxiv.org/abs/2205.06175) or decision transformers. To that end, it would have been useful to see a broader set of baselines in the paper. Some baselines to consider include: RAP: https://arxiv.org/abs/2305.14992 (uses MCTS + LLM dynamics model similar to VLP's tree search+dynamics model), Visual Language Planner: https://arxiv.org/pdf/2304.09179.pdf (learns a multimodal dynamics model and policy by finetuning an LLM), VIMA: https://arxiv.org/pdf/2210.03094.pdf (no dynamics model but multimodal planner, could truly bring out the value of video-based dynamics via such a comparison).

Author response:
**Baseline Comparisons.** Since LLMs are unable to accurately simulate the detailed video dynamics for object manipulation, we cannot apply RAP to our setting. In terms of GATO and VIMA, we already compare VLP against a multimodal transformer, RT-2 which directly predict actions (which both GATO and VIMA can be seen as instances of)  and find that our approach substantially outperforms this baseline. We have added references to each of these papers in the baseline section and clarified our comparisons against them.

Reviewer comment (Experiments):
Authors also say HiP is the closest to their work in terms of an approach but do not provide a performance comparison in their evaluation. Why? Would love to see it.

Author response:
**Comparison with HiP.** HiP leverages consensus between a text-to-video and LLM to synthesize a possible language instruction given the current visual observation of the scene. In VLP, we directly learn a VLM to predict text actions to execute given an image observation, which serves as an upper bound on the consensus operation in HiP. Since HiP does not do planning, its performance is upper bounded by the VLP (No Value Function) base which VLP substantially outperforms.  We have clarified this in the main paper.

Reviewer comment (Reproducibility):
It is unclear if the authors are considering open-sourcing the code and the models. Without that, the reproducibility and value of the work for the community reduces. I encourage the authors to discuss open-sourcing plans in their rebuttal.

Author response:
**Code and Models.** We will release an open-source codebase for reproducing the results in VLP by the camera-ready version of the paper. In terms of models – they are trained using a company's internal codebase, so we are unable to directly open-source the models that are trained in the paper. However, we will aim to release pretrained weights for models trained using the released open-source codebase.

Reviewer comment (Reproducibility):
Unclear how long of a video the video models produce.

Author response:
**Duration of Videos.** 30 minutes corresponds to the long horizon video plans shown on the website at https://vlp-iclr2024.github.io/ and corresponds to approximately 15-second videos (for example depicting a robot arranging a set of blocks into a line or moving blocks to a corner of the board).

Reviewer comment (Experiments):
While the paper properly discusses a few limitations, the limitation of the task horizon is unclear in this paper. In the appendix, the authors provided both VLP and other baselines with 1500 timesteps to complete a task. Does this imply that VLP may not be suitable for handling longer-horizon tasks? For example, a long-distance task, such as moving an object over a long distance (e.g., 10 meters) from one location to another.

Author response:
**Long Horizon Tasks.** We found that VLP had no difficulties with longer horizon tasks – we artificially set the task length to 1500 so that the evaluation of an episode did not take too long (as a single environment evaluation with 1500 steps already take 1 hour).

Reviewer comment (Experiments):
In the introduction, the authors emphasize the potential benefits of VLP when working with incomplete videos that lack corresponding language labels. It would be helpful to see more detailed descriptions and accompanying experimental results to further demonstrate this capability, which are not found in this paper.

Author response:
**Incomplete Videos Lacking Language Labels.** We would like to clarify that the videos used to train VLP in both Language Table, as well as the 14-DoF dexterous manipulation setting, already lack full language labels. In both settings, videos of the full long-horizon task are collected by teleop workers. However, there is no way then to directly segment videos into corresponding language labels, as there is no clear language label for each consecutive temporal segment of the video. Instead, we then asked a separate group of workers to go through video and select and label any sub-snippets of the video that had coherent language labels. These sub-snippets would often overlap in time in the video and many parts of the video would not have consistent language instructions.  We have clarified this in the main paper as well as in Appendix A.3.

Reviewer comment (Experiments):
The authors have highlighted the issue of overestimation with the Value Function. It would be interesting to see if out-of-distribution images or goals might also contribute to this problem. If so, could the application of certain offline reinforcement learning algorithms, such as CQL, potentially offer a solution for addressing this issue?

Author response:
**Overestimation of Value Function.** Yes, we indeed sometimes found that out-of-distribution goals would lead to overestimation of value function . We have added a reference to the CQL paper and methods in offline RL as ways to avoid the exploration of out-of-distribution states.

Reviewer comment (Writing):
Minor issues that may need careful proofreading:  
- Sec. 2: a image goal-conditioned -> an  
- as planning submodules Sec. 2.1 -> in Sec. 2.1

Author response:
**Typos.** Thank you, we have fixed the typos you mentioned.

Reviewer comment (Experiments):
It is unclear if the authors have considered optimizing only the text sequence with planning in Algorithm 1, followed by per-clip video generation. This approach seems much more efficient, as it would be easier to search through text instead of pixels. Based on Table 3, there appear to be large gains from a more exhaustive search, which would likely be easier to implement if rendering pixels was not required.

Author response:
**Searching with Text.** While it would be desirable to search first for a text plan and then through video prediction, this is difficult because the precise text action to execute next in the text plans depends heavily on the exact continuous state the previous text plan ends on. As an example, consider the text instruction “push the green circle to the left”. If the final goal is to group blocks by color, if the continuous execution of the text instruction pushes it to the left of the green square the next text instruction becomes “push the green circle slightly to the right”, while if its still to the right of the green square the, the text instruction becomes “push the green circle slightly to the left”. Thus to effectively plan, it is important to plan with both text and videos.

Reviewer comment (Reproducibility):
In Section 2.2 "Vision-Language Models as Heuristic Functions," is the model trained with regression, or the tokenized representation of the number of steps using the standard language modeling objective? If it is with the language modeling objective, do the authors choose the mode of the distribution over the predicted number of steps when using the heuristic to plan in Algorithm 1?

Author response:
**Heuristic Function.** The VLM is trained to predict the number of steps left to execute a task through language prediction on a tokenized representation of the steps left. This allows the VLM to sample a multimodal set of possible number of steps left until task completion. In practice, during search, we sample at 0 temperature from the VLM, corresponding to the mostly likely probability mode of the number of steps left to finish execution of the task. We have clarified this in Appendix A.4

Reviewer comment (Experiments):
In Section 2.3 "Replanning," is there a way to make the re-planning rate dynamic? Ideally, it would be possible to detect when reality has diverged from the video plan in a significant, unrecoverable way and trigger re-planning accordingly.

Author response:
**Replanning.** Yes, replanning can be made dynamic by making replanning occur based off the pixel MSE distance between the current observation and the goal image. We have added a discussion of this in the method section of the paper.

Reviewer comment (Reproducibility):
In A.4 "Video Models," what is the frequency of these videos? For example, how many seconds does 16 frames correspond to?

Author response:
**Video Model Clarification.** The videos are approximately at 15FPS, so 16 frames of video corresponds to one second of video. The long video rollouts shown in the submission cite correspond to between 10-15 video rollouts chained together.