Paper: Video2Demo: Grounding Videos in State-Action Demonstrations (ICLR-2024)

Reviewer comment (Experiments):
**Other LLMs and VLMs**: How do other LLMs and VLMs perform on this task? I am curious to see how this framework is generalized to other models.

Author response:
### **Other VLMs and LLMs:**
Our choice for the generative LLM was straightforward: there are currently no other LLMs that match the context capacity, instruction following and logical coherence capabilities that GPT-4 has [1]

Our generative VLM choice was based on recent literature surveys, underlying simplicity of the architecture, and the choice of pre-trained models in LLaVA [2,3]. Due to time and cost limitations, we could not ablate the choice of the VLMs. However, the method is plug and play for both the choice of LLM and VLM, similar to the Socratic Model approach.

Finally, OpenAI has recently enabled GPT4-vision as part of their API[4]. For future work, we are interested in evaluating this in our setting as:
1. An alternative generative VLM in the `Video2Demo` pipeline, where we can expect better visual grounding compared to LLaVA.
2. A competing method to `Video2Demo`

Reviewer comment (Experiments):
**GPT4-V**: It would be good to include some results of GPT-4V. I know its API is not released yet, but some quick experiments through ChatGPT's UI are sufficient.

Author response:
Hello! 

Thank you again for your feedback and questions, and suggestions to strengthen our paper! 
As per your suggestion, we have added a qualitative comparison against GPT4-V on the same task, in the global response.

As the discussion period ends soon, please let us know if we can provide any additional clarifications or answers that would help you in your evaluation.

Reviewer comment (Experiments):
**Execution-based evaluation**: I wonder whether you can provide some execution-based results of robot code to prove the generated state-action sequences are really useful.

Author response:
### **On execution-based evaluation** 
In the global response, we elaborate on how we primarily focus on the problem of extracting rich information from offline vision-language demonstrations. In future work, we plan to evaluate `Video2Demo`’s generated state-action with task and motion planners [5,11] in simulators [9-10].

Reviewer comment (Novelty):
**Prior works**: It would be good to discuss the paper's relationship to some additional prior works:

[1] ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. Singh et al.

[2] VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. Huang et al.

[3] Voyager: An Open-Ended Embodied Agent with Large Language Models. Wang et al.

Author response:
### **Clarification on Prior Work:**
We thank the reviewer for their suggestion to consider ProgPrompt[5], Voyager[6] and Voxposer[7] as related works. In our next revision, we will duly acknowledge them.

We wish to emphasize that `Video2Demo` serves a different purpose from the mentioned works.

1. ProgPrompt is an LLM-based task planner translating language instructions to code
2. Voxposer defines affordances and rewards through code for motion planning
3. Voyager builds a skill library for playing Minecraft using LLMs. 

`Video2Demo` however, is not a planner. We focus on extracting state-action predicates through vision-language demonstrations. We use Demo2Code[8] as an example of how `Video2Demo`’s output can be used in downstream task planners like [5-7]. We aim to explore this in future work with more embodied planners.

Reviewer comment (Experiments):
The paper motivates the problem of "teach robots everyday tasks". But there are no simulated or real robots experiments which makes it hard to assess the practicality of proposed approach and the possible failure scenarios. For example, how would the generated task plan compare to execute task in simulated environments like ALFRED [1]. The scope and possible future implication can be clear, like the proposed solution seems well suited for video comprehension, that can facilitate task planning.

Author response:
### **On more Evaluations**
In our global response, we expand more on closing the loop with `Video2Demo` and evaluate using simulators [1-3]. We thank the reviewer for suggesting the ALFRED [1] benchmark to test our method. Since they also use high-level actions and egocentric vision, it provides the right testing ground. Our aim with this paper is to motivate the need for perception and generation of state-action sequences from task videos, and in future work, we hope to evaluate `Video2Demo` on the simulator.

Reviewer comment (Experiments):
Video2Demo relies heavily on prompt engineering, which requires considerable effort. It is unclear if the presented prompts are applicable to just EPIC kitchen videos only, or can be applied more broadly to other activity videos.

Author response:
### **On the reliance on prompt engineering**

Most section of `Video2Demo`'s prompt is domain-independent, so it can be applied to other activity videos. For domains significantly different EPIC-Kitchens, the only parts of the prompts that need to be changed are about the domain. Please refer to our answer to your 1st question for details.

Reviewer comment (Experiments):
Can it scale to beyond egocentric videos in EPIC kitchen to third-person tutorial videos? How would the prompt change, especially in the phase 2 where the prompts and state-action predicates seems to be centered on the human in the videos?

Author response:
**Q1: Can it scale to beyond egocentric videos in EPIC kitchen to third-person tutorial videos? How would the prompt change, especially in the phase 2 where the prompts and state-action predicates seems to be centered on the human in the videos?**


While we did not evaluate our system on third-person videos in this paper, `Video2Demo` can easily be applied to generate state and action predicates for third-person tutorial videos.

`Video2Demo`'s VLM and LLM prompts are mostly domain-independent. To gain better performance in new environments, one only needs to modify the following:

* Domain information in the LLM's prompt, which could include
    + Overall setting or theme of the vision language demonstrations (e.g. house chores)
    + Examples of the state predicates
    + Examples of the action predicates
    + Domain-specific rules (e.g. we forbid the LLM from directly referring to the human in its state and action predicates). 
* Domain information in the VLM
    + Whether the images are first-person view or third-person view

Reviewer comment (Novelty):
My main criticism for this paper is its contribution's significance. Using LLM and VLM with text as history seems like an obvious design choice. The techniques the authors introduced over the Socratic Model, namely the way to structure and prompt the LLM / VLM interaction doesn't seem to constitute enough contribution to be an ICLR paper.

Author response:
### **On the contribution in the paper**

The contribution of the paper is not just in the method of LLM/VLM interaction, but also in solving the problem of extracting symbolic states and actions from long vision language demonstration. The baseline result shows that
* The captions that VLM generates are not sufficient in extracting salient states about objects in the frame. (see SM-Caption)
* Despite including history, a predefined, fixed interaction between the LLM and the VLM is also not sufficient in accurately extracting salient information because the LLM has no way to query VLM about inconsistencies in its answer. (see SM-Fixed-Q) 

Naive application of LLM and VLM cannot effectively solve this problem. `Video2Demo`, despite its simplicity, is able to significantly outperform the baselines by extracting symbolic states and actions with higher quality.

Reviewer comment (Writing):
There are also a few misleading claims. Throughout the paper, the authors talks about constructing "state-action" pairs, while in reality what they extract are some loose-form text predicates as well as loose-form text actions. Such abusive use of terms misleads the readers when they read the abstract.

Author response:
### **Clarification on Outputs**
In the global response under “Representation of States and Actions”, we clarify the choice of how we represent states and actions. While they may be text-based, they are structured in predicate format, and a majority of the most commonly used state predicates and actions are provided to GPT-4 as part of its prompt. Examples of predicates:

* state: `is_inside(<A>, <B>)`, `human_is_holding(<A>)`, `is_on_top_of(<A>, <B>)`, etc.
* action: `pick_up(<A>)`, `open(<A>)`, `turn_off(<A>)`, etc.

Since it is taxing to design a comprehensive pddl-like specification for every possible predicate in a complex setting like EPIC-Kitchens, we allow the model to come up with its own predicates similar to the examples. This can be restricted in the prompt if `Video2Demo` is to be used in a simpler enumerable setting.

Reviewer comment (Novelty):
Overall, I think the current status of the paper lacks the significance an ICLR paper would need.

Author response:
### **On the contribution in the paper**

The contribution of the paper is not just in the method of LLM/VLM interaction, but also in solving the problem of extracting symbolic states and actions from long vision language demonstration. The baseline result shows that
* The captions that VLM generates are not sufficient in extracting salient states about objects in the frame. (see SM-Caption)
* Despite including history, a predefined, fixed interaction between the LLM and the VLM is also not sufficient in accurately extracting salient information because the LLM has no way to query VLM about inconsistencies in its answer. (see SM-Fixed-Q) 

Naive application of LLM and VLM cannot effectively solve this problem. `Video2Demo`, despite its simplicity, is able to significantly outperform the baselines by extracting symbolic states and actions with higher quality.

Reviewer comment (Writing):
Could you clearly define "structure and temporal consistency" in the abstract?

Author response:
**Q2: Could you clearly define "structure and temporal consistency" in the abstract?**

The term structure and temporal consistency refers to the importance of understanding past actions and the relevance of objects in past timesteps (i.e. state of the environment) to predict state-actions for the current timestep. Referring to past state-actions to infer the current state and action is evident during video annotations as human annotators as well. For example, if a cook is seen holding a pan in their hand at some time t:
* If the pan was brought over to the stove-top from elsewhere, it is likely to be put down in the next few frames
* If the pan was not in hand earlier, it was picked up right before time t.
* The pan, as part of the environment, may stay relevant in future actions in the demonstration.

If we use a VLM to generate captions without context, it may fail to recognize what is going on in the scene. However, responses from VLMs along with GPT-4’s ability to reason what states precede actions and what actions follow other actions enhances the consistency in actions and the state of the observed environment.

Reviewer comment (Experiments):
**Not enough evaluation**: it's very difficult to get a signal for the method's abilities given that evaluations are performed on contrived code for 7 videos.

Author response:
### **On insufficient evaluation**

In quantitative evaluation, we focus on the quality of the generated state and action predicates. In Table 1 and Figure 3 in our paper, we evaluated the approaches on 19 videos that have human state-action annotation. We show how our generated predicates are closest to human annotations compared to other baselines across all kitchen activity categories. 

Our evaluation for code generation on 7 dishwashing videos is a qualitative example of a downstream application of `Video2Demo`. We show that [2], which takes text-based state-action trajectories as input, can generate expert-like code when it uses the state-actions generated by `Video2Demo`.

Reviewer comment (Experiments):
**Choice of action space perhaps makes this task too easy**: Generating code is generally useful but in this setup, it's difficult to apply to real-life scenarios due to the level of abstraction. For instance, the reference code uses functions like “check_if_dirty(object)”, a level of abstraction for which we do not have good robot behaviors. In a sense, the method performs a kind of task-level planning. But the level of abstraction of this planning makes it impossible to test in a control setup.

Author response:
### **Clarification on the choice of action space**

In global response, we clarify why we picked a high-level action space. We would like to add that despite the high-level resolution of this task, predicting these high-level actions is still not easy, as is evident from the accuracy and recall scores from the baseline VLM-based methods and even `Video2Demo`.

While we acknowledge that code generated by task planners may not be feasible or affordable by the embodiment in that environment, we note that it is not the focus of our work. We aim to be able to generate states and actions that can increase the success rate for language-based planners compared to only providing language instructions. Qualitatively, we see that our demonstrations allow a code-generation pipeline based on [1-2] to generate expert-like code with loops and conditions that are non-trivial to infer from simple language instructions.

Reviewer comment (Novelty):
**Unclear motivation**: The authors claim that this work tries to ground videos into state actions and states for robot demos. Unfortunately, this is simply not true. They describe the state in videos using text and ground actions into pseudo-code. This is far from the promise of a state-action demonstration.

Author response:
### **Clarification on Motivation**

In the global response, we clarify our motivation behind this work and why we represent our state-action sequences in text-predicate form. 

Our method aims to ground the visual information available in hours of offline demonstration data such as EPIC-Kitchens by converting it into text-based states and actions. We hope that language-based task planners [2-5] can learn the high-level task plan from symbolic state-action trajectory—bridging the gap between language instruction and planning, as seen in [1-2].

Reviewer comment (Experiments):
**No robotic evaluations**: The paper does not run any experiments on robots - neither in simulation nor on real robots. Therefore, I believe calling this a method to generate state-action demonstrations is an overclaim.

Author response:
### **On Robotic Evaluation**

In the global response, we elaborate on how the paper focuses on the problem of extracting rich information from offline vision-language demonstrations. In future work, we plan to evaluate downstream applications of `Video2Demo`’s generated state-action in simulations (e.g. ALFRED, AI2-THOR [7-8]) and on real robots. 
On Cost and Speed of Deployment
We acknowledge the importance of fast perception models during the deployment and online rollout of learned policies. However, **`Video2Demo`'s applications are for offline settings, so it does not need to be deployed on a robot.**
 
Concretely, `Video2Demo` provides a framework for learning high-level task plans from few-shot demonstrations. We hope that with future work, we can:
1. Collect one or more demonstrations of a human performing a task.
2. Run `Video2Demo` to extract the high-level states and actions from these demonstrations
3. Rely on task-planning and motion-planning methods to leverage these generated states and actions to replicate the tasks, e.g., generate code to perform this task, conditioned on the robot embodiment, environment affordances etc.

Thus, `Video2Demo` can run offline asynchronously before the policy is learned then run on the robot.

Reviewer comment (Experiments):
What was the cost of running these experiments?

Author response:
**Q1: What was the cost of running these experiments?**

The costs for these experiments are mainly the cost of OpenAI’s GPT-4 API usage.

* The total cost was approximately $2200.
* Each demonstration costs about $15. 

We ran approximately 6 experiments on each of the 26 videos, encompassing several trials with the baselines, `Video2Demo`, ablations, and qualitative experiments for code generation. 

Lower costs for GPT-4 were recently announced by OpenAI [6], and we can accordingly expect the cost of this method to be reduced in the future.

Reviewer comment (Reproducibility):
Do you have one state for every image?

Author response:
**Q2:Do you have one state for every image?**

We do not generate states for every frame in the video because two consecutive frames are highly similar. Instead, `Video2Demo` generates a set of state predicates and one action for each action segment specified in the EPIC-Kitchens data.