Paper: Learning to Act from Actionless Videos through Dense Correspondences (ICLR-2024)

Reviewer comment (Novelty):
The work “Zero-Shot Robot Manipulation from Passive Human Videos” by Bharadhwaj et al. presents very similar (although not identical) ideas and is not discussed in the prior work or cited. Could the authors discuss and ideally perform a comparison to some of the ideas in that work?

Author response:
**A:** We thank the reviewer for suggesting this paper. It is definitely sharing the same spirit and some ideas with our work. We will discuss this paper in the revision in detail. While this paper and our work tackle the problem of learning from passive videos, we notice three differences between our method and Bharadhwaj et al.

First, our video prediction model is conditional on a textual description of the task. By contrast, Bharadhwaj et al. used an unconditional generation or goal-image conditioning.
Second, we predict an object-centric trajectory given the initial frame; by contrast, Bharadhwaj et al. learned to predict hand poses. While hand poses contain more details of manipulator-object interactions, they may have cross-embodiment transfer issues. Finally, Bharadhwaj et al. directly predict pose trajectories, while ours predicts videos first. In our early experiments, we have found that predicting video frames gives better results than predicting object motion (via prediction optical flows). We posit that this is because video prediction is a dense prediction problem, and convolutional neural networks have been designed for this task; by contrast, optical flows are more sparse for most of the scenes because there are usually only a few objects moving in the scene.

Unfortunately, comparing with Bharadhwaj et al. is not possible because the authors have not yet released their code/models at this moment.

Reviewer comment (Evaluation):
The video generation performance is seemingly quite good even when very few training trajectories are provided (just 165 videos for Meta-World). Can you comment about how overfitting can be avoided or provide some intuition?

Author response:
**A:** In general, we find that predicting trajectories with diffusion has less tendency to overfit than single forward models, as the diffusion model must learn the task of denoising multi-modal samples similar to trajectories of the training distribution (which gives much more supervision than directly learning a single state/action mapping). This can also be validated by comparing a simple BC baseline and Diffusion Policy. Here, we show an extended experiment below: a Diffusion Policy (Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion") with a similar network architecture outperforms the BC baseline.

| method | Door Open | Door Close | Basketball | Shelf Place | Button Press | Button Press Topdown | Faucet Close | Faucet Open | Handle Press |  Hammer | Assembly | Overall |
| -------------------------------------- | ---------:| ----------:| ----------:| -----------:| ------------:| --------------------:| ------------ | -----------:| ------------:| -------:| --------:| -------:|
|           BC |        21.3 |         36.0  |            0.0 |           0.0   |           34.7 |                  12.0  |           18.7 |          17.3 |           37.3 |      0.0   |        1.3 |      16.2 |
| Diffusion Policy                          |      45.3 |       45.3 |        8.0 |         0.0 |         40.0 |                 18.7 | 22.7|**58.7**|21.3|4.0|1.3|24.1|
AVDC (Full) |**72.0**| **89.3** | **37.3** | **18.7** | **60.0** | **24.0** | **53.3** | 24.0 | **81.3** | **8.0** | **6.7** | **43.1** | 

However, the task performance of AVDC still surpasses the Diffusion Policy by a large margin. We posit that video prediction is more data-efficient way to learn behavior than behavior cloning-based methods because it only needs to learn to predict object motions (how things should move in a video). By contrast, BC-based methods, including Diffusion Policies, require "joint" learning of object motion and how to control robots.

Reviewer comment (Experiments):
The performance of the UniPi baseline is surprisingly poor. Could you please provide an explanation for the common failure modes or visualizations? Same goes for the BC baselines. Is this due to the low amount of data provided, thus causing action prediction models to overfit? If so, would it be possible to report results with greater number of demonstrations (like 50, or even 15, rather than 5)?

Author response:
**A:** We thank the reviewer for this suggestion. We included an additional experiment of training BC with more data. The results are reported below.

|  [Method]-[# of demos] |   door-open |   door-close |   basketball |   shelf-place |   button-press |   button-press-topdown |   faucet-close |   faucet-open |   handle-press |   hammer |   assembly |   overall |
|------------|------------:|-------------:|-------------:|--------------:|---------------:|-----------------------:|---------------:|--------------:|---------------:|---------:|-----------:|----------:|
AVDC-165 |72.0| 89.3 | 37.3 | 18.7 | 60.0 | 24.0 | 53.3 | 24.0 | 81.3 | 8.0 | 6.7 | 43.1 | 
|           BC-165 |        21.3 |         36.0  |            0.0 |           0.0   |           34.7 |                  12.0  |           18.7 |          17.3 |           37.3 |      0.0   |        1.3 |      16.2 |
|         BC-330|        21.3 |         65.3 |            0.0 |           0.0   |           45.3 |                   21.3 |           44.0 |          29.3 |           29.3 |      2.7 |        0.0   |      23.5 |
|         BC-660|        61.3 |         72.0 |            0.0 |           1.3 |           77.3 |                   49.3 |           77.3 |          77.3 |           62.7 |     10.7 |        0.0   |      44.5 |
|        BC-1650|        96.0 |         81.3 |            0.0 |           0.0   |           96.0 |                   85.3 |           93.3 |          94.7 |           86.7 |     12.0 |        0.0   |      58.7 |

The experiment results above show that BC needs 20 videos per view (in total 660 videos with action labels, compared to AVDC trained with 165 videos without action labels) to perform similarly to our method AVDC (43.1% overall). That said, BC needs around four times more videos and action labels than our method, highlighting our proposed method's efficiency. Moreover, BC cannot solve most tasks that require grasping (e.g., pick and place, use tools) even with 50 demonstrations. We believe this indicates that the bottleneck of BC's performance is the number of demonstrations (or action labels). That said, BC's poor performance can be attributed to the low-data setting, causing the BC to overfit unwanted features and limiting it to generalizing to different initial configurations on various tasks.

Similarly, while UniPi also predicts videos, it relies on an inverse dynamics model trained with action labels to recover actions. Therefore, it has the same limitations when training with a limited amount of data. We would like to emphasize that we included the performance of BC and UniPi to calibrate the difficulty of the tasks, and BC and UniPi have access to action labels that are not accessible to our proposed method; therefore, the comparison is not fair.

Reviewer comment (Writing):
In the appendix H.1.2, the authors say "We calculate such direction by extrapolating the line between the grasp point and the first subgoal more than 10cm away from the grasp" Should it be push point?

Author response:
**A:** Yes. We thank the reviewer for catching this and will fix it in the revision.

Reviewer comment (Writing):
In Sec. H.1.3, "For the push mode, we re-initialize the gripper as described above" is not clear. What does the re-initialization refer to?

Author response:
**A:** When replanning is triggered, we re-initialize the gripper by (1) synthesizing a video plan, (2) sampling the grasp and calculating subgoals, and (3) calculating the direction for placing the gripper by extrapolating the line between the grasp point and the first subgoal. Once the re-initialization is done, we can start the robot execution. We will revise the paper to make this re-initialization procedure clear.

Reviewer comment (Experiments):
When learning the diffusion model for the IThor environment, why not apply the adaptable frame sampling technique in this case?

Author response:
**A:** We thank the reviewer for raising this question. We did not use adaptive sampling because iTHOR uses discrete actions such as moving and rotating for a constant distance or angle. We will clarify this in the paper.

Reviewer comment (Experiments):
The paper lacks comparison to a strong BC baseline. Since the AVDC method uses diffusion to predict images, it seems natural to compare against a diffusion policy (e.g., [B, C]). R3M is a fairly old representation, and something more recent like Voltron [D] would be a better choice. The simple MLP-based BC policy would likely underperform compared to diffusion policy, which is reflected in the poor baseline performance in Tables 1 and 2. In Sec. 4.3, the authors mention that 'R3M is pretrained on robot manipulation tasks ... it might not be suitable for visual navigation tasks'. A better baseline for visual navigation might be [E].  
[D] Karamcheti, Siddharth, et al. "Language-driven representation learning for robotics." arXiv preprint arXiv:2302.12766 (2023).  
[E] Shah, Dhruv, et al. "ViNT: A Foundation Model for Visual Navigation." arXiv preprint arXiv:2306.14846 (2023).

Author response:
**Scaling up and distilling down** does not learn from videos; instead, it focuses on 6 DoF robotic skill learning, which differs from our problem formulation.

**Language-driven representation learning for robotics** and **ViNT**: We thank the reviewer for suggesting these works that learn representations. To validate our BC implementation, we included an additional experiment of training BC with more data. The results are reported below.

|  [Method]-[# of demos] |   door-open |   door-close |   basketball |   shelf-place |   button-press |   button-press-topdown |   faucet-close |   faucet-open |   handle-press |   hammer |   assembly |   overall |
|------------|------------:|-------------:|-------------:|--------------:|---------------:|-----------------------:|---------------:|--------------:|---------------:|---------:|-----------:|----------:|
AVDC-165 |72.0| 89.3 | 37.3 | 18.7 | 60.0 | 24.0 | 53.3 | 24.0 | 81.3 | 8.0 | 6.7 | 43.1 | 
|           BC-165 |        21.3 |         36.0  |            0.0 |           0.0   |           34.7 |                  12.0  |           18.7 |          17.3 |           37.3 |      0.0   |        1.3 |      16.2 |
|         BC-330|        21.3 |         65.3 |            0.0 |           0.0   |           45.3 |                   21.3 |           44.0 |          29.3 |           29.3 |      2.7 |        0.0   |      23.5 |
|         BC-660|        61.3 |         72.0 |            0.0 |           1.3 |           77.3 |                   49.3 |           77.3 |          77.3 |           62.7 |     10.7 |        0.0   |      44.5 |
|        BC-1650|        96.0 |         81.3 |            0.0 |           0.0   |           96.0 |                   85.3 |           93.3 |          94.7 |           86.7 |     12.0 |        0.0   |      58.7 |

The experiment results above show that plain BC needs 20 videos per view (in total 660 videos with action labels, compared to AVDC trained with 165 videos without action labesls) to perform similarly to our method AVDC (43.1% overall). That said, BC needs around 4 times more videos and action labels than our method, which highlights the efficiency of our proposed method. More importantly, BC still cannot learn most tasks that require grasps (e.g., pick and place, use tools) even with 50 demonstrations per view (BC-1650). We believe this indicates that the bottleneck of BC's performance is the number of demonstrations (or action labels), instead of learned representations.

We would like to emphasize that we included the performance BC mainly for calibrating the difficulty of the tasks, and BC has access to action labels that are not accessible to our proposed method; therefore, the comparison is not fair.

Reviewer comment (Presentation):
In Sec. 4.5 'Results', the paper states that Fig. 10 presents screenshots of robot trajectories, but it appears that these are shown in Fig. 9. Fig. 10 actually shows human predicted trajectories.

Author response:
**Q2** Figure 10 clarifications.
> In Sec. 4.5 'Results', the paper states that Fig. 10 presents screenshots of robot trajectories, but I believe that is Fig. 9? Fig. 10 shows human predicted trajectories.

**A:** We thank the reviewer for pointing this out. Figure 10 visualizes "robot planned trajectories" instead of "robot trajectories." We will revise the paper to fix this. The real-world robot execution trajectories can be found in section B or on our [supplementary website](https://flow-for-action-from-video.github.io/supplements/#realresults).

Reviewer comment (Writing):
Typo — Page 3: 'Unipi'.

Author response:
**Q3** Typos.
> - Page 3 - 'Unipi'.
> - Sec. 4.1: 'compare AVDC to its [variants] that also predict dense correspondence'.
> - Sec. 4.2: 'maximum number of planning affects' -> 'maximum number of replanning steps affects'.

**A:** We thank the reviewer and will fix the typos in the revision.

Reviewer comment (Writing):
Wording issue in Sec. 4.1 — the phrase 'compare AVDC to its [variants] that also predict dense correspondence' is unclear.

Author response:
**Q3** Typos.
> - Page 3 - 'Unipi'.
> - Sec. 4.1: 'compare AVDC to its [variants] that also predict dense correspondence'.
> - Sec. 4.2: 'maximum number of planning affects' -> 'maximum number of replanning steps affects'.

**A:** We thank the reviewer and will fix the typos in the revision.

Reviewer comment (Writing):
Typo in Sec. 4.2 — 'maximum number of planning affects' should be changed to 'maximum number of replanning steps affects'.

Author response:
**Q3** Typos.
> - Page 3 - 'Unipi'.
> - Sec. 4.1: 'compare AVDC to its [variants] that also predict dense correspondence'.
> - Sec. 4.2: 'maximum number of planning affects' -> 'maximum number of replanning steps affects'.

**A:** We thank the reviewer and will fix the typos in the revision.

Reviewer comment (Experiments):
I think one of the main limitations is the setting: AVDC needs videos of robots performing the task. I believe this is a contrived setting as it is very likely that if video + 3D information is available, then this was a robot demonstration, and one can just collect action data. To me, it is unclear how this approach will scale beyond robot data.

Author response:
**A:** We would like to clarify the following two points:
- **Our work does not need any 3D information during training**, and therefore, our proposed method learns from only videos with RGB frames. The 3D information is only required during inference for rendering robot actions. 
- **Our work can learn from videos of humans or robots that are different from the hardware setup performing actions.** Specifically, to show the ability of our proposed method to transfer across different embodiments, we have shown successful results with visual pusher experiments in Section 4.4, where the robot learns from *human video* (whose actions are not available). Additionally, in our real-world Franka Emica Panda arm experiment presented in Section 4.5, the robot learns from a large offline dataset from *other robots*, whose actions are not directly usable, and a small number of *human data* without action labels. AVDC only needs videos of an arbitrary agent performing tasks, such as a human or a different robot. This ability to learn from diverse data sources underscores its potential for real-world applications.

Reviewer comment (Experiments):
I am concerned by the reported results for the BC baseline. Due to action data being available, as well as the robot data being in-domain for the task a simple BC or kNN baseline should work very well. There are many cases where the results are < 5% success. This should be addressed. I would be willing to increase my score if this weakness is addressed.

Author response:
**A:** Thank you for the suggestion. To validate our BC implementation, we included an additional experiment of training BC with more data. The results are reported below.

|  method-total_demos |   door-open |   door-close |   basketball |   shelf-place |   button-press |   button-press-topdown |   faucet-close |   faucet-open |   handle-press |   hammer |   assembly |   overall |
|------------|------------:|-------------:|-------------:|--------------:|---------------:|-----------------------:|---------------:|--------------:|---------------:|---------:|-----------:|----------:|
AVDC-165 |72.0| 89.3 | 37.3 | 18.7 | 60.0 | 24.0 | 53.3 | 24.0 | 81.3 | 8.0 | 6.7 | 43.1 | 
|           BC-165 |        21.3 |         36.0  |            0.0 |           0.0   |           34.7 |                  12.0  |           18.7 |          17.3 |           37.3 |      0.0   |        1.3 |      16.2 |
|         BC-330|        21.3 |         65.3 |            0.0 |           0.0   |           45.3 |                   21.3 |           44.0 |          29.3 |           29.3 |      2.7 |        0.0   |      23.5 |
|         BC-660|        61.3 |         72.0 |            0.0 |           1.3 |           77.3 |                   49.3 |           77.3 |          77.3 |           62.7 |     10.7 |        0.0   |      44.5 |
|        BC-1650|        96.0 |         81.3 |            0.0 |           0.0   |           96.0 |                   85.3 |           93.3 |          94.7 |           86.7 |     12.0 |        0.0   |      58.7 |

The experiment results above show that plain BC needs 20 videos per view (in total 660 videos with action labels, compared to AVDC trained with 165 videos without action labels) to perform similarly to our method AVDC (43.1% overall). That said, BC needs around four times more videos and action labels than our method, which highlights the efficiency of our proposed method. More importantly, BC still cannot learn most tasks that require grasping (e.g., pick and place, use tools) even with 50 demonstrations per view (BC-1650). 

We would like to emphasize that we included the performance of BC to calibrate the difficulty of the tasks, and BC has access to action labels that are not accessible to our proposed method; therefore, the comparison is not fair.

Reviewer comment (Experiments):
AVDC relies on object/robot masks - a simple baseline would be to use those as a proxy for the actions. One could get pseudo action labels from the videos and train a policy.

Author response:
**A:** We thank the reviewer for the suggestion. However, this is not directly applicable to our setting because computing robot actions requires depth, and we do not assume depth information during training, and none of the Bridge and Visual Pusher datasets have depth data. 

Still, to provide an idea of the performance of using object masks as a proxy for actions as suggested by the reviewer. We have conducted experiments with the following setting: We trained a model that takes in a segmented object and directly predicts optical flow within the segmentation (without diffusion). Then, we used the procedure as AVDC to calculate actions. The results are presented as follows.

||   door-open |   door-close |   basketball |   shelf-place |   button-press |   button-press-topdown |   faucet-close |   faucet-open |   handle-press |   hammer |   assembly |   overall |
|---|------------:|-------------:|-------------:|--------------:|---------------:|-----------------------:|---------------:|--------------:|---------------:|---------:|-----------:|----------:|
|Object mask proxy|1.3|20.0|0.0|0.0|12.0|2.7|25.3|9.3|17.3|2.7|0.0|8.2|
|AVDC (Full) |72.0| 89.3 | 37.3 | 18.7 | 60.0 | 24.0 | 53.3 | 24.0 | 81.3 | 8.0 | 6.7 | 43.1 |

The results show that our proposed AVDC outperforms the method that predicts the object masks as a proxy for the actions. We will revise the paper to include this experiment.

Reviewer comment (Theory):
AVDC assumes that all objects are going to be directly manipulated by the robot directly but this is not the when one object as a tool.

Author response:
**A:** We would like to clarify that tasks requiring using tools can be indeed achieved with our proposed method. In particular, our method can achieve this by predicting the movements of a tool and rendering actions that induce such movements. For example, in Meta-World, the hammer task requires using a hammer as a tool.

Reviewer comment (Experiments):
Navigation approaches have many action free baselines which should be explored as well.

Author response:
**A:** We thank the reviewer for the suggestion. However, we are unsure what the "action-free" navigation baselines mean. Can the reviewer kindly provide specific papers or instruct on conducting additional comparisons?

Reviewer comment (Experiments):
It would be good to see real world experiments.

Author response:
**A:** We have shown real-world experiments with a Franka Emika Panda arm in Section 4.5; a further discussion can be found in Appendix H.3. Please also kindly see our videos in the [supplementary website](https://flow-for-action-from-video.github.io/supplements/#realresults).

Reviewer comment (Evaluation):
It would be good to have more of an analysis on the quality of the video prediction model. I suspect it has a

Author response:
**A:** We thank the reviewer for the suggestion. To measure the quality of synthesized videos, we provide qualitative results in Figure 5, Figure 7, Figure 8, Figure 10, and Appendix G.4. Also, we quantify the quality of synthesized videos by reporting the MSE between the last frames of synthesized videos and ground truth videos, presented in Appendix E and on the [supplementary website](https://flow-for-action-from-video.github.io/supplements/#vidgencomparison). We believe this sufficiently justifies the quality of the synthesized videos.

Since the reviewer asked for additional analyses, we further quantitatively compared the synthesized videos to the ground truth videos regarding PSNR, SSIM, MSE, and LPIPS. Specifically, we synthesized videos with our trained Meta-World video model given unseen first frames, and compared every synthesized video frame to the corresponding ground truth video frame. We report average PSNR, SSIM, MSE, and LPIPS (AlexNet) scores over 15 videos for each view, totaling 15\*3\*11=495 videos being evaluated. Also, we report the scores comparing the last frames of synthesized videos and ground truth videos. The following table summarizes the result.

||PSNR ↑|SSIM ↑|MSE ↓|LPIPS ↓|
|---|---:|---:|---:|---:|
|last frame|25.46|0.8920|0.0050|0.0525|
|whole video|25.02|0.8847|0.0057|0.0557|

The results show that our proposed video diffusion model can reliably synthesize videos of task execution.