Paper: Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR-2024)

Reviewer comment (Reproducibility):
Is there a plan to open-source the code and checkpoints? Making them available to the community would greatly facilitate efforts to scale up pretraining for robotics.

Author response:
Sure! We will release the code and checkpoints. We hope this project can serve as an effective approach to robot pre-training for the community.

Reviewer comment (Evaluation):
Comparing Table 1 and Table 2, the performance on the Calvin Benchmark significantly falls when only using 10% of robotic dataset for finetuning (2k trajectories) compared to using 100% of finetuning dataset (20k trajectories). According to the table, it looks like the primary reason is that the success rate primitive skills (i.e., success rate for "1 task completed in a row") falls from 94.9% to 77.8%. Thus it would be helpful to provide a further analysis into this phenomenon along with the failure modes.

Author response:
We visualized the success rate of each task in Tab. 7 of the paper and added discussions in Appendix A.7.  
It can be concluded that the performance on tasks which interacts with blocks (e.g., lift red block slider, place in slider, lift red block table) drops substantially when the data size decreases to 10%.  
These tasks are difficult in a sense that the robot needs to first grasp the correct block and then manipulate it according to the language instruction.  
In addition, the performance on turning on/off the lightbulb also decreases.

Reviewer comment (Experiments):
For zero-shot generalization experiments conducted in the paper, looks like the experiments are primarily conducted on object position variations, color variations, and background variations. It would be helpful to provide an analysis on whether the policy can (1) generalize to longer horizon tasks than the length of the tasks seen during training (e.g., train the policy on 1-5 tasks in a row, test on 6-10 tasks completed in a row); (2) generalize to novel language instructions that are semantically similar to those seen during training but with different wordings (e.g., instead of "lift the red cube", use "pick up the red cube" as instructions; instead of "push the sliding door to the right", use "open the door to the right side" as instructions).

Author response:
Regarding the task length, we believe there is a misunderstanding here.  
We strictly follow the evaluation protocol in [CALVIN](http://calvin.cs.uni-freiburg.de) [1].  
Each trajectory in the robot training data only contains one single task.  
Therefore, the robot is not trained on "1-5 tasks in a row".  
The evaluation is conducted by instructing the robot to perform up to 5 tasks in a row.  
Thus, the evaluation is already testing the capability of generalizing to longer horizon tasks than the length of the tasks seen during training.  
GR-1 improves the success rate of completing 5 tasks in a row from 38.3% to 73.1%.  
This significant improvement shows its strong long-horizon multi-tasking capability.  

Regarding the novel language instructions, we added an experiment to test the generalization capability to unseen languages.  
Specifically, we use GPT-4 to generate 50 synonymous instructions for each of the 34 tasks and randomly sampled from them during evaluation.  
See the following table for some examples.  
More examples can be found in Tab. 6 in the updated paper.  

|                    Original                     |                     Generated                     |  
|:-----------------------------------------------:|:-------------------------------------------------:|  
|   "use the switch to turn off the light bulb"   |     "use the switch to stop the light source"     |  
| "slide the block that it falls into the drawer" | "Move the block ensuring it goes into the drawer" |  
| "take the red block and rotate it to the right" |        "Twist the red object to the right"        |  
|         "grasp and lift the blue block"         |      "grasp firmly and raise the blue block"      |  
|    "push the sliding door to the right side"    |   "Guide the sliding door to move on the right"   |  

For GR-1 and all the comparing baseline methods, we use the model trained on ABCD->D split for evaluation and test on the same sampled set of unseen language instructions.  
Results are shown in the following table.  

| Method     |   1   |   2   |   3   |   4   |   5   | Avg. Len. |  
|------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|  
| MT-R3M     | 0.512 | 0.249 | 0.106 | 0.040 | 0.017 |   0.92    |  
| RT-1       | 0.494 | 0.222 | 0.086 | 0.036 | 0.017 |   0.86    |  
| HULC       | 0.715 | 0.470 | 0.308 | 0.199 | 0.130 |   1.82    |  
| GR-1       | 0.764 | 0.555 | 0.381 | 0.270 | 0.196 |   2.17    |  

GR-1 outperforms all the comparing methods, improving the success rate from 71.5% to 76.4%.  
We hypothesize that this generalization capability attributes to 1) being exposed to diverse languages in the large video dataset during pre-training and 2) freezing the strong CLIP text encoder during the whole training process to retain its powerful text encoding capability.  
The strong performance on zero-shot generalization to unseen languages highlights GR-1's potential to be applied in daily life where human languages are diverse.  

[1] Mees, Oier, et al. "Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks." IEEE Robotics and Automation Letters 7.3 (2022): 7327-7334.

Reviewer comment (Experiments):
It would be helpful to provide an ablation on the effect of different quantities of Ego4D pretraining datasets on downstream robot manipulation task performance. Also is there a (possibly smaller) subset of Ego4D dataset for pretraining that is especially helpful for downstream robot manipulation tasks?

Author response:
Thanks for your advice.  
We perform experiments on pre-training GR-1 on 1/4 and 1/2 of the full data used for pre-training in the paper respectively.  
We then finetune the pre-trained models on 10% data from ABCD-D split.  
Results are shown in the following table.  

| Pre-Training Data |   1   |   2   |   3   |   4   |   5   | Avg. Len. |  
|-------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|  
| 1/4               | 0.655 | 0.395 | 0.247 | 0.144 | 0.077 |   1.52    |  
| 1/2               | 0.694 | 0.422 | 0.252 | 0.152 | 0.096 |   1.62    |  
| Full              | 0.778 | 0.533 | 0.332 | 0.218 | 0.139 |   2.00    |  

The above results show that the performance increases with the increase of pre-training data size.  
We are also interested in exploring whether there is a smaller subset of Ego4D dataset that is especially helpful for robot manipulation task.  
However, given the limited rebuttal time and computation resources, we are not able to perform systematic experiments to investigate this point.  
As indicated in the future work, we plan to explore more on this topic and compare the difference of pre-training on videos of any kind v.s. only videos that are more relevant to manipulation.

Reviewer comment (Experiments):
The number of visual features per image from the MAE encoder can be large, and as a result, the current policy might not generalize to very long horizon tasks. It would be helpful to explore using e.g., Perceiver Resampler, to downsample the number of input image features.

Author response:
Yes, it is correct.  
And this is exactly what we have done in GR-1.  
We downsample the visual feature outputs from MAE with a perceiver resampler to 9 tokens.  
For more details, please refer to Fig. 2(c) and Sec. 3.2.1 in the paper.

Reviewer comment (Experiments):
The literature review is missing a number of relevant works. The baselines considered are also not necessarily state-of-the-art. This year, a number of works have come out that would be prudent to compare against.

Author response:
We have added citations to all these relevant works in the paper.
As for baseline comparison, V-PTR is a concurrent work and its code is not open-sourced.
We added extensive experiments to compare with R3M in both simulation and the real world since it is also pre-trained on Ego4D dataset.
- We first performed an experiment using a similar method proposed in the original R3M paper [1].
We use the pre-trained R3M image encoder to encode the static image and the hand image of the current timestep into two image embeddings.
In order to solve multiple tasks, we use the text encoder to encode the language instruction into text embedding.
Text and image embeddings are passed through linear layers to align the dimension and then through an MLP to output actions.
Following [1], the R3M image encoder is frozen during training.
We denote this multi-task version of R3M as *MT-R3M-MLP*.
- However, we find the performance of this method poor in both simulation and real robot experiments.
We hypothesize the reason is that 1) the model does not have access to history 2) the number of training parameters is too small. We further implemented another version with R3M denoted as *MT-R3M*.
Specifically, *MT-R3M* employs a gpt-style transformer to model the history information. The trainable parameters are the same as that in GR-1.
- We also include Voltron results for reference. Here Voltron* means fine-tuning the entire model to get better results.

Results on CALVIN ABCD split are shown in the following table. More results on CALVIN can be found in Tab. 1 in the paper.

| Method     |   1   |   2   |   3   |   4   |   5   | Avg. Len. |
|------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|
| MT-R3M-MLP | 0.085 | 0.005 | 0.001 | 0.000 | 0.000 |   0.09    |
| MT-R3M     | 0.752 | 0.527 | 0.375 | 0.258 | 0.163 |   2.08    |
| Voltron    | 0.101 | 0.300 | 0.100 | 0.010 | 0.000 |   0.11    |
| Voltron*   | 0.837 | 0.566 | 0.352 | 0.208 | 0.115 |   2.08    |
| RT-1       | 0.844 | 0.617 | 0.438 | 0.323 | 0.227 |   2.45    |
| HULC       | 0.889 | 0.733 | 0.587 | 0.475 | 0.383 |   3.06    |
| GR-1       | 0.949 | 0.896 | 0.844 | 0.789 | 0.731 |   4.21    |

We also compare with MT-R3M in two real robot experiments.
Results on object transport experiment can be found in the following table.

| Method | Seen Objects | Unseen Instances | Unseen Objects |
|--------|:------------:|:----------------:|:--------------:|
| RT-1   |     0.27     |       0.13       |      0.00      |
| MT-R3M |     0.15     |       0.13       |      0.10      |
| GR-1   |     0.79     |       0.73       |      0.30      |

Results on articulated object manipulation experiment can be found in the following table.

| Method | Average Success Rate |
|--------|:--------------------:|
| MT-R3M |         0.30         |
| RT-1   |         0.35         |
| GR-1   |         0.75         |

On CALVIN benchmark, GR-1 outperforms MT-R3M in 1) multi-task learning on ABCD->D split 2) zero-shot generalization on ABC->D split 3) small dataset setting on 10% ABCD->D data.
We also performed experiments on zero-shot generalization to unseen language instructions.
And GR-1 achieves a success rate of 76.4% while MT-R3M achieves 51.2%.
In real-robot experiments, GR-1 outperforms MT-R3M in both object transportation and articulated object manipulation.
**These results highlight the strength of the video generative pre-training in GR-1 on visual manipulation learning.**

[1] Nair, Suraj, et al. "R3M: A Universal Visual Representation for Robot Manipulation." arXiv preprint arXiv:2203.12601 (2022).

Reviewer comment (Novelty):
Diffusion policy has shown very good results in terms of multi-task, low-data regime performance. [B] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." arXiv preprint arXiv:2303.04137 (2023). [C] Ha, Huy, Pete Florence, and Shuran Song. "Scaling up and distilling down: Language-guided robot skill acquisition." arXiv preprint arXiv:2307.14535 (2023).

Author response:
We have added citations to all these relevant works in the paper.
As for baseline comparison, V-PTR is a concurrent work and its code is not open-sourced.
We added extensive experiments to compare with R3M in both simulation and the real world since it is also pre-trained on Ego4D dataset.
- We first performed an experiment using a similar method proposed in the original R3M paper [1].
We use the pre-trained R3M image encoder to encode the static image and the hand image of the current timestep into two image embeddings.
In order to solve multiple tasks, we use the text encoder to encode the language instruction into text embedding.
Text and image embeddings are passed through linear layers to align the dimension and then through an MLP to output actions.
Following [1], the R3M image encoder is frozen during training.
We denote this multi-task version of R3M as *MT-R3M-MLP*.
- However, we find the performance of this method poor in both simulation and real robot experiments.
We hypothesize the reason is that 1) the model does not have access to history 2) the number of training parameters is too small. We further implemented another version with R3M denoted as *MT-R3M*.
Specifically, *MT-R3M* employs a gpt-style transformer to model the history information. The trainable parameters are the same as that in GR-1.
- We also include Voltron results for reference. Here Voltron* means fine-tuning the entire model to get better results.

Results on CALVIN ABCD split are shown in the following table. More results on CALVIN can be found in Tab. 1 in the paper.

| Method     |   1   |   2   |   3   |   4   |   5   | Avg. Len. |
|------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|
| MT-R3M-MLP | 0.085 | 0.005 | 0.001 | 0.000 | 0.000 |   0.09    |
| MT-R3M     | 0.752 | 0.527 | 0.375 | 0.258 | 0.163 |   2.08    |
| Voltron    | 0.101 | 0.300 | 0.100 | 0.010 | 0.000 |   0.11    |
| Voltron*   | 0.837 | 0.566 | 0.352 | 0.208 | 0.115 |   2.08    |
| RT-1       | 0.844 | 0.617 | 0.438 | 0.323 | 0.227 |   2.45    |
| HULC       | 0.889 | 0.733 | 0.587 | 0.475 | 0.383 |   3.06    |
| GR-1       | 0.949 | 0.896 | 0.844 | 0.789 | 0.731 |   4.21    |

We also compare with MT-R3M in two real robot experiments.
Results on object transport experiment can be found in the following table.

| Method | Seen Objects | Unseen Instances | Unseen Objects |
|--------|:------------:|:----------------:|:--------------:|
| RT-1   |     0.27     |       0.13       |      0.00      |
| MT-R3M |     0.15     |       0.13       |      0.10      |
| GR-1   |     0.79     |       0.73       |      0.30      |

Results on articulated object manipulation experiment can be found in the following table.

| Method | Average Success Rate |
|--------|:--------------------:|
| MT-R3M |         0.30         |
| RT-1   |         0.35         |
| GR-1   |         0.75         |

On CALVIN benchmark, GR-1 outperforms MT-R3M in 1) multi-task learning on ABCD->D split 2) zero-shot generalization on ABC->D split 3) small dataset setting on 10% ABCD->D data.
We also performed experiments on zero-shot generalization to unseen language instructions.
And GR-1 achieves a success rate of 76.4% while MT-R3M achieves 51.2%.
In real-robot experiments, GR-1 outperforms MT-R3M in both object transportation and articulated object manipulation.
**These results highlight the strength of the video generative pre-training in GR-1 on visual manipulation learning.**

[1] Nair, Suraj, et al. "R3M: A Universal Visual Representation for Robot Manipulation." arXiv preprint arXiv:2203.12601 (2022).

Reviewer comment (Experiments):
Pretrained video representations, such as R3M, VIP, and Voltron consider transformers, MAEs, and temporal video frames similarly to the proposed work. Taking a policy learning approach with a pre-trained representation such as the ones listed would have been another good option for a baseline. [D] Nair, Suraj, et al. "R3M: A universal visual representation for robot manipulation." arXiv preprint arXiv:2203.12601 (2022). [E] Ma, Yecheng Jason, et al. "VIP: Towards universal visual reward and representation via value-implicit pre-training." arXiv preprint arXiv:2210.00030 (2022). [F] Karamcheti, Siddharth, et al. "Language-driven representation learning for robotics." arXiv preprint arXiv:2302.12766 (2023).

Author response:
We have added citations to all these relevant works in the paper.
As for baseline comparison, V-PTR is a concurrent work and its code is not open-sourced.
We added extensive experiments to compare with R3M in both simulation and the real world since it is also pre-trained on Ego4D dataset.
- We first performed an experiment using a similar method proposed in the original R3M paper [1].
We use the pre-trained R3M image encoder to encode the static image and the hand image of the current timestep into two image embeddings.
In order to solve multiple tasks, we use the text encoder to encode the language instruction into text embedding.
Text and image embeddings are passed through linear layers to align the dimension and then through an MLP to output actions.
Following [1], the R3M image encoder is frozen during training.
We denote this multi-task version of R3M as *MT-R3M-MLP*.
- However, we find the performance of this method poor in both simulation and real robot experiments.
We hypothesize the reason is that 1) the model does not have access to history 2) the number of training parameters is too small. We further implemented another version with R3M denoted as *MT-R3M*.
Specifically, *MT-R3M* employs a gpt-style transformer to model the history information. The trainable parameters are the same as that in GR-1.
- We also include Voltron results for reference. Here Voltron* means fine-tuning the entire model to get better results.

Results on CALVIN ABCD split are shown in the following table. More results on CALVIN can be found in Tab. 1 in the paper.

| Method     |   1   |   2   |   3   |   4   |   5   | Avg. Len. |
|------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|
| MT-R3M-MLP | 0.085 | 0.005 | 0.001 | 0.000 | 0.000 |   0.09    |
| MT-R3M     | 0.752 | 0.527 | 0.375 | 0.258 | 0.163 |   2.08    |
| Voltron    | 0.101 | 0.300 | 0.100 | 0.010 | 0.000 |   0.11    |
| Voltron*   | 0.837 | 0.566 | 0.352 | 0.208 | 0.115 |   2.08    |
| RT-1       | 0.844 | 0.617 | 0.438 | 0.323 | 0.227 |   2.45    |
| HULC       | 0.889 | 0.733 | 0.587 | 0.475 | 0.383 |   3.06    |
| GR-1       | 0.949 | 0.896 | 0.844 | 0.789 | 0.731 |   4.21    |

We also compare with MT-R3M in two real robot experiments.
Results on object transport experiment can be found in the following table.

| Method | Seen Objects | Unseen Instances | Unseen Objects |
|--------|:------------:|:----------------:|:--------------:|
| RT-1   |     0.27     |       0.13       |      0.00      |
| MT-R3M |     0.15     |       0.13       |      0.10      |
| GR-1   |     0.79     |       0.73       |      0.30      |

Results on articulated object manipulation experiment can be found in the following table.

| Method | Average Success Rate |
|--------|:--------------------:|
| MT-R3M |         0.30         |
| RT-1   |         0.35         |
| GR-1   |         0.75         |

On CALVIN benchmark, GR-1 outperforms MT-R3M in 1) multi-task learning on ABCD->D split 2) zero-shot generalization on ABC->D split 3) small dataset setting on 10% ABCD->D data.
We also performed experiments on zero-shot generalization to unseen language instructions.
And GR-1 achieves a success rate of 76.4% while MT-R3M achieves 51.2%.
In real-robot experiments, GR-1 outperforms MT-R3M in both object transportation and articulated object manipulation.
**These results highlight the strength of the video generative pre-training in GR-1 on visual manipulation learning.**

[1] Nair, Suraj, et al. "R3M: A Universal Visual Representation for Robot Manipulation." arXiv preprint arXiv:2203.12601 (2022).

Reviewer comment (Novelty):
Given the above works, the statement "GR-1 for the first time shows that a unified GPT-style transformer, augmented with large-scale video generative pre-training is able to effectively generalize to multi-task visual robot manipulation." may not be accurate or precise.

Author response:
We are sorry for causing the confusion. What differs GR-1 from the mentioned relevant works is 1) its pre-training objective--*video generative pre-training* and 2) its model architecture--*GPT style transformer*.
- V-PTR, concurrent to our work, leverages *video latent dynamics modeling* to learn value functions and refine the visual features by offline RL on robot data.
- VIP uses large-scale human videos (Ego4D) to perform value-implicit pre-training which is able to provide visual reward and representation for downstream robotic tasks. VIP can be viewed as an *implicit time contrastive objective* as indicated in the paper.
- The two papers on Diffusion Policy did not exploit video pre-training.
- R3M and MVP also uses Ego4D dataset for pre-training. The pre-training objective for R3M and MVP are *contrastive learning* and *masked image reconstruction* respectively.

While some of the mentioned relevant works use video for pre-training, none performed *generative pre-training* for videos.
While some works leverages transformers as the model architecture, none of them uses GPT-style transformers.
Therefore, we believe *GR-1 is the first work which shows that a unified GPT-style transformer, augmented with large-scale video generative pre-training, is able to effectively generalize to multi-task visual robot manipulation*. Furthermore, inspired by the successes of the GPTs in language and multi-modal applications, we hope that GR-1 can modestly contribute to the expansion of these achievements.

Reviewer comment (Writing):
Section 3.2.2 is confusing in describing the masked tokens. The OBS token is masked to predict the future video frames—is that correct? The sentence "all the tokens, including the [OBS] tokens, can only attend to all the language and observation tokens but not the [OBS] in the past" is particularly unclear. Why specifically the tokens in the past? The notation of [OBS] and observation tokens adds to the confusion.

Author response:
We are sorry for causing the confusion.
GR-1 follows the causal attention mechanism typically used in generative pre-trained models, except that all the [ACT] and [OBS] tokens are masked.
The [OBS] tokens are masked to predict future video frames; the [ACT] tokens are masked to predict the actions.
During pre-training, a token can attend to all the tokens in the previous positions except the [OBS] tokens;
during finetuning, all the tokens can attend to all the tokens in the previous positions except the [ACT] and [OBS] tokens. We have updated the writing of this part for clarification in Sec. 3.2.2 of the updated paper.

Reviewer comment (Experiments):
In the experimental setup, the actual size of the training set is unclear. The assumption is that GT-1 does not handle data both with and without language.

Author response:
The training set of ABCD->D split contains 22,966 trajectories with language annotations.
It is true that GR-1 does not handle data without language.
In the future, we plan to extend GR-1 to handle both types of data and make use of more data in training.

Reviewer comment (Writing):
There is a typo where 'causal' is often written as 'casual'.

Author response:
We appreciate the reviewer for pointing out these typos.
In the updated paper, we have corrected all the mentioned typos in the paper and the reference.

Reviewer comment (Experiments):
How different is the D simulation setting from A, B, and C?

Author response:
As shown in Fig. 4 in the paper, the four environments are different in table textures and colors. 
Also, the positions of the sliding door, LED, bulb, light switch, and button are different across the four environments.
These large variations require the model to possess strong semantic understanding capabilities and accurate visual-motor control.
GR-1 outperforms the comparing baseline methods by a large margin in the zero-shot generalization setting, improving the averaged task success rate from 53.3% to 85.4%.
This shows the strong zero-shot generalization capability of GR-1 to unseen environments.

Reviewer comment (Reproducibility):
What is the $\Delta_t$ used in the experiments (e.g., 0.1 s)?

Author response:
The $\Delta t$ in the paper specifies the target image for a prediction in the sequence.
In pre-training, we sampled frames in which the duration between adjacent frames is 1/3s.
GR-1 is trained to predict the next frame in the sampled sequence, which is 1/3s in the future.
For CALVIN data and real robot data, GR-1 is trained to predict the 3rd frame in the future as the robot data is denser compared to the down-sampled Ego4D videos.

Reviewer comment (Experiments):
Why is only one action predicted at a time rather than a receding horizon style prediction? [B] found the latter to work well.

Author response:
We appreciate the reviewer for bringing up the receding horizon prediction.
The proposed method GR-1 is a principled method and can be easily extended to support receding horizon control (RHC) by attaching a policy head which is able to predict actions for multiple steps.
During the limited rebuttal time, we work diligently to implement an RHC-style GR-1 by changing the prediction of the action head from predicting one action to multiple actions.
Results on CALVIN ABCD split are shown in the following table.

| Method      |   1   |   2   |   3   |   4   |   5   | Avg. Len. |
|-------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|
| GR-1        | 0.949 | 0.896 | 0.844 | 0.789 | 0.731 |   4.21    |
| GR-1 w/ RHC | 0.950 | 0.902 | 0.857 | 0.816 | 0.761 |   4.29    |

We plan to investigate more on this point in the future.

Reviewer comment (Theory):
Is there multimodality in the distribution that can be modeled for the video frame prediction task?

Author response:
Currently, the video prediction of GR-1 does not support multi-modal prediction given that it uses an L2 loss to reconstruct the target frames.
However, GR-1 can be easily extended to involve multi-modality in video prediction.
One possible approach is to attach a VAE/VQ-GAN decoder for predicting the image.
And the video prediction can be trained by the corresponding generative loss to support multi-modality in video prediction.
We will investigate more on this point in the future.

Reviewer comment (Experiments):
The baselines are missing all relevant papers that also pre-train on human videos / image datasets. Many of these are cited in the related works, but some comparison is necessary to actually demonstrate that pre-training for video prediction has benefits compared to pre-training for other objects like masked prediction, contrastive learning etc. (which is a claim of the paper). For example, a multi-task version of R3M [3] can be a relevant baseline that pre-trains on Ego4D but to learn visual representations, and then fine-tunes with behavior cloning.

Author response:
We have added experiments to compare with R3M in the updated paper.
We use the pre-trained R3M image encoder to encode the static image and the hand image of the current timestep into two image embeddings.
In order to solve multiple tasks, we use the text encoder to encode the language instruction into text embedding.
These embeddings are passed through linear layers to align the dimension and then through an MLP to output actions.
Follwing [2], the R3M image encoder is frozen during training.
We denote this multi-task version of R3M as *MT-R3M-MLP*.
However, we find the performance of this method poor.
We hypothesize the reason is that 1) the model does not have access to history 2) the number of training parameters is too small.
We further implemented another version, denoted as *MT-R3M*, using R3M as the image encoder.
MT-R3M employs a GPT-style transformer to model the history information. The number of trainable parameters is the same as that in GR-1.
Results on CALVIN ABCD->D split are shown in the following table.
More results on CALVIN benchmark can be found in Tab. 1 of the updated paper.

| Method     |   1   |   2   |   3   |   4   |   5   | Avg. Len. |
|------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|
| MT-R3M-MLP | 0.085 | 0.005 | 0.001 | 0.000 | 0.000 |   0.09    |
| MT-R3M     | 0.752 | 0.527 | 0.375 | 0.258 | 0.163 |   2.08    |
| RT-1       | 0.844 | 0.617 | 0.438 | 0.323 | 0.227 |   2.45    |
| HULC       | 0.889 | 0.733 | 0.587 | 0.475 | 0.383 |   3.06    |
| GR-1       | 0.949 | 0.896 | 0.844 | 0.789 | 0.731 |   4.21    |

We also compare R3M in real robot experiments as shown in the response to **Q2**.
On CALVIN benchmark, GR-1 outperforms MT-R3M in 1) multi-task learning on ABCD->D split 2) zero-shot generalization on ABC->D split and 3) small dataset setting on 10% ABCD->D data.
We also performed experiments on zero-shot generalization to unseen language instructions.
And GR-1 achieves a success rate of 76.4% while MT-R3M achieves 51.2%.
In real-robot experiments, GR-1 outperforms MT-R3M in both object transportation and articulated object manipulation.

**These results all highlight the strength of the video generative pre-training in GR-1 on visual manipulation learning.**

[2] Nair, Suraj, et al. "R3M: A Universal Visual Representation for Robot Manipulation." arXiv preprint arXiv:2203.12601 (2022).

Reviewer comment (Experiments):
The baselines are severely lacking. For example, the authors claim that "compared with VIMA, our method is capable of dealing with much more complex 6-DoF long-horizon language-conditioned robot manipulation tasks". A statement like that should be supported by a comparison to some sort of VIMA-like method (i.e., an encoder-decoder architecture with cross-attention). There is also *tons* of prior work on video pretraining for robotics, much of which is cited in Section 2.3: e.g., R3M, VIP, MVP, and many more. Many of these also pretrain on Ego4D and finetune on robot data. It is impossible to evaluate the strength of GR-1 without comparison to these other methods.

Author response:
Thanks for your advice. We have included comparison with R3M since it is also pre-trained on Ego4D.
We use the pre-trained R3M image encoder to encode the static image and the hand image into two image embeddings.
In order to solve multiple tasks, we use the text encoder to encode the language instruction into text embedding.
These embeddings are passed through linear layers to align the dimension and then through an MLP to output actions.
We denote this multi-task version of R3M as *MT-R3M-MLP*.
However, we find the performance of this method poor.
We hypothesize the reason is that 1) the model does not have access to history and 2) the number of training parameters is too small.
We further implemented another version, denoted as *MT-R3M*, using R3M as the image encoder.
MT-R3M employs a GPT-style transformer to model the history information. The number of trainable parameters is the same as that in GR-1.
Results are shown in the following.
- On CALVIN benchmark, GR-1 outperforms MT-R3M in 1) multi-task learning on ABCD->D split 2) zero-shot generalization on ABC->D split 3) small dataset setting on 10% ABCD->D data. The following table shows the results trained on ABCD->D split. More results on CALVIN can be found in Tab. 1 in the paper.

    | Method     |   1   |   2   |   3   |   4   |   5   | Avg. Len. |
    |------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|
    | MT-R3M-MLP | 0.085 | 0.005 | 0.001 | 0.000 | 0.000 |   0.09    |
    | MT-R3M     | 0.752 | 0.527 | 0.375 | 0.258 | 0.163 |   2.08    |
    | RT-1       | 0.844 | 0.617 | 0.438 | 0.323 | 0.227 |   2.45    |
    | HULC       | 0.889 | 0.733 | 0.587 | 0.475 | 0.383 |   3.06    |
    | GR-1       | 0.949 | 0.896 | 0.844 | 0.789 | 0.731 |   4.21    |

- We also compare with MT-R3M in real robot experiments. Results on object transportation and articulated object manipulation can be found in response to **Q2**. GR-1 outperforms MT-R3M in both object transportation and articulated object manipulation.

- We also performed experiments on zero-shot generalization to unseen language instructions. To do so, we use GPT-4 to generate 50 synonymous instructions for each of the 34 tasks and randomly sampled from them during evaluation.
See Tab. 6 in the paper for some examples.
For GR-1 and all the comparing baseline methods, we use the model trained on ABCD->D split for evaluation and test on the same set of sampled unseen language instructions.
Results are shown in the following table. 

    | Method     |   1   |   2   |   3   |   4   |   5   | Avg. Len. |
    |------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|
    | MT-R3M     | 0.512 | 0.249 | 0.106 | 0.040 | 0.017 |   0.92    |
    | RT-1       | 0.494 | 0.222 | 0.086 | 0.036 | 0.017 |   0.86    |
    | HULC       | 0.715 | 0.470 | 0.308 | 0.199 | 0.130 |   1.82    |
    | GR-1       | 0.764 | 0.555 | 0.381 | 0.270 | 0.196 |   2.17    |

    The performance of all the methods drops when evaluated on unseen language instructions.
    GR-1 outperforms all the comparing baseline methods.
    We hypothesize that this generalization capability attributes to being exposed to diverse languages in the large video dataset during pre-training.

**These results all highlight the strength of the video generative pre-training in GR-1 on visual manipulation learning.**