Paper: Vision-Language Foundation Models as Effective Robot Imitators (ICLR-2024)

Reviewer comment (Experiments):
The experimental section of the paper lacks comparison to (A) co-training with the current parameter-freezing scheme, (B) full model finetuning with and without co-training, to support the claim that their partial finetuning scheme is actually key to enable good performance without co-finetuning.

Author response:
**Added full model finetuning to the 3B version** (cannot do bigger size model for now) on the $ABCD\rightarrow D$ setting. The current version of RoboFlamingo trained 1B parameters (as shown in Table 2) while keeping all others frozen on a single 8 A100 GPU server, which takes a batch size of 6 on each 80G A100 GPU (data parallel). Therefore, we temporarily have not extended to the 9B variant due to the development of the model parallel. The results are listed below, which indicate that the performance will deteriorate so much if we do so.
|  | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Full model fine-tuned (3B) | 41.5% | 7.0% | 0.9% | 0.2% | 0.1% | 0.497 |

**Added the co-train verion model with both robot data and vision-language data (COCO caption, VQA)**. The results shown below indicate that co-training could preserve most ability of the VLM backbone on vl tasks, while losing a bit of performance on robot tasks. These results have been updated in Appendix B.1. 
|  | Split | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Co-Train | ABC->D | 82.9% | 63.6% | 45.3% | 32.1% | 23.4% | 2.473 |
| Fine-tune | ABC->D | 82.4% | 61.9% | 46.6% | 33.1% | 23.5% | 2.475 |
| Co-Train | ABCD->D | 95.7% | 85.8% | 73.7% | 64.5% | 56.1% | 3.758 |
| Fine-tune | ABCD->D | 96.4% | 89.6% | 82.4% | 74.0% | 66.2% | 4.086 |
| Co-Train | ABCD->D (Enrich) | 67.8% | 45.2% | 29.4% | 18.9% | 11.7% | 1.73 |
| Fine-tune | ABCD->D (Enrich) | 72.0% | 48.0% | 29.9% | 21.1% | 14.4% | 1.854 |

|  | coco  |  |  |  |  |  |  |  | VQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE_L | CIDEr | SPICE | Acc |
| Fine-tune (3B, zero-shot) | 0.157  | 0.052 | 0.018 | 0.008 | 0.038 | 0.147 | 0.005 | 0.006 | 4.09  |
| Fine-tune (3B, 4-shot) | 0.168  | 0.057 | 0.020 | 0.008 | 0.043 | 0.161 | 0.005 | 0.007 | 3.87 |
| OpenFlamingo (3B, zero-shot)  | 0.580 | 0.426 | 0.301  |  0.209 |  0.208  |  0.464  |  0.757  |  0.153  |  40.92
| OpenFlamingo (3B, 4-shot)  | 0.612 | 0.461 | 0.332 | 0.234 | 0.220 | 0.491 | 0.822 | 0.162 | 43.86 |
| Co-Train (3B, zero-shot) | 0.223 | 0.157 | 0.106 | 0.071 | 0.124 | 0.334 | 0.346 | 0.084 | 36.37 |
| Co-Train (3B, 4-shot) | 0.284 | 0.204 | 0.142 | 0.098 | 0.142 | 0.364 | 0.426 | 0.100 | 38.73 |
| Original Flamingo (80B, fine-tuned) | - | - | - | - | - | - | 1.381 | - | 82.0 |

Reviewer comment (Experiments):
Another comparison that would be good to add is to a simpler, pre-trained visual representation, like VC-1, Voltron etc. These models also use internet data to train good representations for imitation learning, but are arguably easier to use than the billion-parameter scale models introduced here, so it would be good to experimentally show the benefits.

Author response:
**Added the comparison results with pre-trained robotics representation work Voltron and R3M**. In our implementation, we freeze the representation layers of R3M and Voltron. We also tried to fine-tune the entire model of Voltron to improve its performance. Please see the results in the table for comparison and Appendix B.2 of our paper for more details and further analysis:
|  | Split | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Voltron  | ABC->D | 2.6% | 0.1% | 0.0% | 0.0% | 0.0% | 0.027 |
| Voltron (Unfreeze) | ABC->D | 56.9% | 27.2% | 10.5% | 3.8% | 1.4% | 0.998 |
| Ours | ABC->D | 82.4% | 61.9% | 46.6% | 33.1% | 23.5% | 2.475 |
| R3M | ABCD->D | 8.5% | 0.5% | 0.1% | 0.0% | 0.0% | 0.098 |
| Voltron | ABCD->D | 10.1% | 0.3% | 0.1% | 0.0% | 0.0% | 0.105 |
| Voltron (Unfreeze) | ABCD->D | 83.7% | 56.6% | 35.2% | 20.8% | 11.5% | 2.078 |
| Ours | ABCD->D | 96.4% | 89.6% | 82.4% | 74.0% | 66.2% | 4.086 |

Reviewer comment (Experiments):
One notable difference to prior work is that instead of predicting actions as tokens in the VLM's output vocabulary, the proposed method trains a separate action head. It would be good to analyze this choice and compare to directly outputting actions as "text tokens".

Author response:
**Q1**: `...It would be good to analyze this choice and compare to directly outputting actions as "text tokens".`

**A1**: We highly agree with you, and we indeed have thought about doing such experiments. However, since the major difference between RT-2 and RoboFlamingo comes from the foundation model side, we find it hard to do so under the architecture of Flamingo, which is simply a VL model, not a VLA model (the Palm-E used in RT-2). Therefore, if we cannot use the VLM to model history action tokens from the input side, but only predict discrete action tokens instead, the difference will only come from regression loss or discrete token classification loss, which can not specify any advantage of utilizing action tokens or not. 
In fact, in our initial submitted version, the MLP w/ hist version is similar to the implementation that directly outputs text tokens. However, as stated, this version still uses LSTM to encode history observations. We further implement a version that is closer to RT-2 by discretizing each dim of the action into 256 bins and decoding the action for each DoF. However, we still can not get a reasonable performance for this model (the performance for the single-task is lower than 10%, and the Avg. Len. Is less than 0.1). We are still trying our best to make it work.

Reviewer comment (Experiments):
Since the paper is mainly an empirical study, it would be good to evaluate the policy on more than one environment, e.g. the IKEA Furniture assembly environment could be a nice and challenging testbed with approximately photorealistic rendering.

Author response:
**Q4**: `The IKEA dataset.`

**A4**: We sincerely appreciate your valuable suggestion. We acknowledge that extending our model's validation to additional benchmarks, such as the photo-realistic IKEA dataset, would indeed strengthen our findings. However, the IKEA dataset, primarily an online resource lacking linguistic annotations, presents significant challenges in accumulating sufficient offline data for effective model training. Presently, constraints in time and computational resources preclude us from undertaking this extensive data collection process. Consequently, we consider this an important avenue for future research and intend to explore it as soon as the necessary resources are available.

Reviewer comment (Reproducibility):
The paper lacks details on the computational requirements for training policies with the VLM backbones (required GPU + training time), which seems crucial given the focus on making VLM policies more accessible.

Author response:
**Added computational requirements for training policies** in Appendix A.3. "All experiments involved in this paper are conducted on a single GPU server with 8 NVIDIA Tesla A100 GPUs, and the default batch size is 6 on each GPU. The MPT-3b model takes 13 hours of training per epoch and achieves the best performance at the 3rd epoch, while the MPT-9b model also takes 26 hours of training per epoch and achieves the best performance at the 3rd epoch."

Reviewer comment (Experiments):
Section 5.5 on open-loop execution lacks some detail on what exactly was tried, so I was a bit confused about these experimental results (see questions below).

Author response:
**Q3**: `can you explain in more detail the experiment on open-loop execution in Section 5.5? How can you open-loop execute the policy without re-training?`

**A3**: Sorry for your confusion! The policy head is trained to predict action sequences (stacked actions) instead of a single-step action. In our closed-loop control, only the next action will be executed and then we give the new observation to the policy head to predict the following actions. In contrast, for open-loop control, we execute an action sequence before providing a new observation to the policy head after a few timesteps. We've revised the related context and Figure 2 to make it more clear.
"Instead of taking only the next action to execute and performing VLM inference every time for new observations to predict future actions, open-loop control can be achieved by predicting an action sequence (stacked actions) with only one inference given the current observation, therefore alleviating the delay and the test-time computing requirement."

Reviewer comment (Presentation):
According to Section 4.2.2, the finetuned language model backbone will output a fused vision-language representation \(X_t^L=\{x_{t,1}^L,\dots, x_{t,M}^L\}\) for *each* time step \(t \in [1...T]\), where \(M\) is the length of the input language instruction \(l\). To produce such outputs, it seems necessary that the same language instruction needs to be tiled \(T\) times and fed into the language model for a sequence of \(T\) images. However, such detail is not illustrated in Figure 2, and Figure 2 only illustrates the model behavior when \(T=1\). It would be a lot more helpful if Figure 2 illustrates model behavior when \(T>1\).

Author response:
1) The VLM backbone only models single-step observation and the same language instruction is modeled in every single step. 
2) Thanks for your suggestion! We've added the caption for Fig. 2 **"The Flamingo backbone models single-step observations, and the temporal features are modeled by the policy head."**, and added a figure in Appendix C.1, Fig.9 to illustrate the policy head in detail.

Reviewer comment (Experiments):
In Section 5.4.3, what is the setup for the "instruction fine-tuning" experiment? The descriptions for the instruction finetuning designs are also missing in the methodology section.

Author response:
Sorry for your confusion! In this subsection we study the critical factors in VL pre-training (OpenFlamingo) instead of the fine-tuning of RoboFlamingo. For clarity, we change the context to the following:
Instruction fine-tuning is a specialized technique to help LLMs perform specific tasks according to explicit instructions. We find that LLMs with such a training stage can improve the performance of the policy in both seen and unseen scenarios, revealed by the performance improvements of M-3b-IFT against M-3b, and G-4b-IFT against G-4b shown in Table 2.

Reviewer comment (Experiments):
The authors choose to only train RoboFlamingo on the CALVIN benchmark throughout the paper. Even though they claim that they want to showcase RoboFlamingo's ability to produce good language-conditioned manipulation policies with a small amount of fine-tuning data, there is a concern that by only fine-tuning on the CALVIN benchmark, the model overfits to the dataset and loses some crucial abilities like spatial reasoning and object relation understanding (which may be crucial for other robot manipulation tasks that are not present in the CALVIN benchmark). Training RoboFlamingo on a mixture of downstream robot datasets and large-scale datasets used to pretrain, e.g., OpenFlamingo, InstructBLIP, IDEFICS, LLaVA, might alleviate such a phenomenon and even improve CALVIN benchmark performance. Therefore, there is disagreement with the authors' claim that "only a minimal amount of data is required to adapt the model to downstream manipulation tasks".

Author response:
Thanks for your kind review and enlightening suggestions. We agree that joint training with VL datasets may help to alleviate catastrophic forgetting. However, we would like to say that learning with limited manipulative demonstrations is also important since it is expensive to collect large amounts of data on robotic platforms.
In this paper, we focused more on the second part and found that adapting large-scale VL models to robotic tasks is not too "expensive" in the aspect of collecting robotic demonstrations. But surely, we would be excited to take the first part as the ongoing future work. 
We have revised the claim so that it is more clear and more rigorous: "With such a decomposition, we only need to combine a small amount of robotics demonstration to adapt the model to downstream manipulation tasks". We also conduct experiments on co-training RoboFlamingo with existing open-source visual-language datasets, including coco caption and visual question answering. 
Furthermore, as you ask, we implement a co-training framework for both VL data (COCO image caption, VQA) and robotics data (CALVIN), and the performance on both VL tasks and robot tasks is reported as follows:
|  | Split | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Co-Train | ABC->D | 82.9% | 63.6% | 45.3% | 32.1% | 23.4% | 2.473 |
| Fine-tune | ABC->D | 82.4% | 61.9% | 46.6% | 33.1% | 23.5% | 2.475 |
| Co-Train | ABCD->D | 95.7% | 85.8% | 73.7% | 64.5% | 56.1% | 3.758 |
| Fine-tune | ABCD->D | 96.4% | 89.6% | 82.4% | 74.0% | 66.2% | 4.086 |
| Co-Train | ABCD->D (Enrich) | 67.8% | 45.2% | 29.4% | 18.9% | 11.7% | 1.73 |
| Fine-tune | ABCD->D (Enrich) | 72.0% | 48.0% | 29.9% | 21.1% | 14.4% | 1.854 |

|  | coco  |  |  |  |  |  |  |  | VQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE_L | CIDEr | SPICE | Acc |
| Fine-tune (3B, zero-shot) | 0.157  | 0.052 | 0.018 | 0.008 | 0.038 | 0.147 | 0.005 | 0.006 | 4.09  |
| Fine-tune (3B, 4-shot) | 0.168  | 0.057 | 0.020 | 0.008 | 0.043 | 0.161 | 0.005 | 0.007 | 3.87 |
| OpenFlamingo (3B, zero-shot)  | 0.580 | 0.426 | 0.301  |  0.209 |  0.208  |  0.464  |  0.757  |  0.153  |  40.92
| OpenFlamingo (3B, 4-shot)  | 0.612 | 0.461 | 0.332 | 0.234 | 0.220 | 0.491 | 0.822 | 0.162 | 43.86 |
| Co-Train (3B, zero-shot) | 0.223 | 0.157 | 0.106 | 0.071 | 0.124 | 0.334 | 0.346 | 0.084 | 36.37 |
| Co-Train (3B, 4-shot) | 0.284 | 0.204 | 0.142 | 0.098 | 0.142 | 0.364 | 0.426 | 0.100 | 38.73 |
| Original Flamingo (80B, fine-tuned) | - | - | - | - | - | - | 1.381 | - | 82.0 |

From the result, we observe that co-training could indeed help the model preserve its original ability over VL tasks. This provides a solution for fine-tuning VLMs to robotics models while preserving the ability on vision-language tasks, even though it may slightly deteriorate the performance on robotic tasks.

Reviewer comment (Theory):
On Page 3, the statement `"Compared to other works, the controlling policies do not require any ability to understand instructions, but rely on the pre-trained frozen LLM to select necessary skills."` is inaccurate. The proposed approach still needs to understand language instructions, as the LLM backbone still needs to fuse language representations with visual input representations. The authors' actual intended meaning might be that the policy head does not explicitly take language instructions as input.

Author response:
We think you may misunderstand our statement. **This part talks about the related works (not our work)**, which utilize LLMs to plan the language goals into detailed, pre-defined low-level goals. Those low-level goals will be handled by a pre-trained skill policy, which does not need to understand the instructions (because LLM handles that part).

Reviewer comment (Experiments):
Other ways of incorporating VL pre-training are not considered, such as utilizing VL representations like R3M or VOLTRON or MVP. These baselines are relevant given the frozen-backbone + robot finetuning setup in RoboFlamingo. Essentially, a baseline should study different ways of incorporating "web data", which the current baselines do not study.

Author response:
Thanks for your great advice! We have supplemented the experiment results of R3M and Voltron on CALVIN benchmark, which are both pre-trained robotics representation models. 
In our implementation, we freeze the vision and text encoder of R3M and Voltron. We also fine-tune the entire model of Voltron to improve its performance. Please see the results in the table for comparison:

|  | Split | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Voltron  | ABC->D | 2.6% | 0.1% | 0.0% | 0.0% | 0.0% | 0.027 |
| Voltron (Unfreeze) | ABC->D | 56.9% | 27.2% | 10.5% | 3.8% | 1.4% | 0.998 |
| Ours | ABC->D | 82.4% | 61.9% | 46.6% | 33.1% | 23.5% | 2.475 |
| R3M | ABCD->D | 8.5% | 0.5% | 0.1% | 0.0% | 0.0% | 0.098 |
| Voltron | ABCD->D | 10.1% | 0.3% | 0.1% | 0.0% | 0.0% | 0.105 |
| Voltron (Unfreeze) | ABCD->D | 83.7% | 56.6% | 35.2% | 20.8% | 11.5% | 2.078 |
| Ours | ABCD->D | 96.4% | 89.6% | 82.4% | 74.0% | 66.2% | 4.086 |

Reviewer comment (Experiments):
A core claim of RT-2 was the benefit of co-fine-tuning on robotics data in addition to the original VL data. This core claim is not studied in RoboFlamingo.

Author response:
Thanks for your reminder! The main point and contribution of this paper is to show a way of fine-tuning open-source VLMs to robotics models. We believe RT2's conclusions should also benefit RoboFlamingo. In our implementation, we co-train our RoboFlamingo model with both robot data and vision-language data (COCO image caption, VQA). From the results, we observe that co-training could preserve most ability of the VLM backbone on VL tasks, while losing some performance on robot tasks. We have the results performed in the following table and updated in Appendix B.1 of our paper.
|  | Split | Epoch | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CoTrain | ABC->D | 1 | 52.8% | 19.2% | 6.6% | 2.4% | 0.5% | 0.815 |
|  | ABC->D | 2 | 79.3% | 57.4% | 38.1% | 26.1% | 15.9% | 2.168 |
|  | ABC->D | 3 | 80.7% | 59.9% | 42.3% | 30.7% | 20.7% | 2.343 |
|  | ABC->D | 4 | 79.8% | 56.7% | 40.0% | 28.3% | 20.3% | 2.251 |
|  | ABC->D | 5 | 82.9% | 63.6% | 45.3% | 32.1% | 23.4% | 2.473 |
| Finetune | ABC->D | 5 | 82.4% | 61.9% | 46.6% | 33.1% | 23.5% | 2.475 |
| CoTrain | ABCD->D | 1 | 80.2% | 43.8% | 18.7% | 7.5% | 3.0% | 1.532 |
|  | ABCD->D | 2 | 91.8% | 76.3% | 60.0% | 46.7% | 34.5% | 3.093 |
|  | ABCD->D | 3 | 95.7% | 85.8% | 73.7% | 64.5% | 56.1% | 3.758 |
|  | ABCD->D | 4 | 93.3% | 83.6% | 73.3% | 65.4% | 56.7% | 3.723 |
|  | ABCD->D | 5 | 93.0% | 82.7% | 63.9% | 63.9% | 54.7% | 3.662 |
| Finetune | ABCD->D | 3 | 96.4% | 89.6% | 82.4% | 74.0% | 66.2% | 4.086 |

Reviewer comment (Experiments):
Another claim of RT-2 was the benefit of mixing robot action tokens explicitly with VL tokens. In contrast, RoboFlamingo introduces a new policy head that directly only predicts action tokens. It would be interesting to compare an explicit action-only policy head with the multi-modal output token prediction setting in RT-2.

Author response:
We highly agree with you, and we indeed have thought about doing such experiments. However, since the major difference between RT-2 and RoboFlamingo comes from the foundation model side, we find it is hard to do so under the architecture of Flamingo, which is simply a VL model, not a VLA model (the Palm-E used in RT-2). Therefore, if we cannot use the VLM to model history action tokens from the input side, but only predict discrete action tokens instead, the difference will only come from regression loss or discrete token classification loss, which can not specify any advantage of utilizing action tokens or not.

For your concern, we have attempted to apply token mixing configurations like RT-2. However, this approach did not converge when using token cross-entropy loss. We suspect this issue stems from the architectural differences in the VLM backbone between RoboFlamingo and RT-2. Given that RT-2 is not publicly accessible, reproducing its results is beyond our current scope due to time constraints.

Reviewer comment (Reproducibility):
Will checkpoints be released as well?

Author response:
Certainly! The whole project is trying our best to contribute to the open-source community.

Reviewer comment (Evaluation):
How does performance on pre-training tasks change during finetuning? That is, is there catastrophic forgetting occurring, where the base foundation capabilities are lost?

Author response:
We appreciate your insightful reviews. We've revealed some evidence that the model lost some foundation capabilities, as shown in Table 1, Enriched Lang setting (the last four rows), where we use GPT-4 to generate new instructions with the same meaning. The performance loss indicates that there is over-fitting during fine-tuning. To further understand the phenomenon, we conduct further experiments by testing the fine-tuned RoboFlamingo model (the M-3b-IFT variant) on the COCO and VQAv2, which verifies our conjecture that it does overfit the robotics dataset and loses many original VL abilities. 
To prevent such problems, we choose to co-train RoboFlamingo (the M-3b-IFT variant) with VQA and COCO datasets during fine-tuning of the robotics dataset. We test the co-train model on both CALVIN and the COCO and VQAv2 tasks as well. This provides a solution for fine-tuning VLMs to robotics models while preserving the ability on vision-language tasks, even though it may slightly deteriorate the performance on robotic tasks. One interesting observation is that under the Enriched Lang setting, the performance of the co-trained model also drops. This may indicate the difference between understanding different sentences and aligning vision-language representations (as the pre-trained tasks do).
See the following tables for the detailed numerical results. 

**Results on CALVIN**
|  | Split | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Co-Train | ABC->D | 82.9% | 63.6% | 45.3% | 32.1% | 23.4% | 2.473 |
| Fine-tune | ABC->D | 82.4% | 61.9% | 46.6% | 33.1% | 23.5% | 2.475 |
| Co-Train | ABCD->D | 95.7% | 85.8% | 73.7% | 64.5% | 56.1% | 3.758 |
| Fine-tune | ABCD->D | 96.4% | 89.6% | 82.4% | 74.0% | 66.2% | 4.086 |
| Co-Train | ABCD->D (Enrich) | 67.8% | 45.2%  | 29.4% | 18.9% | 11.7% | 1.73 |
| Fine-tune | ABCD->D (Enrich) | 72.0% | 48.0% | 29.9% | 21.1% | 14.4% | 1.854 |


|  | coco  |  |  |  |  |  |  |  | VQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE_L | CIDEr | SPICE | Acc |
| Fine-tune (3B, zero-shot) | 0.157  | 0.052 | 0.018 | 0.008 | 0.038 | 0.147 | 0.005 | 0.006 | 4.09  |
| Fine-tune (3B, 4-shot) | 0.168  | 0.057 | 0.020 | 0.008 | 0.043 | 0.161 | 0.005 | 0.007 | 3.87 |
| OpenFlamingo (3B, zero-shot)  | 0.580 | 0.426 | 0.301  |  0.209 |  0.208  |  0.464  |  0.757  |  0.153  |  40.92
| OpenFlamingo (3B, 4-shot)  | 0.612 | 0.461 | 0.332 | 0.234 | 0.220 | 0.491 | 0.822 | 0.162 | 43.86 |
| Co-Train (3B, zero-shot) | 0.223 | 0.157 | 0.106 | 0.071 | 0.124 | 0.334 | 0.346 | 0.084 | 36.37 |
| Co-Train (3B, 4-shot) | 0.284 | 0.204 | 0.142 | 0.098 | 0.142 | 0.364 | 0.426 | 0.100 | 38.73 |
| Original Flamingo (80B, fine-tuned) | - | - | - | - | - | - | 1.381 | - | 82.0 |

Reviewer comment (Experiments):
Relies on simulated robot environment, may be challenging to transfer to real world.

Author response:
We admit the submitted version of the manuscript limits in the simulation benchmark CALVIN. The CALVIN tasks are implemented based on Pybullet (https://pybullet.org/), which is based on the Bullet Physics SDK and has been used in many previous robotics works (e.g., [1-3]). This benchmark provides diversified robotic manipulation tasks including rotate blocks, turn off lights, open drawers, stack and unstack blocks, and so on, which are still challenging to current state-of-the-art algorithms. Here are some intuitive demonstrations (http://calvin.cs.uni-freiburg.de/images/banner.mp4). We hope that the additional materials will be helpful in understanding the reality, fidelity, and challenges of this benchmark.

Reviewer comment (Experiments):
The evaluation is limited to a single simulated benchmark environment (CALVIN). Testing on more diverse robotic platforms and tasks in simulation could help validate the generalizability of the method.

Author response:
We admit the submitted version of the manuscript limits in the simulation benchmark CALVIN. The CALVIN tasks are implemented based on Pybullet (https://pybullet.org/), which is based on the Bullet Physics SDK and has been used in many previous robotics works (e.g., [1-3]). This benchmark provides diversified robotic manipulation tasks including rotate blocks, turn off lights, open drawers, stack and unstack blocks, and so on, which are still challenging to current state-of-the-art algorithms. Here are some intuitive demonstrations (http://calvin.cs.uni-freiburg.de/images/banner.mp4). We hope that the additional materials will be helpful in understanding the reality, fidelity, and challenges of this benchmark.

Reviewer comment (Experiments):
What steps would be needed to transfer RoboFlamingo to real robotic systems? How realistic are the CALVIN simulations?

Author response:
We tried our best to train the RoboFlamingo on real-world robotics tasks. To do so, we collect real-robot demonstrations for table-top manipulation tasks including pick-and-place, opening drawer and cabinet, etc. As a result, we first collected around 5% CALVIN-size demonstrations using teleoperation and training to finetune RoboFlamingo as we did on the CALVIN benchmark. The robot can hardly manipulate objects successfully due to severe overfitting. To make it work, we then collect an additional set of 5% CALVIN-size demonstrations, based on which we have successfully trained a model that can manipulate some objects in pick-and-place tasks. Nevertheless, we found that this dataset is still not big enough to train a high-performance model at such a scale. We are still working on that, including data collection and model training. However, due to the limited timeline of the review process, we can not provide comprehensive results here. But from performance improvements from 5% CALVIN-size data to 10%CALVIN-size data, we believe that our model can adapt to real-world applications.

Reviewer comment (Experiments):
Is there scope to incorporate offline robotic data to improve sample efficiency?

Author response:
According to your enlightening suggestions, we have verified that more offline robotic data will further improve performance, with less training epochs the performance is improved.
To get more data, we used the script provided by the CALVIN benchmark to generate 4x more data (5x in total), then we trained RoboFlamingo on that, and observed that there is a clear performance gain compared to the model trained on the standard CALVIN dataset. For MPT-3B-IFT, the performance increases to 4.253 on the ABCD -> D setting in the aspect of Avg. Len, with a success rate of 97.1% for the single-task setting and 71.6% for the subsequent 5-task setting. For MPT-9B, the performance increases to 4.095 on the ABCD -> D setting in the aspect of Avg. Len, with a success rate of 95.4% for the single-task setting and 69.4% for the subsequent 5-task setting. 
Due to the time limit, we only evaluate the models' performance for 2 epochs, and we believe that the performance would keep increasing through incorporating more offline robotic data and training more iterations.
This table reports the raw performance and the performance of the same model trained on 5x data:

| Model | Epoch | SR 1 | SR 2 | SR 3 | SR 4 | SR 5 | Avg Len |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MPT-3b-IFT | best in the paper (3 epoch) | 96.4% | 89.6% | 82.4% | 74.0% | 66.2% | 4.086 |
|  | 1 | 96.1% | 87.5% | 80.1% | 72.1% | 65.4% | 4.012 |
|  | 2 | 97.1% | 91.6% | 85.6% | 79.4% | 71.6% | 4.253 |
| MPT-9b | best in the paper (4 epoch) | 95.5% | 87.9% | 78.4% | 71.4% | 63.4% | 3.966 |
|  | 1 | 95.4% | 88.0% | 81.4% | 75.3% | 69.4% | 4.095 |
|  | 2 | 95.1% | 88.6% | 80.5% | 73.9% | 66.9% | 4.05 |

Reviewer comment (Experiments):
Experiments are with visual and language modalities. Robotic manipulation often relies on additional sensing (e.g. force, tactile). How can RoboFlamingo incorporate other modalities?

Author response:
Very nice and interesting point! Currently, our RoboFlamingo is only evaluated on gripper-based manipulation tasks, which do not need to involve force and tactile information. Since such modalities are orthogonal to visual and language modalities, modeling such data with pre-trained VLMs may require much more effort to fine-tune, so we think the most straightforward and simple way is to treat it as low-dimensional states, concatenated with the visual-language features in the policy head input.

Reviewer comment (Experiments):
How does the computational overhead of RoboFlamingo compare to other VLM-based methods?

Author response:
All experiments involved in this paper are conducted on a single GPU server with 8 NVIDIA Tesla A100 GPUs. The training data includes ~2.4M interaction steps, and the trainable parameters for all RoboFlamingo variants are only 1M in total. The MPT-3b model takes 13 hours of training per epoch and achieves the best performance at the 3rd epoch, while the MPT-9b model also takes 26 hours of training per epoch and achieves the best performance at the 3rd epoch. For comparison, RT2 uses the PaLI-X 5B & 55B model, PaLI 3B model and the PaLM-E 12B model, and is deployed on a multi-TPU cloud server.
We have supplemented this information in the updated revised version, Appendix A.3.