Paper: AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents (ICLR-2024)

Reviewer comment (Novelty):
This paper lacks of technical novelty. This is a simple extension of [1, 2] to real-world robotic scenarios. I wonder if there is unique technical contribution over these existing works to deal with real-world robotic scenarios.

Author response:
Thank you for taking the time to review the paper. AutoRT aims to be a data collection system that is agnostic to the specifics of collection and learned policies such that it can deploy a wide variety of collection policies to mine data that may be useful for multiple learnable policies. As such, the focus of this paper is not on designing new collect or learned policies, but to develop a framework that orchestrates a number of potential collection policies (automated or teleoperated) to increase the diversity of the collected data -- which in turn can enhance policy learning.

A main concern of the reviewer is whether there is technical novelty given prior methods for simulated tasks. The real world is significantly different from sim. In real it is harder to scale, act autonomously, act safely, or know the ground truth state of the world, and there is technical novelty in how to tackle these real-world specific challenges, which we discuss below. Additionally, we’d like to note that there is significant interest in the robotics community for improving real world data collection. RT-2, a model using Internet-scale image data, still fails to generalize due to a lack of real world robotics data. AutoRT is a step towards addressing some of these key challenges by enabling real robot data collection while having the potential to improve state of the art robot learning algorithms, which goes beyond capabilities shown in simulated environments.

“This paper lacks of technical novelty. This is a simple extension of [1, 2] to real-world robotic scenarios. I wonder if there is unique technical contribution over these existing works to deal with real-world robotic scenarios.”
The main difference between [1,2] and our work is that AutoRT is trying to solve a different problem  –  we are trying to collect open world robotic data as opposed to learning skills in Minecraft. To this end, a significant part of the paper talks about the quality of data collected (section 5.1) and these experiments are not present in either [1,2]. 
We propose a method where robot agents can ask for human help (teleoperation) or collect autonomous policies, which is not a factor in either mentioned work. This lets us scale robot deployment, where 1 human can supervise 3-5 robots in open world mobile manipulation settings due to the fact that autonomy is shared with LLMs( we have edited the paper to emphasize this). We also show how robots can ask for human help (teleoperation) or collect with autonomous policies, leading to a degree of semi autonomous operation and the ability to incorporate rich human feedback into these systems. The papers mentioned are purely autonomous and do not demonstrate sharing autonomy with humans. In fact our findings show that teleoperated data is the most diverse, and therefore, we believe that future robot data collection systems really need to incorporate many manners of collect.
Additionally, unlike systems that work in simulation, systems for real world robotics need to keep safety at the core of its design. We introduce a robot constitution inspired from Asimov's laws, and we also show that this improves safety almost 3x. The focus and experiments on safety or attempts to align robots with human values is not part of either paper mentioned. (we have edited the paper to emphasize the contribution of real world world alignment)

There is clear interest in the community for improving real-world data collection for manipulation, like ORL (https://sites.google.com/corp/view/real-orl), RoboAgent (https://robopen.github.io/), and Vision-Based Manipulators Need to Also See from Their Hands (ICLR 2022 oral, https://openreview.net/forum?id=RJkAHKp7kNZ), and we believe AutoRT builds upon these works and making progress towards addressing challenges that appear in real-world scenarios.

Reviewer comment (Evaluation):
In table 1, the collect policies other than teleoperation have very low success rate which means the collected data might not be useful for action learning. Basically, this method sounds like it is scalable but in a low quality. Can we learn meaningful actions from these data other than visual generalization ability?

Author response:
“In table 1, the collect policies other than teleoperation have very low success rate which means the collected data might not be useful for action learning. Basically, this method sounds like it is scalable but in a low quality. Can we learn meaningful actions from these data other than visual generalization ability?”
We would like to emphasize that AutoRT is a system for orchestrating robots, proposing tasks, and running a diverse suite of collect policies. It is not specific to the environment used or policies used for data collection. As mentioned in the reply to F9MU, the collect policies are primarily from prior work. Our contribution is in defining a way to orchestrate that prior work.

The Picking (Height Generalization) evaluation task from Section 5.4 was deliberately picked as a task whose actions did not appear in teleoperated data (most teleop demos were either not on picking, or picking from a narrow range of heights). The improvement there suggests that meaningful actions can be learned from non-teleoperated data.

Reviewer comment (Experiments):
Does LLM ever propose tasks that require skills not within the robot's scripted policy or teleoperation policy? If so, how does the robot behave in such cases? If such cases do not exist, how does the LLM restrict task proposals to the skills the robot possesses?

Author response:
“Does LLM ever propose tasks that require skills not within the robot's scripted policy or teleoperation policy? If so, how does the robot behave in such cases? If such cases do not exist, how does the LLM restrict task proposals to the skills the robot possesses?”
Most of the tasks are attemptable by the teleop policy. In the case where the LLM does propose one that cannot be done, this is usually rejected by the affordance LLM, which is prompted to reject tasks based on our description of the robot's limitations (i.e. needs 2 hands, object too heavy) and the policy's limitations (i.e. scripted policy can only do specific tasks). See Appendix D for the exact prompts and Table 4 for the quality of task generation.
If both the initial generation and task filtering fails, the behavior depends on the policy used. For teleoperation, a human simply rejects the impossible task. For the scripted policy, we designed it to exit early instead of attempting manipulation. For RT-2, the policy will make its best attempt at the instruction. As mentioned in Section 5.3, we took a sample of 64 scenes, analyzed the failures, and found all failures in the sample were for teleoperation, rejected by the human, and not demonstrated. Our prompting for non-teleoperated policies is more conservative to make human supervision easier.

Reviewer comment (Evaluation):
The success rates are surprisingly low for both RT-1 and the one co-trained with Auto-RT. The gains are not that impressive to justify the pipeline.

Author response:
“The success rates are surprisingly low for both RT-1 and the one co-trained with Auto-RT. The gains are not that impressive to justify the pipeline.”
The base success of RT-1 is in fact low in our new settings. This is because Auto-RT places the robot in diverse scenarios that are significantly out of distribution from settings where the original RT-1 policies were trained. Specifically, the differing height of manipulation surfaces leads to some of the low performance we observe in the base model. However, we would like to emphasize that the performance of this base model in these novel scenarios are in fact not relevant to the AutoRT’s contribution and RT-1 is just an example of a policy one could use to initialize the learning via data collected via AutoRT. 
As for why the performance gain of RT-1 is modest, we suspect the AutoRT distribution is "flatter". It has many more tasks, and fewer episodes per task. Additionally the episodes for each task are significantly more open-world and diverse than prior robotics data, as shown in Section 5.1: Diversity Scoring. Learning quickly from such data is challenging. We posit this potentially calls for a larger scale data collection via AutoRT to observe policy improvements.

As mentioned in the paper (Section 5.4), we would like to emphasize that the primary focus of AutoRT is collecting more diverse real robot data. The co-training improvement is only to show the data can be useful for training an existing method (RT-1), and  the key contribution is on an effective data collection approach, rather than learning efficiency.

Reviewer comment (Experiments):
What would happen if a task is deemed solvable by RT-2 and then RT-2 cannot actually solve it?

Author response:
“What would happen if a task is deemed solvable by RT-2 and then RT-2 cannot actually solve it?”
The episode ends in failure and is not reattempted. We experimented with multiple success detection methods, and found their precision was too low in real-world scenarios for them to be useful. In such scenarios, we rely on human labelers to provide data, which can later be used during policy improvement.

Reviewer comment (Experiments):
Does Auto-RT leverage the failure data as well?

Author response:
“Does Auto-RT leverage the failure data as well?”
The current learning policies of AutoRT are based on imitation learning (BC), and thus do not leverage failure data. However, one could rely on other learning algorithms that can more effectively learn from both positive and negative examples, and we hope to study this in the future.

Reviewer comment (Experiments):
There is a confusion in the paper. Earlier RT2 is mentioned everywhere but all the tables use RT-1 for quantitative numbers. I thought RT-1 is inferior to RT-2. Why not provide quantitative numbers for RT-2 to see the gains of training on Auto-RT data are also observed there as well? I would like to see an experiment on that.

Author response:
“There is a confusion in the paper. Earlier RT2 is mentioned everywhere but all the tables use RT-1 for quantitative numbers. I thought RT-1 is inferior to RT-2. Why not provide quantitative numbers for RT-2 to see the gains of training on Auto-RT data are also observed there as well? I would like to see an experiment on that.”
We have used RT-1 mainly due to practical reasons such as experiment and iteration time. While RT-2 is a more powerful model, its training can be computationally expensive (taking 2+ weeks). We have edited the paper to clarify this point.

Reviewer comment (Reproducibility):
More details are needed for the scripted policy. It seems to be dependent on object pose which is possible to extract from the cached map in the beginning but it does not say anything about the orientation of the object. Any object where the orientation of the object is important would cause failure with this policy. The only exception that I can think of where the orientation does not really matter is to pick up a deformable object such as stuffed toy, sponge, etc or symmetric objects such as bowl/plate.

Author response:
“More details are needed for the scripted policy. It seems to be dependent on object pose which is possible to extract from the cached map in the beginning but it does not say anything about the orientation of the object. Any object where the orientation of the object is important would cause failure with this policy. The only exception that I can think of where the orientation does not really matter is to pick up a deformable object such as stuffed toy, sponge, etc or symmetric objects such as bowl/plate.”
We apologize if this was not clear, we have now  included the pseudocode of the scripted policy in the Appendix for more details. This policy in fact does ignore the orientation of the object. We did not spend too much time optimizing the scripted pick policy, because ultimately it is the learned policy that needs to be robust to orientation of object, and this policy has access to more data streams than the scripted pick. The goal of the scripted pick policy here is to provide episodes that the learning policy can take advantage of while leading to some level of success in many settings we encountered.

Reviewer comment (Experiments):
In AutoRT Operational Details, the authors mentioned that AutoRT was run using stationary robots that skipped navigation and only conducted manipulation. In these scenarios, would the tasks which robots generate be dependent on the objects which are exposed to the robot? Can the authors provide some explanation on how the scenes for such robots were set up to ensure practicality while reducing the inherent bias that could be introduced into tasks generated by humans manually selecting the objects to set up the environment?

Author response:
Yes, generated tasks depend on the objects exposed to the robot. We've edited the paper to include more details on scene setup (in Appendix A).

Some of AutoRT was run in natural environments with people. Objects in those environments were biased towards typical objects (i.e. office supplies are more common in office environments). Other parts of AutoRT were run in stationary scenes or altered environments.
For altered environments, we gathered many (over 100) random objects, like plastic toys, soda cans, stuffed animals, and office supplies. Each day some of the objects would be scattered around the scene, with a new random sample each day. This provides a greater variety of objects for AutoRT's task generation.

We also believe that the language and visual diversity scoring in AutoRT could reduce the inherent bias. Assuming that human biases lead to some data being overrepresented, that data will score worse in diversity scoring, and objects can be rearranged appropriately. Appendix E shows some examples of the human supervisor arranging objects until a higher score is reached, which leads to novel scenes like turned over recycling bins.

Reviewer comment (Experiments):
I would also suggest the author look into if there is a need for a real-robot for the data collection process as there are a few works that look into self-generating tasks or ever collecting demonstrations for a real robot without the need for a flight of robot. Examples:  
-Huang, Wenlong, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.05973 (2023).  
-Wang, Lirui, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. "GenSim: Generating Robotic Simulation Tasks via Large Language Models." arXiv preprint arXiv:2310.01361 (2023).  
-Duan, Jiafei, Yi Ru Wang, Mohit Shridhar, Dieter Fox, and Ranjay Krishna. "AR2-D2: Training a Robot Without a Robot." arXiv preprint arXiv:2306.13818 (2023).

Author response:
The methods mentioned focus on either sim or zero-shot performance. VoxPoser zero-shots affordance maps for motion planning. GenSim uses LLMs to orchestrate sim environments. AR2-D2 generate robot demos via AR from hand tracking.

The question of whether such zero-shot methods remove the need for a real robot is an interesting one, but AutoRT's focus is on real robots. There is clear interest in the community for improving real-world data collection for manipulation, like ORL (https://sites.google.com/corp/view/real-orl), RoboAgent (https://robopen.github.io/), and Vision-Based Manipulators Need to Also See from Their Hands (ICLR 2022 oral, https://openreview.net/forum?id=RJkAHKp7kNZ). It's an interesting question whether sim or zero-shot methods can remove the need for real robot data collection, but this is orthogonal to judging whether AutoRT helps manage real robot data collect.

Our goal in AutoRT is not to replace simulation-driven methods, but to supplement them by making it easier to scale up and deeply understand real-world robot deployments, and study how well LLMs and VLMs work in realistic scenarios.

Reviewer comment (Theory):
The authors specify a robot constitution for filtering tasks, where the embodiment rules involve an understanding of the physical attributes of objects (i.e. heaviness). However, it is uncertain whether the language models are suitable for embodiment style questions (i.e. weight, composition reasoning). Perhaps the authors should look into literature which studies the capability of language models and vision language models in doing physical reasoning such as:  
1.Wang, Yi Ru, Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa. "NEWTON: Are Large Language Models Capable of Physical Reasoning?." arXiv preprint arXiv:2310.07018 (2023).  
2.Gao, Jensen, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. "Physically Grounded Vision-Language Models for Robotic Manipulation." arXiv preprint arXiv:2309.02561 (2023).

Author response:
AutoRT focuses on studying what is possible using the imperfect physical reasoning abilities of current LLMs and VLMs.

In terms of semantic safety of generated tasks, we find that conceptual considerations (i.e. avoid sharp things, things achievable with 1 hand) were still helpful for deploying robots and filtering tasks. Physical considerations (i.e. heaviness) were occasionally wrong, but the existing capabilities were good enough to move forward with AutoRT. We will definitely try to improve these bottlenecks in future work.