Foundation Models for Robotics: Best Known Practices (2023)

Abstract: “Artificial general intelligence (AGI) used to be a sci-fi word but recently the surprising general-ization capability of foundation models have triggered a lot of attention to AGI, in both academiaand industry. Large language models can now answer questions or chat with human beings,using fluent sentences and clear reasoning. Diffusion models can now draw pictures of unprece-dented photo-realism, according to human commands and controls. Researchers have also madesubstantial efforts to explore new possibilities for robotics applications with the help of founda-tion models. Since this interdisciplinary field is still under fast development, there is no clearmethodological conclusions for now. In this tutorial, I will briefly go through best known prac-tices that have shown transformative capabilities in several sub-fields. Specifically, there are fiverepresentative paradigms: (1) Using foundation models to allow human-friendly human-car in-teraction; (2) Using foundation models to equip robots the capabilities of understanding vaguehuman needs; (3) Using foundation models to break down complex tasks into achievable sub-tasks; (4) Using foundation models to composite skill primitives so that reinforcement learningcan work with sparse rewards; (5) Using foundation models to bridge languge commands andlow-level control dynamics. I hope these best known practices to inspire NLP researchers.”

## introduction this is a tutorial paper that summarizes my talk at ccl 2023, on the topic of foundation models for robotics: best known practices. the concept of foundation models emerge in the community of natural language processing (nlp), like gpt @xcite . current foundation models can learn language skills using few shots and notably without fine-tuning the model (known as in-context learning). this human-like behavior has not been demonstrated before and widely considered as an encouraging step towards artificial general intelligence (agi). however, a long-existing problem still lingers around at the age of foundation models, which is known as the moravec's paradox. specifically speaking, high-level human intelligence like language and reasoning actually consumes relatively less computation while low-level control is much more complicated than one may think. this paradox is echoed by the current state of ai research: while language models are getting stronger and stronger, robots still struggle to move and manipulate, at least in the context of artificial general intelligence. despite the disappointing situation as described by the moravec's paradox, we see a line of encouraging research efforts related to foundation models that have definitely expanded the scope of robotics research. as a disclaimer, this does not mean that the paradox is solved. specifically, we observe several promising paradigms that successfully marry robotics with the recent progress of foundation models, among which five representative works are covered in this tutorial: (1) using foundation models to allow human-friendly human-car interaction; (2) using foundation models to equip robots the capabilities of understanding vague human needs; as shown in fig. 1 , we demonstrate five paradigms that foundation models can benefit robotic perception, planning and control. it is far from an extensive list but reasonably representative.