Embodied Executable Policy Learning with Language-based Scene Summarization (2024)

Abstract: Large Language models (LLMs) have shown remarkable success in assisting robot learning tasks, i.e., complex household planning.However, the performance of pretrained LLMs heavily relies on domain-specific templated text data, which may be infeasible in real-world robot learning tasks with image-based observations. Moreover, existing LLMs with text inputs lack the capability to evolve with non-expert interactions with environments.In this work, we introduce a novel learning paradigm that generates robots’ executable actions in the form of text, derived solely from visual observations. Our proposed paradigm stands apart from previous works, which utilized either language instructions or a combination of language and visual data as inputs. We demonstrate that our proposed method can employ two fine-tuning strategies, including imitation learning and reinforcement learning approaches, to adapt to the target test tasks effectively.We conduct extensive experiments involving various model selections, environments, and tasks across 7 house layouts in the VirtualHome environment. Our experimental results demonstrate that our method surpasses existing baselines, confirming the effectiveness of this novel learning paradigm.

## introduction there has been a surge of interest in building large language models (llms) pretrained on largescale datasets and exploring llms' capability in various downstream tasks. llms start from the transformer model @xcite and are first developed to solve different natural language processing (nlp) applications @xcite @xcite . recently, llms have also shown great potential for accelerating learning in many other domains by generating learned embeddings as meaningful representations * equal contribution for downstream tasks and encoding transferable knowledge in large pretraining datasets. in this paper, we focus on the problem of facilitating robot learning by having a llm in the loop. the robot generates actions according to its environment observations, which are, in general, sensory information in the format of images, point clouds, or kinematic states. we identify one key challenge in massively deploying llms to assist robots is that llms lack the capability to understand such non-text-based environment observations. to solve this challenge, @xcite utilize rule-based perception apis to transform image-based observations into text formats, which then serve as inputs to the llm. we instead propose to integrate the multimodal learning paradigm to transform images into texts, which allows more principled and efficient transfer to novel robot learning tasks than rule-based apis. another key challenge is the widely-existing large distribution shifts between the training tasks of large pretrained models and testing tasks in the domain of robot learning. to close the domain gap, @xcite adapt the pretrained llm to downstream tasks via finetuning with observations converted into text descriptions. in the presence of realistic visual observations, an appropriate method to co-adapt pretrained foundation models for testing tasks in robot learning is still being determined. to address the above challenges, we propose a new visual-based robot learning paradigm that takes advantage of embedded knowledge in both multimodal models and llms. to align different modalities in the visual observations and text-based actions, we consider language as the bridge information. we build a scene-understanding model (sum) with a pretrained image captioning model to grant the robot the ability to describe the surrounding environment with natural language. we then build an action prediction model (apm) with a llm to generate execution actions according figure 1 : the overall architecture of our approach, which includes a scene understanding module (sum) and an action prediction module (apm). the agent takes pure visual observations and encodes the information as latent language, then the language is transferred to apm for action generation. apm fine-tuned on virtualhome can generate executable action plans directly. to the scene caption in the format of natural language. to adapt pretrained models in sum and apm to downstream robot learning tasks, we propose to finetune the multimodal model in sum with pre-collected domain-specific image-caption pairs and the language model in apm with corresponding language-action pairs. besides finetuning with expert demonstrations, we further propose a finetuning paradigm of apm based on the sparse environment feedback to endow apm's capability to evolute with non-expert data. an illustration of the proposed framework is figure 1 . our contributions are summarised as follows: visual feedback in robot learning visual feedback is commonly used in robot learning. gothoskar et al. (2020) learned a generative model from actions to image observations of features to control a robot from visual feedback. ma et al. (2022) proposed a self-supervised pretrained visual representation model which is capable of generating dense and smooth reward functions for unseen robotic tasks. strokina et al. (2022) reviewed the methods of reward estimation and visual representations used in learning-based approaches for robotics applications. mohtasib et al. (2021) studied the performance of dense, sparse, visually dense, and visually sparse rewards in deep rl. kurita and cho (2020) proposed the direct navigation approach based on an image captioning model. li et al. (2019) combined image captioning models and planning models, but li et al. ( 2019 ) took pure language instructions as input, while our approach takes pure visual observations as input. pre-training and fine-tuning of language models over the past few years, fine-tuning @xcite has superseded the use of feature extraction of pretrained embeddings @xcite while pretrained language models are favored over models trained on many tasks due to their increased sample efficiency and performance @xcite . the success of these methods has led to the development of even larger models @xcite . fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace @xcite . @xcite examined the sampling effects in reinforcement learning with gpt and bert. more related works can be found in appendix a. ## method in this section, we first introduce our focused problem in section 3.1, which is generating a visualbased policy by leveraging pretrained large models. we then introduce sum, which learns language descriptions of the surrounding environment in section 3.1, and apm which predicts actions based on sum's caption output in 3.2. to grant both sum and apm the capability of making the correct understanding and decision in the target domain, we propose finetuning algorithms in section 3.1 and 3.2. our code and data are provided in the supplementary materials. ## conclusion in this work, we introduce a novel robot learning paradigm with llm in the loop that handles multiple modalities of visual observations and textbased actions in a principled manner. we bridge both modalities with natural language generated by a pretrained multimodal model. our model contains sum and apm, where sum uses image observations as inputs taken by the robot to generate language descriptions of the current scene, and apm predicts the corresponding actions for the next step. we tested our method in the virtual-home under 7 unique environments, and the results demonstrated that our proposed paradigm outperforms baselines in terms of execution rates and shows strong stability across environments. one interesting future direction is extending our proposed framework to solve generalization tasks in a more data and parameter-efficient manner. for the finetuning of sum, we input the visual observation (i.e., the frames gathered during data collection) and output an image caption that serves to describe the scene. we calculate the loss by utilizing the textual description we collected since these textual descriptions are supposed to represent the "action" being partaken during the frame. divergence of text-image pairs and the number of the possible agent actions the divergence of text-image pairs is in the different combinations of action and object pairs for each text-image pair. for example, let us say we have a task of "turn on the light", and for this task, there are some actions, such as [[walk] < @xmath0, [walk] < @xmath1, [switchon] < @xmath2. for each action, there are n images (or frames) and text descriptions. for a given action, each image (frame) is different. however, for the text description, we simply use the same description as the ground truth for describing each image. nevertheless, there is a diverse corpus of action-object combinations (we have 18 different actions and 308 different objects). in our study, there are 18 different actions, including