Original: VIMA: G ENERAL R OBOT M ANIPULATION WITH M ULTIMODAL P ROMPTS
Revised: VIMA: G ENERAL R OBOT M ANIPULATION WITH M ULTIMODAL P ROMPTS

Original: Anonymous authors
Revised: https://vimalabs.github.io

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts.
Revised: Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts.

Original: Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals.
Revised: Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals.

Original: They are often considered different tasks and tackled by specialized models.
Revised: They are often considered different tasks and tackled by specialized models.

Original: This work shows that we can express a wide spectrum of robot manipulation tasks with multimodal prompts , interleaving textual and visual tokens.
Revised: This work shows that we can express a wide spectrum of robot manipulation tasks with multimodal prompts , interleaving textual and visual tokens.

Original: We design a transformer-based generalist robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively.
Revised: We design a transformer-based generalist robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively.

Original: To train and evaluate VIMA, we develop a new simulation benchmark with thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and four levels of evaluation protocol for systematic generalization.
Revised: To train and evaluate VIMA, we develop a new simulation benchmark with thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and four levels of evaluation protocol for systematic generalization.

Original: VIMA achieves strong scalability in both model capacity and data size.
Revised: VIMA achieves strong scalability in both model capacity and data size.

Original: It outperforms prior SOTA methods in the hardest zero-shot generalization setting by up to 2 .
Revised: It outperforms prior SOTA methods in the hardest zero-shot generalization setting by up to 2 .

Original: 9 × task success rate given the same training data.
Revised: 9 × task success rate given the same training data.

Original: With 10 × less training data, VIMA still performs 2 .
Revised: With 10 × less training data, VIMA still performs 2 .

Original: 7 × better than the top competing approach.
Revised: 7 × better than the top competing approach.

Original: Video demos are available at https://iclr3081.github.io/ .
Revised: We open-source all code, pretrained models, dataset, and simulation benchmark at https://vimalabs.github.io .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformers have given rise to remarkable multi-task consolidation across many AI domains.
Revised: Transformers have given rise to remarkable multi-task consolidation across many AI domains.

Original: For example, users can describe a task using natural language prompt to GPT-3 (Brown et al., 2020), allowing the same model to perform question answering, machine translation, text summarization, etc.
Revised: For example, users can describe a task using natural language prompt to GPT-3 (Brown et al., 2020), allowing the same model to perform question answering, machine translation, text summarization, etc.

Original: Prompt-based learning provides an accessible and flexible interface to communicate a natural language understanding task to a general-purpose model.
Revised: Prompt-based learning provides an accessible and flexible interface to communicate a natural language understanding task to a general-purpose model.

Original: We envision that a generalist robot agent should have a similarly intuitive and expressive interface for task specification.
Revised: We envision that a generalist robot agent should have a similarly intuitive and expressive interface for task specification.

Original: What does such an interface for robot learning look like?
Revised: What does such an interface for robot learning look like?

Original: As a motivating example, consider a personal robot tasked with household activities.
Revised: As a motivating example, consider a personal robot tasked with household activities.

Original: We can ask the robot to bring us a cup of water by a simple natural language instruction.
Revised: We can ask the robot to bring us a cup of water by a simple natural language instruction.

Original: If we require more specificity, we can instead instruct the robot to “bring me <image of the cup> ”.
Revised: If we require more specificity, we can instead instruct the robot to “bring me <image of the cup> ”.

Original: For tasks requiring new skills, the robot should be able to adapt preferably from a few video demonstrations (Duan et al., 2017).
Revised: For tasks requiring new skills, the robot should be able to adapt preferably from a few video demonstrations (Duan et al., 2017).

Original: Tasks that need interaction with unfamiliar objects can be easily explained via a few image examples for novel concept grounding (Hermann et al., 2017).
Revised: Tasks that need interaction with unfamiliar objects can be easily explained via a few image examples for novel concept grounding (Hermann et al., 2017).

Original: Finally, to ensure safe deployment, we can further specify visual constraints like “do not enter <image> room”.
Revised: Finally, to ensure safe deployment, we can further specify visual constraints like “do not enter <image> room”.

Original: To enable a single agent with all these capabilities, we make three key contributions in this work: 1) a novel multimodal prompting formulationthat converts a wide spectrum of robot manipulation tasks into one sequence modeling problem; 2) a new robot agent model capable of multi-tasking and zero-shot generalization; and 3) a large-scale benchmark with diverse tasks to systematically evaluate the scalability and generalization of our agents.
Revised: To enable a single agent with all these capabilities, we make three key contributions in this work: 1) a novel multimodal prompting formulation

Original: We start with the observation that many robot manipulationtasks can be formulated by multimodal prompts that interleave language and images or video frames (denoted by “ { image } ” in Fig. 1).
Revised: that converts a wide spectrum of robot manipulation

Original: For example, Rearrangement (Batra et al., 2020), a type of Visual Goal , can be formulated as “Please rearrange objects to match this { scene image } ”; Novel Concept Grounding looks like “This is adax { new object } 1 and this is a blicket { new object } 2 . Put two metal dax on the marble blicket.”
Revised: tasks into one sequence modeling problem; 2) a new robot agent model capable of multi tasking and zero-shot generalization; and 3) a large-scale benchmark with diverse tasks to systematically evaluate the scalability and generalization of our agents.

Original: ; Few-shot Imitation can embed video snippet in the prompt “Follow this motion
Revised: Foundation

Original: Rearrange objects to match this scene:
Revised: Rearrange objects to match this scene:

Original: Follow this motion for
Revised: Follow this motion for

Original: This is a blicket
Revised: This is a blicket

Original: This is a wug
Revised: This is a wug

Original: Put a wug into a blicket
Revised: Put a wug into a blicket

Original: Sweep all into without exceeding
Revised: Sweep all into without exceeding

Original: One-shot Demonstration
Revised: One-shot Demonstration

Original: Visual Goal: Rearrangement
Revised: Visual Goal: Rearrangement

Original: Visual Constraint

Original: Novel Concept

Original: [Figure]
Revised: [Figure]

Original: VIMA
Revised: We start with the observation that many robot manipulation tasks can be formulated by multimodal prompts that interleave language and images or video frames . For example, Rearrangement (Batra et al., 2020), a type of Visual Goal , can be formulated as “Please rearrange objects to match this { scene image } ”; Novel Concept Grounding looks like “This is a dax { new object } 1 and this is a blicket { new object } 2 . Put two metal dax on the marble blicket.”

Original: [Figure] Padding : trajectory for the wooden cube: { frame 1 } , { frame 2 } , { frame 3 } , { frame 4 } ”; and expressing Visual Constraint is as simple as adding the clause “without touching { safety boundary } ”.
Revised: ; Few-shot Imitation can embed video snippet in the prompt “Follow this motion trajectory for the wooden cube: { frame 1 } , { frame 2 } , { frame 3 } , { frame 4 } ”; and expressing Visual Constraint is as simple as adding the clause “without touching { safety boundary } ”.

Original: Multimodal prompts not only have more expressive power than individual modalities, but also enable a uniform sequence IO interface for training generalist robot agents.
Revised: Multimodal prompts not only have more expressive power than individual modalities, but also enable a uniform sequence IO interface for training generalist robot agents.

Original: Previously, different robot manipulation tasks require distinct policy architectures, objective functions, data pipelines, and training procedures (Aceituno et al., 2021; Stengel-Eskin et al., 2022; Lynch & Sermanet, 2021), leading to siloed robot systems that cannot be easily combined for a rich set of use cases.
Revised: Previously, different robot manipulation tasks require distinct policy architectures, objective functions, data pipelines, and training procedures (Aceituno et al., 2021; Stengel-Eskin et al., 2022; Lynch & Sermanet, 2021), leading to siloed robot systems that cannot be easily combined for a rich set of use cases.

Original: Instead, our multimodal prompt interface allows us to harness the latest advances in large transformer models (Lin et al., 2021; Tay et al., 2020; Khan et al., 2021) for developing scalable multi-task robot learners.
Revised: Instead, our multimodal prompt interface allows us to harness the latest advances in large transformer models (Lin et al., 2021; Tay et al., 2020; Khan et al., 2021) for developing scalable multi-task robot learners.

Original: To this end, we design a novel Vi suo M otor
Revised: To this end, we design a novel Vi suo M otor

Original: A ttention model (VIMA).
Revised: A ttention model (VIMA, reads “ v-eye-ma ”).

Original: The architecture follows the encoder-decoder transformer design proven to be effective and scalable in NLP (Raffel et al., 2020).
Revised: The architecture follows the encoder-decoder transformer design proven to be effective and scalable in NLP (Raffel et al., 2020).

Original: VIMA encodes an input sequence of interleaving textual and visual prompt tokens with a pretrained language model (Tsimpoukelli et al., 2021), and decodes robot control actions autoregressively for each environment interaction step.
Revised: VIMA encodes an input sequence of interleaving textual and visual prompt tokens with a pretrained language model (Tsimpoukelli et al., 2021), and decodes robot control actions autoregressively for each environment interaction step.

Original: The transformer decoder is conditioned on the prompt via cross-attention layers that alternate with the usual causal self-attention.
Revised: The transformer decoder is conditioned on the prompt via cross-attention layers that alternate with the usual causal self-attention.

Original: Instead of operating on raw pixels, VIMA adopts an object-centric approach.
Revised: Instead of operating on raw pixels, VIMA adopts an object-centric approach.

Original: We parse all images in the prompt or observation into objects by off-the-shelf detectors (He et al., 2017), and flatten them into sequences of object tokens.
Revised: We parse all images in the prompt or observation into objects by off-the-shelf detectors (He et al., 2017), and flatten them into sequences of object tokens.

Original: All these design choices combined deliver a conceptually simple architecture with strong model and data scaling properties.
Revised: All these design choices combined deliver a conceptually simple architecture with strong model and data scaling properties.

Original: To systematically evaluate our proposed algorithm, we introduce a new benchmark (VIMA-B ENCH ) built on the Ravens simulator (Zeng et al., 2020; Shridhar et al., 2021).
Revised: To systematically evaluate our proposed algorithm, we introduce a new benchmark (VIMA-B ENCH ) built on the Ravens simulator (Zeng et al., 2020; Shridhar et al., 2021).

Original: We provide 17 representative meta-tasks with multimodal prompt templates, which can be procedurally instantiated into thousands of individual tasks by various combinations of textures and tabletop objects.
Revised: We provide 17 representative meta-tasks with multimodal prompt templates, which can be procedurally instantiated into thou- [Figure] sands of individual tasks by various combinations of textures and tabletop objects.

Original: VIMA-B ENCH establishes a 4-level protocol to evaluate progressively stronger generalization capabilities, from randomized object placement to novel tasks altogether (Fig. 2).
Revised: VIMA-B ENCH establishes a 4-level protocol to evaluate progressively stronger generalization capabilities, from randomized object placement to novel tasks altogether (Fig. 2).

Original: To demonstrate the scalability of VIMA, we train a spectrum of 7 models ranging from 2M to 200M parameters.
Revised: To demonstrate the scalability of VIMA, we train a spectrum of 7 models ranging from 2M to 200M parameters.

Original: Our approach outperforms strong prior SOTA methods such as Gato (Reed et al., 2022), Decision Transformer (Chen et al., 2021), and Flamingo (Alayrac et al., 2022) across all 4 levels of zero-shot generalization and all model capacities, sometimes by a large margin (up to 2 .
Revised: Our approach outperforms strong prior SOTA methods such as Gato (Reed et al., 2022), Decision Transformer (Chen et al., 2021), and Flamingo (Alayrac et al., 2022) across all 4 levels of zero-shot generalization and all model capacities, sometimes by a large margin (up to 2 .

Original: 9 × task success rate given the same amount of training data, and 2 .
Revised: 9 × task success rate given the same amount of training data, and 2 .

Original: 7 × better even with 10 × less data).
Revised: 7 × better even with 10 × less data).

Original: Weplan to open-source the simulation environment, training dataset, algorithm code, and pretrained model checkpoints to ensure reproducibility and facilitate future works from the community.We attach supplementary materials as Appendix to this PDF, and present video demos and anonymized code at https://iclr3081.github.io/ .  R ELATED W ORK Multi-task Learning by Sequence Modeling.
Revised: We open-source the simulation suite, training dataset, algorithm implementation, and pretrained model checkpoints to ensure reproducibility and facilitate future works from the community.

Original: Transformers have enabled task unification across many AI domains (Raffel et al., 2020;

Original: Radford et al., 2019; Brown et al., 2020; Chen et al., 2022a;b;

Original: Lu et al., 2022; Wang et al., 2022c;

Original: Alayrac et al., 2022; Reed et al., 2022).

Original: For example, in NLP ,

Original: T5 (Raffel et al., 2020) unifies all language problems into the same text-to-text format.

Original: GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and Megatron-Turing NLG (Shoeybi et al., 2019) demonstrate emergent behaviours of intuitive task specifications by zero-shot prompting without any finetuning.

Original: In computer vision , Florence (Yuan et al., 2021), BiT (Kolesnikov et al., 2020), and MuST (Ghiasi et al., 2021) pre-train a shared backbone model at scale for general visual representations and transfer it across downstream vision tasks.

Original: Pix2Seq (Chen et al., 2022b) casts many vision problems into a unified sequence format.

Original: In multimodal learning , Flamingo (Alayrac et al., 2022) and Frozen (Tsimpoukelli et al., 2021) design a universal API that ingests an interleaving sequence of images and text and generates free-form text.

Original: Gato (Reed et al., 2022) is a massively multitask model across NLP, vision, and embodied agents.

Original: Our work is most similar in spirit to Gato, but we focus primarily on enabling an intuitive, multimodal prompting interface for a generalist robot agent.

Original: Foundation Models for Embodied Agents.

Original: Foundation models (Bommasani et al., 2021; Brown et al., 2020; Raffel et al., 2020; Ramesh et al., 2022; Wei et al., 2022) have demonstrated strong emergent properties like zero-shot prompting and complex reasoning.

Original: There are many ongoing efforts to replicate this success for embodied agents, focusing on 3 aspects: 1) Transformer agent architecture : Decision Transformer (Chen et al., 2021; Janner et al., 2021; Zheng et al., 2022) and

Original: Gato (Reed et al., 2022) leverage the powerful self-attention models for sequential decision making.

Original: CLIPort (Shridhar et al., 2021) and Perceiver-Actor (Shridhar et al., 2022) apply large transformers to robot manipulation tasks.

Original: Pre-training for better representations : Masked ViT (Gupta et al., 2022), R3M (Nair et al., 2022), and Parisi et al.

Original: (2022) pre-train general visual representations for robotic perception.

Original: Li et al.

Original: ; Reid et al.

Original: (2022) finetune from LLM checkpoints to accelerate policy learning.

Original: MineDojo (Fan et al., 2022) and Ego4D (Grauman et al., 2021) provide large-scale multimodal databases to facilitate scalable policy training.

Original: Large language models for robot learning : SayCan (Ahn et al., 2022) leverages the 500B PaLM (Chowdhery et al., 2022) for zero-shot concept grounding.

Original: Socratic Models (Zeng et al., 2022) composes multiple vision and language foundation models (VLMs) for multimodal reasoning in videos.

Original: Huang et al.

Original: (2022a),

Original: Inner Monologue (Huang et al., 2022b) and LM-Nav (Shah et al., 2022) successfully apply LLMs to long-horizon robot planning.

Original: VIMA differs from these works in our novel multimodal prompting formulation, which existing LLMs and VLMs do not easily support.

Original: Robot Manipulation and Benchmarks.

Original: There are a wide range of robot manipulation tasks that require different skills and task specification formats, such as instruction following (Stepputtis et al., 2020; Shridhar et al., 2021; Lynch & Sermanet, 2021), one-shot imitation (Finn et al., 2017; Dasari & Gupta, 2020; Duan et al., 2017), rearrangement (Batra et al., 2020;

Original: Weihs et al., 2021; Szot et al., 2021), constraint satisfaction (Brunke et al., 2021; Srinivasan et al., 2020; Thananjeyan et al., 2021), and reasoning (Shridhar et al., 2020; Gupta et al., 2019;

Original: Ahmed et al., 2021; Toyer et al., 2020; Lim et al., 2021).

Original: Multiple physics simulation benchmarks are introduced to study the above tasks.

Original: For example, iGibson (Shen et al., 2020; Li et al., 2021; Srivastava et al., 2021) simulates interactive household scenarios.

Original: Ravens (Zeng et al., 2020) and Robosuite (Zhu et al., 2020; Fan et al., 2021) design various tabletop manipulation tasks with realistic robot arms.

Original: Our VIMA-B ENCH is the first robot learning benchmark to support multimodal-prompted tasks.

Original: We also standardize the evaluation protocol to systematically measure an agent’s generalization capabilities.

Original: A more extended literature review can be found in Appendix, Sec.

Original: F.

Original: [Figure]

Original:  M ULTIMODAL P ROMPTS FOR T ASK S PECIFICATION
Revised:  M ULTIMODAL P ROMPTS FOR T ASK S PECIFICATION

Original: A central and open problem in robot learning is task specification (Agrawal, 2022).
Revised: A central and open problem in robot learning is task specification (Agrawal, 2022).

Original: Our key insight is that various task specification paradigms (such as goal conditioning, video demonstration, natural language instruction) can all be instantiated as multimodal prompts (Fig. 1).
Revised: Our key insight is that various task specification paradigms (such as goal conditioning, video demonstration, natural language instruction) can all be instantiated as multimodal prompts (Fig. 1).

Original: Concretely, a multimodal prompt P of length l is defined as an ordered sequence of arbitrarily interleaved texts and images
Revised: Concretely, a multimodal prompt P of length l is defined as an ordered sequence of arbitrarily interleaved texts and images

Original: P :=
Revised: P :=

Original: [ x 0 , x 1 , . . .
Revised: [ x 0 , x 1 , . . .

Original: , x l ] , where each element x i ∈ { text , image } .
Revised: , x l ] , where each element x i ∈ { text , image } .

Original: Task Suite.
Revised: Task Suite.

Original: The flexibility afforded by multimodal prompts allows us to specify and build models for a huge variety of task specification formats.
Revised: The flexibility afforded by multimodal prompts allows us to specify and build models for a huge variety of task specification formats.

Original: Here we consider the following six task categories:
Revised: Here we consider the following six task categories:

Original: Simple object manipulation : simple tasks like “put <object> into <container> ”, where each image in the prompt corresponds to a single object; 2.
Revised: Simple object manipulation : simple tasks like “put <object> into <container> ”, where each image in the prompt corresponds to a single object; 2.

Original: Visual goal reaching : manipulating objects to reach a goal configuration, e.g. , Rearrangement (Batra et al., 2020); 3.
Revised: Visual goal reaching : manipulating objects to reach a goal configuration, e.g. , Rearrangement (Batra et al., 2020); 3.

Original: Novel concept grounding : the prompt contains unfamiliar words like “dax” and “blicket”, which are explained by in-prompt images and then immediately used in an instruction.
Revised: Novel concept grounding : the prompt contains unfamiliar words like “dax” and “blicket”, which are explained by in-prompt images and then immediately used in an instruction.

Original: This tests the agent’s ability to rapidly internalize new concepts; 4.
Revised: This tests the agent’s ability to rapidly internalize new concepts; 4.

Original: One-shot video imitation : watching a video demonstration and learning to reproduce the same motion trajectory for a particular object; 5.
Revised: One-shot video imitation : watching a video demonstration and learning to reproduce the same motion trajectory for a particular object; 5.

Original: Visual constraint satisfaction : the robot must manipulate the objects carefully and avoid violating the (safety) constraints; 6.
Revised: Visual constraint satisfaction : the robot must manipulate the objects carefully and avoid violating the (safety) constraints; 6.

Original: Visual reasoning : tasks that require reasoning skills, such as appearance matching “move all objects with same textures as <object> into a container”, and visual memory, “put <object> in container and then restore to their original position”.
Revised: Visual reasoning : tasks that require reasoning skills, such as appearance matching “move all objects with same textures as <object> into a container”, and visual memory, “put <object> in container and then restore to their original position”.

Original: Note that these six categories are not mutually exclusive.
Revised: Note that these six categories are not mutually exclusive.

Original: For example, a task may introduce a previously unseen verb ( Novel Concept ) by showing a video demonstration, or combine goal reaching with visual reasoning.
Revised: For example, a task may introduce a previously unseen verb ( Novel Concept ) by showing a video demonstration, or combine goal reaching with visual reasoning.

Original: More details about the task suite are discussed in Appendix, Sec.
Revised: More details about the task suite are discussed in Appendix, Sec.

Original: B.
Revised: B.

Original:  V IMA -B ENCH : B ENCHMARK FOR MULTIMODAL ROBOT LEARNING
Revised:  V IMA -B ENCH : B ENCHMARK FOR MULTIMODAL ROBOT LEARNING

Original: Simulation Environment.
Revised: Simulation Environment.

Original: Existing benchmarks are generally geared towards a particular task specification.
Revised: Existing benchmarks are generally geared toward a particular task specification.

Original: To our knowledge, there is no benchmark that provides a rich suite of multimodal tasks and a comprehensive testbed for targeted probing of agent capabilities.
Revised: To our knowledge, there is no benchmark that provides a rich suite of multimodal tasks and a comprehensive testbed for targeted probing of agent capabilities.

Original: To this end, we introduce a new benchmark suite for multimodal robot learning thta we call VIMA-B ENCH .
Revised: To this end, we introduce a new benchmark suite for multimodal robot learning that we call VIMA-B ENCH .

Original: We built our benchmark by extending the Ravens robot simulator (Zeng et al., 2020).
Revised: We built our benchmark by extending the Ravens robot simulator (Zeng et al., 2020; Shridhar et al., 2021).

Original: VIMA-B ENCH supports extensible collections of objects and textures to compose multimodal prompts and procedurally generate a large number of tasks.
Revised: VIMA-B ENCH supports extensible collections of objects and textures to compose multimodal prompts and procedurally generate a large number of tasks.

Original: Specifically, we provide 17 meta-tasks with multimodal prompt templates, which can be instantiated into 1000s of individual tasks.
Revised: Specifically, we provide 17 meta-tasks with multimodal prompt templates, which can be instantiated into 1000s of individual tasks.

Original: Each meta-task belongs to one or more of 6 task specification methods mentioned above.
Revised: Each meta-task belongs to one or more of 6 task specification methods mentioned above.

Original: VIMA-B ENCH can generate large quantities of imitation learning data via scripted oracle agents.
Revised: VIMA-B ENCH can generate large quantities of imitation learning data via scripted oracle agents.

Original: More details are elaborated in Appendix, Sec.
Revised: More details are elaborated in Appendix, Sec.

Original: A.
Revised: A.

Original: [Figure]

Original: Observation and Actions.
Revised: Observation and Actions.

Original: The observation space of our simulator includes RGB images rendered from both frontal view and top-down view.
Revised: The observation space of our simulator includes RGB images rendered from both frontal view and top-down view.

Original: Groundtruth object segmentations and bounding boxes are also provided for training object-centric models (Sec. 5).
Revised: Groundtruth object segmentations and bounding boxes are also provided for training object-centric models (Sec. 4).

Original: We inherit the high-level action space from
Revised: We inherit the high-level action space from

Original: Zeng et al.
Revised: Zeng et al.

Original: (2020), which consists of primitive motor skills like “pick and place” and “wipe”.
Revised: (2020), which consists of primitive motor skills like “pick and place” and “wipe”.

Original: These are parameterized by poses of the end effector.
Revised: These are parameterized by poses of the end effector.

Original: Our simulator also features scripted oracle programs that can generate expert demonstrations by using privileged simulator state information, such as the precise location of all objects, and the groundtruth interpretation of the multimodal instruction.
Revised: Our simulator also features scripted oracle programs that can generate expert demonstrations by using privileged simulator state information, such as the precise location of all objects, and the groundtruth interpretation of the multimodal instruction.

Original: Training Dataset.
Revised: Training Dataset.

Original: We leverage the pre-programmed oracles to generate a large offline dataset of expert trajectories for imitation learning.
Revised: We leverage the pre-programmed oracles to generate a large offline dataset of expert trajectories for imitation learning.

Original: Our dataset includes 50 K trajectories per meta-task, and 650 K successful trajectories in total.
Revised: Our dataset includes 50 K trajectories per meta-task, and 650 K successful trajectories in total.

Original: We hold out a subset of object models and textures for evaluation, and designate 4 out of 17 meta-tasks as a testbed for zero-shot generalization.
Revised: We hold out a subset of object geometries and textures for evaluation, and designate 4 out of 17 meta-tasks as a testbed for zero-shot generalization.

Original: Evaluating Zero-Shot Generalization.
Revised: Evaluating Zero-Shot Generalization.

Original: Each task in VIMA-B ENCH has a binary success criterion and does not provide partial reward signals.
Revised: Each task in VIMA-B ENCH has a binary success criterion and does not provide partial reward signals.

Original: During test time, we execute the agent policies in the physics simulator for multiple episodes to compute a success rate in percentage.
Revised: During test time, we execute the agent policies in the physics simulator for multiple episodes to compute a success rate in percentage.

Original: The average success rate over all evaluated meta-tasks will be the final reported metric.
Revised: The average success rate over all evaluated meta-tasks will be the final reported metric.

Original: We design a 4-level evaluation protocol (Fig. 2) to systematically probe the generalization capabilities of learned agents.
Revised: We design a 4-level evaluation protocol (Fig. 2) to systematically probe the generalization capabilities of learned agents.

Original: Each level deviates more from the training distribution, and is thus strictly harder than the previous one — Level 1) placement generalization : all prompts are seen verbatim during training, but only the placement of objects on the tabletop is randomized at testing; Level 2) combinatorial generalization : all materials (adjectives) and 3D objects (nouns) are seen during training, but new combinations of them appear in testing; Level 3) novel object generalization : test prompts and the simulated workspace include novel adjectives and objects; Level 4) novel task generalization : new meta-tasks with novel prompt templates at test time.
Revised: Each level deviates more from the training distribution, and is thus strictly harder than the previous one: Placement generalization : all prompts are seen verbatim during training, but only the placement of objects on the tabletop is randomized at testing. Combinatorial generalization : all materials (adjectives) and 3D objects (nouns) are seen during training, but new combinations of them appear in testing. Novel object generalization : test prompts and the simulated workspace include novel adjectives and objects.

Revised: Novel task generalization : new meta-tasks with novel prompt templates at test time.

Original:  VIMA: V ISUOMOTOR A TTENTION M ODEL
Revised:  VIMA: V ISUOMOTOR A TTENTION M ODEL

Original: Our goal is to build a robot agent which can perform any task specified by multimodal prompts.
Revised: Our goal is to build a robot agent which can perform any task specified by multimodal prompts.

Original: To learn an effective multi-task robot policy, we propose VIMA, a minimalistic multi-task encoder-decoder architecture with object-centric design (Fig. 3).
Revised: To learn an effective multi-task robot policy, we propose VIMA, a minimalistic multi-task encoder-decoder architecture with object-centric design (Fig. 3).

Original: Concretely, we learn a robot policy π ( a t |P , H ) , where H :=
Revised: Concretely, we learn a robot policy π ( a t |P , H ) , where H :=

Original: [ o 1 , a 1 , o 2 , a 2 , . . .
Revised: [ o 1 , a 1 , o 2 , a 2 , . . .

Original: , o t ] denotes the past interaction history, and o t ∈ O , a t ∈ A are observations and actions at each interaction steps.
Revised: , o t ] denotes the past interaction history, and o t ∈ O , a t ∈ A are observations and actions at each interaction step.

Original: We encode the prompt via a frozen pre-trained langauge model and decode the robot motor commands conditioned on the encoded prompt via cross-attention layers.
Revised: We encode the prompt via a frozen pre-trained langauge model and decode the robot motor commands conditioned on the encoded prompt via cross-attention layers.

Original: Tokenization.
Revised: Tokenization.

Original: There are 3 formats of raw input in the prompt — text, image of a single object, and image of a full tabletop scene ( e.g. , for Rearrangement or imitation from video frames).
Revised: There are 3 formats of raw input in the prompt — text, image of a single object, and image of a full tabletop scene ( e.g. , for Rearrangement or imitation from video frames).

Original: For text inputs , we use pre-trained T5 tokenizer and word embedding to obtain word tokens.
Revised: For text inputs , we use pre-trained T5 tokenizer and word embedding to obtain word tokens.

Original: For images of full scenes , we first extract individual objects using off-the-shelf Mask R-CNN (He et al., 2017).
Revised: For images of full scenes , we first extract individual objects using off-the-shelf Mask R-CNN (He et al., 2017).

Original: Each object is represented as a bounding box and a cropped image.
Revised: Each object is represented as a bounding box and a cropped image.

Original: We then compute object tokens  by encoding them with a bounding box encoder and a ViT, respectively.
Revised: We then compute object tokens [Figure] into without touching by encoding them with a bounding box encoder and a ViT, respectively.

Original: For images of single objects , we obtain tokens in the same way except with a dummy bounding box.
Revised: For images of single objects , we obtain tokens in the same way except with a dummy bounding box.

Original: We then follow Tsimpoukelli et al.
Revised: We then follow Tsimpoukelli et al.

Original: (2021) to produce prompt encoding by passing the resulted token sequence to a pre-trained T encoder model.
Revised: (2021) to produce prompt encoding by passing the resulted token sequence to a pre-trained T encoder model.

Original: Our positional embedding is learnable and absolute.
Revised: Our positional embedding is learnable and absolute.

Original: Robot Controller.
Revised: Robot Controller.

Original: A challenging aspect of designing multi-modal multi-task policy is to select a suitable conditioning mechanism.
Revised: A challenging aspect of designing multi-modal multi-task policy is to select a suitable conditioning mechanism.

Original: In our schema (Fig. 3), the robot controller (decoder) is conditioned on the prompt sequence P by a series of cross-attention layers between P and the trajectory history sequence H .
Revised: In our schema (Fig. 3), the robot controller (decoder) is conditioned on the prompt sequence P by a series of cross-attention layers between P and the trajectory history sequence H .

Original: Each cross-attention layer generates an output sequence H ′ =
Revised: Each cross-attention layer generates an output sequence H ′ =

Original: softmax (cid:16)Q
Revised: softmax (cid:16)

Original: [Equation]
Revised: [Equation]

Original: d (cid:17) V P , where d is the embedding dimension.
Revised: V P , where d is the embedding dimension.

Original: This design choice enjoys three advantages, including 1) strengthened connection to prompt, 2) intact and deep flow of the original prompt tokens, and 3) better computational efficiency.
Revised: This design choice enjoys three advantages, including 1) strengthened connection to prompt, 2) intact and deep flow of the original prompt tokens, and 3) better computational efficiency.

Original: VIMA decoder consists of L alternating cross-attention and selfattention layers.
Revised: VIMA decoder consists of L alternating cross-attention and self-attention layers.

Original: Finally, for ease of learning and optimization, we follow common practice (Baker et al., 2022) to map predicted action tokens to discretized coordinates of the robot arm.
Revised: Finally, for ease of learning and optimization, we follow common practice (Baker et al., 2022) to map predicted action tokens to discretized poses of the robot arm.

Original: Training.
Revised: Training.

Original: We follow the standard behavioral cloning to train our models by minimizing the negative log-likelihood of predicted actions.
Revised: We follow the standard behavioral cloning to train our models by minimizing the negative log-likelihood of predicted actions.

Original: Concretely, for a trajectory with T steps, we minimize min θ (cid:80)
Revised: Concretely, for a trajectory with T steps, we minimize min θ (cid:80)

Original: Tt =1 − log π θ ( a t |P , H ) .
Revised: Tt =1 − log π θ ( a t |P , H ) .

Original: The entire training is conducted on an offline dataset with no access to the physics simulator.
Revised: The entire training is conducted on an offline dataset with no access to the physics simulator.

Original: To make VIMA robust to detection inaccuracies and failures, we apply object augmentation by randomly injecting false-positive detection outputs.
Revised: To make VIMA robust to detection inaccuracies and failures, we apply object augmentation by randomly injecting false-positive detection outputs.

Original: After training, we select model checkpoints for evaluation based on the aggregated accuracy on a held-out validation set.
Revised: After training, we select model checkpoints for evaluation based on the aggregated accuracy on a held-out validation set.

Original: The evaluation involves interacting with the physics simulator.
Revised: The evaluation involves interacting with the physics simulator.

Original: We follow the best practices to train
Revised: We follow the best practices to train

Original: Transformer models using the AdamW optimizer (Loshchilov & Hutter, 2019), learning rate warm-up, cosine annealing (Loshchilov & Hutter, 2016), etc.
Revised: Transformer models using the AdamW optimizer (Loshchilov & Hutter, 2019), learning rate warm-up, cosine annealing (Loshchilov & Hutter, 2016), etc.

Original: See Appendix Sec.
Revised: See Appendix Sec.

Original:  D for a comprehensive listing of our model and training hyperparameters.
Revised: C & D for a comprehensive listing of our model and training hyperparameters.

Revised: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we aim to answer three main questions:(1) How does VIMA compare with prior
Revised: In this section [Figure] we aim to answer three main questions:

Original:  SOTA transformer-based agents on a diverse collection of multimodal-prompted tasks?
Revised: How does VIMA compare with prior SOTA transformer-based agents on a diverse collection of multimodal-prompted tasks?

Original: (2) What are the scaling properties of our approach in model capacity and data size? 
Revised: What are the scaling properties of our approach in model capacity and data size? 3.

Original: (3) How do different visual tokenizers, prompt conditioning, and prompt encoding affect decision making? 
Revised: How do different visual tokenizers, prompt conditioning, and prompt encoding affect decision making? 5.

Original: 6. B ASELINES
Revised: B ASELINES

Original: Gato (Reed et al., 2022) introduces a decoder-only model that solves tasks from multiple domains where tasks are specified by prompting the model with the observation and action subsequence.
Revised: Gato (Reed et al., 2022) introduces a decoder-only model that solves tasks from multiple domains where tasks are specified by prompting the model with the observation and action subsequence.

Original: For fair comparison, we provide the same conditioning as VIMA, i.e. , our multimodal embedded prompt.
Revised: For fair comparison, we provide the same conditioning as VIMA, i.e. , our multimodal embedded prompt.

Original: Input images are divided into patches and encoded by a ViT (Dosovitskiy et al., 2020) model to produce observation tokens.
Revised: Input images are divided into patches and encoded by a ViT (Dosovitskiy et al., 2020) model to produce observation tokens.

Original: Flamingo (Alayrac et al., 2022) is a vision-language model that learns to generate textual completion in response to multimodal prompts.
Revised: Flamingo (Alayrac et al., 2022) is a vision-language model that learns to generate textual completion in response to multimodal prompts.

Original: It embeds a variable number of prompt images into a fixed number of tokens via the Perceiver Resampler module, and conditions the language decoder on the[Figure] encoded prompt by cross-attention.
Revised: It embeds a variable number of prompt images into a fixed number of tokens via the Perceiver Resampler module, and conditions the language decoder on the encoded prompt by cross-attention.

Original: Flamingo does not work with embodied agents out of the box.
Revised: Flamingo does not work with embodied agents out of the box.

Original: We adapt it to support decision-masking by replacing the output layer with robot action heads
Revised: We adapt it to support decision-masking by replacing the output layer with robot action heads.

Original: [Figure] Decision Transformer (DT) (Chen et al., 2021; Janner et al., 2021) is among the first works to reinterpret the RL problem as transformer sequence modeling.
Revised: Decision Transformer (DT) (Chen et al., 2021; Janner et al., 2021) is among the first works to reinterpret the RL problem as transformer sequence modeling.

Original: In visual RL domains like Atari games,
Revised: In visual RL domains like Atari games,

Original: DT is prompted on the desired reward value and outputs actions autoregressively given the RGB observation embedding.
Revised: DT is prompted on the desired reward value and outputs actions autoregressively given the RGB observation embedding.

Original: We replace DT’s initial reward prompt with our multimodal task prompt embeddings, and remove all subsequent reward tokens. 
Revised: We replace DT’s initial reward prompt with our multimodal task prompt embeddings, and remove all subsequent reward tokens. [Figure]

Original: 6. E VALUATION R ESULTS
Revised: 5. E VALUATION R ESULTS

Original: We compare VIMA against other SOTA methods on the four levels of generalization provided in our benchmark for different model and training dataset sizes.
Revised: We compare VIMA against other SOTA methods on the four levels of generalization provided in our benchmark for different model and training dataset sizes.

Original: Model scaling.
Revised: Model scaling.

Original: We train all methods for a spectrum of model capacities from 2M to 200M parameters, evenly spaced on the log scale.
Revised: We train all methods for a spectrum of model capacities from 2M to 200M parameters, evenly spaced on the log scale.

Original: The encoder size is kept constant (pretrained T5-Base) for all methods and excluded from the parameter count.
Revised: The encoder size is kept constant (pretrained T5-Base) for all methods and excluded from the parameter count.

Original: Across all levels of zero-shot generalization, we find that VIMA strongly outperforms prior work.
Revised: Across all levels of zero-shot generalization, we find that VIMA strongly outperforms prior work.

Original: Although models like Gato and Flamingo show improved performance with bigger model sizes, VIMA consistently achieves superior performance over all model sizes.
Revised: Although models like Gato and Flamingo show improved performance with bigger model sizes, VIMA consistently achieves superior performance over all model sizes.

Original: We note that this can only be achieved with both cross-attention and object token sequence representation — altering any component will degrade the performance significantly, especially in the low model capacity regime (ablations in Sec. 6.3).
Revised: We note that this can only be achieved with both cross-attention and object token sequence representation — altering any component will degrade the performance significantly, especially in the low model capacity regime (ablations in Sec. 5.3).

Original: Data scaling.
Revised: Data scaling.

Original: Next we investigate how different methods scale with varying dataset sizes.
Revised: Next we investigate how different methods scale with varying dataset sizes.

Original: We compare model performance at 0 .
Revised: We compare model performance at 0 .

Original: 1% , 1% , 10% and full imitation learning dataset provided in
Revised: 1% , 1% , 10% and full imitation learning dataset provided in

Original: VIMA-B ENCH (Fig. 4).
Revised: VIMA-B ENCH (Fig. 4).

Original: VIMA is extremely sample efficient and with just 1% of the data can achieve performance similar to baseline methods trained with 10 × more data for L1 and L2 levels of generalization.
Revised: VIMA is extremely sample efficient and with just 1% of the data can achieve performance similar to baseline methods trained with 10 × more data for L1 and L2 levels of generalization.

Original: In fact, for L4 we find that with just 1% of training data, VIMA already outperforms prior work trained with entire dataset.
Revised: In fact, for L4 we find that with just 1% of training data, VIMA already outperforms prior work trained with entire dataset.

Original: Finally, across all levels with just 10% of the data, VIMA can outperforms prior work trained with the full dataset by a significant margin.
Revised: Finally, across all levels with just 10% of the data, VIMA can outperform prior work trained with the full dataset by a significant margin.

Original: We hypothesize that the data efficiency can be attributed to VIMA’s object-centric representation, which is less prone to overfitting than learning directly from pixels in the low-data regime.
Revised: We hypothesize that the data efficiency can be attributed to VIMA’s object-centric representation, which is less prone to overfitting than learning directly from pixels in the low-data regime.

Original: This is consistent with findings[Figure] from Sax et al.
Revised: This is consistent with findings from Sax et al.

Original: (2018), which demonstrates that embodied agents conditioned on mid-level visual representations tend to be significantly more sample-efficient than end-to-end control from raw pixels.
Revised: (2018), which demonstrates that embodied agents conditioned on mid-level visual representations tend to be significantly more sample-efficient than end-to-end control from raw pixels.

Original: Progressive Generalization.
Revised: Progressive Generalization.

Original: Finally, we compare the relative degradation in performance as we test the models on progressively challenging zero-shot evaluation levels without further finetuning (Fig. 6).
Revised: Finally, we compare the relative degradation in performance as we test the models on progressively challenging zero-shot evaluation levels without further finetuning (Fig. 6).

Original: Our method exhibits a minimal performance regression, especially between L 1 → L 2 and
Revised: Our method exhibits a minimal performance regression, especially between L 1 → L 2 and

Revised: [Figure]

Original: L 1 → L 3 .
Revised: L 1 → L 3 .

Original: In contrast, other methods can degrade as much as 20% , particularly in the more difficult generalization testing scenarios.
Revised: In contrast, other methods can degrade as much as 20% , particularly in the more difficult generalization testing scenarios.

Original: Although all methods degrade significantly when evaluated on L( Novel Tasks ), the drop in performance for VIMA is only half as severe as all other baselines.
Revised: Although all methods degrade significantly when evaluated on L( Novel Tasks ), the drop in performance for VIMA is only half as severe as all other baselines.

Original: This results suggest that VIMA has developed more generalizable policy and robust representations than the competing approaches.
Revised: This results suggest that VIMA has developed more generalizable policy and robust representations than the competing approaches.

Original: 6. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Revised: [Figure]

Original: Through extensive experiments, we ablate different design choices in VIMA and study their impact on robot decision making.
Revised: Through extensive experiments, we ablate different design choices in VIMA and study their impact on robot decision making.

Original: We focus on the following 3 aspects: visual tokenization, prompt encoding, and prompt conditioning variants.
Revised: We focus on the following 3 aspects: visual tokenization, prompt encoding, and prompt conditioning variants.

Original: Visual tokenization.
Revised: Visual tokenization.

Original: As explained in Sec. 5, VIMA processes the prompt and observation images into a variable number of object tokens with an off-the-shelf Mask RCNN implementation.
Revised: As explained in Sec. 4, VIMA processes the prompt and observation images into a variable number of object tokens with an off-the-shelf Mask R-CNN implementation.

Original: How important is this particular choice of visual tokenizer?
Revised: How important is this particular choice of visual tokenizer?

Original: We study 5 different variants and empirically evaluate their 4 levels of generalization performance on VIMA-B ENCH .
Revised: We study 5 different variants and empirically evaluate their 4 levels of generalization performance on VIMA-B ENCH .

Original: (1) Ours (Oracle) : instead of using Mask R-CNN, we directly read out the groundtruth bounding box from the simulator.
Revised: (1) Ours (Oracle) : instead of using Mask R-CNN, we directly read out the groundtruth bounding box from the simulator.

Original: In other words, we use a perfect object detector to estimate the upper bound on the performance of this study; (2) Object
Revised: In other words, we use a perfect object detector to estimate the upper bound on the performance of this study; (2) Object

Original: Perceiver : we apply a Perceiver module (Jaegle et al., 2021b;a) to convert the variable number[Figure] of objects detected in each frame to a fixed number of tokens.
Revised: Perceiver : we apply a Perceiver module (Jaegle et al., 2021b;a) to convert the variable number of objects detected in each frame to a fixed number of tokens.

Original: Perceiver is more computationally efficient because it reduces the average sequence length; (3) Image Perceiver : the same architecture as the Perceiver Resampler in Flamingo, which converts an image to a small, fixed number of tokens; (4) Image patches : following Gato, we divide an RGB frame into square patches, and extract ViT embedding tokens.
Revised: Perceiver is more computationally efficient because it reduces the average sequence length; (3) Image Perceiver : the same architecture as the Perceiver Resampler in Flamingo, which converts an image to a small, fixed number of tokens; (4) Image patches : following Gato, we divide an RGB frame into square patches, and extract ViT embedding tokens.

Original: The number of patches is more than the output of Image Perceiver; (5) Single image : Decision Transformer’s tokenizer, which encodes one image into a single token.
Revised: The number of patches is more than the output of Image Perceiver; (5) Single image : Decision Transformer’s tokenizer, which encodes one image into a single token.

Original: Fig.
Revised: Fig.

Original: 5 shows the ablation results.
Revised: 5 shows the ablation results.

Original: We highlight a few findings.
Revised: We highlight a few findings.

Original: First, we note that our Mask R-CNN detection pipeline (Appendix, Sec. A.20) incurs a minimal performance loss compared to the oracle bounding boxes, thanks to the object augmentation (Sec. 5) that boosts robustness during training.
Revised: First, we note that our Mask R-CNN detection pipeline (Appendix, Sec. A.20) incurs a minimal performance loss compared to the oracle bounding boxes, thanks to the object augmentation (Sec. 4) that boosts robustness during training.

Original: Second, tokenizing from raw pixels (Image Perceiver, patches, or single embedding) consistently underperforms our object-centric format.
Revised: Second, tokenizing from raw pixels (Image Perceiver, patches, or single embedding) consistently underperforms our object-centric format.

Original: We hypothesize that these tokenizers have to allocate extra internal capacity to parse the objects from low-level pixels, which likely impedes learning.
Revised: We hypothesize that these tokenizers have to allocate extra internal capacity to parse the objects from low-level pixels, which likely impedes learning.

Original: Sax et al.
Revised: Sax et al.

Original: (2018) echoes our finding that using mid-level vision can greatly improve agent generalization compared to an end-to-end pipeline.
Revised: (2018) echoes our finding that using mid-level vision can greatly improve agent generalization compared to an end-to-end pipeline.

Original: Third, even though Ours and Object Perceiver both use the same object bounding box inputs, the latter is significantly worse in decision making.
Revised: Third, even though Ours and Object Perceiver both use the same object bounding box inputs, the latter is significantly worse in decision making.

Original: We conclude that it is important to pass the variable sequence of objects directly to the robot controller rather than downsampling to a fixed number of tokens.
Revised: We conclude that it is important to pass the variable sequence of objects directly to the robot controller rather than downsampling to a fixed number of tokens.

Original: Prompt Conditioning.
Revised: Prompt Conditioning.

Original: VIMA conditions the robot controller (decoder) on the encoded prompt by cross-attention.
Revised: VIMA conditions the robot controller (decoder) on the encoded prompt by cross-attention.

Original: A simple alternative is to concatenate the prompt P and interaction history H into one big sequence, and then apply a decoder-only transformer like GPT (Radford et al., 2018) to predict actions.
Revised: A simple alternative is to concatenate the prompt P and interaction history H into one big sequence, and then apply a decoder-only transformer like GPT (Radford et al., 2018) to predict actions.

Original: In this ablation, we keep the object tokenizer constant, and only switch the conditioning mechanism to causal sequence modeling.
Revised: In this ablation, we keep the object tokenizer constant, and only switch the conditioning mechanism to causal sequence modeling.

Original: Fig.
Revised: Fig.

Original: 7 shows the comparison of VIMA ( xattn ) and the gpt-decoder variant across 4 generalization levels.
Revised: 7 shows the comparison of VIMA ( xattn ) and the gpt-decoder variant across 4 generalization levels.

Original: While GPT achieves comparable performance in larger models, cross-attention still dominates in the small-capacity range and generalizes better in the most challenging L4 ( Novel Task ) setting.
Revised: While GPT achieves comparable performance in larger models, cross-attention still dominates in the small-capacity range and generalizes better in the most challenging L4 ( Novel Task ) setting.

Original: Our hypothesis is that crossattention helps the controller stay better focused on the prompt instruction at each interaction step.
Revised: Our hypothesis is that cross-attention helps the controller stay better focused on the prompt instruction at each interaction step.

Original: This bears resemblance to the empirical results in Sanh et al.
Revised: This bears resemblance to the empirical results in Sanh et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2022b), which show that well-tuned encoder-decoder architectures can outperform GPT-3 in zero-shot generalization.
Revised: (2022b), which show that well-tuned encoder-decoder architectures can outperform GPT-3 in zero-shot generalization.

Original: Prompt Encoding.
Revised: Prompt Encoding.

Original: We vary the size of the pre-trained T5 encoder (Raffel et al., 2020) to study the effect of prompt encoding.
Revised: We vary the size of the pre-trained T5 encoder (Raffel et al., 2020) to study the effect of prompt encoding.

Original: We experiment with three T5 model capacities: t5-small (30M), t5-base (111M), to t5-large (368M).
Revised: We experiment with three T5 model capacities: t5-small (30M), t5-base (111M), to t5-large (368M).

Original: For all T5 variants, we fine-tune the last two layers and freeze all other layers.
Revised: For all T5 variants, we fine-tune the last two layers and freeze all other layers.

Original: We findthat no significant difference among the variants (Appendix, Sec.
Revised: We find no significant difference among the variants (Appendix, Sec.

Original: E.2), thus we set t5-base as default for all our models.
Revised: E.2), thus we set t5-base as default for all our models.

Revised:  R ELATED W ORK

Revised: Multi-task Learning by Sequence Modeling.

Revised: Transformers have enabled task unification across many AI domains (Raffel et al., 2020;

Original: C ONCLUSION
Revised: Radford et al., 2019; Brown et al., 2020; Chen et al., 2022a;b; Lu et al., 2022; Wang et al., 2022c; Alayrac et al., 2022; Reed et al., 2022). For example, in NLP ,

Revised: T5 (Raffel et al., 2020) unifies all language problems into the same text-to-text format.

Revised: GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and Megatron-Turing NLG (Shoeybi et al., 2019) demonstrate emergent behaviours of intuitive task specifications by zero-shot prompting without any finetuning.

Revised: In computer vision , Florence (Yuan et al., 2021), BiT (Kolesnikov et al., 2020), and MuST (Ghiasi et al., 2021) pre-train a shared backbone model at scale for general visual representations and transfer it across downstream vision tasks.

Revised: Pix2Seq (Chen et al., 2022b) casts many vision problems into a unified sequence format.

Revised: In multimodal learning , Flamingo (Alayrac et al., 2022), Frozen (Tsimpoukelli et al., 2021), and VL-T5 (Cho et al., 2021) design a universal API that ingests an interleaving sequence of images and text and generates free-form text.

Revised: Gato (Reed et al., 2022) and HighMMT (Liang et al., 2022) are massively multi-task models across NLP, vision, and embodied agents.

Revised: Our work is most similar in spirit to Gato, but we focus primarily on enabling an intuitive, multimodal prompting interface for a generalist robot agent.

Revised: Foundation Models for Embodied Agents.

Revised: Foundation models (Bommasani et al., 2021; Brown et al., 2020; Raffel et al., 2020; Ramesh et al., 2022; Wei et al., 2022) have demonstrated strong emergent properties like zero-shot prompting and complex reasoning.

Revised: There are many ongoing efforts to replicate this success for embodied agents, focusing on 3 aspects: 1) Transformer agent architecture : Decision Transformer (Chen et al., 2021; Janner et al., 2021; Zheng et al., 2022; Xu et al., 2022) and

Revised: Gato (Reed et al., 2022) leverage the powerful self-attention models for sequential decision making.

Revised: CLIPort (Shridhar et al., 2021) and Perceiver-Actor (Shridhar et al., 2022) apply large transformers to robot manipulation tasks.

Revised: Less is More (Wang et al., 2021) generates navigation instructions from a mixture of visual landmarks and text.

Revised: Pre-training for better representations : MVP (Xiao et al., 2022), R3M (Nair et al., 2022), and Parisi et al.

Revised: (2022) pre-train general visual representations for robotic perception.

Revised: Li et al.

Revised: ; Reid et al.

Revised: (2022) finetune from LLM checkpoints to accelerate policy learning.

Revised: MineDojo (Fan et al., 2022) and

Revised: Ego4D (Grauman et al., 2021) provide large-scale multimodal databases to facilitate scalable policy training.

Revised: Large language models for robot learning : SayCan (Ahn et al., 2022) leverages the 500B PaLM (Chowdhery et al., 2022) for zero-shot concept grounding.

Revised: Socratic Models (Zeng et al., 2022) composes multiple vision and language foundation models (VLMs) for multimodal reasoning in videos.

Revised: Huang et al.

Revised: (2022a),

Revised: Inner Monologue (Huang et al., 2022b) and LM-Nav (Shah et al., 2022) successfully apply LLMs to long-horizon robot planning.

Revised: VIMA differs from these works in our novel multimodal prompting formulation, which existing LLMs and VLMs do not easily support.

Revised: Robot Manipulation and Benchmarks.

Revised: There are a wide range of robot manipulation tasks that require different skills and task specification formats, such as instruction following (Stepputtis et al.,

Revised: 2020; Shridhar et al., 2021; Lynch & Sermanet, 2021), one-shot imitation (Finn et al., 2017; Dasari & Gupta, 2020; Duan et al., 2017), rearrangement (Batra et al., 2020; Weihs et al., 2021; Szot et al., 2021), constraint satisfaction (Brunke et al., 2021; Srinivasan et al., 2020; Thananjeyan et al., 2021), and reasoning (Shridhar et al., 2020; Gupta et al., 2019; Ahmed et al., 2021; Toyer et al., 2020; Lim et al., 2021).

Revised: Multiple physics simulation benchmarks are introduced to study the above tasks.

Revised: For example, iGibson (Shen et al., 2020; Li et al., 2021; Srivastava et al., 2021) simulates interactive household scenarios.

Revised: Ravens (Zeng et al., 2020) and Robosuite (Zhu et al., 2020; Fan et al., 2021) design various tabletop manipulation tasks with realistic robot arms.

Revised: Our VIMA-B ENCH is the first robot learning benchmark to support multimodal-prompted tasks.

Revised: We also standardize the evaluation protocol to systematically measure an agent’s generalization capabilities.

Revised: A more extended literature review can be found in Appendix, Sec.

Original:  Similar to GPT-3, a generalist robot agent should have an intuitive and expressive interface for human users to convey their intent.
Revised: F.  C ONCLUSION Similar to GPT-3, a generalist robot agent should have an intuitive and expressive interface for human users to convey their intent.

Original: In this work, we introduce a novel multimodal prompting formulation that converts diverse robot manipulation tasks into a uniform sequence modeling problem.
Revised: In this work, we introduce a novel multimodal prompting formulation that converts diverse robot manipulation tasks into a uniform sequence modeling problem.

Original: We propose
Revised: We propose

Original: VIMA, a conceptually simple transformer-based agent capable of solving tasks like visual goal, one-shot video imitation, and novel concept grounding with a single model.
Revised: VIMA, a conceptually simple transformer-based agent capable of solving tasks like visual goal, one-shot video imitation, and novel concept grounding with a single model.

Original: VIMA exhibits superior model and data scaling properties, and provides a strong starting point for future work.
Revised: VIMA exhibits superior model and data scaling properties, and provides a strong starting point for future work.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  A CKNOWLEDGEMENTS

Original: We provide comprehensive details to reproduce our work in the Appendix.
Revised: We are extremely grateful to Shyamal Buch, Jonathan Tremblay, Ajay Mandlekar, Chris Choy, De-An

Original: Concretely, the specifications of each meta-task in the benchmarking suite are explained in Sec.
Revised: Huang, Silvio Savarese, Fei Xia, Josiah Wong, Abhishek Joshi, Soroush Nasiriany, Fangchen Liu, and many other colleagues and friends for their helpful feedback and insightful discussions. We also thank the anonymous reviewers for offering us highly constructive advice and kind encouragement during the review period. NVIDIA provides the necessary computing resource and infrastructure for this project.

Original: B. Model architectures are elaborated in Sec. C. Hyperparameter configurations are listed in Sec.
Revised: This work is done during Yunfan Jiang and Guanzhi Wang’s internships at NVIDIA.

Original: D. Furthermore, we host anonymized code at https://iclr3081.github.io/ for review.
Revised: Guanzhi Wang is supported by the Kortschak fellowship in Computing and Mathematical Sciences at Caltech.

Original: A S IMULATOR D ETAILS
Revised: A S IMULATOR D ETAILS

Original: We build our VIMA-B ENCH simulation suite upon the Ravens physics simulator (Zeng et al., 2020).
Revised: We build our VIMA-B ENCH simulation suite upon the Ravens physics simulator (Zeng et al., 2020;

Original: Our benchmark can be represented as an eight-element tuple (cid:0) S , O , A , Γ , T P , F T , Ω , Ψ (cid:1) , where S is the state space, O is the observation space, A is the action space, Γ ( s t +1 , | s t , a t ) is the transition function.
Revised: Shridhar et al., 2021).

Original: We denote T P as a task distribution with 17 task templates that can be procedurally instantiated from. Each drawn task is paired with a multimodal prompt P .
Revised: Specifically, it is supported by PyBullet (Coumans & Bai, 2016–2021) with a Universal Robot UR5 arm.

Original: F T defines a set of success criteria for all tasks within distribution T P .
Revised: The size of the tabletop workspace is 0 .

Original: Ω and Ψ denote sets of object geometries and textures.
Revised: 5 × 1 m. Our benchmark contains extensible sets of object geometries and textures.

Original: Instantiated from Ω × Ψ , all object instances can be rendered as RGB images appeared in multimodal prompts.
Revised: Instantiated from an object-texture combination, all object instances can be rendered as RGB images appeared in multimodal prompts.

Original: Figure A.1 displays all object geometries.
Revised: Figure A. displays all object geometries.

Original: Figure A.2 displays all textures.
Revised: Figure A.2 displays all textures.

Original: Note that these two sets are easily extensible to include more diverse visual appearances.

Original: The observation spaceO of VIMA-B ENCH includes RGB images from both frontal and topdown views.
Revised: The observation space of VIMA-B ENCH includes RGB images from both frontal and topdown views.

Original: It also includes a one-hot vector ∈ { 0 , 1 } 2 to indicate type of the end-effector ∈ { suction cup, spatula } .
Revised: It also includes a one-hot vector ∈ { 0 , 1 } 2 to indicate type of the end-effector ∈ { suction cup, spatula } .

Original: While a suction cup is employed in most manipulation tasks, a spatula is used in particular for visual constraint tasks, where an agent is asked to “wipe” objects.
Revised: While a suction cup is equipped in most manipulation tasks, a spatula is used in particular for visual constraint tasks, where an agent is asked to “wipe” objects.

Original: VIMA-B ENCH inherits the same action spaceA from Zeng et al.
Revised: VIMA-B ENCH inherits the same action space from Zeng et al.

Original: (2020) and Shridhar et al.
Revised: (2020) and Shridhar et al.

Original: (2021), which consists of primitive actions of “pick and place” for tasks with a suction cup as the end effector, and “push” for tasks with a spatula.
Revised: (2021), which consists of primitive actions of “pick and place” for tasks with a suction cup as the end effector, or “push” for tasks with a spatula.

Original: Both primitive actions contain two coordinates ∈ SE (2) specifying target poses of the end effector.
Revised: Both primitive actions contain two poses ∈ SE (2) specifying target poses of the end effector.

Original: Inherited from Zeng et al.
Revised: For the “pick and place” primitive, they represent the pick pose and the place pose. Fir the “push” primitive, they represent the push starting pose and push ending pose.

Original: (2020) and Shridhar et al. (2021), VIMA-B ENCH provides scripted oracles to generate successful demonstrations for all tasks.
Revised: Similar to prior work (Zeng et al., 2020; Shridhar et al., 2021), VIMA-B ENCH provides scripted oracles to generate successful demonstrations for all tasks.

Original: We leverage them to construct an offline imitation dataset for behavioral cloning.
Revised: We leverage them to construct an offline imitation dataset for behavioral cloning.

Revised: Given a prompt, these pre-programmed bots can access privileged information such as the correct object to pick and target location to place.

Original: B T ASK S UITE
Revised: B T ASK S UITE

Original: We develop6 task categories with 17 task templates paired with multimodal prompts. Note that task categories are not mutually exclusive.
Revised: We develop 17 meta tasks that belong to 6 diverse categories. Thousands of individual tasks and their corresponding multimodal prompts can be procedually generated from these meta-task templates.

Original: See Fig.
Revised: We use PyBullet (Coumans & Bai, 2016–2021) as our backend and the default renderer to produce the

Original: A.1 for an overview of object shapes and Fig. A.2 for all textures that are applied in our task suite.
Revised: RGB frames for training data and interactive test environments.

Original: Demonstrations are interacted in PyBullet (Coumans & Bai, 2016–2021) environment, but we further render them by NVISII (Morrical et al., 2020) for better illustration purpose.
Revised: For demonstration purpose, we apply the NVISII (Morrical et al., 2020) raytracing renderer to enhance the visual quality.

Original: We present task templates in each category in the following subsections.
Revised: We elaborate each meta task in the following subsections.

Original: B. S IMPLE O BJECT M ANIPULATION
Revised: B. S IMPLE O BJECT M ANIPULATION

Original: Tasks for following the basic instructions provided by multimodal prompts with objects specifications.
Revised: This task category asks agents to follow basic instructions specified by multimodal prompts.

Original: Task 01: Pick the specified object(s) and place it into the specified object based on the
Revised: Task 01: Pick the specified object(s) and place it into the specified object.

Original: prompt. • Prompt : Put the { object } 1 into the { object } 

Revised: • Prompt : Put the { object } 1 into the { object }

Original: . • Description : The image placeholder { object } 1 is the object to be picked and the { object } 2 is the object at which to be placed.
Revised: . • Description : The image placeholder { object } 1 is the object to be picked and the { object } 2 is the container object.

Original: The agent requires to recognize the objects with the correct color-shape combinations.
Revised: The agent requires to recognize the objects with the correct color-shape combinations.

Original: To extend the difficulties, it supports more than one object to be picked or placed.
Revised: To extend the difficulties, it supports more than one object to be picked or placed.

Original: For example, the prompt Put the { object } 1 and { object } 2 into the { object } 3 .
Revised: For example, the prompt Put the { object } 1 and { object } 2 into the { object } 3 .

Original: asks to pick two different objects and place into a target base object.
Revised: asks to pick two different objects and place into a target container.

Original: Distractors with different color-shape combo fordragged objects  and base objects are sampled with equal probability as well.
Revised: We uniformly sample different color-shape combos for objects to be picked and containers.

Original: • Success Criteria : All target dragged object(s)  are within the bounds of the base object(s), with specified shapes and textures provided in the prompt.
Revised: • Success Criteria : All specified object(s) to pick are within the bounds of the container object(s), with specified shapes and textures provided in the prompt.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.3 with its multimodal prompt.
Revised: A.3 with its multimodal prompt.

Original: Task 02:
Revised: Task 02:

Original: In the workspace, put the objects with a specified texture shown in the scene image in the prompt into base object(s) with a specified color.
Revised: In the workspace, put the objects with a specified texture shown in the scene image in the prompt into container object(s) with a specified color.

Original: This task requires the agent to find the correct object to manipulate by grounding the color attributes in both natural language  and the visual scene image.
Revised: This task requires the agent to find the correct object to manipulate by grounding the textural attributes from both natural language descriptions and the visual scene images.

Original: • Prompt : Put the { texture } 1 object in { scene } into the { texture }object.
Revised: • Prompt : Put the { texture } 1 object in { scene } into the { texture }object.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: (cid:5)(cid:0)(cid:2)(cid:2)(cid:16)(cid:24)((cid:29)(cid:24)(cid:13)(cid:8)(cid:2) (cid:11)(cid:8)(cid:0)(cid:0)(cid:4)((cid:29)(cid:24)(cid:13)(cid:8)(cid:2) (cid:3)(cid:2)(cid:1)(cid:0)((cid:29)(cid:24)(cid:13)(cid:8)(cid:2) (cid:20)(cid:1)(cid:8)(cid:20)(cid:2)(cid:0)((cid:29)(cid:24)(cid:13)(cid:8)(cid:2) (cid:8)(cid:0)(cid:21)((cid:20)(cid:6)(cid:13)(cid:29)(cid:2)(cid:0)(cid:5) (cid:5)(cid:0)(cid:2)(cid:2)(cid:16)(cid:24)((cid:20)(cid:6)(cid:13)(cid:29)(cid:2)(cid:0)(cid:5) (cid:11)(cid:8)(cid:0)(cid:0)(cid:4)((cid:20)(cid:6)(cid:13)(cid:29)(cid:2)(cid:0)(cid:5) (cid:3)(cid:2)(cid:1)(cid:0)((cid:20)(cid:6)(cid:13)(cid:29)(cid:2)(cid:0)(cid:5) (cid:20)(cid:1)(cid:8)(cid:20)(cid:2)(cid:0)((cid:20)(cid:6)(cid:13)(cid:29)(cid:2)(cid:0)(cid:5)
Revised: (cid:3)(cid:2)(cid:1)(cid:0)(

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Put the into the .
Revised: Put the into the .

Original: • Description : The text placeholder { texture } 1 and { texture } 2 are sampled textures fordragged objects to be picked and the base objects to be placed respectively.
Revised: • Description : The text placeholder { texture } 1 and { texture } 2 are sampled textures for objects to be picked and the container objects, respectively.

Original: The number of dragged objects with the same texture can be varied.
Revised: The number of dragged objects with the same texture can be varied.

Original: { scene } is the workspace-likescene placeholder which is an image containing the objects with the equivalent texture as the dragged texture.
Revised: { scene } is the workspace-like image placeholder.

Original: There is a designated number of distractors with different textures (and potentially different shapes) in the scene.
Revised: There is a designated number of distractors with different textures (and potentially different shapes) in the scene.

Original: For each distractor in the workspace, it has 50% chance to be either dragged or base distractor object with different textures from those specified in the prompt.
Revised: For each distractor in the workspace, it has 50% chance to be either dragged or container distractor object with different textures from those specified in the prompt.

Original: • Success Criteria : All objects in the workspace withthe { texture } 1 are within the bounds of any base object withthe { texture } [-
-]
Revised: • Success Criteria : All objects in the workspace with { texture } 1 are within the bounds of the container object with { texture } [-
-]2 .

Original: . • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.4 with its multimodal prompt.
Revised: A.4 with its multimodal prompt.

Original: Put the green blue stripe in into yellow paisley
Revised: [Figure]

Original: Task 03: Rotate objects clockwise by certain degrees along z -axis.
Revised: Task 03: Rotate objects clockwise by certain degrees along z -axis.

Original: The rotational symmetry is defined for which the object can be rotated to coincide with itself.

Original: Therefore, only the objects without rotational symmetry can be added to this task
Revised: Only rotationally asymmetric objects are considered in this task.

Original: • Prompt :
Revised: • Prompt :

Original: Rotate the { object } 1 { angles } degrees.
Revised: Rotate the { object } 1 { angles } degrees.

Original: • Description : The agent is required to rotate all objects in the workspace same as the dragged object in the image placeholder { object } 1 .
Revised: • Description : The agent is required to rotate all objects in the workspace specified by the image placeholder { object } 1 .

Original: To extend the difficulty for this task, more than one kind of dragged object is supported, such that the prompt would be Rotate the { object } 1 and { object } 2 { angles } degrees.

Original: if there are two dragged objects of interest for instance.

Original: There are also objects with different color-shape combinations in the workspace for distraction.
Revised: There are also objects with different color-shape combinations in the workspace as distractors.

Original: { angles } is the sampled degreeangle that the dragged object needs to be rotated.
Revised: { angles } is the sampled degree that the dragged object needs to be rotated.

Original: For consistency, possible angles for rotation are specified as clockwise, and by default, the target angle is sampled from 30 ◦ , 60 ◦ , 90 ◦ , 120 ◦ , and 150 ◦ .
Revised: A target angle is sampled from 30 ◦ , 60 ◦ , 90 ◦ , 120 ◦ , and 150 ◦ .

Original: • Success Criteria :
Revised: • Success Criteria :

Original: The position of the specifieddragged object matches its original position, and the orientation matches the orientation after rotating specific angles.
Revised: The position of the specified object matches its original position, and the orientation matches the orientation after rotating specific angles.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.5 with its multimodal prompt.
Revised: A.5 with its multimodal prompt.

Original: Rotate the 120 degrees.
Revised: B. V ISUAL G OAL R EACHING

Original: Tasks require the agent to manipulate objects in the workspace to reachthe goal state illustrated in the prompt.
Revised: This task category requires agents to manipulate objects in the workspace to reach goal states represented as images shown in prompts.

Original: Task 04: Rearrange the objectsof interest in the workspace based on the given configuration  in the prompt.
Revised: Task 04: Rearrange target objects in the workspace to match goal configuration shown in prompts

Original: Note that to achieve the goalof rearranged configuration, distractors may need to be moved away first.
Revised: Note that to achieve the goal configuration, distractors may need to be moved away first.

Original: • Prompt : Rearrange to this { scene } .
Revised: • Prompt : Rearrange to this { scene } .

Original: • Description : Objects in the scene placeholder { scene } are target objectsof interest to be manipulated and rearranged.
Revised: • Description : Objects in the scene placeholder { scene } are target objects to be manipulated and rearranged.

Original: In the workspace, the same  objectsof interest are sampled randomly, and distractor objects can also be sampled.
Revised: In the workspace, the same target objects are spawned randomly, potentially with distractors randomly spawned as well.

Original: With a defined distractor conflict rate, the position of each distractor has this probability occupied the position of any  objectof interest in the scene such that the rearrangementonly can be succeeded if moving awayfrom that distractor first.
Revised: With a defined distractor conflict rate, the position of each distractor has this probability to occupy the position of any target object such that the rearrangement can only succeed if moving away that distractor first.

Original: • Success Criteria : All poses of  objectsshown in the prompt scene match the poses of corresponding objects in the scene.
Revised: • Success Criteria : The configuration of target objects in the workspace matches that specified in the prompt.

Original: Manipulating the distractor to any position is allowed (sometimes required) in this task.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.6 with its multimodal prompt. .
Revised: A.6 with its multimodal prompt. .

Revised: Rearrange to this .

Original: [Figure]
Revised: Task

Original: Task 05: Extend the task 04 by requiring the agent to restore  objectsin the scene to the initial setup afterfinishing the phase 1 rearrangement.
Revised: : Extend the task 04 by requiring the agent to restore rearranged objects to the initial setup after the “rearranging” phase.

Original: • Prompt : Rearrange objects to this setup { scene } and then restore.
Revised: • Prompt : Rearrange objects to this setup { scene } and then restore.

Original: • Description :
Revised: • Description :

Original: Same as the task 04 , except introducing the instruction ”restore”.
Revised: Same as the task 04 , except introducing the instruction ”restore”.

Original: Note that moving away the distractor is possible for both the ”rearrange” and ”restore” phase.

Original: • Success Criteria : Meet the success criteria of the task 04 , and then within the allowed max steps, the poses of objects in the workspace that is also shown in the prompt match theirposes at the initial state.
Revised: • Success Criteria : Meet the success criteria of the task 04 , and then within the allowed max steps restore all target objects to their initial configurations.

Original: Note that only conflicted distractors that manipulated in the rearrange phase are allowed to (and have to) manipulate in the restore phase.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.7 with its multimodal prompt.
Revised: A.7 with its multimodal prompt.

Original: [Figure]
Revised: [Figure]

Original: B. NOVEL CONCEPT GROUNDING
Revised: B. NOVEL CONCEPT GROUNDING

Original: In this category, tasks require the agent to groundthe new concepts of adjectives, nouns, or verbs via visual perception and language in the prompt.
Revised: This task category requires agents to ground new concepts of adjectives, nouns, or verbs via visual perception and language understanding.

Original: Prompts consist of two phases: a definition phase followed by the instruction phase.
Revised: Prompts consist of two parts: a definition part followed by an instruction part.

Original: In the definition phase, novel conceptions are defined by multimodal illustrations with multiple supporting examples, and in the instruction phase, the agent is instructed to achieve the goal while applying concepts in the definition phase accordingly.
Revised: In the definition part, novel conceptions are defined by multimodal illustrations with multiple support examples. In the instruction part, agents are asked to achieve the goal by properly applying concepts from the definition part.

Original: The assignment of unique nonsense words is varied and independent for each task instance such that tasks can  be solvedconsistently only if the agent applies the mapping between the words’ conception in the definition phase to the decision-making in the instruction phase.
Revised: The assignment of unique nonsense words is varied and independent for each task instance such that tasks can only be solved if the agent applies the reasoning correctly.

Original: This ability is also referred to as fast-mapping (Heibeck & Markman, 1987)).
Revised: This ability is also referred to as fast-mapping (Heibeck & Markman, 1987).

Original: Task 06: Groundthe comparative adjectives by comparing the size or the lightness of objects and manipulating the correct object
Revised: Task 06: Ground comparative adjectives by comparing the size or the textural saturation of objects and manipulating the correct object(s) instructed in the prompt.

Original: (s ) instructed in the prompt. • Prompt : { demo object } 1 is { novel adj } than { demo object } 2 .
Revised: • Prompt : { demo object } 1 is { novel adj } than { demo object } 2 .

Original: Put the { adv } { novel adj } { object } 1 into the { object } 

Revised: Put the { adv } { novel adj } { object } 1 into the { object }

Original: . • Description :This task supports multiple examples with the format { demo object } i is { novel adj } than { demo object } i + The color-shape combo of objects in different examples is sampled differently. The sampled adjective { novel adj } is a nonsense adjective  for agent grounding.
Revised: . • Description : The sampled adjective { novel adj } is a dummy adjective placeholder for agent to ground.

Original: By default, the novel adjective set is { daxer, blicker, modier, kobar } .
Revised: By default, the novel adjective set is { daxer, blicker, modier, kobar } .

Original: Equivalently, the real compassion is sampled from the size (smaller/larger), height (shorter/taller), or lightness (lighter/darker texture).
Revised: The real meaning can be related to size (smaller/larger) or textural saturation (lighter/darker texture).

Original: The image placeholders { demo object } 1 and { demo object } 2 illustrate how the novel adjective is defined per-episode.
Revised: The image placeholders { demo object } 1 and { demo object } 2 illustrate how the novel adjective is defined.

Original: For example, if the real comparison is ”taller”, then the sampled object in { demo object } 1 is taller than { demo object } 2 .
Revised: For example, if the real comparison is ”taller”, then the sampled object in { demo object } 1 is taller than { demo object } 2 .

Original: The choices of the novel adjective and the real comparison are independently sampledmapping for different episodes.
Revised: The choices of the novel adjective and the real meaning are independently sampled for different task instances.

Original: In the instruction phase, the task is performed similarly tothe task 01 , where the agent is required to pick the specified dragged object (s)
Revised: For the instruction part, this task is similar to task 01 , where the agent is required to pick the specified dragged object(s) with the novel adjective attribute and then place it into the specified container object.

Original: with the novel adjective attribute and then place it into the specified base object. To avoid spoiling which objects to be picked, objects in image placeholders fortarget dragged objects are normal size and
Revised: To avoid revealing the correct object to manipulate, we use a neutral texture for objects appeared in the instruction part.

Original: gray.

Original: With 50% chance the target object to be picked has the flipped adjective attribute as the adjective defined in the definition phase, with extra adv ”less” added in the prompt (e.g. if the target comparison is for ”taller” and the dragged object is the block, then the object to be picked should be shorter block).

Original: The primary distractor would sampled as constructing the comparison, while keeping other attributes the same.

Original: For example, if the target dragged object is a taller block then the first distractor is the shorter block.

Original: Due to the limited size of the workspace we only

Original: use

Original: ”shorter/taller” and ”lighter color/darker color” in experiments • Success Criteria : All targetdragged objects with the specified adjective attribute implicitly illustrated in the prompt is within the bounds of the specified base object.
Revised: • Success Criteria : All target objects with the specified adjective attribute are within the bounds of the specified container object.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.8 with its multimodal prompt.
Revised: A.8 with its multimodal prompt.

Original: is kobar than .

Original: is kobar than . is kobar than . Put the kobar into the . Task 07: Extend the task 01 by requiring to learn mappings between novel nouns and dragged objects and base objects instead.
Revised: Task 07: Orthogonal to task 06 by requiring to learn mappings of novel nouns.

Original: • Prompt : This is a { novel name } 1 { object } 1 .
Revised: • Prompt : This is a { novel name } 1 { object } 1 .

Original: This is a { novel name }{ object } 2 .
Revised: novel name }{ object } 2 .

Original: Put { novel name } 1 into a { novel name } 2 . • Description : The novel noun words are defined with the text placeholders { novel name }and { novel name } 2 , following with its mapped image placeholders { object } 1 and { object } 2 , for the targetdragged object and base object respectively.Multiple dragged objects are supported like the task 01 , with separate novel names and the same definition prompt format.
Revised: • Description : Novel noun words are defined with the text placeholders { novel name }and { novel name } 2 , following their image placeholders { object } 1 and { object } 2 , for the target object and container object, respectively.

Original: The default novel nouns are sampled from { dax, blicket, wug, zup } .
Revised: Novel nouns are sampled from { dax, blicket, wug, zup } .

Original: In the instruction phase, objects are expressed as novel nouns defined in the previous definition phase.
Revised: In the instruction part, objects are expressed as novel nouns defined in the previous definition part.

Original: Distractors are defined the same as task 01 .
Revised: Distractors are defined the same as task 01 .

Revised: is kobar than .

Original:  • Success Criteria : All targetdragged object(s) are within the bounds of the base object(s), with specified novel name and visual illustration mappings in the prompt.
Revised: is kobar than . is kobar than . Put the kobar into the . • Success Criteria : All target object(s) are within the bounds of the container object(s).

Original: The color-shape combos of target objects are the same as the objects in the prompt.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.8 with its multimodal prompt.
Revised: A.8 with its multimodal prompt

Original:  Task08:
Revised: [Figure] Task

Original: Combine with the task 06 and task 07 .

Original: • Prompt : This is a { novel name } 1 { object } 1 .
Revised: • Prompt : This is a { novel name } 1 { object } 1 .

Original: novel { object } 2 .adj
Revised: novel name }{ object } 2 .

Original:  demo object 
Revised: { demo object } 1 is { adj } than { demo object } 2 .

Original:  the { adv } { novel adj } { novel name } 1 into the  novel name 
Revised: Put the { adv } { novel adj } { novel name } 1 into the { novel name } 2 .

Original: 2 . • Description : see task description for task 06 and task 07 .
Revised: • Description : see task description for task 06 and task 07 .

Original: • Success Criteria : Objects to be picked and placed follow the success criteria of the taskwhile others follow criteria of the task 06  .
Revised: • Success Criteria : Similar as tasks 06 and 07 .

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.10 with its multimodal prompt.
Revised: A.10 with its multimodal prompt.

Original: This is a blicket .

Original: This is a zup .

Original: Put a zup into a blicket.

Original: This is a wug .
Revised: This is a wug .

Original: This is a zup .
Revised: This is a zup .

Original: is blicker than . is blicker than .
Revised: is blicker than . is blicker than .

Original: is blicker than .
Revised: is blicker than .

Original: Put the blicker zup into the wug.
Revised: Put the blicker zup into the wug.

Original: Task 09: Ground the verb ”twist”which is defined as rotating a specific angle corresponding with several examples for each task instance.
Revised: Task 09: A novel verb ”twist” is defined as rotating a specific angle conveyed by several examples.

Original: This task is similar tothe task 03 , but it needs to learn what is the exact angle to twist from each task prompt and to ground  verbs that are semantically similar but different in exact definitions.
Revised: This task is similar to task 03 , but it requires the agent to infer what is the exact angle to rotate from the prompt and to ground novel verbs that are semantically similar but different in exact definitions.

Original: • Prompt : "Twist" is defined as rotating object a specific angle.
Revised: • Prompt : "Twist" is defined as rotating object a specific angle.

Original: For examples: From { before twist } i to { after twist } i .
Revised: For examples: From { before twist } i to { after twist } i .

Original: Now twist all { texture } objects.
Revised: Now twist all { texture } objects.

Original: • Description : This task supports multiple examples with the format From { before twist } i to { after twist } i . where i corresponds with the order of examples.

Original:  { before twist } i and { after twist } i areboth scene placeholders where { before twist } i is a workspace-like scene with a randomly sampled objectpose before ”twist” and { after twist } i is the scene showing the  object pose after ”twist”.
Revised: • Description : Both { before twist } i and { after twist } i are scene placeholders where { before twist } i shows a randomly sampled object before ”twist” and { after twist } i shows the same object pose after ”twist”.

Original: All examples illustrate the same sampled angle of the rotation.
Revised: All examples illustrate the same sampled angle of the rotation.

Original: In the workspace, the targetmanipulated objects have thesame texture as { texture }, andshapes are randomly sampled.
Revised: In the workspace, the target objects have the texture specified by { texture } and randomly sampled shapes.

Original: Instead of defining the specific number of the manipulated object as the task 03 , the total number of objects is defined in this task, where the number of targets manipulated objects and distractor objects (at least one) with a different texture(s) from { texture } are randomly sampled.

Original: • Success Criteria :
Revised: • Success Criteria :

Original: Same as the task 03 .
Revised: Same as the task 03 .

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.11 with its multimodal prompt.
Revised: A.11 with its multimodal prompt.

Revised: (cid:2)(cid:23)(cid:24)(cid:9)(cid:19)(cid:20)(cid:2)(cid:18)(cid:9)(cid:19)(cid:18)(cid:11)(cid:15)(cid:13)(cid:9)(cid:12)(cid:15)(cid:11)(cid:18)(cid:8)(cid:19)(cid:18)(cid:7)(cid:22)(cid:20)(cid:8)(cid:20)(cid:9)(cid:12)(cid:4)(cid:18)(cid:22)(cid:25)(cid:14)(cid:15)(cid:21)(cid:20)(cid:18)(cid:8)(cid:18)(cid:19)(cid:5)(cid:15)(cid:21)(cid:9)(cid:13)(cid:9)(cid:21)(cid:18)(cid:8)(cid:12)(cid:4)(cid:3)(cid:15)(cid:10)(cid:18)(cid:17)(cid:22)(cid:7)(cid:18)(cid:15)(cid:1)(cid:8)(cid:6)(cid:5)(cid:3)(cid:15)(cid:19)(cid:0)(cid:16)

Revised: [Table]

Original: B. O NE - SHOT V IDEO I MITATION
Revised: B. O NE - SHOT V IDEO I MITATION

Original: Tasks require the agent to imitate the moves with the strict manipulation sequence in the given one-shot video trajectory. We define video trajectories by the frames in the prompt. [Table]
Revised: This task category requires agents to imitate motions demonstrated through videos shown in prompts.

Original: Task
Revised: Task

Original: 10: Follow motions from the video trajectory as a specified walk along a polygon that approximates a circle around a central object.
Revised: Follow motions for specific objects

Original: • Prompt : Follow this motion for { object } : { frame } 1 ... { frame } i ... { frame } n .
Revised: • Prompt : Follow this motion for { object } : { frame } 1 ... { frame } i ... { frame } n .

Original: • Description : Image placeholder { object } is the target object to be manipulated and {{ frame } i } is set of workspace-like scene placeholders for the imitation video trajectory frames, where n is the trajectory length and.
Revised: • Description : Image placeholder { object } is the target object to be manipulated and {{ frame } i } is set of workspace-like scene placeholders to represent a video trajectory, where n is the trajectory length.

Original: There is an object sampled at the center in both the workspace and trajectory frames but with different textures, which naturally becomes a distractor.
Revised: There is an object spawned at the center in both the workspace and the prompt video but with different textures as a distractor.

Original: In each frame of the video trajectory, the target dragged object is sampled at random vertices of an invisible polygon centered at the distracting object.

Original: Note that the position of the targetdragged object at the initial state is the same as it is inthe { frame } 1 . 
Revised: The initial position of the target object matches that in { frame } 1 . • Success Criteria :

Original: The color-shape combos of the target dragged object in the workspace and frames are the same.

Original: • Success Criteria : For each step, the pose of the targetdragged object matches the poseof it in each corresponding frame in the prompt.
Revised: In each step, the pose of the target object matches the pose in the corresponding video frame.

Original: The incorrect manipulation sequence at a time step is considered a failure.
Revised: Incorrect manipulation sequences are considered as failures.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.12 with its multimodal prompt.
Revised: A.12 with its multimodal prompt.

Original: [Figure] Task 11: Stack objects with the order illustrated in the video trajectory in the prompt.
Revised: Task 11: Stack objects with the order illustrated in the prompt video.

Original: • Prompt : Stack objects in this order { frame } 1 ... { frame } i ... { frame } n .
Revised: • Prompt : Stack objects in this order { frame } 1 ... { frame } i ... { frame } n .

Original: • Description : There are multiple objects with the same shape but different textures sampled in the workspace without any stacking initially.
Revised: • Description : There are multiple objects with the same shape but different textures spawned in the workspace without any stacking initially.

Original: For convenience, they are sampled as a line at the center of the workspace.

Original: Distractor objects with different shapes can be sampled in the workspace but not for the trajectory frames.
Revised: Distractor objects with different shapes are spawned in the workspace but not in the prompt video.

Original: Objects without the distractor(s) in the workspace in the initial state share the same positions in { frame } 1 .

Original: For the rest of the trajectory frames in the prompt, one of the (uppermost) objects is stacked over another object or  at the empty position, in comparison with the previous frame.
Revised: At each step of the prompt video, one of the top objects is stacked over another object or put at an empty position.

Original:  • Success Criteria : For each step, poses of all objects without the distractor(s) in the workspace match the poses of the same objects in each corresponding frame. The incorrect manipulation sequence at a time step is considered a failure.
Revised: Follow this motion for : . • Success Criteria : Similar as task 10.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.13 with its multimodal prompt.
Revised: A.13 with its multimodal prompt.

Original: [Figure]
Revised: [Figure]

Original: B. V ISUAL C ONSTRAINT S ATISFACTION
Revised: B. V ISUAL C ONSTRAINT S ATISFACTION

Original: Tasks for manipulating a specific number of objects in the workspace to achieve the goal  while alsorequired to satisfy the given constraint in the prompt.
Revised: This task category requires agents to wipe a specific number of objects in the workspace to a goal region while also satisfy the given visual constraint.

Original: Task 12:
Revised: Task 12:

Original: Sweep the designated number of objects into the specified bounds of the zone without 
Revised: Sweep the designated number of objects into a specified region without exceeding the

Original: exceeding the constraint area in the workspace.
Revised: boundary

Original: • Prompt : Sweep { quantifier } { object } into { bounds } without exceeding { constraint } .
Revised: • Prompt : Sweep { quantifier } { object } into { bounds } without exceeding { constraint } .

Original: • Description : { object } is the image placeholder of the targetswept object which is sampled with the random number and same texture in the workspace.
Revised: • Description : { object } is the image placeholder of the target object to be swept spawned with a random amount in the workspace.

Original: Distractors with the same number, same shape, and different color from target swept objects are sampled as well.
Revised: Distractors have the same amount, same shape, but different color from target objects.

Original: { quantifier } is the text placeholder to determine the target number ofswept objects to be actually manipulated, sampled from any , one , two , three , and all .
Revised: { quantifier } is the text placeholder to determine the target quantity of objects to be wiped, sampled from any , one , two , three , and all .

Original: { bounds }in experiments is the image placeholder for a three-sided rectangle with
Revised: { bounds } is the image placeholder for a three-sided rectangle as the goal region.

Original: sampled color for blocks being swept in, and { constraint } is theline with sampled color that is a constraint for exceeding by sweeping blocks.
Revised: { constraint } is the constraint line.

Original: • Success Criteria :
Revised: • Success Criteria :

Original: The exact number of targetswept objects that should be manipulated are all inside the specified bounds of the zone.
Revised: The exact number of target objects to be swept are all inside the specified region.

Original: If any distractor is swept, the  target swept objects are exceeding the constraint area, or the number ofobjects in the zone is exceeded the targetnumber of objects to be manipulated, the task is considered as failed.
Revised: Failure reasons include 1) any distractor being wiped into the region, 2) target object exceeding the constraint, or 3) incorrect number of target objects being swept into the goal region.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.14 with its multimodal prompt.
Revised: A.14 with its multimodal prompt.

Original: Task 13:
Revised: Task 13:

Original: Sweep the designated number of objects into the specified bounds of the zone without touching the constraint area in the workspace.
Revised: Sweep the designated number of objects into a specified region without touching the constraint.

Original: • Prompt : Sweep { quantifier 
Revised: • Prompt : Sweep { quantifier } { object } into { bounds } without touching { constraint } .

Original: • Description : Same asthe task 12 but providing not directly accessible goal states with the constraint.
Revised: • Description : Similar as task 12 but requiring different way to satisfy the constraint.

Original: The agent has to learn to avoid ”touching” the constraint line in this case.
Revised: The agent has to learn to avoid ”touching” the constraint line in this case.

Original:  • Success Criteria : Same asthe task 12 except  the constraint is touching in this case.
Revised: Sweep any into [Figure] • Success Criteria : Similar as task 12 except that the constraint is to not touch the red line.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.15 with its multimodal prompt.
Revised: A.15 with its multimodal prompt.

Original: [Figure]
Revised: Sweep two into

Original: [Figure]
Revised: [Figure]

Original: B. V ISUAL R EASONING
Revised: B. V ISUAL R EASONING

Original: Tasks require the agent to make decisions by reasoningfrom the implicitly instructional prompt. Inclusively, tasks may require the agent to memorize over a long horizon as well.
Revised: This task category requires agents to make decisions by reasoning over or memorizing information conveyed through multimodal prompts.

Original: Those tasks may have two phases, where the first phase requires the agent to act with the explicit instruction in the prompt while the second phase requires the agent to act with the instruction with implicit information from the previous or initial state.

Original: The action sequence also has to follow the order of two phases.

Original: Task 14:
Revised: Task 14:

Original: By reasoning the ”sametexture”, the agent is required to pick all objects in the workspace with the same
Revised: By reasoning the ”same

Original: texture as
Revised: texture”

Original:  the target base objects specified in the prompt and place them 
Revised: , the agent is required to pick all objects in the workspace with the same texture as the container objects specified in the prompt and place them into it

Original: into it. • Prompt : Put all objects with the same texture as { object } into it.
Revised: • Prompt : Put all objects with the same texture as { object } into it.

Original: • Description : { object } is the sampled target base object to be placed.
Revised: • Description : { object } is the sampled goal container object.

Original: In the workspace, there is a predefined number of objects withpotentially different shapes and the same texture as the target base objects.
Revised: In the workspace, there are objects with the same texture as the container but potentially different shapes.

Original: For the distraction, several numbers of distractors with different textures are sampled.
Revised: Distractors with different textures are spawned.

Original: • Success Criteria : All objects with the same texture as the target base object are within the bounds of the target base object illustrated in the prompt.
Revised: • Success Criteria : All objects with the same texture as the goal container are within the bounds of the container.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.16 with its multimodal prompt.
Revised: A.16 with its multimodal prompt.

Original: Task15: By reasoning the ”same profile”, the agent is required to pick all objects in the workspace with the same profile as the target base objects specified in the prompt and place them into it.Profile here is defined as the shape attribute of an object.
Revised: Task By reasoning the ”same shape”, the agent is required to pick all objects in the workspace with the same top-down shape as the goal container specified in the prompt and place them into it.

Original: For example, the object block and pallet are both considered as square-like profile.
Revised: For example, blocks and boxes have the same rectangular shape.

Original: • Prompt : Put all objects with the same profile as { object } into it.
Revised: • Prompt : Put all objects with the same profile as { object } into it.

Original: [Figure]
Revised: Put all objects with the same texture as into it.

Original: • Description : Similar to the task 14 except the objects to be picked and placed are with the same profile as the specified base object in the prompt.
Revised: • Description : Similar to the task 14 except the objects to be picked and placed are with the same shape.

Original: Without further elaboration, there are three different profiles in our object set used in experiments for sampling: square-like (e.g. block and pallet), circle-like (e.g. ring and bowl), and undetermined for the rest.
Revised: There are three different shapes: rectangular-like (e.g. block and pallet), circle-like (e.g. ring and bowl), and undetermined for the rest.

Original: • Success Criteria : All objects with the same profile as the target base object are within the bounds of the target base object illustrated in the prompt.
Revised: • Success Criteria : All objects with the same shape as the container are within the container.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.17 with its multimodal prompt.
Revised: A.17 with its multimodal prompt.

Original: [Figure]
Revised: [Figure]

Original: Task  Put the target object into the target base object and then put  its old neighbor from a sampled direction into that base object
Revised: Task 16: Put the target object into the container, and then put one of its old neighbors into the same container.

Original: • Prompt : First put { object } 1 into { object } 2 then put the object that was previously at its { direction } into the same { object } 

Revised: • Prompt : First put { object } 1 into { object } 2 then put the object that was previously at its { direction } into the same { object }

Original: . • Description : Objects in image placeholders { object } 1 and { object } 2 are thefirst dragged target object to be picked and the base object to be placed respectively.
Revised: . • Description : Objects in image placeholders { object } 1 and { object } 2 are the target object to be picked and the container, respectively.

Original: There is an invisible r × h gridworld for objects to be sampled initially. Besides the target dragged object, the second neighbor object is implicitly required to be picked and placed at the possible direction { direction } in terms of the  target (first) dragged object. By default, directions can be sampled from { north, south, west, east } , corresponding with ( − 1 , 0) , (1 , 0) , (0 , − 1) , and (0 , 1) coordinates change in the gridworld respectively. Distractors in this task are defined as the real distractor and the null distractor.
Revised: We then ask the agent to put one of old neighbors of the previous target object into the same container.

Original: The real distractors are sampled with different color-shape combos from the target dragged object, and the null distractors are just the empty space in the gridworld. Note that the total number of distractors is defined as r × h − 2 .As positions of distractors are randomly sampled, positional configurations of each task instance are different for the general purpose.
Revised: The old neighboring object is specified through cardinal directions { north, south, west, east } .

Original: • Success Criteria :
Revised: • Success Criteria :

Original: The targetdragged objectis within the bounds of the base object, and the  object previously at the specified direction of the target dragged object is also within the bounds of the base object. The moving sequence is constrained by manipulating the target dragged object to the base object first and then the second neighbor object.
Revised: The target object and the correct neighboring object are inside the container.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.18 with its multimodal prompt.
Revised: A.18 with its multimodal prompt.

Revised: [Figure]

Revised: into

Original: [Figure]
Revised: [Figure]

Original:  Task 17: Pick and place the target dragged objects illustrated in the prompt  to thetarget base objects with specified colors accordingly then put it into its initial base object.
Revised: put the object that was previously at its west into the same . Task 17: Pick and place the target object specified in the prompt into different containers in order then restore to the initial container.

Original:  Prompt : Put { object } 1 into { object } 2 .
Revised: • Prompt : Put { object } 1 into { object } 2 .

Original: Finallyrestore it into its original container
Revised: Finally it into its original container.

Original: • Description : The object in the image placeholder { object } 1 is the targetdragged object to be picked and placed across the task.
Revised: • Description : The object in the image placeholder { object } 1 is the target object to be manipulated across the task.

Original: In the workspace, there is also an invisible gridworld that is placed with sampled base objects with the same shape but different textures for all grid points.

Original: There can be more than one target base objects to be placed (e.g.
Revised: There are more than one target containers (e

Original:  Put { object } 1 into { object } 2 then { object } 3 . Finally restore it into its original container.
Revised: g. Put { object } 1 into { object } 2 then { object } 3 .Finally restore it into its original container.

Original: for two target base objects to be placed in order).
Revised: for two target base objects to be placed in order).

Original: The rest of sampled base objects are naturally becoming distractors.
Revised: The rest of spawned containers naturally becomes distractors.

Original: • Success Criteria :
Revised: • Success Criteria :

Original: The targetdragged object is within the bounds of each target base object first.
Revised: The target object are first put into multiple containers following the specific order.

Original: If there are more than one target base objects, the manipulation sequence is constrained by the order in the prompt. After meeting the before criteria, the target dragged object is within the bounds of the first base object that contains it in the initial state.
Revised: Finally it should be restored into its original container.

Original: • Oracle Trajectory : Shown in Fig.
Revised: • Oracle Trajectory : Shown in Fig.

Original: A.19 with its multimodal prompt.
Revised: A.19 with its multimodal prompt.

Original: [Figure]
Revised: [Figure]

Original: C M ODEL A RCHITECTURE
Revised: C M ODEL A RCHITECTURE

Original: In this section, we provide comprehensive details about VIMA model architecture as well as other adapted baseline methods.
Revised: In this section, we provide comprehensive details about VIMA model architecture as well as other adapted baseline methods.

Original: We implement all models in PyTorch (Paszke et al., 2019) and adapt
Revised: We implement all models in PyTorch (Paszke et al., 2019) and adapt

Original: Transformer-related implementation from Wolf et al.
Revised: Transformer-related implementation from Wolf et al.

Original: C. VIMA A RCHITECTURE C.1. M ULTIMODAL P ROMPT T OKENIZATIONS
Revised: C. VIMA A RCHITECTURE C.1. M ULTIMODAL P ROMPT T OKENIZATIONS

Original: As introduced in Section 5, there are 3 types of input formats in multimodal prompts, namely (1) text inputs , (2) images of full scenes , and (3) images of single objects .
Revised: As introduced in Section 4, there are 3 types of input formats in multimodal prompts, namely (1) text inputs , (2) images of full scenes , and (3) images of single objects .

Original: For text inputs , we follow the standard pipeline in NLP to first tokenize raw languages to discrete indices through pre-trained t5-base tokenizer.
Revised: For text inputs , we follow the standard pipeline in NLP to first tokenize raw languages to discrete indices through pre-trained t5-base tokenizer.

Original: We then obtain corresponding word tokens from the[Table] embedding look-up of the pre-trained t5-base model.
Revised: We then obtain corresponding word tokens from the embedding look-up of the pre-trained t5-base model.

Original: For images of full scenes , we first parse the scene through a fine-tuned mask R-CNN detection model  to extract individual object.
Revised: For images of full scenes , we first parse the scene through a fine-tuned mask R-CNN detection model (He et al., 2017; Wu et al., 2019) to extract individual objects.

Original: Each object representation contains a bounding box and a cropped image.
Revised: Each object representation contains a bounding box and a cropped image.

Original: The bounding box is in the format of [ x center , y center , height , width ] .
Revised: The bounding box is in the format of [ x center , y center , height , width ] .

Original: We normalize it to be within [0 , 1] by dividing each dimension with corresponding upper-bound value.
Revised: We normalize it to be within [0 , 1] by dividing each dimension with corresponding upper-bound value.

Original: We then pass it through a bounding box encoder MLP and obtain a feature vector.
Revised: We then pass it through a bounding box encoder MLP and obtain a feature vector.

Original: To process the cropped image, we first pad non-square image to a square by padding along the shorter dimension.
Revised: To process the cropped image, we first pad non-square image to a square by padding along the shorter dimension.

Original: We then resize it to a configured size and pass it through a ViT (trained from scratch) to obtain the image feature.
Revised: We then resize it to a pre-configured size and pass it through a ViT (trained from scratch) to obtain the image feature.

Original: Finally, an object token is obtained by concatenating the bounding box feature and the image feature and mapping to the embedding dimension.
Revised: Finally, an object token is obtained by concatenating the bounding box feature and the image feature and mapping to the embedding dimension.

Revised: For images of single objects , we obtain tokens in the same way except with a dummy bounding box.

Original: Detailed model hyperparameters about tokenizations are listed in Table 1.
Revised: Detailed model hyperparameters about tokenizations are listed in Table 1.

Original: After obtaining a sequence of prompt tokens [Table] we follow Tsimpoukelli et al.
Revised: After obtaining a sequence of prompt tokens, we follow Tsimpoukelli et al.

Original: (2021) to pass it through a pre-trained t5-base encoder to obtain encoded prompt.
Revised: (2021) to pass it through a pre-trained t5-base encoder to obtain encoded prompt.

Original: Note that we add adapter NLP between object tokens and the T5 encoder.
Revised: Note that we add adapter MLP between object tokens and the T5 encoder.

Original: We adopt learned absolute positional embedding.
Revised: We adopt learned absolute positional embedding.

Original: Model hyperparameters are listed in Table 1 as well. 
Revised: Model hyperparameters are listed in Table 1 as well. [Table]

Original: C.1. O BSERVATION E NCODING
Revised: C.1. O BSERVATION E NCODING

Original: Since all RGB observations are images of full scenes, we follow the same procedure discussed above to obtain flattened object tokens.
Revised: Since all RGB observations are images of full scenes [Table] we follow the same procedure discussed above to obtain flattened object tokens.

Original: Because we provide RGBs from two views (frontal and top-down), we order object tokens by following the order of [ frontal, top-down ] .
Revised: Because we provide RGBs from two views (frontal and top-down), we order object tokens by following the order of [ frontal, top-down ] .

Original: We one-hot encode the state of the end effector.
Revised: We one-hot encode the state of the end effector.

Original: We then concatenate object tokens with the end-effector state and transform to observation tokens.
Revised: We then concatenate object tokens with the end-effector state and transform to observation tokens.

Original: We adopt learned absolute positional embedding.
Revised: We adopt learned absolute positional embedding.

Original: Detailed model hyperparameters about observation encoding is provided in Table2.
Revised: Detailed model hyperparameters about observation encoding is provided in Table

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C.1. A CTION E NCODING
Revised: C.1. A CTION E NCODING

Original: Since our model is conditioned on observation-action interleaved trajectory history, we also tokenize past actions.
Revised: Since our model is conditioned on observation-action interleaved history [Table] we also tokenize past actions.

Original: We follow common practice in Chen et al.
Revised: We follow common practice in Chen et al.

Original: ; Zheng et al.
Revised: ; Zheng et al.

Original: (2022) to encode past actions with a two-layer MLP.
Revised: (2022) to encode past actions with a two-layer MLP.

Original: It has a hidden dimension of 256.
Revised: It has a hidden dimension of 256.

Original: We then map outputs to token dimension and obtain action tokens.
Revised: We then map outputs to token dimension and obtain action tokens

Original:  C.1. S EQUENCE M ODELING
Revised: . C.1. S EQUENCE M ODELING

Original: The robot controller in VIMA is a causal decoder that autoregressively predicts actions.
Revised: The robot controller in VIMA is a causal decoder that autoregressively predicts actions.

Original: To condition the decoder on prompt tokens, we perform cross-attention betweentrajectory history tokens and prompt tokens (Figure 3).
Revised: To condition the decoder on prompt tokens, we perform cross-attention between history tokens and prompt tokens (Figure 3).

Original: Concretely, we passtrajectory history tokens as the query sequence and prompt tokens as the key-value sequence into cross-attention blocks.
Revised: Concretely, we pass history tokens as the query sequence and prompt tokens as the key-value sequence into cross-attention blocks.

Original: The output prompt-aware trajectory tokens then go through causal self-attention blocks.
Revised: The output prompt-aware trajectory tokens then go through causal self-attention blocks.

Original: We alternate cross-attention and self-attention L times.
Revised: We alternate cross-attention and self-attention L times.

Original: This procedure is technically described in Pseudocode 1.
Revised: This procedure is technically described in Pseudocode 1.

Original: 1 def xattn_sequence_modeling( prompt_tokens, # the [L, d] prompt tokens (L=prompt length) obs_tokens, # the [T, d] obs tokens (T=time step) act_tokens, # the [T-1, d] action tokens traj_pos_embd, # learned positional embedding for trajectory prompt_pos_embd, # learned positional embedding for prompt7 ): # interleave obs and action tokens
Revised: def xattn_sequence_modeling(prompt_tokens, #the [L, d] prompt tokens (L=prompt length) obs_tokens, # the [T, d] obs tokens (T=time step) act_tokens, # the [T-1, d] action tokens traj_pos_embd, # learned positional embedding for trajectory prompt_pos_embd, # learned positional embedding for prompt ): # interleave obs and action tokens

Original: traj_tokens = interleave(obs_tokens, act_tokens) #
Revised: traj_tokens = interleave(obs_tokens, act_tokens) #

Original: [2T-1, d] # add positional embedding to trajectory tokens x
Revised: [2T-1, d] # add positional embedding to trajectory tokens x

Original: = traj_tokens + traj_pos_embd # add positional embedding to prompt tokens
Revised: = traj_tokens + traj_pos_embd # add positional embedding to prompt tokens

Original: prompt_tokens =
Revised: prompt_tokens =

Original: prompt_tokens + prompt_pos_embd 14# apply xattn and causal self-attn for i in range(num_layers): token
Revised: prompt_tokens + prompt_pos_embd # apply xattn and causal self-attn for i in range(num_layers): # cross-attention x = x + attn_i(q=x, kv=

Original:  return 
Revised: prompt_tokens) # feed forward x = x + ffw_xattn_i(x) # self-attention x = x + causal_attn_i(q=x, kv=x) # feed forward x = x + ffw_i(x) # the last token is the predicted action token predicted_act_token = x[-1] return predicted_act_token

Original: Pseudocode 1: Cross-attention operation that conditions thetrajectory history on prompt.
Revised: Pseudocode 1: Cross-attention operation that conditions the history on prompt.

Original: We repetitively alternate cross-attention and self-attention to model the trajectory given a specific task.
Revised: We repetitively alternate cross-attention and self-attention to model the trajectory given a specific task.

Original: C.1. A CTION D ECODING
Revised: C.1. A CTION D ECODING

Original: After obtaining the predicted action token, we map it to the action space A and obtain the predicted action.
Revised: After obtaining the predicted action token, we map it to the action space A and obtain the predicted action.

Original: This is achieved though a group of action heads.
Revised: This is achieved though a group of action heads.

Original: Since the action space consists of two
Revised: Since the action space consists of two

Original: SE (2) poses, for each pose we use six independent heads to decode discrete actions (two for xy coordinate and four for rotation represented in quaternion).
Revised: SE (2) poses, for each pose we use six independent heads to decode discrete actions (two for xy coordinate and four for rotation represented in quaternion).

Original: These discrete actions are then dediscretized and mapped to continuous actions through affine transformation.
Revised: These discrete actions are then dediscretized and mapped to continuous actions through affine transformation.

Original: The two poses are modeled independently.
Revised: The two poses are modeled independently.

Original: Early ablations show that this independent modeling has no difference with, for example, auto-regressive modeling.
Revised: Early ablations show that this independent modeling is equally good as alternatives techniques like autoregressive decoding (Vinyals et al., 2019; OpenAI et al., 2019).

Original: Detailed model hyperparameters are listed in Table 3.[Table]
Revised: Detailed model hyperparameters are listed in Table 3.

Original: [Table]
Revised: [Table]

Original: C. B ASELINES A RCHITECTURES
Revised: C. B ASELINES A RCHITECTURES

Original: In this section, we elaborate model architectures for baseline methods.
Revised: In this section, we elaborate model architectures for baseline methods.

Original: Some components such as the action decoder are same across all baseline methods and ours.
Revised: Some components such as the action decoder are same across all baseline methods and ours.

Original: Therefore, we only discuss unique model components.
Revised: Therefore, we only discuss unique model components.

Revised: [Table]

Original: C.2. G ATO
Revised: C.2. G ATO

Original: Gato (Reed et al., 2022) introduces a decoder-only model that solves tasks from multiple domains where tasks are specified by prompting the model with the observation and action subsequence.
Revised: Gato (Reed et al., 2022) introduces a decoder-only model that solves tasks from multiple domains where tasks are specified by prompting the model with the observation and action subsequence.

Original: There are two differences between Gato and our method.
Revised: There are two differences between Gato and our method.

Original: (1) Different visual tokenizers .
Revised: (1) Different visual tokenizers .

Original: In Gato, observation tokens are represented as sequence of image patches instead of object tokens as in our method.
Revised: In Gato, observation tokens are represented as sequence of image patches instead of object tokens as in our method.

Original: We deviate from the original Gato implementation by replacing the ResNet with a ViT to encode image patches.
Revised: In other words, Gato is not object-centric.

Original: (2) Different prompt conditioning .
Revised: (2) Different prompt conditioning .

Original: In Gato, prompt tokens are directly prepended totrajectory history tokens.
Revised: In Gato, prompt tokens are directly prepended to history tokens.

Original: This sequence of temporally concatenated tokens are then passed through a decoder-only model.
Revised: This sequence of temporally concatenated tokens are then passed through a decoder-only model.

Revised: In contrast, VIMA adopts cross-attention to condition the robot controller on the prompt (Fig. 3).

Original: Hyperparameters of Gato’s ViT is listed in Table 4.
Revised: Hyperparameters of Gato’s ViT is listed in Table 4.

Original: The decoder-style sequence modling is technically illustrated in Pseudocode 2.
Revised: The decoder-style sequence modling is technically illustrated in Pseudocode 2.

Original: 1 def causal_sequence_modeling( prompt_tokens, # the [L, d] prompt tokens (L=prompt length) sep_token, # the [1, d] learned  tokento separate prompt and trajectory history obs_tokens, # the [T, d] obs tokens (T=time step) act_tokens, # the [T-1, d] action tokens pos_embd, # learned positional embedding7 ): # interleave obs and action tokens
Revised: def causal_sequence_modeling(prompt_tokens, #the[L, d] prompt tokens (L=prompt length) sep_token, # the [1, d] learned separator token obs_tokens, # the [T, d] obs tokens (T=time step) act_tokens, # the [T-1, d] action tokens pos_embd, # learned positional embedding ): # interleave obs and action tokens

Original: traj_tokens = interleave(obs_tokens, act_tokens) #
Revised: traj_tokens = interleave(obs_tokens, act_tokens) #

Original: [2T-1, d] # assemble input tokens x = concat([prompt_tokens, sep_token, traj_tokens])
Revised: [2T-1, d] # assemble input tokens x = concat([prompt_tokens, sep_token, traj_tokens])

Original: x
Revised: x

Original: = x + pos_embd 13# apply GPT layers with causal mask for i in range(num_layers): # self-attention x = x + causal_attn_i(q=x, kv=x) # feed forward x = x + ffw_i(x) 20# the last token is the predicted action token predicted_act_token = x[-1] return predicted_act_token
Revised: = x + pos_embd # apply GPT layers with causal mask for i in range(num_layers): # [Equation] # feed [Equation]

Revised: [Table]

Original:  C.2. F LAMINGO
Revised: return [Table] Pseudocode 2: Plain sequence modeling that temporally concatenates prompt and trajectory history and repetitively perform causal self-attention operation. C.2. F LAMINGO

Original: Flamingo (Alayrac et al., 2022) is a vision-language model that learns to generate textual completion in response to multimodal prompts.
Revised: Flamingo (Alayrac et al., 2022) is a vision-language model that learns to generate textual completion in response to multimodal prompts.

Original: It embeds a variable number of prompt images into a fixed number of tokens via the Perceiver Resampler module (Jaegle et al., 2021b), and conditions the language decoder on the encoded prompt by cross-attention.
Revised: It embeds a variable number of prompt images into a fixed number of tokens via the Perceiver Resampler module (Jaegle et al., 2021b), and conditions the language decoder on the encoded prompt by cross-attention.

Original: Therefore, the only difference between Flamingo and our method lies at the visual tokenizer.
Revised: Because the original Flamingo does not support embodied agents altogether, we have made major changes to adapt the architecture to robot learning.

Original: Concretely, in Flamingo we first use a ViT (hyperparameters  in Table 4) to obtain an encoded sequence of image patches.
Revised: First, we swap the LLM component of Flamingo with our robot controller, which is a transformer decoder that outputs robot arm actions at every environment step (hyperparameters listed in Table 3).

Original: We then apply a Perceiver Resampler to down-sample these image patches to a fixed number of tokens.
Revised: Second, VIMA uses an object-centric tokenizer instead of Perceiver Resampler that operates directly on raw RGB.

Original: We refer interested readers to Section 3.1.1 in Alayrac et al.
Revised: We refer interested readers to Section 3.1 in Alayrac et al.

Original: for  more  technical details.
Revised: for a more detailed technical discussion on Flamingo’s architecture design.

Original: Model hyperparameters for  the Perceiver Resampler is listed in Table 5.
Revised: Model hyperparameters for our reimplementation of the Perceiver Resampler is listed in Table 5.

Original: Because Flamingo does not work with embodied agents out of the box.

Original: We adapt it to support decision-masking by replacing the original word prediction head with robot action heads with same hyperparameters listed in Table 3.

Original: [Table]
Revised: [Table]

Original: C.2. D ECISION T RANSFORMER
Revised: C.2. D ECISION T RANSFORMER

Original: Decision Transformer (DT) (Chen et al., 2021; Janner et al., 2021) is among the first works to reinterpret the RL problem as transformer sequence modeling.
Revised: Decision Transformer (DT) (Chen et al., 2021; Janner et al., 2021) is among the first works to reinterpret the RL problem as transformer sequence modeling.

Original: In visual RL domains like Atari games,
Revised: In visual RL domains like Atari games,

Original: DT is prompted on the desired reward value and outputs actions autoregressively given the RGB observation embedding.
Revised: DT is prompted on the desired reward value and outputs actions autoregressively given the RGB observation embedding.

Original: We replace DT’s initial reward prompt with our multimodal task prompt embeddings, and remove all subsequent reward tokens.
Revised: We replace DT’s initial reward prompt with our multimodal task prompt embeddings, and remove all subsequent reward tokens.

Original: For visual tokenizer, we employ a learned
Revised: For visual tokenizer, we employ a learned

Original: ViT with hyperparameters listed in Table 4 to map an observed image to a single “state” token.
Revised: ViT with hyperparameters listed in Table 4 to map an observed image to a single “state” token.

Original: C. M ASK R-CNN D ETECTION M ODEL
Revised: C. M ASK R-CNN D ETECTION M ODEL

Original: Finally, we elaborate the mask R-CNN model (He et al., 2017) for scene parsing and object extraction.
Revised: Finally, we elaborate the mask R-CNN model (He et al., 2017) for scene parsing and object extraction.

Original: We fine-tuned a pre-trained lightweight mask R-CNN  from Wu et al.
Revised: We fine-tuned a pre-trained lightweight mask R-CNN ( mask rcnn R 50 FPN 3x ) from Wu et al.

Original: (2019) to adapt to scenes and images in our tabletop environment.
Revised: (2019) to adapt to scenes and images in our tabletop environment.

Original: A visualization of its output is provided in Figure A.20.
Revised: A visualization of its output is provided in Figure A.20.

Original: We do not use the predicted object names in our models.
Revised: We do not use the predicted object names in our models.

Original: D VIMA T RAINING D ETAILS
Revised: D VIMA T RAINING D ETAILS

Original: We follow the best practices to train Transformer models using the AdamW optimizer (Loshchilov & Hutter, 2019), learning rate warm-up, cosine annealing (Loshchilov & Hutter, 2017), etc.
Revised: We follow the best practices to train Transformer models using the AdamW optimizer (Loshchilov & Hutter, 2019), learning rate warm-up, cosine annealing (Loshchilov & Hutter, 2017), etc.

Original: Training hyperparameters are provided in Table 6.
Revised: Training hyperparameters are provided in Table 6.

Original: We use GEGLU activation (Shazeer, 2020) within
Revised: We use GEGLU activation (Shazeer, 2020) inside

Original: Transformer models across all methods.
Revised: Transformer models across all methods.

Original: [Table] To make trained models robust to detection inaccuracies and failures, we apply object augmentation by randomly injecting false-positive detection outputs.
Revised: To make trained models robust to detection inaccuracies and failures, we apply object augmentation by randomly injecting false-positive detection outputs.

Original: Concretely, for observation at each time step, we sample number of augmented objects i.i.d.
Revised: Concretely, for observation at each time step, we sample number of augmented objects i.i.d.

Original: n augmented objects ∼ Cat ( K, p ) , where Cat ( · ) denotes a multi-categorical distribution with K supports parameterized by p .
Revised: n augmented objects ∼ Cat ( K, p ) , where Cat ( · ) denotes a multi-categorical distribution with K supports parameterized by p .

Original: For each augmented object, we then randomly sample a bounding box and corresponding cropped image to add to object tokens.
Revised: For each augmented object, we then randomly sample a bounding box and corresponding cropped image to add to object tokens.

Original: In our experiments, we set p = { 0 : 0 .
Revised: In our experiments, we set p = { 0 : 0 .

Original: 95 , 1 : 0 . 05 } with K = 2 .
Revised: 95 , 1 : 0 . 05 } with K = 2 .

Revised: [Table]

Original: D. V ARY M ODEL C APACITY
Revised: D. V ARY M ODEL C APACITY

Original: We train a spectrum of 7 models ranging from 2M to 200M parameters.
Revised: We train a spectrum of 7 models ranging from 2M to 200M parameters.

Original: To vary the model capacity, we follow prior work (Chowdhery et al., 2022) to change embedding dimension and number of layers.
Revised: To vary the model capacity, we follow prior work (Chowdhery et al., 2022) to change embedding dimension and number of layers.

Original: We list configurations for methods with cross-attention prompt conditioning (i.e., ours and Flamingo) in Table7, and configurations for methods only with causal self-attention in Table 8.
Revised: We list configurations for methods with cross-attention prompt conditioning (i [Table] e., ours and Flamingo) in Table

Original: E M ORE E XPERIMENT R ESULTS E. B REAKDOWN R ESULTS

Original: We show breakdown results for Figure 4 in Tables 9, 10, 11, and 12, respectively. [Table] [Table]
Revised: [Table] and configurations for methods only with causal self-attention (i.e., Gato and DT) in

Original: [Table]
Revised: Table 8.

Original: [Table]
Revised: [Table]

Original: DT

Original: [Table]
Revised: [Table]

Original: Flamingo
Revised: We show breakdown results for Figure 4 in Tables 9, 10,

Original: [Table]
Revised: [Table]

Revised: and 12, respectively.

Original: E. V ARY T5 E NCODER S IZES
Revised: E. V ARY T5 E NCODER S IZES

Original: We vary the size of the pre-trained T5 encoder (Raffel et al., 2020) to study the effect of prompt encoding.
Revised: We vary the size of the pre-trained T5 encoder (Raffel et al., 2020) to study the effect of prompt encoding.

Original: We experiment with three T5 model capacities: t5-small (30M), t5-base (111M), to t5-large (368M).
Revised: We experiment with three T5 model capacities: t5-small (30M), t5-base (111M), to t5-large (368M).

Original: For all T5 variants, we fine-tune the last two layers and freeze all other layers.
Revised: For all T5 variants, we fine-tune the last two layers and freeze all other layers.

Original: We fix the parameter count of the decision-making part to be 200M. As shown in Table 13, we findthat no significant difference among the variants.
Revised: We fix the parameter count of the decision-making part to be 200M. As shown in Table 13, we find no significant difference among the variants.

Original: Thus we set the standard t5-base as default for all our models. 
Revised: Thus we set the standard t5-base as default for all our models. [Table] [Table] [Table]

Revised: 1 Flamingo 41.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: DT 48.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 20M

Revised: Ours 98 .100 .

Revised: 5 Gato 46.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Flamingo

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: DT 50.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 200M

Revised: Ours 99 .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: 5 Gato 51.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: F E XTENDED R ELATED W ORK
Revised: Flamingo 49. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: DT 52.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: F E XTENDED R ELATED W ORK

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide an extended review of related work as complementary to Section2.
Revised: we provide an extended review of related work as complementary to Section

Revised: [Table]

Revised: [Table]

Original: Multi-task Learning by Sequence Modeling.
Revised: Multi-task Learning by Sequence Modeling.

Original: In NLP domain, T5 (Raffel et al., 2020) and

Original: GPT (Radford et al., 2019; Brown et al., 2020) are among the earliest architectures to use a single model for generation, reasoning, translation, etc.

Original: The Natural Language Decathlon (McCann et al., 2018) adopts a consistent question-answering format for a suite of10 NLP tasks.
Revised: In NLP domain, the Natural Language Decathlon (McCann et al., 2018) adopts a consistent question-answering format for a suite of NLP tasks.

Original: In computer vision, Mask R-CNN (He et al., 2017), UberNet (Kokkinos, 2016),  12-in-1 (Lu et al., 2020) leverages a single backbone model with multiple independent heads for different tasks.
Revised: In computer vision, Mask R-CNN (He et al., 2017), UberNet (Kokkinos, 2016), and 12-in-1 (Lu et al., 2020) leverage a single backbone model with multiple independent heads for different tasks.

Original: Pix2Seq (Chen et al., 2022a;b) casts object detection, instance segmentation, and keypoint detection into a common sequence setting.

Original: UVim (Kolesnikov et al., 2022) is another unified approach for vision that uses a language model(LM) to generate the guiding code for a second model to predict raw vision outputs.
Revised: UVim (Kolesnikov et al., 2022) is another unified approach for vision that uses a language model to generate the guiding code for a second model to predict raw vision outputs.

Original: In multimodal learning, Unified-IO (Lu et al., 2022), BeiT (Wang et al., 2022c),
Revised: In multimodal learning, numerous works (Lu et al., 2022; Wang et al., 2022a; Zellers et al., 2021; 2022;

Original: Flamingo (Alayrac et al., 2022), and Gato (Reed et al., 2022) are able to unify a wide range of vision, language, and policy learning problems. OFA (Wang et al., 2022a) leverages a sequence-to-sequence learning framework with a unified vocabulary for all the image and text tokens.
Revised: Buch et al., 2022; Fu et al., 2021; Yang et al., 2022) investigate the unification of image, video, audio, and/or language modalities to deliver multi-purpose foundation models, though most of which are not equipped with decision-making facilities.

Original: Perceivers (Jaegle et al., 2021b;a) propose an efficient architecture to handle general-purpose inputs and outputs.
Revised: Perceivers (Jaegle et al., 2021b;a) propose an efficient architecture to handle general-purpose inputs and outputs.

Original: BEiT-3 (Wang et al., 2022c) performs masked data modeling on images, texts and image-text pairs to pre-train the backbone then transfer to various downstream tasks.
Revised: BEiT-3 (Wang et al., 2022c) performs masked data modeling on images, texts and image-text pairs to pre-train a backbone for various downstream tasks.

Revised: MetaMorph (Gupta et al., 2022) learns a universal controller over a modular robot design space.

Original: Foundation Models for Embodied Agents.
Revised: Foundation Models for Embodied Agents.

Original: Embodied agent research (Duan et al., 2022; Batra et al., 2020; Ravichandar et al., 2020; Collins et al., 2021) isalso adopting the large-scale pre-training paradigm. From the aspect of pre-training for better representations , LaTTe (Bucker et al., 2022),
Revised: Embodied agent research (Duan et al., 2022; Batra et al., 2020; Ravichandar et al., 2020; Collins et al., 2021) is adopting the large-scale pre-training paradigm, powered by a collection of learning environments (Abramson et al., 2020; Shridhar et al., 2020; Savva et al., 2019; Puig et al., 2018; Team et al., 2021; Toyama et al., 2021; Shi et al., 2017).

Original: CLIPort (Shridhar et al., 2021), and Embodied-CLIP (Khandelwal et al., 2021) leverage the frozen visual and textual representations of CLIP (Radford et al., 2021) for robotic manipulation.When using pre-trained models for better policies , Li et al. and Reid et al. (2022) use pre-trained
Revised: From the aspect of pre-training for better representations , LaTTe (Bucker et al., 2022) and Embodied-CLIP (Khandelwal et al., 2021) leverage the frozen visual and textual representations of CLIP (Radford et al., 2021) for robotic manipulation.

Original: LMs as policy backbone and outperform baselines without pre-trained models.

Original: Robot Manipulation and Benchmarks.
Revised: Robot Manipulation and Benchmarks.

Original: There are more prior works that are not mentioned in the main paper that study different robotic manipulation tasks, such asinstruction following (Stepputtis et al., 2020; Shridhar et al., 2021; Lynch & Sermanet, 2021), constraint satisfaction (Bharadhwaj et al., 2021), one-shot imitation (Paine et al., 2018; Huang et al., 2019; Aceituno et al., 2021;
Revised: There are many prior works that are not mentioned in the main paper that study different robotic manipulation tasks, such as constraint satisfaction (Bharadhwaj et al., 2021), one-shot imitation (Paine et al., 2018; Huang et al., 2019; Aceituno et al., 2021;

Original: Zhao et al., 2022), and rearrangement (Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021;
Revised: Zhao et al., 2022), and rearrangement (Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021;

Original: Stengel-Eskin et al., 2022).
Revised: Stengel-Eskin et al., 2022).

Original: Multiple simulation benchmarks are introduced to study the above tasks: 1) Indoor simulation environments : Habitat (Savva et al., 2019; Szot et al., 2021) is equipped with a high-performance 3D simulator for fast rendering and proposes a suite of common tasks for assistive robots.
Revised: Multiple simulation benchmarks are introduced to study the above tasks: 1) Indoor simulation environments : Habitat (Savva et al., 2019; Szot et al., 2021) is equipped with a high-performance 3D simulator for fast rendering and proposes a suite of common tasks for assistive robots.

Original: AI2-THOR (Ehsani et al., 2021; Deitke et al., 2022) is a framework that supports visual object manipulation and procedural generation of environments.
Revised: AI2-THOR (Ehsani et al., 2021; Deitke et al., 2022) is a framework that supports visual object manipulation and procedural generation of environments.

Original: Tabletop environments :
Revised: Tabletop environments :

Original: Meta-World (Yu et al., 2019) and RLBench (James et al.,  2019) are widely used simulator benchmarks studying robotics manipulation with tabletop settings.
Revised: Meta-World (Yu et al., 2019), RLBench (James et al., 2019), and SURREAL (Fan et al., 2018; 2019) are widely used simulator benchmarks studying robotics manipulation with tabletop settings.

Original: CausalWorld
Revised: CausalWorld

Original: (Ahmed et al., 2021) is a benchmark for causal structure and transfer learning in manipulation, requiring long-horizon planning and precise low-level motor control.
Revised: (Ahmed et al., 2021) is a benchmark for causal structure and transfer learning in manipulation, requiring long-horizon planning and precise low-level motor control.

Original: However, these simulators and benchmarksstill lack task unification and do not  support task prompts with multiple modalities.
Revised: However, these simulators and benchmarks do not natively support task specification and prompting with multiple modalities.