I NSTRUCTION -F OLLOWING A GENTS WITH J OINTLY P RE -T RAINED V ISION -L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans are excellent at understanding language and vision to accomplish a wide range of tasks.
In contrast, creating general instruction-following embodied agents remains a difficult challenge.
Prior work that uses pure language-only models lack visual grounding, making it difficult to connect language instructions with visual observations.
On the other hand, methods that use pre-trained vision-language models typically come with divided language and visual representations, requiring designing specialized network architecture to fuse them together.
We propose a simple yet effective model for robots to solve instruction-following tasks in visionbased environments.
Our Instruct RL method consists of a multimodal transformer that encodes visual observations and language instructions, and a policy transformer that predicts actions based on encoded representations.
The multimodal transformer is pre-trained on millions of image-text pairs and natural language text, thereby producing generic cross-modal representations of observations and instructions.
The policy transformer keeps track of the full history of observations and actions, and predicts actions autoregressively.
We show that this unified transformer model outperforms all state-of-the-art pre-trained or trained-from-scratch methods in both single-task and multi-task settings.
Our model also shows better model scalability and generalization ability than prior work.
I NTRODUCTION
Humans are able to understand language and vision to accomplish a wide range of tasks.
Many tasks require language understanding and vision perception, from driving to whiteboard discussion and cooking.
Humans can also generalize to new tasks by building upon knowledge acquired from previously-seen tasks.
Meanwhile, creating generic instruction-following agents that can generalize to multiple tasks and environments is one of the central challenges of reinforcement learning (RL) and robotics.
Driven by significant advances in learning generic pre-trained models for language understanding (Devlin et al., 2018; Brown et al., 2020; Chowdhery et al., 2022), recent work has made great progress towards building instruction-following agents (Lynch & Sermanet, 2020; Mandlekar et al., 2021;
Ahn et al., 2022; Jang et al., 2022; Guhur et al., 2022; Shridhar et al., 2022b).
For example,
SayCan (Ahn et al., 2022) exploits PaLM models (Chowdhery et al., 2022) to generate language descriptions of step-by-step plans from language instructions, then executes the plans by mapping the steps to predefined macro actions.
HiveFormer (Guhur et al., 2022) uses a pre-trained language encoder to generalize to multiple manipulation tasks.
However, a remaining challenge is that pure language-only pre-trained models are disconnected from visual representations, making it difficult to differentiate vision-related semantics such as colors.
Therefore, visual semantics have to be further learned to connect language instructions and visual inputs.
Another category of methods use pre-trained vision-language models, which have shown great success in joint visual and language understanding (Radford et al., 2021).
This has made tremendous progress towards creating a general RL agent (Zeng et al., 2022; Khandelwal et al., 2022; Nair et al., 2022b;
Khandelwal et al., 2022; Shridhar et al., 2022a).
For example, CLIPort (Shridhar et al., 2022a) uses CLIP (Radford et al., 2021) vision encoder and language encoder to solve manipulation tasks.
However, a drawback is that they come with limited language understanding compared to pure language-only pre-trained models like BERT (Devlin et al., 2018), lacking the ability to follow long and detailed instructions.
In addition, the representations of visual input and textual input are often disjointly learned, so such methods typically require designing specialized network architectures on top of the pre-trained models to fuse them together.
To address the above challenges, we introduce Instruct RL, a simple yet effective method based on the multimodal transformer (Vaswani et al., 2017; Tsai et al., 2019).
It first encodes fine-grained cross-modal alignment between vision and language using a pre-trained multimodal encoder (Geng et al., 2022), which is a large transformer (Vaswani et al., 2017; He et al., 2022) jointly trained on image-text (Changpinyo et al., 2021; Thomee et al., 2016) and text-only data (Devlin et al., 2018).
The generic representations of each camera and instructions form a sequence, and are concatenated with the embeddings of proprioception data and actions.
These tokens are fed into a multimodal policy transformer, which jointly models dependencies between the current and past observations, and cross-modal alignment between instruction and views from multiple cameras.
Based on the output representations from our multimodal transformer, we predict 7-DoF actions, i.e. , position, rotation, and state of the gripper.
We evaluate Instruct RL on RLBench (James et al., 2020), measuring capabilities for single-task learning, multi-task learning, multi-variation generalization, long instructions following, and model scalability.
On all 74 tasks which belong to 9 categories (see Figure 1 for example tasks), our
Instruct RL significantly outperforms state-of-the-art models (Shridhar et al., 2022a; Guhur et al., 2022; Liu et al., 2022), demonstrating the effectiveness of joint vision-language pre-trained representations.
Moreover, Instruct RL not only excels in following basic language instructions, but is also able to benefit from human-written long and detailed language instructions.
We also demonstrate that Instruct RL generalizes to new instructions that represent different variations of the task that are unseen during training, and shows excellent model scalability with performance continuing to increase with larger model size.
R ELATED W ORK
Language-conditioned RL with pre-trained language models.
Pre-trained language models have been shown to improve the generalization capabilities of language-conditioned agents to new instructions and to new low-level tasks (Lynch & Sermanet, 2020; Hill et al., 2020; Nair et al., 2022a;
Jang et al., 2022; Ahn et al., 2022; Huang et al., 2022).
Some prior work use prompt engineering with large language models (Brown et al., 2020; Chowdhery et al., 2022) to extract temporally extended plans over predefined skills (Huang et al., 2022; Ahn et al., 2022; Jiang et al., 2022), similar to work that decomposes high-level actions into sub-goals (Team et al., 2021). These work rely purely on language models to drive agents and require converting observations into language through predefined APIs.
Others combine pre-trained language representations with visual inputs (Jang et al.,
[Figure]
2022; Lynch & Sermanet, 2020; Team et al., 2021; Khandelwal et al., 2022) using specialized architectures such as UNet (Ronneberger et al., 2015) and FiLM (Perez et al., 2018).
Such approaches have demonstrated success in solving challenging robotic manipulation benchmarks (Guhur et al., 2022; Shridhar et al., 2022b).
In this work, we argue that using jointly pre-trained vision-language representations with RL can achieve superior performance in solving complex language-specified tasks, and show that our proposed approach enjoys better scalability and simpler architecture.
Our work is also complementary to prompt-based methods, e . g .,
SayCan (Ahn et al., 2022).
This work focuses on improving the mapping from language instructions to robot actions, and we expect that combining our approach with prompt-based methods can achieve greater success.
Language-conditioned RL with pre-trained vision-language models.
There have been strong interests in leveraging pre-trained vision-language models for language-conditioned RL (Shridhar et al., 2022a; Zeng et al., 2022; Khandelwal et al., 2022), motivated by the effectiveness of visionlanguage models such as CLIP (Radford et al., 2021).
However, these methods use disentangled pipelines for visual and language input, with the language primarily being used to guide perception.
Our work uses jointly pre-trained vision-language models that comes with better grounding text-tovisual content (Geng et al., 2022).
The effectiveness of such jointly pre-trained models enables a simple final model, which is a jointly pre-trained vision-language transformer followed by a policy transformer.
While using pretrained vision-language models has been explored in grounded navigation (Guhur et al., 2021;
Hao et al., 2020; Majumdar et al., 2020; Shah et al., 2022), our work focuses on manipulation tasks which have combinatorial complexity (composition of objects/actions).
Moreover, in contrast to these methods, our method Instruct RL is a simple architecture that scales well to large-scale tasks and can directly merge long complex language instructions with visual input.
RL with Transformers.
Transformers (Vaswani et al., 2017) have led to significant gains in natural language processing (Devlin et al., 2018;
Brown et al., 2020), computer vision (Dosovitskiy et al., 2020;
He et al., 2022) and related fields (Lu et al., 2019; Radford et al., 2021;
Geng et al., 2022).
They have also been used in the context of supervised reinforcement learning (Chen et al., 2021a;
Reed et al., 2022), vision-language navigation (Chen et al., 2021b; Shah et al., 2022), robot learning and behavior cloning from noisy demonstrations (Shafiullah et al., 2022; Cui et al., 2022), and language-conditioned RL (Guhur et al., 2022; Shridhar et al., 2022a).
Inspired by their success, we leverage the transformer architecture to extract pre-trained representations from language and vision and learn a language-conditioned policy.
P ROBLEM D EFINITION
We consider the problem of robotic manipulation from visual observations and natural language instructions.
We assume the agent receives a natural language instruction x := { x 1 , . . .
, x n } con- [Figure] sisting of n text tokens.
At each timestep t , the agent receives a visual observation o t ∈ O and takes an action a t ∈ A in order to solve the task specified by the instruction.
We parameterize the policy π (cid:0) a t | x , { o i } ti =1 , { a i } t − 1 i =1 (cid:1) as a transformer model, which is conditioned on the instruction x , observations { o i } ti =1 , and previous actions { a i } t − 1 i =1 .
For robotic control, we use macro steps (James & Davison, 2022), which are key turning points in the action trajectory where the gripper changes its state (open/close) or the joint velocities are set to near zero.
Following James & Davison (2022), we employ an inverse-kinematics based controller to find a trajectory between macro-steps.
In this way, the sequence length of an episode is significantly reduced from hundreds of small steps to typically less than 10 macro steps.
Observation space : Each observation o t consists of images { c kt } Kk =1 taken from K different camera viewpoints, as well as proprioception data o P t ∈ R 4 .
Each image c kt is an RGB image of size 256 × 256 × 3 .
We use K = 3 camera viewpoints located on the robot’s wrist, left shoulder, and right shoulder.
The proprioception data o P t consists of 4 scalar values: gripper open, left finger joint position, right finger joint position, and timestep of the action sequence.
Note that we do not use point cloud data in order for our method to be more flexibly applied to other domains.
Since RLBench consists of sparse-reward and challenging tasks, using point cloud data can benefit performance (James & Davison, 2022; Guhur et al., 2022), but we leave this as future work.
Action space : Following the standard setup in RLBench (James & Davison, 2022), each action a t :=
( p t , q t , g t ) consists of the desired gripper position p t =
( x t , y t , z t ) in Cartesian coordinates and quaternion q t = ( q 0 t , q 1 t , q 2 t , q 3 t ) relative to the base frame, and the gripper state g t indicating whether the gripper is open or closed.
An object is grasped when it is located in between the gripper’s two fingers and the gripper is closing its grasp.
The execution of an action is achieved by a motion planner in RLBench.
I NSTRUCT RL
We propose a unified architecture for robotic tasks called Instruct RL, which is shown in Figure 3.
It consists of two modules: a pre-trained multimodal masked autoencoder (He et al., 2022;
Geng et al., 2022) to encode instructions and visual observations, and a transformer-based (Vaswani et al., 2017) policy that predicts actions.
First, the feature encoding module (Sec. 4.1) generates token embeddings for language instructions { x j } nj =1 , observations { o i } ti =1 , and previous actions { a i } t − 1 i =1 .
Then, given the token embeddings, the multimodal transformer policy (Sec. 4.2) learns relationships between the instruction, image observations, and the history of past observations and actions, in order to predict the next action a t .
4. F EATURE E NCODING WITH P RE - TRAINED T RANSFORMER
We encode the instruction and visual observations using a pre-trained vision-language encoder, as shown in Figure 3.
Specifically, we use a pre-trained multimodal masked autoencoder (M3AE) (Geng et al., 2022) encoder, which is a large transformer-based architecture based on ViT (Dosovitskiy et al., 2020) and BERT (Devlin et al., 2018). Specifically M3AE
(Geng et al., 2022) is a transformer-based architecture that learns a unified encoder for both vision and language data masked token prediction on a large-scale image- dataset(
- pinyo et al.,
corpus (Devlin et al., 2018) and is able to generalizable representations that transfer well to downstream tasks.
Encoding Instructions and Observations.
Following the practice of M3AE, we first tokenize the language instructions { x j } nj =1 into embedding vectors and then apply 1D positional encodings.
We denote the language instructions as E x ∈ R n × d , where n is the length of language tokens and d is the embedding dimension.
We divide each image observation in { c kt } Kk =1 into image patches, and use a linear projection to convert them to image embeddings that have the same dimension as the language embeddings.
Then, we apply 2D positional encodings.
Each image is represented as E c ∈ R l c × d e where l c is the length of image patches tokens and d e is the embedding dimension.
The image embeddings and text embeddings are then concatenated along the sequence dimension:
E = concat ( E c ,
E x ) ∈ R ( l c + n ) × d e .
The combined language and image embeddings are then processed by a series of transformer blocks to obtain the final representation ˆ o kt ∈ R ( l c + n ) × d e .
Following the practice of VIT and M3AE, we also apply average pooling on the sequence length dimension of ˆ o kt to get o kt ∈ R d e as the final representation of the k -th camera image c kt and the instruction.
We use multi-scale features h kt ∈ R d which are a concatenation of all intermediate layer representations, where the feature dimension d
= L ∗ d e equals the number of intermediate layers L times embedding dimension d e .
Finally, we can get the representations over all K camera viewpoints h t = { h 1 t , · · · , h Kt } ∈ R K × d as the representation of the vision-language input.
Encoding Proprioceptions and Actions.
The proprioception data o P t ∈ R 4 is encoded with a linear layer to upsample the input dimension to d ( i .
e ., each scalar in o P t is mapped to R d ) to get a representation z t = { z 1 t , · · · , z 4 t } ∈ R 4 ×
d all each state in o P t .
Similarly, the action is projected to feature space f t ∈ R d .
4. T RANSFORMER P OLICY
We consider a context-conditional policy, which takes all encoded instructions, observations and actions as input, i.e., { ( h i , z i ) } ti =1 and { f i } t − 1 i =1 .
By default, we use context length 4 throughout the paper ( i .
e
., 4( K + 5) embeddings are processed by the transformer policy).
This enables learning relationships among views from multiple cameras, the current observations and instructions, and between the current observations and history for action prediction.
The architecture of transformer policy is illustrated in Figure 4.
We pass the output embeddings of the transformer into a feature map to predict the next action a t =
[ p t ; q t ; g t ] .
We use behavioral cloning to train the models.
In RLBench, we generate D , a collection of N successful demonstrations for each task.
Each demonstration δ ∈ D is composed of a sequence of (maximum) T macro-steps with observations { o i } Ti =1 , actions { a ∗ i } Ti =1 and instructions { x l } n l =1 .
We minimize a loss function L over a batch of demonstrations B = { δ j } | B | j =1 ⊂ D .
The loss function is the mean-square error (MSE) on the gripper’s action:
[Equation]
E XPERIMENTAL S ETUP
To evaluate the effectiveness of our method, we run experiments on RLBench (James et al., 2020), a benchmark of robotic manipulation task (see Figure 1).
We use the same setup as in Guhur et al.
(2022), including the same set of 74 tasks with 100 demonstrations per task for training, and the same set of instructions for training and inference, unless stated otherwise.
We group the 74 tasks into 9 categories according to their key challenges (see Appendix A.7 for each category’s description and list of tasks).
For each task, there are a number of possible variations , such as the shapes, colors, and ordering of objects; the initial object positions; and the goal of the task.
These variations are randomized at the start of each episode, during both training and evaluation.
Based on the task and variation, the environment generates a natural language task instruction using a language template (see Appendix A.3).
We compare Instruct RL with strong baseline methods from three categories:
• RL with pre-trained language model : HiveFormer (Guhur et al., 2022) is a state-of-the-art method for instruction-following agents that uses a multimodal transformer to encode multicamera views, point clouds, proprioception data, and language representations from a pre-trained CLIP language encoder (Radford et al., 2021).
We report the published results of HiveFormer unless otherwise mentioned.
• RL with pre-trained vision-language model : CLIP-RL is inspired by CLIPort (Shridhar et al., 2022a), which demonstrates the effectiveness of CLIP for robotic manipulation.
CLIP-RL uses concatenated visual- and language-representations from a pre-trained CLIP model.
Similar to Instruct RL, CLIP-RL uses multi-scale features by concatenating intermediate layer representations, and is trained using the same hyperparameters.
• RL trained from scratch :
Auto- λ (Liu et al., 2022) is a model trained from scratch that uses the UNet network (Ronneberger et al., 2015) and applies late fusion to predictions from multiple views.
We report the published results of Auto- λ unless mentioned otherwise. Instruct RL uses the
official pre-trained multimodal
masked
autoencoder
(M3AE) model Geng et al. 2022), which was jointly pre-trained on a combination of image-text datasets (Changpinyo et al.,
Thomee et al., 2016) and text-only corpus (Devlin et al., 2018).
CLIP-RL uses the official pre-trained CLIP models.
See Appendix A.2 for more details about the pre-training datasets.
All models are trained for 100K iterations.
For evaluation, we measure the per-task success rate for 500 episodes per task.
Further implementation and training details can be found in Appendix A.1.
E XPERIMENTAL R ESULTS
We evaluate the methods on single-task performance (Figure 5), multi-task performance (Figures 6, 7), generalization to unseen instructions and variations (Figure 8a), and scalability to larger model size (Figure 8b).
In all metrics, we find that Instruct RL outperforms state-of-the-art baselines despite being a simpler method.
Single-task performance.
Figure 5 shows that, across all 9 categories of tasks, Instruct RL performs on par or better than all state-of-the-art baselines.
On average, Instruct RL significantly outperforms prior work despite being much simpler.
Multi-task performance.
In the multi-task setting, each model is trained on a set of tasks and then evaluated on each of the training tasks.
In Figure 6, we compare multi-task performance on
RLBench tasks considered in HiveFormer (Guhur et al., 2022) and Auto- λ (Liu et al., 2022), with [Figure] [Figure] 100 training demonstrations per task.
Instruct RL exhibits strong multi-task performance compared to other methods.
In particular, even though both Instruct RL and HiveFormer use a transformer policy with language instructions, Instruct RL outperforms HiveFormer by a large margin.
In Figure 7, we further evaluate multi-task performance on all 74 RLBench tasks, and see that Instruct RL outperforms CLIP-RL in most categories.
These results demonstrate the transferability of jointly pre-trained vision-language models to diverse multi-task settings.
Generalization to unseen instructions and variations.
In Figure 8a, we report the performance on the Push Buttons task, which requires the agent to push colored buttons in a specified sequential order given in the instruction.
In this task, instructions are necessary to solve the unseen task variations correctly ( e .
g ., pushing the buttons in a blue-red-green vs. red-green-blue order cannot be inferred from only looking at the scene).
We evaluate the models on instructions that are both seen and unseen during training; unseen instructions can contain new colors, or an unseen ordering of colors.
We see that Instruct RL achieves higher performance on both seen and unseen instructions, even in the most challenging case where only 10 demonstrations are available per variation.
Scalability to larger model size.
One of the key benefits of pre-trained models is that we can use a huge amount of pre-training data that is typically not affordable in RL and robot learning scenarios.
In Figure 8b, we evaluate different model sizes on multi-task performance across 14 selected tasks (listed in Appendix A.5).
For fair comparison with training-from-scratch, we fix the size of the policy transformer, and only vary the size of the transformer encoder.
We compare four model sizes: B/32, B/16, L/16, and H/16, where “B” denotes ViT-base; “L” denotes ViT-large; “H” denotes
ViT-huge; and “16” and “32” denote patch sizes 16 and 32, respectively.
Both CLIP-RL and Instruct RL improve with larger model size, but Instruct RL shows better scalability.
On the other hand, learning-from-scratch is unable to benefit from larger model capacity; in fact, a strong weight decay was needed to prevent this baseline from overfitting to the limited data. [Figure] [Figure]
A BLATION S TUDY
Instructions are more effective with joint vision-language model.
Table 1 shows models trained with and without instructions, on 10 RLBench tasks from Guhur et al.
; Liu et al.
While using instructions improves the performance of all methods, they are most effective with
Instruct RL.
This is not surprising, as the CLIP language encoder in HiveFormer was not pre-trained on large text corpora.
One can pair CLIP with a large language model such as BERT, but doing so can often hurt performance (Guhur et al., 2022) due to the language representations not being aligned with the visual representations.
On the other hand, Instruct RL shows the strongest performance due to using the M3AE encoder, which was jointly pre-trained on both image-text and text-only data.
Detailed instructions require language understanding.
In Table 2, we also evaluate the methods on longer and more detailed instructions, which include step-by-step instructions specific to the task (see Table 3 for examples).
The detailed instructions are automatically generated using a template that contains more details than the default instruction template from RLBench (“Default”).
Each “Tuned Short Instruction” has maximum token length 77 (to be compatible with CLIP’s language encoder), while each “Tuned Long Instruction” can have up to 256 tokens (to be compatible with
Instruct RL).
We compare Instruct RL with CLIP-RL, which uses concatenated visual- and languagerepresentations from CLIP, as well as a variant of CLIP-RL called BERT-RL, which uses a BERT language encoder and a trained-from-scratch vision encoder.
Generally across all methods, we can see that performance increases with longer and more detailed instructions, which implies that the pre-trained language encoders can extract task-relevant information from language instructions.
Instruct RL achieves the best performance, especially with longer instructions ( e .
g .
, Instruct RL achieves 63.7 vs. 46.1 from BERT-RL, on the Push Buttons task with Tuned Long Instructions).
Fusion strategy.
How to fuse representations from multimodal data is one of the key design choices in Instruct RL.
In Figure 9a, we compare variants of Instruct RL with other fusion strategies: Concat [Table] [Table] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] runs the encoder twice to obtain vision and language representation separately, then concatenates them together.
FiLM (Perez et al., 2018) fuses vision and language features layer-wisely before concatenating them together.
Default refers to Instruct RL where language and vision are encoded once to obtain joint representations.
We see that using a joint representation of language and vision inputs is critical for performance.
History context encoding.
The flexibility of the transformer architecture allows us to encode multiple past steps.
To investigate the benefit of utilizing historical information, we ablate the history context length.
Figure 9b shows that there is a steady increase in performance when we increase the context length.
However, using a longer context requires more compute and memory ( e .
g ., increasing the context length from 4 to 8 adds about 20% longer training time on one task).
Thus, we choose a context length of 4 for computational efficiency.
Multi-scale features.
In Figure 9c, we compare the performance of Instruct RL with features selected from different layers.
The results show that using a combination of intermediate representations is most effective.
C ONCLUSION
In this paper we propose Instruct RL, a novel transformer-based instruction following method.
Instruct RL has a simple and scalable architecture which consists of a pre-trained multimodal transformer and a policy transformer.
Experimental results show that Instruct RL significantly outperforms the state-of-the-art pre-trained and train-from-scratch instruction following methods.
As Instruct RL achieves state-of-the-art results and scales well with model capacity, applying our approach to a larger scale of problems would be an interesting future work
A A PPENDIX A. I MPLEMENTATION AND COMPUTE RESOURCES
We use the AdamW (Kingma & Ba, 2015; Loshchilov & Hutter, 2017) optimizer with a learning rate of 5 ×
10 − 4 and weight decay 5 e − 5 .
All models are trained on TPU v3-128 using cloud TPU.
Since
TPU does not support headless rendering with PyRep 3 simulators that RLBench is built upon, we evaluate the models on NVIDIA Tesla V100 SXM2 GPU using headless rendering.
Each training batch per device consists of 32 demonstrations sequence with length 4, in total the batch size is 512.
All models are trained for 100K iterations.
We apply data augmentation in training, including jitter over the RGB images c kt .
Our implementation of Instruct RL is built upon the official JAX implementation of multimodal masked autoencoder (M3AE)
(Geng et al., 2022) 4 ,
and we use the official pre-trained M3AE models that were pre-trained on CC12M (Changpinyo et al., 2021) and text corpus (Devlin et al., 2018).
Our implementation of CLIP-RL is built upon a JAX CLIP implementation 5 , and we use the official pre-trained CLIP models 6 for the visual- and language-representations.
Both CLIP-RL and Instruct RL use ViT-B/16 in all experiments unless otherwise specified.
As we demonstrated in experiment section, while the larger models ViT-L/16 and ViT-H/16 can further boost Instruct RL’s results, we use ViT-B/16 to reduce computation cost and to have apple to apple comparison with baselines.
Our code is available at at https://sites.google.com/view/instructrl/ .
A. P RE - TRAINING DATASETS A.2. M3AE
M3AE is trained on a combination of image-text datasets and text-only datasets.
The image-text data includes the publicly available Conceptual Caption 12M (CC12M) (Changpinyo et al., 2021),
Redcaps (Desai et al., 2021), and a 15M subset of the YFCC100M (Thomee et al., 2016) selected according to (Radford et al., 2021).
The text-only data includes the publicly available Wikipedia and the Toronto BookCorpus (Zhu et al., 2015).
For language data, we use the BERT tokenizer from Huggingface 7 to tokenize the text.
Following Geng et al.
(2022), we use 0.5 for text loss weight and 0.75 for text mask ratio.
We use 0.1 for text-only loss.
A.2. CLIP
CLIP is trained on the publicly available YFCC100M (Thomee et al., 2016) which consists of 100M image-text pairs.
A. RLB ENCH I NSTRUCTIONS
We use the task instructions provided in RLBench.
For each task, there are a number of possible variations , such as the shapes and colors of objects, the initial object positions, and the goal of the task.
Based on the task and variation, the environment generates a natural language task instruction.
For example, in the task “Put groceries in cupboard” , the generated instruction is of the form “ pick up the OBJECT and place it in the cupboard ” where OBJECT is one of 9 possible grocery items.
A full list of RLBench task instructions can be found at https://github.com/stepjam/
RLBench/tree/master/rlbench/tasks .
3 https://github com/stepjam/PyRephttps://github.com/young-geng/m3ae_publichttps://github.com/google-research/scenic/tree/main/scenic/projects/ baselines/clip 6 https://github.com/openai/CLIPhttps://huggingface.co/docs/transformers/main_classes/tokenizer [Table]
A. E VALUATION DETAILS
In multi-task setting, models are trained on multiple tasks simultaneously and subsequently evaluated on each task independently.
In the generalization experiment setting, models are trained on a subset of all variations, then evaluated on unseen task variations
. A. M ULTI - TASK MULTI - VARIATION TASKS DETAILS
For experiments including model scaling (Figure 8b) and ablation studies (Figure 9), we selected 14 difficult tasks that have multiple variations:
Stack wine , Open drawer , Meat off grill , Put item in drawer , Turn tap , Put groceries in cupboard , Sort shape , Screw bulb in , Close jar , Stack blocks , Put item in safe , Insert peg , Stack cups , and Place cups .
Each task comes with multiple variations that include objects colors, objects shapes, and the ordering of objects to be interacted with.
To make a fair comparison with baselines, we use the slightly modified version of RLBench from HiveFormer (Guhur et al., 2022).
The changes include adding new tasks and improving the motion planner.
The full details of each task can be found in RLBench Github repo 8 and the HiveFormer Github repo 9 .
A. S AMPLE EFFICIENCY ABLATION
In the experiments, we used 100 demonstrations following the same setup as in prior work, and Instruct RL outperforms prior state-of-the-arts significantly. However, getting 100 demonstrations can be expensive and difficult in many real world tasks.
We hypothesize that Instruct RL can achieve better sample-efficient learning thanks to the joint language-vision encoder pretrained on large-scale passive datasets.
To study this, we randomly choose a subset of 8 tasks and evaluate the performance using onlydemonstrations.
The results from Table 4 show that Instruct RL outperforms baselines when using 100 demonstrations or 10 demonstrations.
Moreover, using only 10 demonstrations, Instruct RL achieves competitive or higher success rates than baselines that use 100 demonstrations.
For example, on the stack blocks task, Instruct RL using 10 demos achieves 32% success rate while
HiveFormer gets 31% using 100 demos.
Similarly, on the open drawer task, Instruct RL using 10 demos gets 81% while HiveFormer gets 86% using 100 demos.
The results show that Instruct RL is more sample efficient than prior state-of-the-arts.
A. T ASK CATEGORIES DETAILS
To compare with baselines including Guhur et al.
(2022), our experiments are conducted on the same 74 out of 106 tasks from RLBench 8 .
The 74 tasks can be categorized into 9 task groups according to their key challenges as shown in Table 5. Task diversity.
The 74 tasks RLBench cover a wide range of challenges that are essential for robot learning, including planning over multiple sub goals ( e .
g ., picking a basket ball and then throwing the ball) and domains where there are multiple possible trajectories to solve a task due to a large affordance area of the target object ( e . g ., the edge of a cup). RLBench employs 3 keys terms: Task, Variation, and Episode. For each task there are one or more variations, and from each variation, an infinite number of episodes can be drawn.
Each variation of a task comes with a list of textual descriptions that verbally summarise this variation of the task.
An example showing the distinction between task and variation is Figure 1 and Figure 4 of James et al. Task difficulty. RLBench is a manipulation benchmark that is significantly more difficult than locomotion benchmarks. Some tasks involve precise object manipulation such as ‘put knife on chopping board‘ and ‘take usb out of computer‘, some tasks require preceise grasping such as ‘close laptop lid‘ and ‘open and close drawer‘, some tasks require solving long horizon tasks that involve many composed sets of actions, for example, the ‘empty dishwasher’ task involves opening the washer door, sliding out the tray, grasping a plate, and then lifting the plate out of the tray.
In this work we consider tasks that can be grouped into 9 categories, as shown in Table 5, to comprehensively evaluate the effectiveness of Instruct RL.
In addition to manipulation and perception challenges, RLBench comes with a suite of diverse task instructions that require natural language understanding.
RLBench has a large and diverse vocabulary, it has over 100 content words vocabulary size ( e . g ., table, cup, open, grasp, box, etc) with function words removed, combing with the diverse visual input and complex interactions leads to a suite of challenging instruction following manipulation tasks. B I NSTRUCTION TEMPLATES AND GENERATIONS Default instructions generation.
The default instructions templates are shown in Table 6.
Long instructions generation. The tuned instructions that contain step-by-step and detailed descriptions of objects are shown in Table 7. Group Challenges
Tasks
Planning multiple sub-goals ( e . g ., picking a basket ball and then throwing the ball) basketball in hoop, put rubbish in bin, meat off grill, meat on grill, change channel, tv on, tower3, push buttons, stack wine
Tools a robot must grasp an object to interact with the target object slide block to target, reach and drag, take frame off hanger, water plants, hang frame on hanger, scoop with spatula, place hanger on rack, move hanger, sweep to dustpan, take plate off colored dish rack, screw nail
Long term requires more than 10 macro-steps to be completed wipe desk, stack blocks, take shoes out of box, slide cabinet open and place cups
Rotationinvariant can be solved without changes in the gripper rotation reach target, push button, lamp on, lamp off, push buttons, pick and lift, take lid off saucepan
Motionplanner requires precise grasping toilet seat down, close laptop lid, open box, open drawer, close drawer, close box, phone on base, toilet seat up, put books on bookshelf
Multimodal can have multiple possible trajectories to solve a task due to a large affordance area of the target object ( e .
g ., the edge of a cup) pick up cup, turn tap, lift numbered block, beat the buzz, stack cups
Precision involves precise object manipulation take usb out of computer, play jenga, insert onto square peg, take umbrella out of umbrella stand, insert usb in computer, straighten rope, pick and lift small, put knife on chopping board, place shape in shape sorter, take toilet roll off stand, put umbrella in umbrella stand, setup checkers
Screw requires screwing an object turn oven on, change clock, open window, open wine bottle
VisualOcclusion involves tasks with large objects and thus there are occlusions from certain views close microwave, close fridge, close grill, open grill, unplug charger, press switch, take money out safe, open microwave, put money in safe, open door, close door, open fridge, open oven, plug charger in power supply Task Variation Type # Variations Language Tempalte basketball in hoop NA put the ball in the hoop put rubbish in bin
NA put rubbish in bin meat off grill
Name take the { chicken, steak } off the grill meat on grill
Name take the { chicken, steak } on the grill change channel
Up / Down turn the channel up, down tv on
NA turn on the TV push buttons
Ordering & Colors push the { color name } button, then push the { color name } button, then ...
stack wine
NA stack wine bottle slide block to target
NA slide the block to targe reach and drag
Color use the stick to drag the cube onto the { color name } target take frame off hanger
NA hang frame on hanger water plants
NA pick up the watering can by its handle and water the plant hang frame on hanger
NA hang frame on hanger scoop with spatula
NA take frame off hanger place hanger on rack
NA pick up the hanger and place in on the rack move hanger
NA move hanger onto the other rack sweep to dustpan
NA sweep dirt to dustpan take plate off colored dish rack
Colors put the plate between the { color name } pillars of the dish rack screw nail
NA screw the nail in to the block wipe desk
NA wipe dirt off the desk stack blocks
Number of blocks Colors.
stack blocks to stack { color name } blocks take shoes out of box
NA take shoes out of box slide cabinet open and place cups Left / Right put cup in { option } cabinet reach target
Colors reach the { color name } target push button
Colors push the { button color name } lamp on
NA turn on the light lamp off
NA turn off the light pick and lift
Colors pick up the { block color name } block and lift it up to the target take lid off saucepan
NA take lid off the saucepan toilet seat down
NA toilet seat down close laptop lid
NA close laptop lid open box
NA open box open { option } drawer
Bottom / Middle / Top close { option } drawer close { option } drawer
Bottom / Middle / Top close option drawer close box
NA close box phone on base
NA put the phone on the base toilet seat up
NA toilet seat up put books on bookshelf
Number put { index } books on bookshelf pick up cup
Colors pick up the { target color name } cup turn tap
Left / Right turn { option } tap lift numbered block
Number pick up the block with the number { block num } beat the buzz
NA beat the buzz stack cups
Number of cups.
Colors.
3x stack { blocks to stack } { color name } cups take usb out of computer
NA take usb out of computer play jenga
NA play jenga insert onto square peg
Colors put the ring on the { color name } spoke take umbrella out of umbrella stand
NA take umbrella out of umbrella stand insert usb in computer
NA insert usb in computer straighten rope
NA straighten rope pick and lift small
Shapes (’cube’, ’cylinder’, ’triangular prism’, ’star’, ’moon’) pick up the { shape name } and lift it up to the target put knife on chopping board
NA put the knife on the chopping board place shape in shape sorter
Shapes (’cube’, ’cylinder’, ’triangular prism’, ’star’, ’moon’) put the { shape } in the shape sorter take toilet roll off stand
NA take toilet roll off stand put umbrella in umbrella stand
NA put umbrella in umbrella stand setup checkers
Number place the remaining { number } checker in its initial position on the board turn oven on
NA turn on the oven change clock
NA change the clock to show time 12.15 open window
NA open left window open wine bottle
NA open wine bottle close microwave
NA close microwave close
[Table]
T a s k
V a r i a ti on
T yp e
N u m o f
V a r i a ti on s
D e f a u lt
L a ngu a g e
T e m p
l
a t e
S ho r t
T un e d
L a ngu a g e T e m p
l
a t e
L ong
T un e d
L a ngu a g e T e m p
l
a t e s t ac k b l o c k s
N u m b e r o f
b l o c k s , c
o l o r s s t ac k {
b l o c k
s t o s t ac k } { c o l o r n a m e } b l o c k s
M ov e t
h e g r
i pp e r c
l
o s e t o {s t c o l o r n a m e } bu tt on , t h e n pu s h t
h e { 2nd c
o l o r n a m e } bu tt on , a f t e r t h a t , m ov e t
h e g r
i pp e r c l o s e t o t
h e { 3 r d c o l o r n a m e } bu tt on t
o pu s h t
h e { 4 t h c
o l o r n a m e } bu tt on , fi n a ll y , ..., m ov e t
h e g r
i pp e r c l o s e t o t
h e { n - t h c o l o r n a m e } bu tt on t
o pu s h t
h e { n - t h c o l o r n a m e } bu tt on .
M ov e t
h e w h
it e g
r
i pp e r c
l
o s e r t o { 1 s t c
o
l
o r n a m e } bu tt on , t h e n pu s h
{ 2nd c o l o r n a m e } bu tt on do w n , a f t e r pu s hi ng { 3 r d c o l o r n a m e } bu tt on , pu ll t h e w h
it e g
r
i pp e r up a n d m ov e t h e g
r
i pp e r c
l
o s e r t o { 4 t h c o l o r n a m e } bu tt on , ..., pu ll t h e w h it e g
r
i pp e r up a nd m ov e t
h e g r
i pp e r c
l
o s e r t o t h e {
n t h c
o l o r n a m e } bu tt on , t h e n pu s h { n t
h c o l o r n a m e } bu tt on do w n pu s h bu tt on s
O r d e r i
ng o f bu tt on s , c
o l o r s pu s h t
h e { c o l o r n a m e } bu tt on , t h e n pu s h t
h e { c o l o r n a m e } bu tt on , t h e n ...
C hoo s e a { c o l
o
r n a m e } c ub ea s s t ac k b a s e , t
h e np
i c k a no t
h e r { c o l o r n a m e } c ub ea ndp l ace it on t o t
h e { c o l o r n a m e } s t ac k , r e p ea t un til t h e s t ac
k h
a s 3 { c o l o r n a m e } c ub e s .
C hoo s e a { c o l
o
r n a m e } c ub e a s t
h e s t ac k b a s e , t
h e n p
i
c
k a no t
h e r { c o l o r n a m e } c ub e a nd p l ace it on t o t h e
{ c o l o r n a m e } c ub e s t ac k , t h e n m ov e t h e w h
it e g
r
i pp e r t
o p i
c
k a no t
h e r { c o l o r n a m e } c ub e a nd p l ace it on t o t h e
{ c o l o r n a m e } c ub e s t ac k , r e p ea t un til t h e s t ac k h
a s 3 { c o l o r n a m e } c ub e s .