Instruction-driven history-aware policies for robotic manipulations
Abstract:
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions.
Yet, robotic manipulation is extremely challenging as it requires  ne-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments.
To address these challenges, we propose a uni  ed transformer-based approach that takes into account multiple inputs.
In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions.
Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views.
We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art.
We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.
Introduction
People can naturally follow language instructions and manipulate objects to accomplish a wide range of tasks from cooking to assembly and repair.
It is also easy to generalize to new tasks by building upon skills learned from previously seen tasks.
Hence, one of the long-term goals for robotics is to create generic instruction-following agents that can generalize to multiple tasks and environments.
Thanks to signi  cant advances in learning generic representations for vision and language [2, 3, 4, 5], recent work has made great progress towards this goal [6, 7, 8, 9, 10].
For example, CLIPort [9] exploits CLIP models [5] to encode single-step visual observations and language instructions and to learn a single policy for 10 simulated tasks.
BC-Z [10] uses a pre-trained sentence encoder [11] to generalize to multiple manipulation tasks.
However, several challenges remain underexplored.
One important challenge is that sequential tasks require to track object states that may be hidden from current observations, or to remember previously executed actions.
This behaviour is hard to model with recent methods that mainly rely on current observations [9, 10].
Another challenge concerns manipulation tasks that require precise control of the robot end-effector to reach target locations.
Such tasks can be dif  cult to solve with single-view approaches [12], especially in situations with visual occlusions and objects of different sizes, e .
g .
see put money in safe
Figure 1 (left).
While several recent approaches combine views from multiple cameras by converting multi-view images into a uni  ed 2D/3D space [13, 14] or through a late fusion of multi-view predictions [15], learning representations for multiple camera views is an open research problem.
Furthermore, cross-modal alignment between vision, action, and text is challenging, in particular when training and test tasks differ in terms of objects and the order of actions, see Figure 1 (right).
Most of existing methods [9, 10, 16, 17] condense instructions into a global vector to condition policies [18] and are prone to lose  ne-grained information about different objects.
To address the above challenges, we introduce Hiveformer - a H istory-aware i nstruction-conditioned multi- v i e w trans former .
It converts instructions into language tokens given a pre-trained language encoder [5], and combines visual tokens for both past and current visual observations and proprioception.
These tokens are concatenated and fed into a multimodal transformer which jointly models dependencies between the current and past observations, spatial relations among views from multiple cameras, as well as  ne-grained cross-modal alignment between vision and instruction.
Based on the output representations from our multimodal transformer, we predict 7-DoF actions, i .
e ., position, rotation and state of the gripper, with a UNet [20] decoder.
We carry out extensive experiments on RLBench [19] in three setups: single-task learning, multitask learning, and multi-variation generalization 1 .
Our Hiveformer signi  cantly outperforms stateof-the-art models for all three settings, demonstrating the effectiveness of encoding instruction, history and views from multiple cameras with the proposed transformer.
Moreover, we evaluate our model on 74 tasks of RLBench, which goes beyond the 10 tasks used by Liu et al .
We manually group all the tasks into 9 categories according to their main challenges and analyze results per category for a better understanding.
Hiveformer not only excels in the multiple task setting with seen instructions in training, but also enables generalization to new instructions that represent different variations of the task, even with human-written language instructions.
Finally, we evaluate our model deployed on a real robot and show excellent performance.
Interestingly, pretraining the model in the RLBench simulator results in signi  cant performance gains when only a small number of real robot demonstrations are available.
To summarize, our contributions are three-fold:
• We introduce a new model, Hiveformer, to solve various challenges in robotics tasks.
It jointly models an instruction, multiple views, and history via a multimodal transformer for action prediction in robotic manipulation.
• We perform extensive ablations of our model on RLBench with 74 tasks grouped into 9 distinct categories.
The history improves long-term tasks and the multi-view setting is most helpful for tasks requiring high precision or in the presence of visual occlusions.
• We demonstrate that Hiveformer outperforms the state of the art in three RLBench setups, namely single-task, multi-task and multi-variation.
A single Hiveformer trained with synthetic instructions is able to solve multiple tasks and task variations, can generalize to unseen humanwritten instructions and shows excellent performance on a real robot after  netuning.
Our code, pre-trained models and additional results are available on the project webpage [1].
Related Work
Vision-based robotic manipulation.
While earlier methods for solving robotics tasks such as visual servoing [21, 22] were designed manually, the need to cope with large variations of objects and environments led to the emergence of learning-based neural approaches [23, 24, 25, 26].
Deep neural networks [27, 28] have achieved impressive results in manipulation for single tasks [29], and recently led to more challenging setups such as multi-task learning [30, 31, 32, 33].
Different multi-task approaches are explored by discovering which tasks should be trained together [15, 34], determining shared features across tasks [35, 36], meta-learning [37, 38, 39], goal-conditioned learning [40, 41], or inverse reinforcement learning [42].
These approaches can be generally split in two categories according to the training algorithm: reinforcement learning (RL) methods [43, 44, 45, 46] which learn policies from rewards provided by environments and behavioral cloning methods [47, 48, 49] that learn from demonstrations using supervised learning.
Demonstrations can be obtained from humans [50], robots [23, 51] or play interactions [49].
The emergence of robotic simulators, such as Gym [52], manipulaTHOR [53], dm control [54], Sapien [55], CausalWorld [56], and RLBench [19], also greatly accelerated the development of manipulation methods.
In this work, we use behavioral cloning to train policies given scripted demonstrations from RLBench [19] which covers many challenging manipulation tasks.
Instruction-driven vision-based robotic manipulation has received growing attention for manipulations in 2D planar [57, 58] or recent 3D environments [8, 59, 60], and has been transferred to the real world [9, 10].
As grounding the language in visual scenes is important, existing works have focused on challenges in object grounding, such as localizing objects based on referring expressions [61, 62, 63] and grounding spatial relationships [7, 64, 65].
Since language describes high-level actions, several works [60, 66, 67] consider a hierarchical approach to decompose a task into sub-goals.
Because natural language is rich and diverse, while training resources are limited, further works learn from collected of  ine data with instructions [10, 17] or leverage pre-trained vision-language models [4, 5] for action prediction [9, 68].
To further improve the precision of manipulation skills, Mees et al .
[8] align instructions with multiple cameras by fusing input images with known camera parameters.
Most of these works [6, 8, 9, 10] are stateless, since they only employ current observations to predict next actions.
Instead, our work proposes to jointly model language instructions, history, and multi-view observations.
Transformers [69] have led to signi  cant gains in natural language processing [2], computer vision [70] and related  elds [4, 5, 71].
They have also been used in the context of supervised reinforcement learning, such as Decision Transformer [72] or Trajectory Transformer [73].
Recent works in Vision-and-Language Navigation (VLN) [74, 75, 76] further demonstrate that the Transformer allows to better leverage previous observations to improve multi-modal action prediction.
Transformers are also used to build a multi-modal, multi-task, multi-embodiment generalist agent,
GATO [77].
Inspired by the success of transformers, we explore the transformer architecture for instruction-driven and history-aware robotic manipulation.
Problem De  nition
Our goal is to train a policy π  a [Equation] nl = 1 , { o i } ti = 1 , { a i } ti =  for robotic manipulation conditioned on a natural language instruction { x l } nl = 1 , visual observations { o i } ti = 1 , and previous actions { a i } ti = 1 where n is the number of words in the instruction and t is the current step.
For robotic control, we use macro steps [12] – key turning points in action trajectories where the gripper changes its state (open/close) or velocities of joints are close to zero.
We employ an inverse-kinematics based controller to  nd a trajectory between macro-steps.
In this way, the sequence length of an episode is signi  cantly reduced from hundreds of small steps to typically less than 10 macro steps.
The observation o t at step t consists of RGB images I t and point clouds P t aligned with the RGB images.
I t is composed of { I kt } Kk = 1 RGB images from K cameras, with each I kt being of size H ×
W × 3 (height, width, 3 channels).
Following [15], we use K = 3 with cameras on the wrist, left shoulder and right shoulder of the agent, and H = W = 128.
Similarly, P t represents point clouds { P kt } Kk = 1 from K = 3 cameras.
A point cloud P kt
∈
R
H
× W × 3 is obtained by projecting a single channel depth image H × W from the k -th camera in world coordinates using known camera intrinsics and extrinsics.
Each point in P kt has thus 3D coordinates and is aligned with a pixel in I kt .
The action space a t consists of the gripper pose and its state following the standard setup in RLBench [12].
The gripper pose is composed of the Cartesian coordinates p t =
( x t , y t , z t ) and its rotation described by a quaternion q t = ( q 0 t , q 1 t , q 2 t , q 3 t ) relative to the base frame.
The gripper’s state c t is boolean and indicates whether the gripper is open or closed.
An object is grasped when it is located in between the gripper’s two  ngers and the gripper is closing its grasp.
The execution of an action is achieved by a motion planner in RLBench.
Our Model: Hiveformer
We propose a uni  ed architecture for robotic tasks called Hiveformer ( H istory-aware i nstructionconditioned multi-
v i e w trans former ), see Figure 2 for an overview.
It consists of three modules: feature encoding, multimodal transformer and action prediction.
The feature encoding module (Sec. 4.1) generates token embeddings for instructions { x l } n l = 1 , visual observations { o i } t i = 1 and previous actions { a i } ti = 1 .
Then, the multimodal transformer (Sec. 4.2) learns relationships between the instruction, current multi-camera observations and history.
Finally, the action prediction module (Sec. 4.3) utilizes a convolutional network (CNN) to predict the next rotation q t + 1 and gripper state c t + 1 , and adopts a UNet decoder [20] to predict the next position p t + 1 .
4.1 Feature Encoding
We encode the instruction, visual observations, and actions as a sequence of tokens.
Instructions.
We employ a pre-trained language encoder to tokenize and encode the sentence instruction.
Speci  cally, we use the language encoder in the CLIP model [5].
Thanks to its vision-andlanguage pre-training, it is better at differentiating vision-related semantics such as colors compared to pure language-only pre-trained models like BERT [2], see Table 7 in the supplementary material.
We freeze the pre-trained language encoder and use a linear layer on top of it to obtain embeddings ˆ x l ∈ R d for each word token:
[Equation]
with ˜ x l the l -th embedding output by the language encoder, LN layer normalization [78], W x a projection matrix, and E xT a type embedding which differentiates instructions from visual observations.
Observations and Proprioception.
We encode the RGB image I kt , point clouds P kt , and proprioception A kt for each camera k separately.
A kt ∈ { 0 , 1 } H × W is a binary attention map used to encode the position of the gripper p t .
It takes value one at the location of the gripper center and zero elsewhere.
We concatenate I kt and A kt in the channel dimension and use a UNet encoder to obtain a feature map ˆ F kt ∈ R H v × W v × d v , where H v , W v , d v are the height, width, and number of channels of the feature map.
More details about the CNN architecture are presented in Section A of the supplementary material.
Next, we concatenate ˆ F k t with point cloud representations in the channel dimension to indicate the spatial location of each patch in the feature map.
To match the size of P kt and ˆ F kt , we apply mean-pooling to P kt .
The  nal encoded feature map F kt ∈ R H v × W v × ( d v + 3 ) is computed as follows:
[Equation]
We use patches f kt , h , w ∈ F kt , h ∈ [ 1 , H v ] , w ∈ [ 1 , W v ] as separate visual tokens.
We further encode f kt , h , w using embeddings of the camera id E kC , of the step id E tS , and of the patch location E h , w L as well as an embedding to indicate the visual nature of the tokens E vT as follows:
[Equation]
The encoded visual tokens of the k -th camera at step t are denoted as G kt =
{ g kt , h , w } H v , W v h = 1 , w = 1 ∈
R H v × W v × d .
We concatenate the encoded tokens for all cameras as G t =
( G 1 t , ··· , G Kt ) .
4.2 Multimodal transformer
Given the encoded tokens at the current macro step t , the multimodal transformer aims to obtain a contextualized representation for G t conditioned on the encoded instruction { ˆ x l } nl = 1 and history { G i } t − 1 i = 1 .
This enables learning relationships among views from multiple cameras, the current observations and instructions, and between the current observations and history for action prediction.
We use the transformer’s attention mechanism [79] to learn such relationships:
[Equation]
where W Q , W K , W V are learnable parameters.
Unlike previous work [75] that uses self-attention layers to capture all relationships, we employ different attention layers to capture different types of relationships, in order to reinforce the importance of the context.
First, we use a cross-attention layer to learn the inter-modal relationships between G t and its conditioned contexts C t consisting of tokens in the instruction { ˆ x l } nl = 1 and history { G i } t − 1 i = 1 , which is:
[Equation]
Then we learn the intra-modal relationships among patch tokens obtained from the views from multiple cameras through a self-attention layer, i .
e .
SA ( ˜ G t ) =
Attn ( ˜ G t , ˜ G t , ˜ G t ) .
Finally, a feedforward network consisting of two linear layers W 1 and W 2 is applied as follows:
[Equation]
4.3 Action Prediction
We concatenate the output embeddings of the transformer ˆ G t in Eq (6) and the original encoded visual representations ˆ F t in Sec.
4.1 in the channel dimension and reshape the  attened sequence into a feature map H t ∈ R K
× H v × W v × ( d + d v ) to predict the next action a t +
=
[ p t + 1 ; q t + 1 ; c t + 1 ] .
As some RLBench tasks require accurate  ne-grained positioning, different from the rotation q t + 1 and gripper state c t + 1 , the position p t + 1 is predicted through a separate module that uses point clouds P t .
Rotation and gripper’s state.
We transform H t into R H v × W v × K ( d + d v ) and feed it into a CNN decoder, described in Section A of the supplementary material.
We then apply average pooling across spatial dimensions and employ a linear layer to regress a 5-dimension vector [ q t + 1 ; c t + 1 ] .
Position.
The prediction of the gripper position p t + 1 is decomposed into an expected point on point clouds p et + 1 and an offset p ot + 1 , i .
e .
p t + 1 = p et + 1 + p ot + 1 .
The offset allows us to predict a virtual point outside the convex hull of the point cloud, e .
g .
when a robotic arm reaches  rst above the object and then touch the object.
For each camera k , a CNN with an upsampling layer predicts an attention map
B kt ∈ R H
× W over the point clouds P kt .
Each value B kt , h , w ∈ B kt corresponds to the probability of reaching the point P kt , h , w ∈ P kt .
Therefore, we compute p et + 1 as the expected position over all cameras:
[Equation]
[Equation]
The offset p ot + 1 is computed from the instruction and the current step id. Let E O ∈ R N τ × T × 3 be a learnable embedding, where N τ is the number of tasks and T is the maximum length of episodes.
We predict the task id from the instruction: Pr ( m ) =
Softmax 
[Equation]
n ∑ n l = 1 ˜ x l  , where Pr ( m ) ∈ [ 0 , 1 ] N τ , and we obtain the offset as: p ot + 1 =
∑ m Pr ( m ) · E O ( m , t , : ) .
4.4 Training and Inference
Losses.
We use behavioral cloning to train the models.
In RLBench, we generate D , a collection of
N successful demonstrations for each task.
Each demonstration δ ∈ D is composed of a sequence of (maximum) T macro-steps with observations { o δ i } Ti = 1 , actions { a ∗ i } Ti = 1 , task m ∗ and instruction { x l } nl = 1 .
We minimize a loss function L over a batch of demonstrations B = { δ j } | B | j = 1 ⊂ D .
The loss function is the sum of two losses: a mean-square error (MSE) on the gripper’s action and a cross-entropy (CE) over the task classi  cation:
[Equation]
[Table]
Masking current observation.
To ensure that the model uses past information { o i } t − 1 i = 1 , { a i } t − 1 i =instead of only relying on the current observation o t
[Table]
we randomly mask the current observation with a probability of 0.1.
The masking zeros out randomly selected patch features in the current observation.
Therefore, even if the unmasked current observations contain suf  cient information, the model still requires to complete the masked observations from the history for action prediction.
Experiments
In this section we present experiments on RLBench [19] tasks to demonstrate the effectiveness of our
Hiveformer model in three settings: single-task, multi-task, and multi-variation.
In the single-task setup, a separate model is trained and tested for each task with no variations of the task.
Multi-task refers to a setting where one model is trained for multiple tasks (but each task has a unique variation).
In the multi-variation case we train a single model to solve multiple variations of a single task and test it on new variations of the task unseen during training.
5.1 Experimental Setup
Dataset setups.
RLBench [19] is a benchmark of robotic tasks.
To compare our method with previous work [15], we use the same 10 tasks with 100 demonstrations for training unless stated otherwise.
We further evaluate our model on 74 tasks for which RLBench provides successful demonstrations.
We manually group these tasks into 9 categories according to their challenges.
More details on the split of the tasks are given in Section B of the Supplementary material.
We evaluate models by measuring the per task success rate for 500 unseen episodes.
Implementation details.
We use the Adam optimizer with a learning rate of 5 × 10 − 5 .
Each batch consists of 32 demonstrations.
Models were trained for 100,000 iterations.
We apply data augmentation in training including jitter over RGB images I kt , and a random crop of I kt , P kt , and A kt while keeping them aligned.
Models are trained on one NVIDIA Tesla V100 SXM2 GPU using a Singularity container with headless rendering.
Auto- λ
[15] uses a UNet network and applies late fusion to predictions from multiple views.
5.2 Ablations
To demonstrate the effectiveness of the proposed model architecture, we ablate the impact of its components in Table 1.
The model in R1 (row 1) is a UNet architecture similar to Auto- λ [15] except that it is conditioned on instructions rather than task ids.
This baseline only uses visual observations at the current step and already achieves promising results with a success rate of 73 .
On top of R1’s architecture, a multimodal transformer with self-attention is added in R2 to improve the modeling of multi-view images.
Visual tokens { G it } Ki = 1 are different channels in the feature map instead of spatial patches used in our  nal model.
In R3 and R4, we further add point clouds P t and gripper position A t in the feature encoding, which leads to 3% improvement in total.
The impact of history, i .
e .
the use of observations from previous steps, ( { G ij } K , t − 1 i = 1 , j = 1 ) is studied in R5 and R6.
The history information brings 4 .
5% absolute gains and the masking of observations during training further improves the performance by 0 .
In R7, we replace the tokenization of feature maps from channels g kt , c to patches g kt , h , w , and obtain another 2 .
2% gain.
This improvement can be attributed
[Table]
[Table] Ours [Table] to patch tokens that help encode  ne-grained spatial information.
Finally, we use cross-attention instead of self-attention (Eq. 5) to condition on the instruction and history context.
It further boosts the performance with a 3 . 8% gain.
5.3 Comparison with State of the Art
Single-task evaluation.
The upper block in Table 2 presents results of different models on 10 single tasks in RLBench.
We compare our model with ARM [12] and Auto- λ [15], two state-of-the-art methods on RLBench and observe a consistent improvement for all tasks.
Extending tasks in a single-task evaluation setup.
In Table 3, we further compare Auto- λ and
Hiveformer’s variants across 74 RLBench tasks grouped into 9 categories.
The variant without history removes the history tokens in Hiveformer, while the variant with one view only uses one camera at each step (we take the best among the 3 cameras for each task).
The full Hiveformer achieves consistently better performance compared to Auto- λ [15] on all types of tasks.
Among them, the
Long-term, Tools and Planning task groups assess the use of history, where our model brings improves signi  cantly over the variant without history.
Compared to the one view variant, our full model performs signi  cantly better on tasks requiring  ne-grained control or with large occlusions such as Screw, Precision and Visual Occlusion categories.
Yet, our method performs relatively poorly for Long-term tasks with more than 10 steps, such as “take shoes out of box”.
As Long-term tasks have an average number of steps 2-4 times higher than others, they are more prone to distribution shift issues and accumulated errors.
Hierarchical modeling or better training algorithms such as reinforcement learning and dagger [80] could be helpful, but are left as future work.
Multi-task evaluation.
The lower half in Table 2 shows the results in a multi-task setting.
Notably,
Auto- λ uses a training algorithm which dynamically adjusts the weights of different tasks, while our model simply treats all tasks with equal weights.
Nevertheless, our model outperforms Auto- λ by 14%, demonstrating the improvements due to our architecture.
We further compare our model with a variant without instructions in the input sequence (since p ot + 1 is predicted from instructions, we modify the model such as it is predicted from H k t ).
The results show that instructions are important in the multi-task setting.
Moreover, the performance of our single model trained for all tasks is only slightly worse than the performance of individual models for each task.
[Table]
Generalization to multi-variations.
Table 4 shows results of Hiveformer trained on different variations of the two tasks Tower and Push Buttons .
The Tower (resp.
Push Buttons ) task requires the robot to sequentially stack colored cubes (resp.
push colored buttons) using the order speci  ed in the instruction, see Figure 1 (right).
We use 100 variations in training and test models for both the 100 seen variations and 100 unseen variations.
In this setting, instructions are necessary to generalize to unseen variations ( e .
g .
it is impossible to distinguish the order of pushing buttons red-green-blue vs. blue-red-green by only looking at the scene).
We compare the models trained with different numbers of demonstrations per variation.
Even in the most challenging case where only 10 demonstrations are available per variation, Hiveformer achieves a success rate of 71 .
1% for the push buttons task and 49 .
8% for the tower task in unseen variations.
Furthermore, besides tests on synthetic instructions (Synt), we also test the generalization to real instructions.
Despite being only trained on synthetic instructions with limited vocabulary and diversity, our model performs well on instructions generated by humans (Real).
Finetuning Hiveformer on human instructions [75] is expected to result in further improvements.
Details of human-generated instructions are presented in Section C of the Supplementary material.
5.4 Real-robot Experiments
Setup.
We conduct real-robot experiments for the push buttons task on a 6-DoF UR5 robotic arm equipped with a 2-  nger Robotiq RG2 gripper and two cameras on each side of the scene.
As there exists a large difference between simulated and real environments, we  netune the simulatortrained policy on real-robot demonstrations.
We use 10 variations of the task and 10 real-robot demonstrations per variation.
More details are presented in Section E of the Supplementary material.
[Table]
Results.
We report success rates for real-robot experiments in Table 5 on the “push buttons” task using synthetic instructions.
The models are tested on 10 seen and 10 unseen variations.
We compare two models: one trained from scratch using real-robot demonstrations; and the other pretrained on RLBench and then  netuned using real-robot demonstrations.
As shown in Table 5, the pretraining signi  cantly improves the performance especially for unseen variations.
The model without pretraining is prone to over  tting on seen variations.
Although the domain gap between the real robot and
RLBench environments is large, our model bene  ts from pretraining in the simulator.
More analysis and examples are presented in Section E of the Supplementary material.
Conclusion
We introduced Hiveformer, a multimodal transformer that jointly models instructions, views from multiple cameras, and history for instruction-driven robotics manipulation.
We evaluated the model on RLBench in three settings: single-task learning, multi-task learning, and multi-variation generalization and demonstrated its effectiveness outperforming state of the arts. We deployed our model on a real robot which is able to generalize to unseen variations and human-written instructions.
Limitations.
The computational cost quadratically increases with the input sequence length due to the transformer.
Furthermore, our model is trained with behavioral cloning, which may suffer from exposure bias.
Future works could improve the ef  ciency for long-term tasks with hierarchical models and also incorporate reinforcement learning.
Moreover, our model is trained on only synthetic instructions and performs worse on the human-written instructions.
Training on human-written automatically generated instructions could help improve the performance.
Acknowledgments
This work was granted access to the HPC resources of IDRIS under the allocation 101002 made by GENCI.
This work is funded in part by the French government under management of Agence
Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR19P3IA-0001 (PRAIRIE 3IA Institute), the ANR project VideoPredict (ANR-21-FAI1-0002-01) and by Louis Vuitton ENS Chair on Arti  cial Intelligence.