CLIP ORT : What and Where Pathways for Robotic Manipulation
Abstract: How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts?
Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks.
In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for ﬁne-grained manipulation.
To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation.
Speciﬁcally, we present CLIP ORT , a language-conditioned imitationlearning agent that combines the broad semantic understanding ( what ) of CLIP [1] with the spatial precision ( where ) of Transporter [2].
Our end-to-end framework is capable of solving a variety of language-speciﬁed tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures.
Experiments in simulated and real-world settings show that our approach is data efﬁcient in few-shot settings and generalizes effectively to seen and unseen semantic concepts.
We even learn one multi-task policy for 10 simulated andreal-world tasks that is better or comparable to single-task policies.
Introduction
Ask a person to “get a scoop of coffee beans” or “fold the cloth in half” and they can naturally take concepts like scoop or fold and ground them in concrete physical actions within an accuracy of a few centimeters.
We humans do this intuitively, without explicit geometric or kinematic models of coffee beans or cloths.
Moreover, we can generalize to a broad range of tasks and concepts from a minimal set of examples on what needs to be achieved.
How can we imbue robots with this ability to efﬁciently ground abstract semantic concepts in precise spatial reasoning?
Recently, a number of end-to-end frameworks have been proposed for vision-based manipulation [2, 3, 4, 5].
While these methods do not use any explicit representations of object poses, instance segmentations, or symbolic states, they can only replicate demonstrations with a narrow range of variability and have no notion of the semantics underlying the tasks.
Switching from packing red pens to blue pens involves collecting a new training set [2], or if using goal-conditioned policies, involves the user providing a goal-image from the scene [5, 6].
In realistic human-robot interaction settings, collecting additional demonstrations or providing goal-images is often infeasible and unscalable.
A natural solution to both these problems is to condition policies with natural language.
Language provides an intuitive interface for specifying goals and also for implicitly transferring concepts across tasks.
While language-grounding for manipulation has been explored in the past [7, 8, 9, 10], these pipelines are limited by object-centric representations that cannot handle granular or deformable objects and often do not reason about perception and action in an integrated manner.
In parallel, there has been great progress in learning models for visual representations [11, 12] and aligning representations of vision and language [13, 14, 15] by training on large-scale internet data.
However, these models lack a ﬁne-grained understanding on how to manipulate objects, i.e. physical affordances.
[Table]
[Figure]
To this end
[Figure]
we propose the ﬁrst framework that combines the best of both worlds: end-to-end learning for ﬁne-grained manipulation with the multi-goal and multi-task generalization capabilities of vision-language grounding systems.
We introduce a two-stream architecture for manipulation with semantic and spatial pathways broadly inspired by (or vaguely analogous to) the two-stream hypothesis in cognitive psychology [16, 17, 18].
Speciﬁcally, we present CLIP ORT , a languageconditioned imitation-learning agent that integrates the semantic understanding ( what ) of CLIP [1] with the spatial precision ( where ) of Transporter [2].
Transporter has been applied to a wide range of rearragement tasks from industrial packing [2] to manipulating deformable objects [6].
The key insight of the approach is formulating tabletop manipulation as a series of pick-and-place affordance predictions, where the objective is to detect actions rather than detect objects and then learn a policy.
This action-centric approach to perception [19] is data efﬁcient and effective at circumventing the need for explicit “objectness” in learnt representations.
However, Transporter is a tabula rasa system that learns all visual representations from scratch and so every new goal or task requires collecting a new set of demonstrations.
To address this problem, we bake in a strong semantic prior while learning policies.
We condition our semantic stream with visual and language-goal features from a pre-trained CLIP model [1].
Since CLIP is pre-trained to align image and language features from millions of image-caption pairs from the internet, it provides a powerful prior for grounding semantic concepts that are common across tasks like categories, parts, shapes, colors, texts, and other visual attributes, all without a top-down pipeline that requires bounding boxes or instance segmentations [13, 14, 15, 20].
This allows us to formulate tabletop rearrangement as a series of language-conditioned affordance predictions, a predominantly vision-based inference problem, and thus beneﬁt from the strengths of data-driven paradigms like scale and generalization.
To study these beneﬁts, we conduct large-scale experiments in the Ravens [2] framework with a simulated suction-gripper robot.
We propose 10 language-conditioned tasks with 1000s of unique instances per task that require both semantic and spatial reasoning (see Figure 1 a-j).
CLIP ORT is not only effective at solving these tasks, but surprisingly, it can even learn a multi-task model for all 10 tasks that achieves better or comparable performance to single-task models.
Further, our evaluations indicate that our multi-task model can effectively transfer attributes like “pink block” across tasks, having never seen pink blocks or the word ‘pink’ in the context of the evaluation task.
We also demonstrate our approach on a Franka Panda manipulator with one multi-task model forreal-world tasks trained with just 179 image-action pairs (see Figure 1 k-o).
In summary, our contributions are as follows:
• An extended benchmark of language-grounding tasks for manipulation in Ravens [2].
• Two-stream architecture for using internet pre-trained vision-language models for conditioning precise manipulation policies with language goals.
• Empirical results on a broad range of manipulation tasks, including multi-task models, validated with real-robot experiments.
The benchmark, code, and pre-trained models are available at: cliport.github.io .
Related Work
Vision-based Manipulation.
Traditionally, perception for manipulation has centered around object detectors, segmentors, and pose estimators [21, 22, 23, 24, 25, 26].
These methods cannot handle deformable objects, granular media, or generalize to unseen objects without object-speciﬁc training data.
Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects.
On the other hand, end-to-end perception-toaction models can learn precise sequential policies [2, 4, 6, 33, 34, 35], but these methods have limited understanding of semantic concepts and rely on goal-images to condition policies.
In contrast,
Yen-Chen et.
al [36] showed that pre-training on semantic tasks like classiﬁcation and segmentation helps in improving efﬁciency and generalization of grasping predictions.
Semantic Models.
With the advent of large-scale models [37, 38, 39], a number of methods for learning joint vision and language representations have been proposed [13, 14, 15, 20, 40].
However, these methods are restricted to bounding boxes or instance segmentations, which make them inapplicable for detecting things like piles of coffee beans or squares on a chessboard.
Alternatively, works in contrastive learning forgo top-down object-detection and learn continuous representations by pre-training on unlabeled data [11, 12].
Recently, CLIP [1] applied a similar approach to align vision and language representations by training on millions of image-caption pairs from the internet.
Language Grounding for Robotics.
Several works have proposed systems for instructing robots with natural language [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47].
However, these methods use disentangled pipelines for perception and action with the language primarily being used to guide the perception.
As such, these pipelines lack the spatial precision necessary for tasks like folding cloths.
Recently, Lynch et.
al [48] proposed an end-to-end system for grounding language in continuous control, but it requires several hours of human teleoperation data for a single simulated desk setting.
Two-Stream Architectures are prevalent in action-recognition networks [49, 50, 51] and audiorecognition systems [52, 53].
In robotics, Zeng et. al [54] and Jang et.
al [55] have proposed twostream pipelines for affordance predictions of novel objects.
The former requires goal-images and the latter is restricted to one-step grasps with single-category goals.
In contrast, our framework provides a rich and intuitive interface with composable language commands for sequential tasks.
CLIP ORT
CLIP ORT is an imitation-learning agent based on four key principles: (1) Manipulation through a two-step primitive where each action involves a start and ﬁnal end-effector pose.
(2) Visual representations of actions that are equivariant to translations and rotations [56, 57].
(3) Two separate pathways for semantic and spatial information.
(4) Language-conditioned policies for specifying goals and also transferring concepts across tasks.
Combining (1) and (2) from Transporter with (3) and (4) allows us to achieve generalizable policies that go beyond just imitating demonstrations.
Section 3.1 describes the problem formulation, gives an overview of Transporter [2], and presents our language-conditioned model.
Section 3.2 provides details on the training approach.
3.1 Language-Conditioned Manipulation
We consider the problem of learning a goal-conditioned policy ⇡ that outputs actions a t given input  t =
( o t , l t ) consisting of a visual observation o t and an English language instruction l t :
[Equation]
[Figure]
RGB-D
S e m a n t
i c
V e n t r a l
S p a t i a l
RGB HxWx
[Figure]
D o r s a l
[Figure]
The actions a =
( T pick , T place ) specify the end-effector pose for picking and placing, respectively.
We consider tabletop tasks where T pick , T place 2 SE (2) .
The visual observation o t is a top-down orthographic RGB-D reconstruction of the scene where each pixel corresponds to a point in 3D space.
The language instruction l t either speciﬁes step-by-step instructions e.g. “pack the scissors” ! “pack the purple tape” ! etc., or a single goal description for the whole task e.g “pack all the blue and yellow boxes in the brown box” .
See Figure 4 for speciﬁc examples.
We assume access to a dataset D = { ⇣ 1 , ⇣ 2 , . . .
⇣ n } of n expert demonstrations with associated discrete-time input-action pairs ⇣ i = { ( o 1 , l 1 , a 1 ) , ( o 2 , l 2 , a 2 ) , . . .
} where a t =
( T pick , T place ) corresponds to expert pick-and-place coordinates at timestep t .
These expert demonstrations are used to supervise the policy ⇡ .
Transporter for Pick-and-Place.
The policy ⇡ is trained with Transporter [2] to perform spatial manipulation.
The model ﬁrst (i) attends to a local region to decide where to pick, then (ii) computes a placement location by ﬁnding the best match through cross-correlation of deep visual features.
Following Transporter [2, 6], the policy ⇡ is composed of two action-value modules (Q-functions):
The pick module Q pick decides where to pick, and conditioned on this pick action the place module
Q place decides where to place.
These modules are implemented as Fully-Convolutional-Networks (FCNs) that are translationally equivariant by design.
As we will describe in more detail below, we extend these networks to two-stream architectures that can handle language input.
The pick
FCN f pick takes input  t =
( o t , l t ) and outputs a dense pixelwise prediction Q pick 2 R H ⇥ W of action-values, where are used to predict the pick action T pick :
[Equation]
Since o t is an orthographic heightmap, each pixel location ( u, v ) can be mapped to a 3D picking location using the known camera calibration.
f pick is trained in a supervised manner to predict the pick action T pick that imitates the expert demonstration with the speciﬁed language instruction at timestep t .
The second FCN  query takes in  t [ T pick ] , which is a c ⇥ c crop of o t centered at T pick along with the language instruction l t , and outputs a query feature embedding of shape R c ⇥ c ⇥ d .
The third FCN  key consumes the full input  t and outputs a key feature embedding of shape R H ⇥ W ⇥ d .
The place action-values Q place are then computed by cross-correlating the query and key features:
[Equation]
where  ⌧ 2 SE (2) represents a potential placement pose.
Since o t is an orthographic heightmap, rotations in the placement pose  ⌧ can be captured by stacking k discrete angle rotations of the crop before passing it through the query network  query .
Then T place = argmax  ⌧ Q place (  ⌧ |  t , T pick ) , where the place module is trained to imitate the placements in the expert demonstrations.
For all models, we use c = 64 , k = 36 and d = 3 .
As in Transporter [2, 6], our framework can be extended to handle any motion primitive like pushing, sliding, etc.
that can be parameterized by two end-effector poses at each timestep.
For more details, we refer the reader to the original paper [2].
Two-Stream Architecture.
In CLIP ORT , we extend the network architecture of all three FCNs f pick ,  query and  key from Transporter [2] to allow for language input and reasoning about high-level semantic concepts.
We extend the FCNs to two-pathways: semantic (ventral) and spatial (dorsal).
The semantic stream is conditioned with language features at the bottleneck and fused with intermediate features from the spatial stream.
See Figure 2 for an overview of the architecture.
The spatial stream is identical to the ResNet architecture in Transporter – a tabula rasa network that takes in RGB-D input o t and outputs dense features through an hourglass encoder-decoder model.
The semantic stream uses a frozen pre-trained CLIP ResNet50 [1] to encode the RGB input 2 ˜ o t up until the penultimate layer ˜ o t ! v (0) t : R 7 ⇥ 7 ⇥ 2048 , and then introduces decoding layers that upsample the feature tensors to mimic the spatial stream v ( l  1) t ! [Equation] ⇥
[Equation]
⇥ [Equation] at each layer l . The language instruction l t is encoded with CLIP’s Transformer-based sentence encoder to produce a goal encoding l t !
g t : R 1024 . This goal encoding g t is downsampled with fully-connected layers to match the channel dimension C and tiled to match the spatial dimensions of the decoder features such that g t ! g ( l ) t : R h ⇥ w ⇥ C .
The decoder features are then conditioned with the tiled goal features through an element-wise product v ( l ) t  g ( l ) t (Hadamard product).
Since CLIP was trained with contrastive loss on the dot-product alignment between pooled image features and language encodings, the element-wise product allows us to use this alignment while the tiling preserves the spatial dimensions of the visual features.
This language conditioning is repeated for three subsequent layers after the bottleneck inspired by LingUNet [58].
We also add skip connections to these layers from the CLIP ResNet50 encoder to utilize different levels of semantic information from shapes to parts to object-level concepts [59].
Finally, following existing two-stream architectures in videoaction recognition [51], we add lateral connections from the spatial stream to the semantic stream.
These connections involve concatenating two feature tensors and applying 1 ⇥ 1 conv to reduce the channel dimension [ v ( l ) t  g ( l ) t ; d ( l ) t ] : R h ⇥ w ⇥ C v + C d !
R h ⇥ w ⇥ C v , where v ( l ) t and d ( l ) t are the semantic and spatial tensors at layer l , respectively.
For the ﬁnal fusion of dense features, addition for f pick and 1 ⇥ 1 conv fusion for  query and  key worked the best empirically.
See
Appendix C for details on the exact architecture.
3.2 Implementation Details
Training from demonstrations.
Similar to Transporter [2] we train CLIP ORT through imitation learning from a set of expert demonstrations D = { ⇣ 1 , ⇣ 2 , . . .
, ⇣ n } consisting of discrete-time inputaction pairs ⇣
i = { ( o 1 , l 1 , a 1 ) , ( o 2 , l 2 , a 2 ) , . . . } .
During training, we randomly sample an inputaction pair from the dataset and supervise the model end-to-end with one-hot pixel encodings of demonstration actions Y pick : R H ⇥ W ⇥ k and Y place : R H ⇥ W ⇥ k with k discrete rotations.
In simulated experiments with the suction-gripper, we use k = 1 for pick actions and k = 36 for place actions.
The model is trained with cross-entropy loss: L =  E Y pick [ log V pick ]  E Y place [ log V place ] where
V pick = softmax ( Q pick (( u, v ) |  t )) and V place = softmax ( Q place (( u 0 , v 0 , ! 0 ) |  t , T pick )) .
Com- pared to the original Transporter models that were trained for 40K iterations, we train our models for 200K iterations (with data augmentation; see Appendix E) to account for additional semantic variation in tasks – randomized colors, shapes, objects.
All models are trained on a single commodity
GPU for 2 days with a batch size of 1.
Training multi-task models.
Multi-task training is nearly identical to single-task training except for the sampling of training data.
First, we randomly sample a task, and then select a random inputaction pair from that task in the dataset.
Using this strategy, all tasks are equally likely to be sampled but longer horizon tasks are less likely to reach full coverage of input-action pairs available in the dataset.
To compensate for this, we train all multi-task models 3 ⇥ longer for 600K iterations or
GPU days.
Results
We perform experiments both in simulation and hardware aimed at answering the following questions: 1) How effective is the language-conditioned two-stream architecture for ﬁne-grained manipulation compared to one-stream alternatives and other simpler baselines?
Is it possible to train a multi-task model for all tasks, and how well does it perform and generalize? 3)
How well do these models generalize to seen and unseen semantic attributes like colors, shapes, and object categories?
4.1 Simulation Setup
Environment.
All simulated experiments are based on a Universal Robot UR5e with a suction gripper.
The setup provides a systematic and reproducible environment for evaluation, especially for benchmarking the ability to ground semantic concepts like colors and object categories.
The input observation is a top-down RGB-D reconstruction from 3 cameras positioned around a rectangular table: one in the front, one on the left shoulder, and one on the right shoulder, all pointing towards the center.
Each camera has a resolution of 640 ⇥ 480 and is noiseless.
Language-Conditioned Manipulation Tasks.
We extend the Ravens benchmark [2] set in PyBullet [60] with 10 language-conditioned manipulation tasks.
See Figure 1 for examples and Table 3 for challenges associated with each task.
Each task instance is constructed by sampling a set of objects and attributes: poses, colors, sizes, and object categories.
8 of the 10 tasks have two variants, denoted by seen and unseen , depending on whether the task has unseen attributes (e.g. color) at test time.
For colors: T seen colors =
{ yellow, brown, gray, cyan } and T unseen colors =
{ orange, purple, pink, white } with 3 overlapping colors T all = { red, green, blue } used in both the seen and unseen spilts.
For packing objects, we use 56 tabletop objects from the
Google Scanned Objects dataset [61] and split them into 37 seen and 19 unseen objects.
The language instructions are constructed from templates for simulated experiments, and human-annotated for real-world experiments.
For more details about individual tasks, see Appendix A.
Evaluation Metric.
We adopt the 0 (fail) to 100 (success) scores proposed in the Ravens benchmark [2].
The score assigns partial credit based on the task, e.g. 3 / 5 ) 60 . 0 for packing 3 out of 5 objects speciﬁed in the instructions, or 30 / 56 ) 53 . 6 for pushing 30 out of 56 particles into the correct zone.
See Appendix A for the speciﬁc evaluation metric used in each task.
During an evaluation episode, an agent keeps interacting with the scene until an oracle indicates task-completion.
We report scores on 100 evaluation runs for agents trained with n = 1 , 10 , 100 , 1000 demonstrations.
4.2 Simulation Results
Table 1 presents results from our large-scale experiments in
Ravens [2] and Figure 3 summarizes these results with average scores across seen and unseen splits.
Baseline Methods. To
study the effectiveness of our two-stream architecture, we broadly compare against two baselines:
Transporter-only and CLIP-only.
Transporter-only is the original Transporter [2], or equivalently, the spatial stream of CLIP ORT with RGB-D input.
Although Transporter-only does not receive any language goals, it shows what can be achieved through chance by exploiting the most likely actions seen during training.
On the other hand, CLIP-only is just the semantic stream of CLIP ORT with
RGB and language input.
CLIP-only shows what can be achieved by ﬁne-tuning a pre-trained semantic model for manipulation without spatial information, particularly depth.
Two-Stream Performance.
Figure 3 (seen) captures the essence of our main claims.
The performance of Transporter-only saturates at 50% since it doesn’t use the language instruction to ground the desired goal.
CLIP-only does have a goal, but lacks the spatial precision to go the last mile and thus saturates at 76% .
Only CLIP ORT (single) achieves more than 90% , which indicates that both the semantic and spatial streams are crucial for ﬁne-grained manipulation.
Further, CLIP ORT (single) achieves 86% on most tasks with just 100 demonstrations, showcasing its efﬁciency.
In addition to these baselines, we present various ablations and alternative one-stream and twostream models in Appendix F.
To brieﬂy summarize these results, CLIP is essential for few-shot
[Table]
learning (i.e. n  10 ) in lieu of semantic stream alternatives like ImageNet-trained ResNet50 [62] with BERT [38].
Image-goal models outperform CLIP ORT (single) in packing Google objects, but this is only because they do not have to solve the language-grounding problem.
Multi-Task Performance.
In realistic scenarios, we want the robot to be capable of any task, not just one task.
We investigate this through CLIP ORT (multi) in Table 1 with one multi-task model trained on all 10 tasks.
CLIP ORT (multi) models are trained only on seen-splits of tasks, so an unseen attribute like ‘pink’ is consistent throughout single and multi-task settings.
Surprisingly,
CLIP ORT (multi) outperforms single-task CLIP ORT (single) models in 41 / 72 = 57% of the evaluations in Table 1.
This trend is also evident in Figure 3 (seen) , especially in instances withdemonstrations or less.
Although CLIP ORT (multi) is trained on more diverse data from other tasks, both CLIP ORT (multi) and CLIP ORT (single) have access to the same amount of data per task.
This supports our premise that language is a strong conditioning mechanism for reusing concepts from other tasks without learning them from scratch.
It also validates a trait of data-driven approaches where training on lots of diverse data leads to more robust and generalizable representations [1, 63].
However, CLIP ORT (multi) performs worse on longer-horizon tasks like align-rope .
We hypothesize that this is because longer-horizon tasks get less coverage of input-action pairs in the dataset.
Future works could use better sampling methods that balance tasks according to their average time horizon.
Generalizing to Unseen Attributes.
Tasks that require generalizing to novel colors, shapes, and objects are more difﬁcult and all our agents achieve relatively lower performance on these tasks, as shown in Figure 3 ( unseen ).
However, CLIP ORT (single) models do substantially better than chance, i.e., Transporter-only.
The lower performances are due to the difﬁculty of grounding unseen attributes such as ‘pink’ and ‘orange’ in the language instruction “put the pink block on the orange bowl”, when the agent has never encountered words ‘orange’, ‘pink’ or their corresponding visual characteristics in the context of the physical environment.
Although pre-trained CLIP has been exposed to the attribute ‘pink’, it could correspond to different concepts in the physical setting depending on factors like lighting condition, and thus requires at least few examples to condition the trainable semantic decoder layers.
Additionally, we notice that CLIP ORT (single) is also less prone to overﬁtting compared to Transporter-only.
As evidenced in towers-of-hanoi-seq-unseen-colors task in Table 1, Transporter-only suffers from a performance drop because of rings with unseen colors despite the fact that Tower of Hanoi can be solved without attending to the colors and simply focusing on the ring size.
We hypothesize that since CLIP was trained on diverse internet data, it enables our agent to focus on task-relevant concepts while ignoring irrelevant aspects of the task.
[Figure]
I n p u t
P i c k
P l a c e t= t= t= t= t=
Transferring Attributes across Tasks.
One solution for dealing with unseen attributes is to explicitly learn these attributes from other tasks.
We study this with CLIP ORT (multi-attr) in Table 1 and
Figure 3 ( unseen ).
For these models, CLIP ORT (multi) is trained on both seen -and- unseen splits from all tasks except for the task being evaluated on, for which it was only trained on the seen split.
As such, this evaluation measures whether having seen pink blocks in put-blocks-in-bowl-unseen-colors helps solve “pack all the pink and cyan boxes” in packing-box-pairs-unseen-colors .
Results indicate that such explicit transfers result in signiﬁcant improvements.
For instance, on the put-blocks-inbowls-unseen-colors task for n = 1000 , CLIP ORT (multi)’s performance increases from 45 .
8 to 75 .
4.3 Real-Robot Experiments
[Figure]
[Table]
We validated our results in hardware with a Franka
Panda manipulator.
See Appendix D for setup details.
Table 2 reports success rates for a multi-task model trained and evaluated on 9 real-world tasks.
Due to COVID restrictions, we could not conduct largescale user-studies, so we report on small train (5-10 demos) and test sets (5-10 runs) per task.
Overall, CLIP ORT (multi) is effective at few-shot learning with just 179 samples, and the performances roughly correspond to those in simulated experiments, with simple block manipulation tasks achieving ⇠ 70% .
We estimate that for more robust real-world performance at least 50 to 100 training demonstrations are necessary, as evident in Figure 3.
Interestingly, we observed that the model sometimes exploits biases in the training data instead of learning to ground instructions.
For instance, in Put Blocks in Bowl , the training set consisted of only one datapoint on “yellow blocks” being placed inside a “blue bowl”.
This made it difﬁcult to condition the model to place “yellow blocks” in non-blue bowls.
But instances with just one or two examples where a colored block went to different colored bowls was sufﬁcient to make the model pay attention to the language.
In summary, unbiased datasets containing both a good coverage of expected skills and invariances, and a decent number of training demonstrations, are crucial for good real-world performance.
Conclusion
We introduced CLIP ORT , an end-to-end framework for language-conditioned ﬁne-grained manipulation.
Our experiments, speciﬁcally with multi-task models, indicate that data-driven approaches to generalization have yet to be fully-exploited in robotics.
Coupled with the right action abstraction and spatio-semantic priors, end-to-end methods can quickly learn new skills without requiring top-down pipelines that need task-speciﬁc engineering.
While CLIP ORT can solve a range of tabletop tasks, extending it to dexterous 6-DOF manipulation that goes beyond the two-step primitive remains a challenge.
As such, it cannot handle complex partially-observable scenes, or output continuous control for multi-ﬁngered hands, or predict task-completion (see Appendix I for an extended discussion).
But overall, we are excited by the conﬂuence of data and structural priors for building scalable and generalizable robotic systems.
Acknowledgments
All simulated experiments were facilitated through the Hyak computing cluster funded by the STF at the University of Washington.
We thank Mohak Bhardwaj for help with the Franka setup at UW.
We are also grateful to our colleagues Chris Xie, Jesse Thomason and Valts Blukis for providing feedback on the initial draft.
This work was funded in part by ONR under award #1140209-