SORNet: Spatial Object-Centric Representations for Sequential Manipulation
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Sequential manipulation tasks require a robot to perceive the state of anenvironment and plan a sequence of actions leading to a desired goal state, wherethe ability to reason about spatial relationships among object entities from rawsensor inputs is crucial.
Prior works relying on explicit state estimation or endto-end learning struggle with novel objects.
In this work, we propose SORNet( S patial O bject-Centric R epresentation Net work), which extracts object-centricrepresentations from RGB images conditioned on canonical views of the objectsof interest.
We show that the object embeddings learned by SORNet generalize zero-shot to unseen object entities on three spatial reasoning tasks: spatialrelationship classiﬁcation, skill precondition classiﬁcation and relative directionregression, signiﬁcantly outperforming baselines.
Further, we present real-worldrobotic experiments demonstrating the usage of the learned object embeddings intask planning for sequential manipulation.
[Figure]
Introduction
A crucial question for complex multi-step robotic tasks is how to represent relationships betweenentities in the world, particularly as they pertain to preconditions for various skills the robot mightemploy.
In goal-directed sequential manipulation tasks with long-horizon planning, it is common touse a state estimator followed by a task and motion planner or other model-based system [1, 2, 3,4, 5, 6].
A variety of powerful approaches exist for explicitly estimating the state of objects in theworld, e.g. [7, 8, 9, 10].
However, it is challenging to generalize these approaches to an arbitrarycollection of objects.
In addition, the objects are often in contact in manipulation scenarios, whereworks explicitly addressing the problem of generalizing to unseen objects [11, 12] still struggle.
Fortunately, knowing exact poses of objects may not be necessary for manipulation. End-to-endmethods [13, 14, 15] leverage that fact and build networks that generates actions directly from sensorinputs without explicitly representing objects.
Nevertheless, these networks are very speciﬁc to the tasks they are trained on.
For example, it is non-trivial to use a network trained on stacking blocksto unstack blocks.
In this work, we take an important step towards a manipulation framework that generalizes zero-shotto novel tasks with unseen objects.
Speciﬁcally, we propose a neural network that extracts objectcentric embeddings directly from raw RGB images, which we call SORNet , or S patial O bjectCentric R epresentation Net work.
The design of SORNet allows it to generalize to novel objectsand tasks without retraining or ﬁnetuning.
Trained from large amounts of simulated robotic manipulation data, the object-centric embeddings produced by SORNet can be used to inform a task andmotion planner with implicit object state information relevant to goal-directed sequential manipulation tasks, e.g. logical preconditions for primitive skills or continuous 3D directions between theentities in the scene.
To summarize, our contribution are: (1) a method for extracting object-centric embeddings from
RGB images that generalizes zero-shot to different number and type of objects; (2) a framework forlearning object embeddings that capture continuous spatial relationships with only logical supervision; (3) a dataset containing sequences of RGB observations labeled with spatial predicates duringvarious tabletop rearrangement manipulation tasks. We empirically evaluate the object-centric embeddings produced by SORNet on three differentdownstream tasks: 1) classiﬁcation of logical predicates for action preconditions; 2) visual questionand answering for spatial relationships; 3) prediction of relative 3D direction between entities inmanipulation scenes, in all cases testing on held-out objects that did not appear in training data.
In alltasks, SORNet obtains signiﬁcant improvements over the baseline methods.
Finally, we evaluated SORNet with real-world robot experiments to showcase the transfer of the learned object-centricembeddings to real-world observations.
Related Work
Sequential Manipulation In goal-directed sequential manipulation tasks with long-horizon planning, a class of work uses a pipeline approach requiring a state estimator followed by a task andmotion planner [1, 2, 3, 4, 5, 6], but the state estimator with an explicit state representation is oftenrestricted to an environment or a set of objects.
Another class of work employs neural networks to learn motor controls directly from raw sensordata, such as RGB images and joint encoder readings [14, 16, 15, 17, 18, 19], but they are oftennot transferable to new tasks, especially tasks involving long-horizon planning.
Our work combinesa network that estimates implicit states with a symbolic planner to obtain the best of both worlds:generalization to new objects and to new tasks.
Learning Spatial Relationships Learning spatial relationships between object entities have beenstudied in the ﬁeld of 3D vision and robotics.
Methods such as [20, 21, 22] predict discrete or continuous pairwise object relations from 3D inputs such as point clouds or voxels, assuming completeobservation of the scene and segmented objects with identities.
In contrast, our approach does notmake any assumptions regarding the observability of the objects and does not require pre-processingof the sensor data.
The learning framework by Kase et al.
[23] is most related to our approach, whichtakes a sequence of sensor observations and classiﬁes a set of pre-deﬁned relational predicates
whichis then used by a symbolic planner to produce a suitable operator.
Compared to our approach, theirsis limited by the number of objects in the scene and a ﬁxed set of spatial predicates.
Visual Reasoning
Recently, several advancements have been made on visual reasoning benchmarks[24, 25, 26] using transformer networks [27].
Toward solving spatio-temporal reasoning task from
CLEVRER [25] and CATER [26], Ding et al.
[28] proposed an object-based attention mechanismand Zhou et al.
[29] proposed a multi-hop transformer model.
Both works assume a segmentationmodel to produce object segments and performs language grounding to the segments to perform reasoning.
Our SORNet architecture is simpler and can solve spatial-reasoning tasks for unseen objectinstances without requiring a segmentation or object detection module.
Furthermore, our work focuses on a relatively complex manipulation task domain involving manipulator in the observations.
Although our current work focuses on predicting spatial relations from a single RGB frame, theobject-centric embeddings could potentially be used for solving temporal-reasoning tasks.
Methods
[Figure]
3.1 SORNet: Spatial Object-Centric Representation Network
Our object embedding network ( SORNet ) (Fig. 2) takes an RGB image and an arbitrary number ofcanonical object views and outputs an embedding vector corresponding to each input object patch.
The architecture of the network is based on the Visual Transformer (ViT) [30].
The input imageis broken into a list of ﬁxed-sized patches, which we call context patches .
The context patchesare concatenated with the canonical object views to form a patch sequence.
Each patch is ﬁrstﬂattened and then linearly projected into a token vector, then positional embedding is added to thesequence of tokens.
Following [30], we use a set of learnable vectors with the same dimension as thetoken vectors as positional embeddings.
The positional-embedded tokens are then passed througha transformer encoder, which includes multiple layers of multi-head self-attention.
The transformerencoder outputs a sequence of embedding vectors.
We discard the embedding for context patchesand keep those for the canonical object views.
We apply the same positional embedding to the canonical object views to make the output embeddings permutation equivariant.
We also mask out the attention among canonical object views andthe attention from context patches to canonical object views to ensure the model uses informationfrom the context patches to make predictions.
In this way, we can pass in an arbitrary number ofcanonical object views in arbitrary order without changing model parameters during inference.
Intuitively, the canonical object views can be viewed as queries where the context patches serve askeys to extract the spatial relationships’ values.
Note that the canonical object views are not cropsfrom the input image, but arbitrary views of the objects that may not match the objects’ appearancein the scene. Our model learns to identify objects even under drastic change in lighting, pose andocclusion.
Fig. 3 shows some examples of canonical object views used in our experiments.
3.2 Predicate Classiﬁer
[Figure] The predicate classiﬁer (Fig. 4) is responsible for predicting a list of predicates using object embeddings.
The predicates can be logical statements about spatial relationships, e.g., whether the blueblock is on the left part of the table, or continuous quantities such as which direction should theend effector move to reach the red block.
The predicate classiﬁer consists of a collection of 2-layer MLPs, one for each type of relationship.
Here we focus on unary and binary predicates.
Unarypredicates involve a single object or an object and the environment, which could be the robot ora region on the table.
Binary predicates involve two objects and, optionally, the environment.
Inprinciple, our framework is extensible to predicates involving more than two objects, but we leavethat for future work.
The MLP for unary predicates takes the list of object embeddings produced by SORNet and outputspredicates pertaining to the object that the embedding is conditioned on.
Taking the top is clearclassiﬁer for an example, if the input embedding is conditioned on the blue block, the MLP willoutput whether there is any object on top of the blue block.
If the input embedding is conditionedon the red block, the MLP will output whether there is any object on top of the red block.
The MLP for binary predicates takes a list of binary object embeddings created by concatenatingpairs of object embeddings produced by SORNet and outputs predicates corresponding to a pair ofobjects, e.g., whether the blue block is on top of the red block.
Thus, with N object embeddings,there will be N ( N − 1) binary object embeddings.
Parameters of the predicate classiﬁer are independent of the number of objects.
The number ofoutput predicates dynamically changes with the number of input object embeddings.
For example,when are 7 unary MLPs and 2 binary MLPs, with 4 objects, the model predicts 7 × 4+2 × 4 × (4 − 1) =
52 predicates; with 5 objects, the model predicts 7 × 5 + 2 × 5 × (5 − 1) = 75 predicates.
In thisway, our model generalizes zero-shot to scenes with an arbitrary number of objects.
The Leonardo Dataset
To test the generalization of our model to new objects and tasks, we created a simulated tabletopenvironment named Leonardo, where a Franka Panda robot manipulates a set of randomly coloredblocks.
The robot is given a goal formulated as a list of predicates to be satisﬁed, and then use asimple task planner [5] to ﬁnd a sequence of actions to achieve that goal and a sample-based motionplanner [31] to generate trajectories and choose grasps.
As we know the ground truth poses of theblocks in the simulator, we can compute ground-truth logical predicates at every step of the planning process.
We used NVISII [32] to render the RGB observations.
Domain randomization includingrandom lighting, background and perturbations to the camera position is applied while rendering.
The training data contains 133796 sequences of a single task - stacking 4 blocks in a tower.
Theblock colors are randomly chosen from 405 xkcd colors 1 .
The testing data contains 9526 sequenceswith 4-7 blocks, chosen from colors that are not included in the training data: red, green, blue,yellow, aqua, pink, purple . The testing tasks consists of 7 tasks different from training.
Fig.
5 showssome examples.
Please see the supplement for full description of the testing tasks.
Results
[Figure]
5.1 CLEVR-CoGenT
[Figure] We ﬁrst evaluate our approach on a variant of the CLEVR dataset [24], a well established benchmarkfor visual reasoning.
CLEVR contains rendered RGB images with at most 10 objects per image.
There are 96 different objects in total (2 sizes, 8 colors, 2 materials, 3 shapes).
Each image islabeled with 4 types of spatial relationships (right, front, left, behind) for each pair of objects.
Speciﬁcally, we use the CoGenT version of the dataset, which stands for Compositional Generalization Test, where the data is generated in two different conditions.
In condition A, cubes are gray,blue, brown, or yellow and cylinders are red, green, purple, or cyan.
Condition B is the opposite:cubes are red, green, purple, or cyan and cylinders are gray, blue, brown, or yellow.
Spheres can beany color in both conditions.
The models are trained on condition A and evaluated on condition B.
The training set (trainA) contains 70K images and the evaluation set (valB) contains 15K images.
Several prior works [33, 34] show signiﬁcant generalization gap on CLEVR-CoGenT caused by thevisual model learning strong spurious biases between shape and color.
We generate a question for each spatial relationship in the image, e.g. “ Is the large red rubber cubein front of the small blue metal sphere? ”
We ﬁlter out any query that is ambiguous, e.g. if therewere two large red cubes, one in front and one behind the small blue sphere.
This results in around2 million questions for both valA and valB sets.
We compare against MDETR [34], which reportsstate-of-the-art zero-shot result on CLEVR-CoGenT, i.e. there is no ﬁne-tuning on any examplefrom condition
B. The results are summarized in Table 1.
Our model performs drastically betteron
classifying spatial relationships of unseen objects and shows a much smaller generalization gapbetween valA and valB sets.
Unlike MDETR which takes text queries, our model takes visual queries in the form of canonicalobject views, and the set of questions we generate are different from the ones provided with CLEVRCoGenT. To eliminate the inﬂuence of those factors, we report the performance of the MDETRmodel trained on the full CLEVR dataset, denoted as MDETR-oracle.
We can see that although
SORNet is only trained on condition A, it is able to achieve similar performance to MDETR-oracle.
[Table]
The zero-shot generalization ability of our model can potentially be combined with other reasoningpipelines to improve generalization performance on other types of queries as well.
5.2 Predicate Classiﬁcation
Next, we evaluate the task of predicate classiﬁcation on the Leonardo dataset.
We compare against 3baselines that do not use object conditioning.
The ﬁrst two baselines use a ResNet18 and a ViT-B/32respectively to directly predict 52 predicates.
The last baseline uses the same architecture as ours, butthe embedding tokens come from 4 ﬁxed class embedding vectors instead of being conditioned onthe canonical object views. We report 3 metrics: accuracy, F-1 score and all-match accuracy for allpredicates as well as predicates within a certain category.
The accuracy and F-1 score are computedper predicate and averaged.
For all-match accuracy, we consider the output predicates as vectors.
If a single predicate is wrong, we consider the prediction for the entire category to be wrong.
Theall-match accuracy most accurately reﬂect real world performance of the predicate classiﬁer, sincethe planner uses a vector of predicates instead of a single predicate.
As mentioned in Sec. 4, the models are tested on images where the objects are completely unseenduring training.
No ﬁne-tuning on the test data is performed.
All models except the single-view
SORNet model are trained on 3 different views of the scene, which serves as a sort of data augmentation.
During testing, we aggregate the predictions from 3 views by adding the logits to minimizethe effect of occlusion.
We denote this multi-view prediction by M-View in Table 2.
We alsoconcatenate the gripper state to the object embedding in a variant of SORNet , denoted by (G) in
Table 2.
The opening/closing of the gripper is hard to see visually but provides useful informationfor predicate classiﬁcation.
The results are summarized in Table 2.
We can see that the non-object-conditioned baselines faildrastically when applied to unseen objects zero-shot. Even after ﬁne-tuning on 100 examples, theystill signiﬁcantly underperform zero-shot SORNet.
This demonstrates the generalizability of ourmodel obtained via conditioning on canonical object views.
Further, we tested our model on scenes with 5 to 7 objects, while it has only been trained on 4-objectscenes.
The performance of our model drops for some types of predicates (e.g. stacked) but remainsthe same for most predicates.
Note that the number of binary predicates increases quadraticallywith the number of objects in the scene, which also increases complexity of the scene.
None ofthe baselines can even be applied to these scenes with more objects without introducing additionalmodel parameters and retraining the model.
Finally, we run our best performing model on 30 real-world images of a robot performing variousmanipulation tasks.
The quantitative results are in Table 2 and Fig.
6 shows two qualitative examplesof the predicates predicted by our model.
Our model transfers to real-world without losing muchaccuracy.
It does make some mistakes when encountered with novel scenarios never seen duringtraining, such as one block stacked in between two blocks (right plot in Fig. 6).
5.3 Skill Executability
We further evaluate the predicate prediction in the context of task planning.
Each frame in the
Leonardo dataset is labeled with a primitive skill that will be executed, e.g. grasp the red block.
Each skill has a list of preconditions that needs to be satisﬁed before it can be executed, whichcan be formulated as a vector of predicate values.
In other words, if all of the predicates in thepreconditions are classiﬁed correctly, we can determine the executability of that skill correctly.
In
Table 3, we report the accuracy for classifying the executability of skills in the Leonardo test setusing the predicate predictions. [Table] [Table]
[Table]
This evaluation puts more emphasis on the predicates relevant to the manipulation of objects, e.g.approach , aligned , which are rarely true in the training data.
Our model is able to identify thesepredicates correctly whereas the baselines fail completely on skills relevant to object manipulation,i.e. grasp , align , place and lift .
5.4 Open Loop Planning
In this experiment, we incorporate SORNet as a part of an open-loop planning pipeline.
Speciﬁcally,given an initial frame, we use the predicates predicted by SORNet M-View (G) to populate a statevector.
A task and motion planner takes the state vector and desired goal (formulated as a list ofpredicate values to be satisﬁed), and outputs a sequence of primitive skills.
The robot then executesthis sequence of skills in an open loop fashion.
This demonstrate that how SORNet can be appliedto sequential manipulation of unseen object in a zero-shot fashion, i.e. without any ﬁne-tuning onthe test objects.
Please refer to the supplement for the demo video.
5.5 Relative Direction Prediction
Although the training objective of SORNet is purely logical, with large-scale pre-training, the objectembeddings learned by SORNet contains continuous spatial information.
We demonstrate this factby using SORNet embeddings to predict the relative 3D direction between entities. Speciﬁcally,we trained a regressor (same architecture as the classiﬁer in Sec. 3.2) on top of frozen SORNetembeddings to predict the continuous direction between two objects (Obj-Obj) or the direction theend effector should move to reach a certain object (EE-Obj).
The regressor is trained using L2 losson a 1000 examples with unseen objects.
The results are summarized in Table 4. SORNet is able to outperform models trained from scratchwith pose supervision (ResNet18) as well as models initialized with weights from large-scalelanguage-image pretraining [35] (CLIP-ViT).
This demonstrates that our representation learningtechnique is more suited to manipulation scenarios where more precise spatial information is crucial. In our supplement video, we showed that by predicting the direction for movement in an onlinefashion, our model can be used for visual servoing to guide the robot to reach a target object.
Conclusion
We proposed SORNet ( S patial O bject-Centric R epresentation Net work) that learns object-centric representations from RGB images.
We show that the object embeddings produced by SORNet capture spatial relationships which can be used in a downstream tasks such as spatial relationship classiﬁcation, skill precondition classiﬁcation and relative direction regression.
Our method workson scenes with an arbitrary number of unseen objects in a zero-shot fashion.
With real-world robot experiments, we demonstrate how SORNet can be used in manipulation of novel objects.