SORNet: Spatial Object-Centric Representations for Sequential Manipulation
//wentaoyuan.github.io/sornet
Abstract: Sequential manipulation tasks require a robot to perceive the state of an environment and plan a sequence of actions leading to a desired goal state, where the ability to reason about spatial relations among object entities from raw sensor inputs is crucial.
Prior works relying on explicit state estimation or end-toend learning struggle with novel objects or new tasks.
In this work, we propose SORNet ( S patial O bject-Centric R epresentation Net work), which extracts objectcentric representations from RGB images conditioned on canonical views of the objects of interest.
We show that the object embeddings learned by SORNet generalize zero-shot to unseen object entities on three spatial reasoning tasks: spatial relation classiﬁcation, skill precondition classiﬁcation and relative direction regression, signiﬁcantly outperforming baselines.
Further, we present real-world robotic experiments demonstrating the usage of the learned object embeddings in task planning for sequential manipulation.
Introduction
A crucial question for complex multi-step robotic tasks is how to represent relations between entities in the world, particularly as they pertain to preconditions for various skills the robot might employ.
In goal-directed sequential manipulation tasks with long-horizon planning, it is common to use a state estimator followed by a task and motion planner or other model-based system [1, 2, 3, 4, 5, 6].
A variety of powerful approaches exist for explicitly estimating the state of objects in the world, e.g. [7, 8, 9, 10].
However, it is challenging to generalize these approaches to an arbitrary collection of objects.
In addition, the objects are often in contact in manipulation scenarios, where works explicitly addressing the problem of generalizing to unseen objects [11, 12] still struggle.
Fortunately, knowing exact poses of objects may not be necessary for manipulation.
End-to-end methods [13, 14, 15] leverage that fact and build networks that generates actions directly from sensor inputs without explicitly representing objects.
Nevertheless, these networks are very speciﬁc to the tasks they are trained on.
For example, it is non-trivial to use a network trained on stacking blocks to unstack blocks.
In this work, we take an important step towards a manipulation framework that generalizes zero-shot to novel tasks with unseen objects.
Speciﬁcally, we propose a neural network that extracts objectcentric embeddings from raw RGB images conditioned on object queries, which we call SORNet , or S patial O bject-Centric R epresentation Net work.
The design of SORNet allows it to generalize to novel objects without retraining or ﬁnetuning.
The object-centric embeddings produced by SORNet can be combined with readout networks to inform a task and motion planner with implicit object states relevant to goal-directed sequential manipulation tasks, e.g. logical preconditions for primitive skills or continuous 3D directions from the end effector to objects in the scene.
To summarize, our contribution are: (1) a method for extracting object-centric embeddings from
RGB images that generalizes zero-shot to different number and type of objects; (2) a framework for learning object embeddings that capture continuous spatial relations with only logical supervision; (3) a dataset containing sequences of RGB observations labeled with spatial predicates during various tabletop rearrangement manipulation tasks.
We empirically evaluate the object-centric embeddings produced by SORNet on three different downstream tasks: 1) classiﬁcation of logical predicates for action preconditions; 2) visual question and answering for spatial relations; 3) prediction of relative 3D direction between entities in manipulation scenes. In all three tasks, the models are tested on held-out objects that did not appear in training data.
In all tasks, SORNet obtains signiﬁcant improvements over the baseline methods.
Finally, we evaluated SORNet with real-world robot experiments to showcase the transfer of the learned object-centric embeddings to real-world observations.
Related Work
Sequential Manipulation In goal-directed sequential manipulation tasks with long-horizon planning, a class of work uses a pipeline approach requiring a state estimator followed by a task and motion planner [1, 2, 3, 4, 5, 6], but the state estimator with an explicit state representation is often restricted to an environment or a set of objects.
Another class of work employs neural networks to learn motor controls directly from raw sensor data, such as RGB images and joint encoder readings [14, 16, 15, 17, 18, 19], but they are often not transferable to new tasks, especially tasks involving long-horizon planning.
Our work combines a network that estimates implicit states with a symbolic planner to obtain the best of both worlds: generalization to new objects and to new tasks.
Learning Spatial Relations Learning spatial relations between object entities have been studied in the ﬁeld of 3D vision and robotics.
Methods such as [20, 21, 22] predict discrete or continuous pairwise object relations from 3D inputs such as point clouds or voxels, assuming complete observation of the scene and segmented objects with identities.
In contrast, our approach does not make any assumptions regarding the observability of the objects and does not require pre-processing of the sensor data.
The learning framework by Kase et al.
[23] is most related to our approach, which takes a sequence of sensor observations and classiﬁes a set of pre-deﬁned relational predicates which is then used by a symbolic planner to produce a suitable operator.
Compared to our approach, theirs is limited by the number of objects in the scene and a ﬁxed set of spatial predicates.
Visual Reasoning
Recently, several advancements have been made on visual reasoning benchmarks [24, 25, 26] using transformer networks [27].
Toward solving spatio-temporal reasoning task from
CLEVRER [25] and CATER [26], Ding et al.
[28] proposed an object-based attention mechanism and Zhou et al.
[29] proposed a multi-hop transformer model.
Both works assume a segmentation model to produce object segments and performs language grounding to the segments to perform reasoning.
Our SORNet architecture is simpler and can solve spatial-reasoning tasks for unseen object instances without requiring a segmentation or object detection module.
Furthermore, our work focuses on a relatively complex manipulation task domain involving manipulator in the observations.
Although our current work focuses on predicting spatial relations from a single RGB frame, the object-centric embeddings could potentially be used for solving temporal-reasoning tasks.
Methods 3.1 SORNet: Spatial Object-Centric Representation Network
Our object embedding network ( SORNet ) (Fig. 2) takes an RGB image and an arbitrary number of canonical object views and outputs an embedding vector corresponding to each input object patch.
The architecture of the network is based on the Visual Transformer (ViT) [30].
The input image is broken into a list of ﬁxed-sized patches, which we call context patches .
The context patches are concatenated with the canonical object views to form a patch sequence.
Each patch is ﬁrst ﬂattened and then linearly projected into a token vector, then positional embedding is added to the sequence of tokens.
Following [30], we use a set of learnable vectors with the same dimension as the token vectors as positional embeddings.
The positional-embedded tokens are then passed through a transformer encoder, which includes multiple layers of multi-head self-attention.
The transformer encoder outputs a sequence of embedding vectors.
We discard the embedding for context patches and keep those for the canonical object views.
We apply the same positional embedding to the canonical object views to make the output embeddings permutation equivariant.
We also mask out the attention among canonical object views and the attention from context patches to canonical object views to ensure the model uses information from the context patches to make predictions.
In this way, we can pass in an arbitrary number of canonical object views in arbitrary order without changing model parameters during inference.
Intuitively, the canonical object views can be viewed as queries where the context patches serve as keys to extract the spatial relations’ values.
Note that the canonical object views are not crops from the input image, but arbitrary views of the objects that may not match the objects’ appearance in the scene.
Our model learns to identify objects even under drastic change in lighting, pose and occlusion.
Fig.
3 shows some examples of canonical object views used in our experiments.
3.2 Readout Networks
The readout networks (Fig. 4) are responsible for predicting a list of relations using object embeddings.
The relations can be logical statements, e.g., whether the blue block is stacked onto the green block, or continuous quantities, e.g. which direction should the end effector move to reach the red block.
The readout networks consist of a collection of 2-layer MLPs, one for each type of relations.
Here we focus on unary and binary relations.
Unary relations involve a single object or an object and the environment, which could be the robot or a region on the table.
Binary relations involve two objects and, optionally, the environment.
In principle, our framework is extensible to relations involving more than two objects, but we leave that for future work.
The readout network for unary relations takes the list of object embeddings and outputs relations pertaining to the object that the embedding is conditioned on.
Taking the top is clear classiﬁer for an example, if the input embedding is conditioned on the blue block, the network will output whether there is any object on top of the blue block.
If the input embedding is conditioned on the red block, the network will output whether there is any object on top of the red block.
The readout network for binary relations takes a list of binary object embeddings created by concatenating pairs of object embeddings and outputs relations corresponding to a pair of objects, e.g., whether the blue block is on top of the red block.
Thus, with N object embeddings, there will be N ( N − 1) binary object embeddings and N ( N − 1) output relations.
Parameters of the readout network are independent of the number of objects.
The number of output relations dynamically changes with the number of input object embeddings.
For example, when are 7 unary relations and 2 binary relations, with 4 objects, the network generates 7 × 4+2 × 4 × (4 − 1) =
52 outputs; with 5 objects, the network generates 7 × 5 + 2 × 5 × (5 − 1) = 75 outputs.
In this way, our overall model generalizes zero-shot to scenes with an arbitrary number of objects.
The Leonardo Dataset
To test the generalization of our model to new objects and tasks, we created a simulated tabletop environment named Leonardo, where a Franka Panda robot manipulates a set of randomly colored blocks.
The robot is given a goal formulated as a list of predicates to be satisﬁed, and then use a simple task planner [5] to ﬁnd a sequence of actions to achieve that goal and a sample-based motion planner [31] to generate trajectories and choose grasps.
As we know the ground truth poses of the blocks in the simulator, we can compute ground-truth logical predicates at every step of the planning process.
We used NVISII [32] to render the RGB observations.
Domain randomization including random lighting, background and perturbations to the camera position is applied while rendering.
The training data contains 133796 sequences of a single task - stacking 4 blocks in a tower.
The block colors are randomly chosen from 405 xkcd colors 1 .
The testing data contains 9526 sequences with 4-6 blocks, chosen from colors that are not included in the training data: red, green, blue, yellow, aqua, pink, purple .
The testing tasks consists of 7 tasks different from training.
Fig.
5 shows some examples.
Please see the supplement for full description of the testing tasks.
Results 5.1 CLEVR-CoGenT
We ﬁrst evaluate our approach on a variant of the CLEVR dataset [24], a well established benchmark for visual reasoning.
CLEVR contains rendered RGB images with at most 10 objects per image.
There are 96 different objects in total (2 sizes, 8 colors, 2 materials, 3 shapes).
Each image is labeled with 4 types of spatial relations (right, front, left, behind) for each pair of objects.
Speciﬁcally, we use the CoGenT version of the dataset, which stands for Compositional Generalization Test, where the data is generated in two different conditions.
In condition A, cubes are gray, blue, brown, or yellow and cylinders are red, green, purple, or cyan.
Condition B is the opposite: cubes are red, green, purple, or cyan and cylinders are gray, blue, brown, or yellow.
Spheres can be any color in both conditions.
The models are trained on condition A and evaluated on condition B.
The training set (trainA) contains 70K images and the evaluation set (valB) contains 15K images.
Several prior works [33, 34] show signiﬁcant generalization gap on CLEVR-CoGenT caused by the visual model learning strong spurious biases between shape and color.
We generate a question for each spatial relation in the image, e.g. “ Is the large red rubber cube in front of the small blue metal sphere? ”
We ﬁlter out any query that is ambiguous, e.g. if there were two large red cubes, one in front and one behind the small blue sphere.
This results in aroundmillion questions for both valA and valB sets.
We compare against MDETR [34], which reports state-of-the-art zero-shot result on CLEVR-CoGenT, i.e. there is no ﬁne-tuning on any example from condition
B. The results are summarized in Table 1.
Our model performs drastically better on classifying spatial relations of unseen objects and shows a much smaller generalization gap between valA and valB sets.
Unlike MDETR which takes text queries, our model takes visual queries in the form of canonical object views (i.e. 2 canonical views for the objects mentioned in the question). To eliminate the inﬂuence of those factors, we report the performance of the MDETR model trained on the full CLEVR dataset, denoted as MDETR-oracle.
We can see that although SORNet is only trained on condition A, it is able to achieve similar performance to MDETR-oracle.
The zero-shot generalization ability of our model can potentially be combined with other reasoning pipelines to improve generalization performance on other types of queries as well. [Table]
5.2 Predicate Classiﬁcation
Next, we evaluate the task of predicate classiﬁcation on the Leonardo dataset.
We compare againstbaselines that do not use object conditioning.
The ﬁrst two baselines use a ResNet18 and a ViT-B/ respectively to directly predict 52 predicates.
The last baseline uses the same architecture as ours, but the embedding tokens come from 4 ﬁxed class embedding vectors.
We report 3 metrics: accuracy, F-1 score and all-match accuracy across all predicates and among predicates within a category.
The accuracy and F-1 score are computed per predicate and averaged.
For all-match accuracy, the predicates are considered as a single vector, which most faithfully represents how the predicates are treated by a planner.
If any predicate is classiﬁed incorrectly, the prediction for the entire category is considered as wrong.
The models are tested on images where the objects are completely unseen during training.
The results are summarized in Table 2.
Models labeled with M-View are trained on 3 different views of the scene as data augmentation.
During testing, we aggregate the predictions from 3 views by adding the logits.
The M-View models are better at handling occlusions by leveraging multi-view information.
The gripper state is concatenated to the object embedding in the SORNet variant denoted by (G) in Table 2, which is better at classifying predicates correlated to the opening/closing of the gripper, e.g. has obj .
We can see that the non-object-conditioned baselines fail drastically when applied to unseen objects zero-shot.
Even after ﬁne-tuning on 100 examples, they still signiﬁcantly underperform zero-shot SORNet.
This demonstrates the generalizability of our model obtained via conditioning on canonical object views.
Further, we tested our model on scenes with 5 to 6 objects, while it has only been trained on 4-object scenes.
We ﬁrst treat the additional objects as distractors, which shows that it is not necessary to acquire canonical views of every object in the scene if they are not of interest.
We then treat all objects as objects of interest. In this case, the number of binary predicates increases quadratically with the number of objects in the scene, so the all-match accuracy naturally drops even if the average accuracy and F-1 score remain the same.
Note that none of the baselines can even be applied to these scenes with more objects without introducing additional model parameters and retraining the model.
Finally, we run our best performing model on 30 real-world images of a robot performing various manipulation tasks.
The quantitative results are in Table 2 and Fig.
6 shows two qualitative examples of the predicates predicted by our model.
Our model transfers to real-world without losing much accuracy.
It does make some mistakes when encountered with novel scenarios never seen during training, such as one block stacked in between two blocks (right plot in Fig. 6).
5.3 Skill Executability
We further evaluate the predicate prediction in the context of task planning.
Each frame in the
Leonardo dataset is labeled with a primitive skill that will be executed, e.g. grasp the red block.
Each skill has a list of preconditions that needs to be satisﬁed before it can be executed, which can be formulated as a vector of predicate values.
In other words, if all of the predicates in the preconditions are classiﬁed correctly, we can determine the executability of that skill correctly.
In
Table 3, we report the accuracy for classifying the executability of skills in the Leonardo test set using the predicate predictions.
This evaluation puts more emphasis on the predicates relevant to the manipulation of objects, e.g. approach , aligned , which are rarely true in the training data.
Our model is able to identify these predicates correctly whereas the baselines fail completely on skills relevant to object manipulation, i.e. grasp , align , place and lift .
[Table]
[Table] [Table]
[Table] [Table] [Table]
5.4 Open Loop Planning
In this demo [Table] we incorporate SORNet as a part of an open-loop planning pipeline in a real-world manipulation scenario.
Speciﬁcally, given an initial frame, we use the predicates predicted by SORNet M-View (G) to populate a state vector.
A task and motion planner takes the state vector and desired goal (formulated as a list of predicate values to be satisﬁed), and outputs a sequence of primitive skills.
The robot then executes this sequence of skills in an open loop fashion.
This demonstrate that how SORNet can be applied to sequential manipulation of unseen object in a zero-shot fashion, i.e. without any ﬁne-tuning on the test objects.
Please refer to our online video for the demo.
5.5 Relative Direction Prediction
Although the training objective of SORNet is purely logical, with large-scale pre-training, the object embeddings learned by SORNet contains continuous spatial information.
We demonstrate this fact by using SORNet embeddings to predict the relative 3D direction between entities.
Speciﬁcally, we trained a regressor (same architecture as the classiﬁer in Sec. 3.2) on top of frozen SORNet embeddings to predict the continuous direction between two objects (Obj-Obj) or the direction the end effector should move to reach a certain object (EE-Obj).
The regressor is trained using L2 loss on a 1000 examples with unseen objects, and tested on 3000 examples with the same objects.
The results are summarized in Table 4.
SORNet is able to outperform models trained from scratch with pose supervision (ResNet18) as well as models initialized with weights from large-scale language-image pretraining [35] (CLIP-ViT).
This demonstrates that our representation learning technique is more suited to manipulation scenarios where more precise spatial information is crucial.
In our online video, we showed that by predicting the direction for movement in an online fashion, our model can be used for visual servoing to guide the robot to reach a target object.
Conclusion
We proposed SORNet ( S patial O bject-Centric R epresentation Net work) that learns object-centric representations from RGB images.
We show that the object embeddings produced by SORNet capture spatial relations which can be used in a downstream tasks such as spatial relation classiﬁcation, skill precondition classiﬁcation and relative direction regression.
Our method works on scenes with an arbitrary number of unseen objects in a zero-shot fashion.
With real-world robot experiments, we demonstrate how SORNet can be used in manipulation of novel objects.
Acknowledgments
This work was supported in part by the National Science Foundation under Contract NSF-NRI2024057 and in part by Honda Research Institute as part of the Curious Minded Machine initiative.