Predicting Stable Conﬁgurations for Semantic Placement of Novel Objects
Abstract: Human environments contain numerous objects conﬁgured in a variety of arrangements.
Our goal is to enable robots to repose previously unseen objects according to learned semantic relationships in novel environments.
We break this problem down into two parts: (1) ﬁnding physically valid locations for the objects and (2) determining if those poses satisfy learned, high-level semantic relationships.
We build our models and training from the ground up to be tightly integrated with our proposed planning algorithm for semantic placement of unknown objects.
We train our models purely in simulation, with no ﬁne-tuning needed for use in the real world.
Our approach enables motion planning for semantic rearrangement of unknown objects in scenes with varying geometry from only RGBD sensing.
Our experiments through a set of simulated ablations demonstrate that using a relational classiﬁer alone is not sufﬁcient for reliable planning.
We further demonstrate the ability of our planner to generate and execute diverse manipulation plans through a set of real-world experiments with a variety of objects.
[Figure]
Introduction & Motivation
Many robotics tasks in human environments involve reasoning about the relationships between different objects and their locations in a particular environment.
Imagine a robot tasked with pouring a cup of coffee, it must reason about the relative position of the cup to the coffee pot, such that it pours the coffee into the cup and not onto the counter or ﬂoor.
For a robot attempting to brew the coffee prior to pouring, many more multi-object relations must be reasoned about for the tasks of retrieving the necessary equipment from storage, scooping and pouring beans or grounds, and ﬁlling any required water vessels.
Classically, researchers investigating these sorts of multi-object planning and manipulation problems assume full knowledge of the objects and their poses in the scene [1].
However, for robots acting in homes and other open-world environments, it is unreasonable to assume that a deployed robot will have knowledge and accurate pose estimation of all objects in the environment.
Furthermore, simple hand-engineered classiﬁers for determining logical predicates often exhibit unintended behavior when deployed on real-world systems [2].
For these reasons, rearrangement has recently been identiﬁed as a major challenge for robotics [3].
Given these challenges, its natural to examine the use of learning to improve task and motion planning with real sensing [4, 5, 6, 7, 8, 9].
However, previous methods fail to solve the full problem of unknown object rearrangement with physical robots.
Some only operate on known objects [6, 9], others ignore or signiﬁcantly restrict the space of robot control [4, 5] or relations [7, 8], while still others make assume an explicit goal conﬁguration is given [5].
An alternative approach to solve complex manipulation tasks relies on learning model-free neural net policies instead of explicit models of conditions and effects [10, 11].
Such methods have so far failed to show the level of generalization across objects and environments capable by modern task and motion planners.
In this paper, we focus on a critical subtask of rearrangement planning– semantic placement –where the robot must perform a pick and place operation to move an object into a stable conﬁguration that satisﬁes a desired set of semantic relations.
This allows us to focus on three speciﬁc objectives not addressed in previous work on learning for task and motion planning.
First, we examine the problem of planning and controlling manipulation behaviors to change inter-object relations of potentially previously unseen objects using RGB-D sensing.
Second, we aim to learn the necessary relations from data to avoid the bias introduced from hand-engineered classiﬁers.
Finally, we must learn what realistic scenes look like–so that the robot can ensure that it places objects in reasonable locations which are stable and free of undesired collisions.
Our work relies on the ability to infer inter-object relational predicates between objects proposed from an RGB-D instance segmentation, e.g. [12].
Semantic relations serve an important role in instructing robots [13].
As such, researchers have examined visual prediction of spatial relations [14, 15, 16, 17] including inference of support relations [18, 19].
While these methods can be sophisticated incorporating language [20, 21, 16] or scene graph information [22, 23, 17]; none have demonstrated the ability to integrate prediction with robot planning and control to satisfy new relations between unknown objects.
Indeed, we show semantic prediction alone is insufﬁcient for reliably planning successful semantic rearrangement.
In addition to this semantic prediction, we need to identify physically realistic poses where these objects can be placed.
Stable placement prediction has also received attention from robotics researchers [24, 25, 26, 27]; however, learning-based approaches [24, 27] train directly to maximize stability of object placement.
In contrast our approach simply distinguishes realistic conﬁgurations from invalid ones, which allows us to learn a general-purpose scene-realism discriminator which can capture wide distributions over realistic poses in 3D space.
This provides the beneﬁt of a more general model for use in arrangement planning, while giving up somewhat the level of precision seen in some placement-speciﬁc methods [24, 26].
In addition, it means we can use simulationbased data to train both our relational prediction and scene discriminator directly on raw point cloud and associated segmentation masks, and transfer to the real world.
We embedded our learned relational predictor and scene discriminator within a sampling-based planning framework to change relations in the scene in a goal directed way.
By planning directly over changes in segment pose in a point cloud, we can decouple the goal generation problem from the robot control problem, allowing us to leverage model-based, state-of-the-art motion planning and grasp prediction algorithms [28, 29] to perform the necessary manipulation.
We further accelerate our planner by learning an object pose sampler conditioned on desired relations to initialize the optimization, similar to the grasp planning approach in [28].
Production of these goal states can then be used for semantically-deﬁned placement tasks.
We highlight the advantages of our approach over a variety of baselines and ablations of our full method.
Critically, we demonstrate that both the relational classiﬁer and scene discriminator are necessary for reliably generating successful plans.
We then demonstrate our approach in the real world using a Franka robot (Fig. 1).
Our experiments constitute the ﬁrst physical-robot demonstration that combine learned models for inter-object relations and stability estimation enabling rearrangement of novel objects.
Crucially, we show that the combination of relationship classiﬁer and scene discriminator allows us to plan placements for a variety of relationships in cluttered scenes.
In addition, our models are trained entirely in simulation with no need for real-world ﬁne tuning
Methods
Given a single view of a scene containing objects for which our robot potentially has no previous experience, we wish for the robot to rearrange the scene to satisfy some new set of logical constraints.
For example the robot may be tasked to move the query object i to be on the far right side of anchor object j or to be stacked on top of object k .
Each individual relationship between i and j is referred to as a predicate ρ ij ; we can describe multiple logical relationships as the vector (cid:126)ρ ij .
We assume we are given a partial-view point cloud Z with segment labels for each point to identify the different objects.
Given this point cloud and a set of logical predicates describing the desired relationships (cid:126)ρ ij , the robot must ﬁnd a pose offset δ (3D translation and planar rotation) for object i that satisﬁes (cid:126)ρ ij and is additionally a stable, physically valid placement pose in the environment.
Thus there are two key components in our rearrangement and placement planning approach: predicting which poses objects can be physically placed and predicting which poses satisfy the given high-level instructions.
We formalize predicate planning as the following problem:
[Equation]
where x i and x j are the object point clouds for objects i and j , respectively, and ⊕ denotes the application of the 3D translation and planar rotation.
At the heart of our planning cost, Eq. 1, are two models.
The ﬁrst f ( x i ⊕ δ, Z (cid:48) ) is a neural net trained to determine if the resulting scene is physically realistic and stable.
The second term deﬁnes the cost associated with matching the set of target predicates, where p ρ ( x i ⊕ δ, x j ) estimates the set of predicate relationships between the transformed point cloud x i ⊕ δ and x j .
This cost implies maximizing the likelihood that the resulting scene is both realistic and satisﬁes the desired predicates.
In addition to the cost function, we put minimum bound constraints on the physical feasibility (Eq. 3) and predicate probabilities (Eq. 4).
We use (cid:126)ρ ij as an index in Eq.
4 to extract the subset of predicted relations that must be satisﬁed.
This ensures we never attempt to plan to a scene conﬁguration with low probability of success, even if it deﬁnes a local optimum of the objective.
Eq.
2 models the transition of applying offset δ to x i in the observed scene Z to generate the resulting scene Z (cid:48) , which we evaluate in the cost and other constraints.
Finally Eq.
5 ensures that sampled object offsets are visible in the camera view of the robot, since our cost and constraint evaluations would be illdeﬁned otherwise.
We describe the details of these models and their construction in Sec. 2.1 and give a detailed description of our data generation process for training in Section 2.3.
We use a variant of the cross-entropy method (CEM) [30] to solve this constrained optimization problem.
CEM has previously been applied to robot motion planning [30], including semantic motion planning from learned models [31].
We provide further details of our planner in Section 2.2.
2.1 Rearrangement Planner Models and Training
We train multiple neural networks to compute the values needed to instantiate our planning problem deﬁned by Equation 1: the current set of predicates, and the discriminator score which describes whether or not a particular set of object points x deﬁnes a realistic conﬁguration in the scene.
Object and scene encoders The core piece of the model is a PointNet++ [32] encoder which extracts a lower-dimensional object representation h .
Each network takes two objects i and j represented as point clouds x i and x j , as well as the scene point cloud Z , as input.
Given an observation point cloud x , we learn a mapping e ( x ) → h for the objects, in order to get two latent representations h i and h j for the query and anchor objects.
The object encoder is a Pointnet++ model with three set abstraction layers; see the supplemental material for further details.
We train a separate scene encoder e Z ( x Z ) to capture the objects’ relation to other scene geometry when predicting where it can be placed.
This outputs h Z encoding scene-speciﬁc information, where x Z is centered around the anchor point o j – the centroid of x j .
Points are sampled in a radius of 𝝅 𝒑 ρ 𝝆 ෝ𝝆 𝒉 𝒊 𝒉 𝒋 𝒉 𝒁
[Figure]
Pointnet Encoders
[Figure]
𝒊 𝒋 𝒁 r = 0 .
5 m around o j .
We choose this representation to deﬁne features from the ﬁxed perspective of the anchoring object j .
Relation predictor The predicate classiﬁer network p ρ ( x i , x j ) → ˆ ρ ij estimates which predicates are true for a pair of object point clouds.
It uses the representations from the object encoder, e ( x i )
=
h i and e ( x j ) =
h j , for the query and anchor objects i and j , respectively.
These are passed into an
MLP which predicts a vector of length N predicates .
When used as part of the planning algorithm, we ignore predicted predicates that are not part of the goal speciﬁcation.
Pose prior In addition, we learn a prior distribution over possible poses where the object might satisfy the predicates, relative to the anchoring object.
We implement this prior π ( x i , x j , Z, (cid:126)ρ ij ) → { α k , µ k , σ k } k =1: K as a Mixture Density Network (MDN) [33], which predicts the parameters of a Gaussian mixture model with K components.
This GMM distribution deﬁnes the probability of a speciﬁc pose offsets δ with respect to the anchor point o j .
The relationships are assumed to be spatially deﬁned relative to this object, so that the predicted center of the query object i is ˆ o i
=
o j ⊕ δ .
Once trained we can produce samples from the MDN by evaluating it for the current observations and then applying standard GMM sampling using the output parameters.
As with the relation predictor p ρ , the pose prior uses the object encoder to get lower-dimensional representations h i and h j for each object as well as the scene encoder to predict h Z .
Scene discriminator
The relation predictor and prior on their own are not enough to ﬁnd stable placement poses.
As such, we deﬁne a discriminator network f
( x, Z ) trained to predict if a given conﬁguration results in a placement that is physically realistic w.r.t.
the training data, i.e. it is stable.
This model uses a slightly different architecture from the above models, and unlike them we ﬁrst center the scene on ˆ
o i , the query point and potential new pose for object i .
We look at the local region around ˆ o i with a ﬁxed radius r = 0 .
5 m to classify whether a particular δ would result in a realistic placement pose.
We use a single PointNet++ model to encode points from both objects together, given a label indicating which points belong to the query object (the object whose placement we are attempting to ﬁnd).
In practice, the sphere-query necessary to extract the local context around a particular pose is the same as that used in PointNet++, so this operation can be performed quickly at inference time.
Transform operator We require one additional operator to plan on point clouds: the transformation operator T ( Z, x i ⊕ δ ) .
We assume a deterministic transition function and rigidly transform points associated with the query object according to the relative pose offset δ constructing a new scene Z (cid:48) .
Model Training We jointly train the object and scene encoders e and e Z , relationship predictor p ρ , and the pose prior π .
The relationships can be directly supervised from our training data, given knowledge of the ground truth predicates, and are trained with a binary cross entropy loss.
The prior π is trained to predict the offset δ from the anchor point o j to the observed pose of the query object, o i , in the training data; i.e. δ =
o j (cid:9) o i .
To train the scene discriminator, we ﬁrst note that all of our training data consists of stable object placements, thus there is no negative data nor stability supervision available.
Instead, we create negative data online by applying random δ offsets to the pose of the query object i .
We simply sample a random δ and apply our transform operator to create a new scene Z − =
T ( Z, x i ⊕ δ ) .
These δ were sampled to be between 2 and 15 centimeters in a random direction.
The resulting scenes are highly likely to not be physically realistic nor stable (e.g. object i is ﬂoating or is in collision).
Algorithm
1: function F IND P LACEMENT (object point cloud x i , object point cloud x j , scene Z , goal (cid:126)ρ ij ) 2: for s ∈ range (1 , N ) do 3: (cid:126)δ s = ∅ 4: while length ( (cid:126)δ s ) <
B do (cid:46)
Rejection Sampling 5: if s = 0 then 6: δ ∼ π ( x i , x j , Z, (cid:126)ρ ij ) (cid:46) Sample initial poses from learned prior 7: else 8: δ ∼ N ( µ (cid:48) , Σ (cid:48) )
(cid:46)
Sample subsequent poses from surrogate distribution 9:
Z (cid:48) ← T ( Z, x i ⊕ δ ) (cid:46) Shift object point clouds by δ 10: realistic ← f ( x i ⊕ δ, Z (cid:48) )
> (cid:15) f
(cid:46) Determine if pose is realistic 11: goal ← p ρ ( x i ⊕ δ, x j )
[ (cid:126)ρ ij ] > (cid:15) ρ (cid:46) Classify if goal predicates are true 12: in view ← Π( x i ⊕ δ ) (cid:46)
Ensure it will be visible 13: if realistic and goal and in view then 14: (cid:126)δ s ← (cid:126)δ ∪ δ (cid:46) Add new δ to batch of samples 15:
Compute cost c ( δ ) for each δ ∈ (cid:126)δ s 16:
Sort (cid:126)δ s and take top N elite 17:
Fit surrogate distribution parameters ( µ (cid:48) , Σ (cid:48) ) 18: return lowest-cost δ seen so far, or ∅
2.2 Manipulation Planning with Relationship Models
We now describe how we solve the optimization problem for Equation 1 using the components deﬁned above.
Alg.
1 describes how we can place an object so as to satisfy a particular relationship.
We take as given a set of desired relationships (cid:126)ρ ij in scene Z , and target objects i and j , where i is the query object that we will be moving and j is the anchor object that will be kept stationary.
Initially, we perform rejection sampling to draw a batch of B candidate pose offsets from our MDN prior π ( x i , x j , Z, (cid:126)ρ ij ) (line 6) keeping only those that satisfy the full set of constraints (lines 10–14).
We sample until either a time budget has been reached or B samples have been successfully drawn.
[Figure]
For each sample δ we compute a new query point cloud using the transform function Z (cid:48) =
T ( Z, x i ⊕ δ ) (line 9) and thus satisfy the constraint in Eq. 2 by construction.
We then evaluate the remaining constraints in Eq.
3–5 (lines 10–12) accepting only feasible samples (lines 13–14).
Next, as per the cross-entropy method, we evaluate the cost of the valid samples (line 15) and ﬁt a surrogate distribution to the best scoring N elite samples with mean µ (cid:48) and variance Σ (cid:48) (lines 16–17).
At each subsequent step, we draw B samples δ ∼ N ( µ (cid:48) , Σ (cid:48) ) from our current surrogate distribution, compute the scores again, and re-sample, until we have performed N sampling iterations.
Fig.
3 gives an example for how the planning algorithm works in practice to achieve the goal that the yellow object be aligned to the right of the cyan object.
In both Fig.
2 and Fig.
3, yellow objects are predicted positions of the query object; cyan represents the anchor object, and red points represent downsampled scene geometry.
This is an accurate depiction of the inputs into our model.
The ﬁnal frame of Fig.
3 has been rotated to show the precise alignment with the table surface.
2.3 Dataset Creation
We generated a large-scale dataset in simulation of RGB-D images with associated segmentation masks and relational predicates.
We provide binary labels between all visible pairs of objects for all predicates listed in Table 2.
Each scene consists of 3 to 7 random Shapenet [34] objects in stable conﬁgurations on various surfaces, including in stacks.
We include mugs, bowls, and bottles, as well as boxes and cylinders of random sizes.
Examples of generated scenes are shown in the supplemental material.
Objects were placed in random conﬁgurations, and we ran physics forward to ﬁnd stable arrangements.
We then rendered images both with and without each object.
In order to train the rotation model, we render each object on its own and apply a random rotation.
Directional predicates ( left of , right of , etc.) are computed based on bounding box overlap; others are computed based on distance between meshes or ground-truth position and orientation.
See the supplementary materials for further details.
Experiments
We ﬁrst performed a set of simulation experiments on scenes that resembled our real world objects examining different versions of our model.
First, we show an ablation test of our method on heldout YCB objects [35], with a set of known grasps from Eppner et al.
[36], in order to show that our algorithm with discriminator is better able to ﬁnd stable positions for objects and scenes that did not appear in our training data.
Second, we show a break-down of the predicate results, showing that our learned model is comparable or better at capturing a wide range of difﬁcult relationships even in partially-occluded scenes.
We sampled 100 random scenes in the kitchen environment from Fig.
4, with objects either positioned on top of the counter or in the top drawer.
These objects did not appear in the training set.
Table 1 shows a breakdown of several different versions of the planner after running experiments on 100 random scenes, each with a random predicate goal chosen from (in front, behind, left, right).
Due to the random placements of the objects, not all scenes are feasible, and in many cases the goal pose would be occluded or off of the table to the front, resulting in challenging planning problems.
[Table]
The different baselines look at the effects of the discriminator model, which determines whether or not a scene is realistic and whether or not an object can be placed at a particular pose.
For example, the “no discriminator” case does not use the discriminator at all, and is very good at ﬁnding poses matching the predicate goal but not ﬁnding stable poses.
We also vary the weight of the discriminator λ f in several examples.
For these experiments we use a batch size, B , of 100 .
We ran two experiments that do not use the discriminator in our “full” planning approach.
Mean only draws samples only from the mean of the MDN prior π .
This looks at what performance is like with a learned policy.
MDN Prior uses the learned mixture density function as a cost function in place of using the discriminator, since presumably this might capture much of the same information.
We can see that it actually does a fairly good job at matching the predicates, but is not very discriminative when it comes to ﬁnding stable poses for placement.
Both of these perform notably worse at ﬁnding stable poses in our test environments.
[Table]
[Table]
[Table]
3.1 Scene Discriminator Performance
Here, we examine the performance of our scene discriminator.
To do this
[Table]
we compare placement poses sampled from the MDN prior distribution in randomly-generated kitchen scenarios to placement scores after optimization.
We place the object at the new pose and then run 500 simulation steps to allow the object to settle into its ﬁnal pose.
We then compare the discriminator’s conﬁdence score with how much the object moved.
For these experiments we generated 100 random scenes, each with a random predicate goal so as not to bias it to a particular subset of the problem space.
We ignore scenes if no feasible pose was found according to the discriminator.
We found that in 95% of these scenarios, the discriminator was able to ﬁnd a stable pose to place a particular object, and in 90% of all scenarios the planner was also able to match the speciﬁed predicates.
This shows that not only can we ﬁnd realistic positions, but that the discriminator does not preclude achieving speciﬁed goals.
Note the higher perceived success rates than in the planner comparison in Table 1: this is because we only test scenes where the planner was initially conﬁdent that it could ﬁnd a solution.
[Figure]
Realism vs. Observed Stability
Fig.
4 (left) shows the relationship between the distance the object moved once placed and the output of the scene discriminator model, after rejecting a small number of outliers (distance ≥ 0. meters).
When we compute Pearson’s correlation coefﬁcient r , we see that for a single random sample s , r s = − 0 .
147 with p s = 0 . 161 , and for the ﬁnal predicted placement P , r P = −
.with
p P = 0 .
Individual initial samples s are shown in blue and optimized placements in red.
These results indicate that our scene discriminator learns a metric which correlates with stability, even in challenging, cluttered scenes.
Failures of the discriminator are driven by oddly perceived object or environment geometry.
When parts of the object geometry are missing, the model is prone to making mistakes, such as placing an object so that it intersects with another object, or placing an object so that it is not properly supported, as in shown in Fig.
4 (right).
3.2 Predicate Performance
Table 2 shows the F1 score by predicate on a held-out portion of the dataset containing 3000 examples.
Our model correctly predicates all predicates (minus aligned ) 69.4% of the time, vs. 58.5% of the time for the baseline.
In addition, we learn predicates which are difﬁcult to write manual rules for, such as aligned , on which we have a true positive rate of 83% ( f 1 = 0 . 43 ).
Generally, our model classiﬁes different relationships well, matching or exceeding the baseline in every case but one.
In particular, our learned model has consistently good sensitivity , meaning that we can ﬁnd valid positions for the object when attempting to plan.
Finally, the rule-based approach requires implementing and hand-tuning a range of different predicates; by learning from data, we could in principle scale to a larger number of symbolic relationships.
Our results show that even with very unbalanced, “natural” data, we can learn useful classiﬁers for a range of relationships.
Baseline implementation : First, we compute bounding boxes and means from each object point cloud.
For the centered predicate, we compute xy distance on the table and use a larger threshold that appears in our dataset (2 mm instead of 1 mm).
For touching , we compute minimum distance between point clouds and threshold it with 2.5mm.
For near , the threshold is a 5 cm distance.
For directional predicates, we apply the exact same rules used in data generation to the computed bounding boxes, as seen in the supplementary materials.
3.3 Real World Experiments
Finally, we deployed our system on a Franka Panda robot with an arm-mounted RGB-D camera.
We used objects from the YCB object set [35] augmented with toy kitchen objects.
We also used various cardboard boxes to force the robot to adapt to changes in scene geometry.
We generate grasps using [29] and use RRT-Connect [37] for motion planning.
We used unseen object instance segmentation from [12] to determine segmentation masks for different objects, and use a prompt to choose which object to move when performing experiments.
We compute standoff poses for each object 10cm above the predicted goal position, and release it 2cm above the predicted pose in order to ensure that we do not press into the table.
Figure 1 shows example pairs of before-after images.
To quantify our results in the real world, we conducted a set of experiments with eight different objects.
We report results in Table 3.
Our placement planner was highly successful at ﬁnding manipulation plans with a range of different objects and predicates, including both grasps and placements for the various held-out objects.
Crucially, our method is able to ﬁnd multiple valid solutions for each scene; some examples appear in the supplementary materials.
We saw a high rate of grasp execution failures on the real world, which could easily be improved in future work by re-grasping
[Table]
Conclusions and Future Work
We demonstrated the ability for a robot to learn to accurately infer inter-object relations from point clouds of real-world, unstructured environments, which can be used to perform manipulation planning for previously unseen objects.
Our results show that a model trained purely in simulation, effectively predicts relations on real-world point clouds of objects not seen during training.
Furthermore, our incorporation of a scene-realism discriminator signiﬁcantly improves performance over the predicate goal predictor alone.
In the future, we will add semantic understanding about which objects the robot is interacting with using learned object class and attribute classiﬁers.
We will also expand our method to handle multi-object relations and explore long-horizon manipulation tasks by formally extending our method into a task and motion planner.
Chris Xie was funded by NSF NRI grant IIS-2024057.