Generative Category-Level Shape and Pose Estimation with Semantic Primitives
Abstract: Empowering autonomous agents with 3D understanding for daily objects is a grand challenge in robotics applications.
When exploring in an unknown environment, existing methods for object pose estimation are still not satisfactory due to the diversity of object shapes.
In this paper, we propose a novel framework for category-level object shape and pose estimation from a single RGB-D image.
To handle the intra-category variation, we adopt a semantic primitive representation that encodes diverse shapes into a uniﬁed latent space, which is the key to establish reliable correspondences between observed point clouds and estimated shapes.
Then, by using a SIM(3)-invariant shape descriptor, we gracefully decouple the shape and pose of an object, thus supporting latent shape optimization of target objects in arbitrary poses.
Extensive experiments show that the proposed method achieves SOTA pose estimation performance and better generalization in the real-world dataset.
Code and video are available at https://zju3dv.github.io/gCasp .
Introduction
Estimating the shape and pose for daily objects is a fundamental function which has various applications, including 3D scene understanding, robot manipulation and autonomous warehousing [1, 2, 3, 4, 5, 6].
Early works for this task are focused on instance-level pose estimation [7, 8, 9, 10, 11], which aligns the observed object with the given CAD model.
However, such a setting is limited in real-world scenarios since it is hard to obtain the exact model of a casual object in advance.
To generalize to these unseen but semantically familiar objects, category-level pose estimation is raising more and more research attention [12, 13, 14, 15, 16, 17] since it could potentially handle various instances of the same category in real scenes.
Existing works on category-level pose estimation usually try to predict pixel-wise normalized coordinates for instances within one class [12] or adopt a canonical prior model with shape deformations to estimate object poses [14, 15].
Although great advances have been made, these one-pass prediction methods are still faced with difﬁculties when large shape differences exist within the same category.
To handle the variety of intra-class objects, some works [18, 16] leverage neural implicit representation [19] to ﬁt the shape of the target object by iteratively optimizing the pose and shape in a latent space, and achieve better performance.
However, in such methods, the pose and shape estimation are coupled together and rely on each other to get reliable results.
Thus, their performance is unsatisfactory in real scene (see Sec. 4.2).
So we can see that the huge intra-class shape differences and coupled estimation of shapes and poses are two main challenges for the existing category-level pose estimation methods.
To tackle these challenges, we propose to estimate the object poses and shapes with semantic primitives from a generative perspective.
The insight behind this is that the objects of a category are often composed of components with the same semantics although their shapes are various, e.g. , a cup is usually composed of a semicircular handle and a cylindrical body (see Fig. 1(a)).
This property
Work done during Guanglin Li’s internship at ByteDance.
6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand.
[Figure]
inspired us to handle huge differences in intra-class shapes by fusing this category-level semantic information.
Speciﬁcally, we propose to encode object shapes into semantic primitives, which construct a uniﬁed semantic representation among objects in the same category (see Fig. 1(b)).
Following Hao et al.
[20], we map diverse shapes and the corresponding semantic primitives into the same shared latent space of a generative model.
In this way, we can obtain the correspondence between the observed point cloud and the latent space by dividing the observed point cloud into several semantic primitives with a part segmentation network.
In order to disentangle the shape and pose estimation, we further extract a novel SIM(3)-invariant descriptor from the geometric distribution of the point cloud’s semantic primitives.
By leveraging such SIM(3)-invariant descriptors, we can perform latent optimization of object shapes in arbitrary poses and obtain the resulting shape consistent with the observation.
After obtaining the estimated shape, we use a correspondence-based algorithm to recover the size and pose by aligning semantic primitives of the target object with that of the estimated shape.
In summary, our contributions are as follows: 1) We propose a novel category-level pose estimation method, which uses semantic primitives to model diverse shapes and bridges the observed point clouds with implicitly generated shapes.
We propose an online shape estimation method, which enables object shape optimization in arbitrary poses with a novel SIM(3)-invariant descriptor.
Even when training on the synthetic dataset only, our method shows good generalization and achieves state-of-the-art category-level pose estimation performance on the real dataset.
Related Works
Instance-Level Object Pose Estimation.
Instance-level pose estimation assumes the CAD model of test objects is known in the training phase, and previous methods are mainly divided into three categories.
The ﬁrst category of methods [21, 8, 7, 22] directly regress the 6D object pose from
RGB or RGB-D images.
PoseCNN [7] extends 2D detection architecture to regress 6D object pose from RGB images, and DenseFusion [8] combines color and depth information from the RGB-D input and makes the regressed pose more accurate.
The second are correspondence-based methods [9, 10, 11, 23].
They predict the correspondences between 2D images and 3D models, and then recover the pose by Perspective-n-Point algorithm [24].
For example, PVNet [9] predicts the 3D keypoints on the RGB image by a voting scheme.
CDPN [11] predicts the dense correspondences between image pixels and 3D object surface.
The third category is rendering-based methods [25, 26], which recover the pose by minimizing the re-projection error between the posed 3D model and 2D image through a differentiable render.
Compared with these instance-level methods, our categorylevel method does not need the known CAD model as prior.
Category-Level Object Pose Estimation.
Category-level methods [12, 27, 13] aim at estimating the pose for the arbitrary shape of the same category.
To overcome the intra-category variations,
NOCS [12] introduces normalized object canonical space, which establishes a uniﬁed coordinate space among instances of the same category.
CASS [27] learns a variational auto-encoder to recover object shape and predicts the pose in an end-to-end neural network.
These methods directly regress the pose or coordinates, and thus struggle to form an intra-category representation.
Other methods
g f (
[Figure]
g c (  z )
z g f (  z )
[Figure]
Semantic Primitives
Shape Descriptor
Latent Space d  f dz
Generative Decoder
[Figure]
g c (  z ) (c) Pose and Size Estimation
Scale Rotation Translation ˜ s ˜ R ˜ t
g f (  z )
Scale Rotation Translation ˜ s ˜ R ˜ t (d) Object Mesh Reconstruction
[Figure]
f utilize a prior model of each category for the category-level representation.
For example, SPD [14] and SGPA [15] predict deformation on canonical prior models and recover the 6D pose through correspondence between the deformed model and the observed point cloud.
However, when facing a signiﬁcant difference between the prior model and the target object, the deformation prediction tend to fail.
DualPose [28] takes the consistency between an implicit pose decoder and an explicit one into account and reﬁnes the predicted pose during testing.
Besides, FS-Net [13] proposes a novel data augmentation mechanism to overcome intra-category variations.
SAR-Net [17] recovers the shape by modeling symmetric correspondence of objects, which makes pose and size prediction easier.
To utilize the diverse and continuous shapes in generative models, iCaps[16] and ELLIPSDF[18] introduce the generative models [19, 20] to their shape and pose estimation.
Shapes and poses are jointly optimized in these works.
Unlike these previous works, we establish a semantically consistent representation and decouple the shape and pose estimation, which makes the optimization more robust.
Method
Problem Formulation.
Given observed point cloud P
= { p i | i = 1 , ..., N 0 } of an object with known category, our goal is to estimate the complete shape, scale and 6D pose for this object.
The recovered shape is represented by a shape embedding ˆ z in a latent space.
The estimated scale and 6D pose are represented as a similarity transformation { s, R , t } 2 SIM(3), where scale s 2 R , 3D rotation R 2 SO(3) and 3D translation t 2 R 3 . SIM(3) and SO(3) indicate the Lie group of 3D similarity transformations and 3D rotation, respectively.
Overview.
As illustrated in Fig 2, our method takes the RGB-D image as input.
A Mask-RCNN[29] network is used to obtain the mask and category of each object instance.
In the following stages, our method only processes the observed instance point cloud back-projected from the masked depth image.
To overcome the huge intra-category variations, we utilize a uniﬁed intra-category representation of semantic primitives (Sec 3.1).
Then, we process the observed point cloud with a part segmentation network to extract semantic primitives which establish a connection between the observed object and the latent space (Sec 3.2).
To estimate the shape of the target with its pose unknown, we design a SIM(3)-invariant descriptor to optimize the generative shape in the latent space (Sec 3.3).
Finally, we estimate the 6D pose and scale of the object through the semantic correspondence between the observed point cloud and our recovered generative shape (Sec 3.4).
3.1 Semantic Primitive Representation
The huge intra-category differences make it challenging to represent objects with a standard model.
To overcome this problem, we utilize a semantic primitive representation.
The insight behind the semantic primitive representation is that although different instances in the same category have various shapes, they tend to have similar semantic parts, e.g. , each camera instance has a lens, and each laptop has four corners (see Fig 1(a)).
In semantic primitive representation, each instance is decomposed into several parts, and each part always corresponds to the same semantic part in different instances (see Fig 1(b)).
To generate this representation, we learn a generative model g following the network structure in [20] and map instances of the same category into a latent space.
Speciﬁcally, Fig 1(c) gives an overview of the generative model.
The generative model g expresses shapes at two granular levels, one capturing detailed shapes and the other representing an abstracted shape using semantically consistent shape primitives.
We note the ﬁne-scale model as g f and the coarse-scale one as g c .
g f
and g c are two auto-decoders sharing the same latent code z .
g f
( z, x ) = SDF ( x ) , where x is a 3D position in canonical space and SDF denotes the signed distance function.
g c ( z ) =
{ ↵ i | i = 1 , ..., N c } , where N c is the number of primitives and ↵ i is the parameters of a primitive.
We use sphere primitives here and ↵ =
( c , r ) , where c is the 3D center of a sphere and r is the radius.
Please refer to [20] and Sec.
A.3 in the supp.
material for more details.
3.2 Extract Semantic Primitives from Point Cloud
We use a part segmentation network to obtain the semantic primitives of the observation point cloud in two stages.
First, we predict a semantic label l i for each point p
i in the observed point cloud P 0 , and then a centralization operation is performed on the points to extract the center of each primitive.
Semantic Label Prediction.
For each point p i 2 P 0 , its semantic label l i is deﬁned in terms of closest primitive center c j :
[Equation]
We treat point label prediction as a classiﬁcation problem and follow the part segmentation version of 3D-GCN [30] to perform point-wise classiﬁcation.
Standard cross entropy is used as the loss function.
In addition, because of imperfect segmentation masks produced by Mask R-CNN, the point cloud back-projected from the masked depth image may contain extra background points.
For these outlier points, we add another “dust” label in the point-wise classiﬁcation to ﬁlter out them.
Symmetric Loss.
Many common objects in robot manipulation have symmetrical structures, and most objects ( e.g., bottles ) are symmetrical about an axis.
The unique pre-labeled ground truth cause ambiguity when the object rotates around the axis.
In order to eliminate the inﬂuence of symmetric structure on our classiﬁcation task, we introduce the symmetric loss as the same as [12].
For each symmetric object in the dataset, we deﬁne a symmetric axis and make the loss identical when the object rotates around the axis.
Speciﬁcally, we generate a separate ground truth for each possible angle of rotational symmetry.
Given the rotation angle ✓ , the rotated semantic label l ✓ ,i is deﬁned as:
[Equation]
where c ✓ ,j is the position of c j after ✓ degrees of rotation about the axis.
Then we deﬁne the symmetric loss L sym as:
[Equation]
where L is the standard cross entropy, ˜ l i is the predicted label by the network, and we set ⇥ =
{ 0  } for non-symmetric objects and ⇥ =
{ i · 60  | i = 0 , ..., 5 } for symmetric objects in our experiments.
Primitive centralization.
After predicting the semantic label of each 3D point, we count the semantic labels ˜
L = { l i | i = 1 , ..., ˜ N c } which appear in the partially observed point cloud.
Then, we calculate the primitive centers ˜ C
= { c l | l = l 1 , ..., l ˜ N c } from the labeled points by averaging the points with the same label.
Speciﬁcally,
[Equation]
v v
Transformation
SIM(3)
where N l is the total number of points with predicted label l , I is an indication function indicating whether the predicted label ˜
l
i equals l .
3.3 Shape Optimization with SIM(3)-Invariant Descriptor
We decouple the shape estimation from the pose estimation to perform shape estimation for the observed object when the pose is unknown.
In order to achieve this pose-invariant shape optimization, we propose a novel SIM(3)-invariant descriptor extracted from semantic primitive representation.
SIM(3)-invariant Descriptor.
Given a set of 3D points C
= { c i | i = 1 , ..., N } and a SIM(3)- transformation { s, R , t } , a SIM(3)-invariant descriptor f 0 can be expressed as:
[Equation]
[Equation]
To make our descriptor SIM(3)-invariant, ﬁrst, we obtain a translation-invariant descriptor by calculating the vectors v composed of every two points:
[Equation]
After that, we normalize the vector v to eliminate the scale factor s :
[Equation] [Equation]
Finally, we eliminate the effect of rotation transformation by calculating the dot product between two vectors (see Fig. 3):
[Equation]
where · denotes the dot product of two vectors, and ✓ denotes a function which takes 4 points as input and a SIM(3)-invariant scalar as output (as the angles shown in Fig. 3)
We call ✓ the atomic descriptor.
However, only a single atomic descriptor calculated frompoints (primitives centers) is not enough to describe an object’s shape.
Thus, we design the SIM(3)- invariant shape descriptor through a function f , which concatenates multiple atomic descriptors to form a SIM(3)-invariant shape descriptor.
With our estimated primitives ˜ C and their semantic labels ˜
L , the set ˜ L 4 is denoted as: ˜
[Equation]
and the SIM(3)-invariant shape descriptor f is denoted as:
[Equation]
where  denotes the concatenate operation which concatenates multiple scalars into a vector in order, and ˜ L 4 sub is a subset randomly chosen from ˜ L 4 for each optimization.
We choose the subset of ˜ L 4 because | ˜ L 4 | would be large, which is computational expensive.
In Sec. 4.3, an ablation is conducted to analyze the effect of | ˜
L 4 sub | , which indicates the number of atomic descriptors in a shape descriptor.
Online Shape Optimization.
Given the shape embedding ˆ z in the latent space and estimated primitives ˜ C with semantic labels ˜ L , we can measure the shape difference between the generative shape g c (ˆ z ) and the observation P 0 through the SIM(3)-invariant descriptor.
We denote this difference as ˜ e :
[Equation] [Equation]
where ˆ C ( g c (ˆ z ))
= { ˆ c i | i = 1 , ..., N c } is the primitive centers of g c (ˆ z ) .
Obviously, f is a differentiable function and g c is a differentiable neural network.
We optimize the shape embedding ˆ z via gradient descent in an online iterative way and set ⌘ = 0 .
to weight the regularization term.
[Table]
[Figure]
The red arrows and dotted box point out the inaccurate results of SGPA compared with our method.
3.4 6D Pose and Size Estimation
With the primitive centers ˆ C ( g c (ˆ z )) of the optimized shape and the observed point cloud, we use
Umeyama algorithm [31] to compute the SIM(3) transformation by minimizing following error function:
[Equation]
Experiments 4.1 Datasets and Evaluation Metrics
We evaluate our method on NOCS [12] dataset which consists of six objects categories including bottle, bowl, camera, can, laptop and mug .
The NOCS dataset contains a virtual dataset called
CAMERA25 and a real-world dataset called REAL275.
The CAMERA25 dataset contains 300K synthetic images and there are 25K images for evaluation among them.
The REAL275 contains 4.3K real-scene images for training and 2.75K real-scene images for evaluation.
Two aspects of metrics are adopted for evaluation: 3D IoU and pose error.
3D IoU measures the overlap of two posed 3D bounding boxes, and we compute the average precision at threshold values of 50% and 75% .
The pose error measures the rotation and translation errors between the predicted pose and ground truth pose, and we compute the average precision at threshold values ( 5  , 2 cm ), ( 5  , 5 cm ), ( 10  , 2 cm ) and ( 10  , 5 cm ).
4.2 Comparison with State-of-the-Art Methods
We compared our proposed method with several SOTA methods [12, 14, 15, 17, 16].
NOCS [12] directly regress pixel-level canonical coordinate.
SPD [14] and SGPA [15] are the methods which recover canonical shape by deforming a prior shape.
iCaps [16] and our methods utilize a generative model to reﬁne and recover the canonical model respectively.
Table 1 shows the quantitative results.
Overall, our method needs the minimum parameters learned for the shape and pose estimation task and shows the results of the smallest domain gap between the synthetic dataset and
[Table] [Table]
the real-world dataset.
In detail, our method outperforms other existing methods on all metrics but
IoU 50 on the real-world REAL275 dataset.
Fig.
4 shows qualitative comparisons with SGPA [15]
[Table]
On the synthetic CAMERA25 dataset, our method also shows improvement on most metrics compared with other methods, except for a small gap with SGPA [15] on two metrics, ( 10  , 2 cm ) and ( 10  , 5 cm ).
Besides, Fig. 5 shows the average precision curves of each category under different thresholds on IoU, rotation and translation on REAL275 and CAMERA25 dataset.
Compared to the other generative method iCaps [16], our method shows far better results, indicating the advantage of our decoupled shape and pose estimation.
It should be noted here that our trained generative model (Sec. 3.1) consists of about 4.3M parameters for each category.
The generative models are only pre-trained on ShapeNetCore [32] and need no more ﬁne-tuning for the speciﬁc shape and pose estimation task.
Thus, we compare the learnable parameters with other methods.
Speciﬁcally, our method needs only additionally train a part segmentation network (2.3M) for all categories on the synthetic training data provided by CAMERA25 [12], which leads to our better generalization in real scenes.
4.3 Ablation Study
To justify the design choices of our method, we explore the following issues through experiments:
• Effectiveness of the SIM(3)-invariant descriptor.
Does the descriptor-based shape optimization make the optimized shape more accurate and beneﬁt the pose estimation?
• Effect of different numbers of primitives.
Semantic primitives are the basis of our method.
How many primitives are better for pose estimation?
• Effect of different numbers of atomic descriptors | ˜
L 4 sub | .
Our shape descriptor is concatenated by multiple atomic descriptors.
How many atomic descriptors are more effective for shape optimization?
Effectiveness of SIM(3)-invariant descriptors.
We verify the effectiveness of our SIM(3)- invariant descriptors in two aspects.
First, we verify that descriptor-based shape optimization improves the accuracy of shape reconstruction.
We compare the reconstruction error of our optimized shape with the shape when the shape embedding z = 0 (denoted as “Mean” shape) and a random shape when the shape embedding z ⇠ N ( 0 , 1 ) (denoted as ”Random” shape).
Table 4 shows that our optimized shapes are more accurate.
In Table 4, we also compare our method with SGPA [15], even without shape loss to supervise our method, our coarse-level shape reconstruction shows comparable results with SGPA.
Second, we verify that descriptor-based shape optimization improves the pose estimation results.
We compare the results of the pose estimation under the three canonical shapes (mean, random and optimized) in the Table 2.
The results show that we get far better pose estimation results based on the optimized shape.
Besides, Fig.
6 visualizes cases where shapes become more consistent with observations as optimization progresses.
[Table] [Table]
[Figure]
Effect of different numbers of primitives.
We conduct an ablation study on the number of semantic primitives.
We experiment the number with 64, 128, 256, and 512.
As shown in Table
[Table]
the results show that the accuracy of pose estimation is improved with the increase of the number of primitives, which indicates that more primitives can represent more complex geometry.
The results show a decline when the number is set to 512 because 512 primitives are relatively redundant for our 1024 input points, and more primitive classes also decline the accuracy of part segmentation network (Sec. 3.2).
Based on these observations, we use 256 primitives as the default setting in other experiments.
Effect of different numbers of atomic descriptors.
As illustrated in Sec. 3.3, a single atomic descriptor is insufﬁcient to describe an object’s shape, and concatenating all atomic descriptors to form a shape descriptor would be computationally expensive and unnecessary.
Thus, we conduct experiments and analyze the inﬂuence of the number of atomic descriptors on the shape optimization results.
Speciﬁcally, we randomly choose { 10 1 , 10 2 , 10 4 , 10 6 } atomic descriptors to form the shape descriptor and use the chamfer distance to evaluate the quality of the optimized shape.
As shown in
Table.
5, the reconstruction error of the optimized shape decreases as the number of atomic descriptors increases, and applying RANSAC in the random selection can further reduce this error.
In the main experiments, we choose 10 6 atomic descriptors depending on the computational resources and do not use RANSAC for efﬁciency.
Conclusion and Limitations
We propose a novel category-level shape and pose estimation framework by utilizing the semantic primitive representation.
Leveraging a novel SIM(3)-invariant descriptor, we decouple shape and pose estimation and optimize the implicitly generated shape in an online iterative way.
Experiments results demonstrate the advantages of our method on the real-scene data compared with other SOTA methods.
Limitations.
As other methods [14, 15, 17], our method also relies on the pre-processed masks of object instances by Mask R-CNN and extra annotation of symmetry axes when training.
Furthermore, although we can reconstruct the object mesh (Sec. B in the supp.
material), this by-product has no inﬂuence or contribution to our work.
Future work could consider the mesh reconstruction to get better shape and pose estimation results.
Besides, our method does not consider the occlusion and collision between objects, and may cause ambiguity from single-view observation.
Feasible future work could fuse color information into our input, or expand the proposed method to multi-view observation to address these problems.
Acknowledgments
We would like to thank Bangbang Yang for the useful discussion on this paper.