Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset
Abstract 6D object pose estimation is one of the fundamental problems in computer visionand robotics research.
While a lot of recent efforts have been made on generalizingpose estimation to novel object instances within the same category, namely categorylevel 6D pose estimation, it is still restricted in constrained environments giventhe limited number of annotated data.
In this paper, we collect Wild6D, a newunlabeled RGBD object video dataset with diverse instances and backgrounds.
Weutilize this data to generalize category-level 6D object pose estimation in the wildwith semi-supervised learning.
We propose a new model, called Re ndering for
Po se estimation network ( RePoNet ), that is jointly trained using the free groundtruths with the synthetic data, and a silhouette matching objective function onthe real-world data.
Without using any 3D annotations on real data, our methodoutperforms state-of-the-art methods on the previous dataset and our Wild6D testset (with manual annotations for evaluation) by a large margin.
Our code anddataset will be made publicly available.
Introduction
Estimating the 6D object pose is one of the core problems in computer vision and robotics.
Itpredicts the full configurations of rotation, translation and size of a given object, which has wideapplications including Virtual Reality (VR) [2], scene understanding [27] and [38, 52, 28, 45].
There are two directions in 6D object pose estimation.
One is performing instance-level 6D poseestimation, where a model is trained to estimate the pose of one exact instance with an existing 3Dmodel [12, 31, 19, 46, 29, 5, 13].
However, learning instance-level model restricts its generalizationability to unseen objects.
To achieve generalization to unseen instance, another direction is recentlyproposed to perform category-level 6D pose estimation using one model [44, 3, 37, 26, 6].
However,the large appearance and shape variance across instances largely increase the difficulty in learning.
To overcome this limitation, Wang et al .
[44] take the initial step to collect real-world dataset and annotations for category-level 6D pose estimation.
Combining synthetic data with free ground-truthannotations, they show the learned model can be generalized to unseen objects within the same category.
While this result is encouraging, the generalization ability of the model is still limited by the number and the diversity of the data due to the challenges in annotating 6D object poses.
Specifically,only 8,000 images across 13 scenes are collected and annotated from the dataset proposed in [44].
Thus it is still very challenging to generalize 6D pose estimation on diverse objects in complex scenes.
In this paper, we propose to generalize category-level 6D object pose estimation in the wild.
Toachieve this goal, we introduce a new dataset and a new semi-supervised learning approach.
Our key insight is that, while annotating the 6D object pose is challenging, collecting the RGBD videos for these objects without labels is much easier and more affordable.
On the other hand, there are infinite
[Figure] [Figure]
[Figure]
ground-truth annotations for synthetic data which come for free.
We propose to leverage the benefitsfrom both sides.
We first collect a rich object-centric video dataset with diverse backgrounds andobject instances using an RGBD camera.
We train our model jointly using synthetic data with thefree ground-truth 6D pose annotations and the unlabeled real-world RGBD videos via a silhouettematching objective.
In this way, our pose estimation model can be generalized to in-the-wild datawith minimum human labor.
We collect an RGBD video dataset for 6D object pose estimation in thewild, namely Wild6D .
Each video in the dataset shows multiple views of one or multiple objects (seeexamples in Fig. 1).
In total, there are 5166 videos ( > 1.1 million images) over 1722 object instancesand 5 categories, which is significantly (300x) larger than the previous 6D object pose estimationdataset [44].
For evaluation, we annotate 486 videos over 162 objects.
Given this dataset, we design a novel model for semi-supervised learning, called Re ndering for
Po se estimation Net work (RePoNet).
The RePoNet is composed of two branches of networks witha Pose Network to estimate the 6D object pose and a Shape Network to estimate the 3D objectshape.
Given an RGBD image of an object, our Pose Network first estimates the Normalized Object
Coordinate Space (NOCS) map [44], which will be integrated with lower-layer features to regress theobject 9D pose (rotation, translation, and size parameters).
Meanwhile, the Shape Network takes acategory-level 3D shape prior as an input, and estimates the object shape for the current input instance.
During training, we utilize both the synthetic data with the full ground-truths and the real-world datafrom Wild6D with foreground segmentation masks (obtained by Mask R-CNN [11]).
Given the inputswith synthetic data, we apply the regression losses on both the NOCS map, 6D pose parameters, andobject shape.
Additionally, given the estimated pose and object shape, we perform differentiablerendering to obtain an object mask projected in 2D. A silhouette matching objective is proposed tocompare the difference between the projected mask and the ground-truth mask.
With the real RGBDdata, although we do not have the 3D ground-truths, we can still learn with the silhouette matchingobjective by comparing the projected mask against the foreground segmentation.
In this way, thegradients are back-propagated through the 6D pose and object shape to adjust both Pose Network and
Shape Network for real RGBD data.
During inference, we first apply our Pose Network to estimatethe NOCS map with a given image.
With the NOCS map output, the object pose can be computed bysolving the Umeyama algorithm [39].
Some estimation results on Wild6D can be found in Fig.
In our experiments, we first evaluate our approach with the dataset proposed in [44].
Our RePoNetshows significant improvement over state-of-the-art approaches when trained in a fully-supervisedsetting (using 3D ground-truths from both real and synthetic data).
By using the real-world datawithout its 3D ground-truth with semi-supervised learning, our performance can still be on parwith previous approaches using full annotations.
We then experiment with our Wild6D dataset andconsistently show a large performance gain over the baselines on in-the-wild objects.
We highlightour main contributions as follows:
• A new large-scale dataset Wild6D of object-centric RGBD videos in-the-wild for categorylevel 6D object pose estimation.
• A semi-supervised approach with RePoNet which leverages both synthetic data and realworld data without 3D ground-truths for category-level 6D object pose estimation.•
Our approach outperforms baselines by a large margin, especially when deployed on in-thewild objects.
We commit to release our dataset, annotation and code for benchmarkingfuture research.
Related Work
Category-level 6D Object Pose Estimation.
Recently, researchers have proposed to learn a singlemodel for one category of objects instead of just one instance on pose estimation.
For example,
Wang et al .
[44] propose a canonical shape representation of different objects called Normalized
Object Coordinate Space (NOCS) to handle the instance variations.
The object pose and size arecalculated by the Umeyama algorithm [39] with predicted NOCS map and the observed points.
Follow-up work using the NOCS representation has focused on improving the shape priors [37] andincorporating direct regression of object pose and size [3, 22, 6].
While these approaches can bedeployed on unseen object instances, the generalization ability of these models is still limited by thescale and the diversity of real-world annotated data in the REAL275 dataset [44].
Only 13 scenes with18 real objects in total are presented in REAL275.
In contrast, we propose a semi-supervised learningapproach with RePoNet, which leverages a new large-scale unlabeled dataset Wild6D for training.
Both our method and the new dataset are the key components that lead to our goal of generalizingcategory-level 6D pose estimation in the wild.
Large-scale 3D Object Datasets.
Various large-scale 3D datasets have been proposed for differenttasks including reconstruction and pose estimation.
For example, the Objectron dataset are collectedand studied with large-scale object-centric videos and 3D annotations [1, 14, 23].
However, there areno depth images provided in Objectron, which might lead to ambiguities in 6D object pose estimation.
Similarly, the recent proposed CO3D [33] dataset with diverse instances in different categories isalso collected without recording the depth.
While we can perform 3D reconstruction or generatethe depth map using COLMAP [34], the error in predicted depth makes it difficult to be used for6D pose estimation.
Different from these datasets, the 3DScan [9] dataset records RGBD videos ofdifferent categories of objects.
However, most objects are heavily occluded by hands or only partiallyvisible with large objects like cars.
The diversity of objects and scenes is also relatively low givenlimited human labor.
Thus 3DScan is not suitable for pose estimation tasks.
In this paper, we propose
Wild6D , a dataset with RGBD videos containing diverse objects taken with diverse backgrounds.
While the training set is not labeled, we provide 6D pose annotations for the test videos.
To the bestof our knowledge, Wild6D is the largest RGBD dataset for 6D object pose estimation in the wild.
Rendering in Object Pose Estimation.
One effective refinement method for 6D object poseestimation is via rendering [21, 50, 18, 15].
For instance, Iwase et al .
[15] propose to utilizedifferentiable rendering and learn the texture of a 3D model.
However, it still depends on a pre-defined
CAD model with a high-quality texture map for instance-level pose estimation and it is not feasible forcategory-level.
Additionally, object pose estimation can be conducted via Analysis-by-Synthesis [40,8, 48, 42, 36], but they usually require gradient descent for optimization leading to relatively slowerinference speed.
Critically, these works only focus on the instance-level setting and none of themcan generalize to different instances.
Inspired by recent work on 3D reconstruction [16, 20, 7, 17],our RePoNet learns the object shape and pose (including both NOCS map and R/T/S parameters)simultaneously using differentiable rendering to provide a loss function during learning [25], whichallows the gradients to backprop through the whole network for training.
While Manhardt et al .
[26]also utilize differentiable rendering to improve category-level 6D pose, it is only applied in test timeto adjust the pose, instead of training the network end-to-end.
We show substantial improvement overall baselines training with RePoNet.
Wild6D Dataset
Existing Category-level 6D Object Pose Datasets.
NOCS [44] is the most common dataset forcategory-level 6D object pose estimation, which consists of the synthetic CAMERA25 dataset andthe real-world REAL275 dataset.
The synthetic CAMERA25 dataset contains 300,000 RGBD imagesof 1,085 object instances from 6 categories for training and evaluation.
The REAL275 dataset sharesthe same categories with CAMERA25 but is more challenging under the real-world background.
Itcontains 4,300 RGBD images of 7 scenes for training and 2,750 images of 6 scenes for testing. Due to [Table] [Figure]
[Table] Objectron [1] ✓ 18K [Figure] [44] ✓  300K
[Figure] [Figure]
✓ ✓
[Figure]
✓ ✓ 1.8K 1M
[Figure]
the limited scale and diversity of real data, the learned models from this dataset cannot generalize toin-the-wild scenarios.
Objectron [1] is a large-scale dataset for object pose estimation and tracking.
Different from NOCS [44], it consists of a collection of short object-centric video clips captured inthe real world.
However, no depth maps are provided in Objectron, which might lead to ambiguitiesin pose estimation.
Wild6D Collection.
To achieve a categorical 6D pose for real objects, we collect a new large-scale
RGBD dataset, named Wild6D .
Each video in the Wild6D is recorded via the iPhone front camerashowing multiple views of objects where RGB images and the corresponding depth images andpoint cloud are captured simultaneously.
The videos are captured by different turkers with their owniPhones to guarantee the diversity of instances and background scenes.
Three videos are taken foreach object under different scenes.
In total, Wild6D consists of 5,166 videos ( > 1.1 million images)over 1722 different object instances and 5 categories, i .
e ., bottle , bowl , camera , laptop , and mug .
Among this data, we split 486 videos of 162 instances to use them as the test set.
Table 1 summarizesthe statistics of Wild6D comparing previous datasets: Wild6D significantly improves the number ofimages, object instances, and scene complexity.
Wild6D Annotation.
To annotate more than 10,000 images in the testing set efficiently, we proposea tracking-based annotation pipeline.
Inside a video, we manually annotate the 6D object poses every50 frames as keyframes.
Given the annotation of the keyframe, we implement TEASER++ [47]together with colored ICP [30] to achieve the registration between the keyframe and the followingframe and compute the transformation matrix.
The ground-truth object pose of the following framecan be obtained by applying the transformation matrix to the keyframe annotation.
Following thispipeline, we can obtain accurate ground-truths by only labeling around 5 keyframes per video.
Proposed Method
We propose the Re ndering for Po se estimation network (RePoNet) using both synthetic data andlarge-scale unlabeled real-world data in a semi-supervised manner.
RePoNet overview.
The RePoNet is composed of two networks including the Pose Network to directlyestimate the object 6D pose parameters (with the NOCS map as an intermediate representation) andthe Shape Network to reconstruct the object shape.
The outputs from both networks can go through adifferentiable rendering [25] module which outputs a segmentation mask.
Semi-supervised learning setting .
As shown in Fig.
2, we utilize two sets of data for semi-supervisedlearning: (i) synthetic data with full annotations including NOCS maps, CAD models, foregroundsegmentation, and 6D pose parameters, denoted as D syn ; and (ii) a large-scale real-world RGBDdataset Wild6D with estimated foreground masks (using pre-trained Mask R-CNN [11]), denotedas D real .
Both the synthetic data and the real-world data are utilized to train our RePoNet jointly.
For the synthetic data, all the ground-truths are used as supervision signals for the Pose Network and the Shape Network .
While for the real-world data, we directly feed the outputs from the Pose
Network and the Shape Network into the differentiable rendering module to generate the binary mask,then train the whole RePoNet by comparing the rendered mask with the object foreground mask(silhouette matching loss).
While there can be a more delicate way to deal with the domain gapbetween simulation and real, we find the direct sharing of parameters during training already providesa simple yet effective solution for generalization.
One important contribution of this framework is to make the whole procedure with RePoNet differentiable, using the Shape Network, and a ConvNet to connect NOCS map to 6D pose parameters inthe Pose Network.
This allows the gradients to backprop through the network end-to-end.
We willexplain the architecture and objective details in the following subsections.
4.1 Framework Details
Input data pre-processing.
Given an RGBD image, we first utilize the off-the-shelf Mask RCNN [11] model to infer the segmentation mask for each object instance (See supplementary materialfor segmentation examples).
We crop an object of interest from the RGB image and the correspondingdepth map.
The pixels in the depth map are back-projected as an observed point cloud.
We denotethe observation of an object instance as X ∈ R
𝐻
×
𝑊 × 3 for RGB channels and P ∈ R 𝑁 p × 3 for the pointclouds, where 𝑊, 𝐻 stands for cropped image resolution and 𝑁 p is the number of sampled points.
Feature extraction.
We employ the PSPNet [51] to extract the image feature as 𝑓 x (Fig.
3 green box),and a PointNet [32] to extract the point cloud geometry feature as 𝑓 p (Fig.
3 blue box).
Meanwhile,for each category, we first define a category-level mesh prior M ∈ R 𝑁 v × 3 to represent objects belongto a specific category, where 𝑁 v is the number of vertices of the pre-defined mesh.
We also use the
PointNet to extract the shape prior feature as 𝑓 cate (Fig. 3 purple box).
4.1.1 Pose Network.
Unlike instance-level 6D object pose estimation, where the CAD model of a specific instance isalways given as the reference, the Pose Network is excepted to work well even without any instancemodels.
To achieve this goal, we leverage the NOCS representation proposed in [44].
The architectureof the Pose Network is shown as the bottom part of Fig.
The Pose Network takes the RGB features𝑓 x extracted using PSPNet and the point cloud features 𝑓 p extracted using PointNet as inputs.
Thetwo different modalities are combined as RGBD features using the dense fusion approach proposedin [41].
Specifically, the geometry feature 𝑓 𝑖 p of the 𝑖 th point is concatenated with its correspondingcolor feature 𝑓 𝑖 x
and it is fed into a Graph Convolution Network (GCN) [24] (Fig. 3 yellow box) toobtain the point-wise RGBD feature, denote as 𝑓 𝑖 rgbd .
Meanwhile, recall that we utilize a PointNet to obtain the point-wise geometry feature of the categorylevel shape prior 𝑓 cate (on the Shape Network branch).
The shape prior contains the prior knowledgeof the category, i .
e .coarse canonical shape and pose, but it does not align with the RGBD featureperfectly.
To use the information of categorical shape prior and address the misalignment, we firstapply a max-pooling operation on point dimension to obtain a global representation, then concatenateit with every point-wise RGBD feature, denoted as 𝑓 𝑖 nocs , which can be used to estimate the NOCScoordinate.
As the size and geometry of real-world object can be quite diverse, we introduce animplicit function Φ nocs (·) to deal with objects of the arbitrary shape (Fig. 3 gray box with MLP).
Specifically, the Φ nocs (·) is implemented as a MLP with the point position and the correspondingfeature as inputs, and predicts its NOCS coordinate as,
[Equation]
where P is the set of input point clouds and 𝑁 p is the number of point clouds.
The output is the NOCScoordinate P 𝑖 nocs for each point.
The NOCS map also explicitly reflects the geometric shape information of objects [43].
Thus, the6D pose, i .
e ., rotation ( R ), translation ( T ), and scale ( S ) can be inferred from the NOCS map via asimple neural network.
In detail, we directly take the RGBD feature 𝑓 rgbd and concatenate it with theintermediate results of NOCS coordinates .
This concatenated feature 𝑓 pose will be used to predictthe object pose R , T and S by a 3-layer ConvNet (Fig.
3 gray box with CNN).
Using a neural networkto connect the NOCS map and the 6D pose parameters makes the whole process differentiable, whichis essential for training with the silhouette matching loss.
More details of the network architecturecan be found in the supplementary materials.
[Figure]
: Crop
[Figure]
M
C :
Max pooling : Concatenation
[Figure]
Inference for 6D pose. For 6D object pose estimation, only the NOCS map output of Pose Networkis required for inference. The object pose can be computed by solving the Umeyama algorithm
[39]with the NOCS map. The 6D pose parameters R , T and S are only used to perform differentiablerendering during training.
The reason is that the directly estimated pose parameters from conv moduleis accurate when fitting the training data, but not as accurate as using NOCS map when generalizingto novel test instances.
4.1.2 Shape Network.
The Shape Network (top part of Fig. 3) aims to reconstruct the 3D shape of the input object fromthe given shape prior via mesh deformation.
The major usage of this network is for performingdifferentiable rendering to provide training signals.
The Shape Network is not necessary duringinference time for 6D object pose estimation.
Given the categorical shape prior representation 𝑓 cate , the Shape Network borrows the point-wise
RGBD feature of the input object 𝑓 rgbd from Pose Network and concatenate them together, denotedas 𝑓 shape .
Specifically, we max-pool the features 𝑓 rgbd on point dimension and repeat it back withthe number of shape prior points for concatenation with 𝑓 cate .
We then predict the per-verticesdeformation for the input mesh given 𝑓 shape .
The deformation prediction network Φ deform (·) is alsoan implicit function taking 𝑓 shape and mesh vertices positions M as the inputs, and outputs the 3Ddeformation as,
[Equation]
where m 𝑖 delta represents the deformation for each vertex, 𝑁 v is the number of the vertices, and
M delta = { m 𝑖 delta } 𝑁 v 𝑖 = 1 .
Then the estimated mesh vertices of the given object is M deform =
M + M delta .
4.1.3 Differentiable rendering.
To obtain the supervision from 2D segmentation, we perform differentiable rendering using theoutputs from the Shape Network and the Pose Network.
Specifically, we feed the estimated objectshape M deform and estimated object pose ( R , T , S ) into a differentiable rendering module [25] togenerate the binary mask of the given object, as shown in Fig. 3.
Given the rendering output binarymask, we can compare it against the Mask R-CNN segmentations and supervise the training of bothpose network and shape network without additional 3D annotations.
4.2 Learning Objectives
We define multiple objectives for training the RePoNet, including the loss for 6D pose parameters,
NOCS regression loss, shape reconstruction loss using synthetic ground-truths for supervision, and asilhouette matching loss which can be applied to both synthetic data and real data without 6D poseannotations.
We introduce each loss as the following.
Disentangled 6D Pose Loss.
Inspired by [21, 43, 35], we implement disentangled pose loss viaindividually supervising the rotation R , translation T and scale S .
Instead of directly computingparametric distances based on rotation matrix, we employ the variant of Point-Matching loss 1 [21, 18].
Additionally, we decouple the translation into the 2D location ( 𝑜 𝑥 , 𝑜 𝑦 ) of the 3D centroid projectionon the image plane and the object’s distance to camera 𝑡 𝑧 .
( 𝑜 𝑥 , 𝑜 𝑦 ) can be approximated as thebounding box center of the given object ( 𝑐 𝑥 , 𝑐 𝑦 ) .
Given the camera intrinsics 𝐾 , the translation canbe calculated via Eq. 3,
[Equation]
Therefore, given the ground-truth pose annotations ( R ∗ , T ∗ , S ∗ ) , the objective function of 6D pose isformulated as Eq. 4,
[Equation]
Thereby,
[Equation]
where M CAD is the ground-truth CAD model of the given object and 𝑜 ∗ 𝑥 , 𝑜 ∗ 𝑦 , 𝑡 ∗ 𝑧 can be computed via
Eq. 3.
NOCS Regression Loss.
To guarantee the pose estimation accuracy, we enforce the predicted NOCScoordinates P nocs to be closer to the ground-truth ones P ∗ nocs .
We use smooth-L1 loss [10],
[Equation]
otherwise
[Equation]
where 𝑥 ∈ P ∗ nocs , 𝑦 ∈ P nocs
Reconstruction Loss.
Given the ground-truth CAD model M CAD , the shape estimation is supervisedby minimizing Chamfer Distance(CD) between M deform and M CAD via Eq.
L recon =
[Equation]
Silhouette Matching Loss.
The above three objective functions, L pose , L nocs and L recon , require theground-truth information and cannot work with the real-world data without annotations.
Thus, wefurther implement a silhouette matching objective function based on input foreground segmentationand
the rendering results.
We employ the negative IOU loss [49] to measure the difference betweentwo masks and denote it as L mask .
Therefore, for the fully-supervised training with annotated synthetic data, the total loss function isdescribed in Eq. 8, where the 𝜆 s are balance parameters.
[Equation]
Meanwhile, under the semi-supervised setting with both annotated synthetic data and unlabeleddata, the total loss function is then formulated as:
[Equation]
where 1 𝑑 ∈ D syn is the indicator function.
Experiments
5.1 Implementation Details
Semi-supervised setting.
We use the training data of CAMERA25 [44] along with the correspondingannotations and jointly train the model with images of REAL275 [44] or Wild6D without any 6Dpose annotations.
More configurations of RePoNet have been specified in supplementary materials.
To validate the effectiveness, we conduct experiments on both REAL275 [44] and Wild6D.
Fully-supervised setting.
We share the same architecture and configurations with semi-supervisedtraining but utilize all annotations of CAMERA25 [44] and REAL275 [44] and evaluate the performance on the REAL275 testing set. [Table] [Table]
5.2 Ablation Study
Implicit function and NOCS map.
We first validate the effectiveness of two components in ourapproach: implicit function and intermediate NOCS map as described in Sec. 4.1.1.
We evaluatethe performance under fully-supervised setting and semi-supervised setting on REAL275 [44] andreport the results in Table 2.
Under the fully-supervised setting, by using the implicit function,the performance on IOU 0 .
5 and 5 degree, 5cm is improved by 2.2% and 4.4%, respectively.
Asimilar improvement can also be observed for the model with an intermediate NOCS map.
The bestperformance is achieved when combining the implicit function and NOCS map together And thesame conclusion can also be obtained under the semi-supervised setting as shown in Table 2.
Pose Loss and Silhouette Matching Loss.
Besides the network architecture, we also show theeffectiveness of the disentangled 6D pose loss and silhouette matching loss.
The performance ofthe model w./w.o.
the pose regression loss and w./w.o.
the silhouette matching loss are listed in
Table 3.
Clearly, both pose loss and silhouette matching loss can enhance the object pose estimationperformance under both fully-supervised and semi-supervised settings.
For instance, the silhouettematching loss improves the IOU 0 .
75 and 10 degree 5cm by 5.5% and 9.1% compared with the modelwithout it under the fully-supervised setting.
This significant improvement shows the renderingmodule can effectively encourage communication between two networks, thereby boosting the poseestimation performance.
Leveraging unlabeled real data.
Additionally, we report the performance of the proposed semisupervised approach with training data from different sources in Table 4.
As the baseline model, wefirst train it only with the synthetic data from CAMERA25 in a fully-supervised manner.
Then, byusing the unlabeled Wild3D RGBD videos, the performance is significantly improved by 6.8% and5.4% on IOU 0 .
75 and 5 degree, 2cm.
Similarly, the performance gains can be observed when adoptingthe REAL275 data without its annotations.
Furthermore, with both REAL275 and Wild3D data,our model outperforms the baseline model by a large margin and approaches the fully-supervisedmodel trained on REAL275 which can be served as the upper bound.
More ablation studies about theamount of unlabeled real data used can be found in {+supplementary.5.
+}
Comparison with State-of-the-art Methods
Performance on REAL275.
We split all existing methods into two groups by whether using fullannotations of real data and compare their performance with RePoNet under both fully-supervisedsetting and semi-supervised setting, denoted as “RePoNet-sup" and “RePoNet-semi" respectively.
Aslisted in Table 5, RePoNet-sup can achieve competitive results among all existing approaches underthe fully-supervised setting.
Note our method is complimentary to the techniques proposed in SGPA,and our key contribution and focus lies on the semi-supervised counterpart for generalization.
Asshown in the bottom part of Table 5, the RePoNet-semi outperforms CPS++ [26] by a large marginunder the semi-supervised setting 2 .
Moreover, compared with the existing fully-supervised methods,
RePoNet-semi achieves a comparable or even better performance without any 6D pose annotations. [Table] [Table]
[Table] [Table] Additionally, our approach can perform shape reconstruction, but it’s not our goal and we report itsperformance in the supplementary material. [Table]
Performance on Wild6D.
Finally, we evaluatethe performance of RePoNet on the Wild6D testing set as reported in Table 6.
For some existingworks, we directly test their pre-trained modelstrained on CAMERA75 along with REAL275.
Since FS-Net [6] does not release the model,we cannot experiment on it.
It can be observedthat the pre-trained models cannot generalize to Wild6D due to the limited diversity of real dataduring training.
For example, the performanceof Shape-Prior [37] on three pose estimationmetrics are all lower than 15%.
On the otherhand, the RePoNet with semi-supervised learning on Wild6D achieves 29.5% and 34.4% on5 degree, 2cm and 5 degree, 5cm, which is 10 × better than Shape-Prior [37].
This significantimprovement shows the better generalization ability of RePoNet.
Also, the superior performanceof RePoNet-semi over RePoNet-syn shows our proposed semi-supervised training can effectivelyleverage the in-the-wild data.
Discussion
Conclusion.
In this work, we consider the problem of generalizing 6D pose estimation to in-the-wildobjects.
Most of existing work are restricted in constrained environments due to limited numberof annotated data.
In an effort to resolve this limitation, we collect Wild6D , a new RGBD datasetwith diverse instances and backgrounds.
Instead of annotating them, we also propose RePoNet, anetwork that can leverage both the synthetic data and real-world RGBD data.
Without using any 3Dannotations on real-world data, RePoNet outperforms state-of-the-art methods on existing datasetsand Wild6D test set by a large margin.
Limitations.
Our proposed RePoNet may be hard to generalize to unseen categories.
This is because
RePoNet depends on the categorical mesh prior which is pre-defined for each category leading to thelearnt deformation model fixed to a specific category.
Alternative way to address this problem is torepresent object shape via a deformation of a unit sphere to get rid of the category-level prior.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tary
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See section
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See supplementary
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]