PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation
Abstract: Accurate 6D object pose estimation is an important task for a variety of robotic applications such as grasping or localization.
It is a challenging task due to object symmetries, clutter and occlusion, but it becomes more challenging when additional information, such as depth and 3D models, is not provided.
We present a transformer-based approach that takes an RGB image as input and predicts a 6D pose for each object in the image.
Besides the image, our network does not require any additional information such as depth maps or 3D object models.
First, the image is passed through an object detector to generate feature maps and to detect objects.
Then, the feature maps are fed into a transformer with the detected bounding boxes as additional information.
Afterwards, the output object queries are processed by a separate translation and rotation head.
We achieve state-ofthe-art results for RGB-only approaches on the challenging YCB-V dataset.
We illustrate the suitability of the resulting model as pose sensor for a 6-DoF state estimation task.
Code is available at https://github.com/aau-cns/poet .
Introduction
Accurately estimating the 6D pose of objects from RGB images is essential for robotics tasks such as grasping or localization [1, 2].
Grasping tasks require the robot to know the exact position of the object such that it can place its end-effector effectively.
In autonomous driving, it is critical that the vehicle has sufﬁcient knowledge of its surroundings including the relative 6D pose of all objects in its vicinity.
For unmanned aerial vehicle (UAV) navigation, especially in close proximity to people or infrastructure, realizing precise control depends on the estimation of 6D object poses.
In recent years, vision-based 6D pose estimation with deep learning [3] has been on the rise.
Approaches differ in terms of input data, network architecture, post processing and number of viewpoints [4, 5, 6, 7].
Observing objects from multiple viewpoints introduces constraints to the pose of objects and improves estimation [7, 8].
Availability of 3D object models allows for an iterative reﬁnement of an initial pose estimate by either iterative closest point (ICP)
[9] matching of pointclouds or by matching keypoints with a perspective-n-point (PnP) [10] algorithm.
However, these algorithms are computationally demanding.
Besides being used for post processing, 3D models can be utilized as an additional input to the network [5].
This may include model keypoints and corresponding features or information about the object such as symmetry axes and planes.
Additionally, prior information about the object class or a depth map corresponding to the input RGB image, which improves the networks ability to estimate the objects’ distance to the camera [8], can be passed to the network.
Although additional input information can greatly beneﬁt the accuracy of the ﬁnal estimated pose, apart from requiring a more detailed data base containing accurate 3D models and depth maps corresponding to RGB images, it results in higher computational complexity and thus, longer run time in comparison to the same neural network-based architecture taking only RGB images [8].
While the requirements for real-time performance depend on the speciﬁc application, more often the availability of high-quality, detailed 3D models or depth maps is limited.
Depending on the type of additional input information, additional sensor hardware is also required, which may not be available during inference.
In this work, we focus on a pose estimation framework purely based on RGB images and information provided by a backbone object detector.
To the best of our knowledge, we are the ﬁrst to incorporate global image context information into the pose estimation task by passing multi-scale feature maps to a transformer network and relying only on 2D image information.
Our approach does not depend on the number of objects present.
Our framework, dubbed PoET (Pose Estimation Transformer), can be used on top of any 2D object detector.
We evaluate our approach on the YCB-V [3] dataset and compare our results to state-of-the-art approaches.
Finally, we illustrate the suitability of the obtained model as a pose sensor for a 6-DoF state estimation task, where ”pose sensor” means the combination of a camera and PoET providing information about the camera pose relative to objects, i.e. its relative position and orientation.
Our contributions are the following:
• We present a transformer-based framework that takes a single RGB-image as input, estimates the 6D pose for every object present in the image and can be trained on top of any object detector framework.
A detailed ablation study supports our design choices.
• The framework is independent of any additional information which is not contained in the raw RGB image.
In particular, it does not depend on depth maps, object symmetries or 3D object models.
Hence, our results are achieved without iterative reﬁnement and the whole network can be trained using 3D model independent loss functions.
• We achieve state-of-the-art results on the YCB-V [3] dataset for RGB-only methods and competitive results in comparison to approaches utilizing 3D models.
• We show the feasibility of the resulting model as a pose sensor in a 6-DoF localization task.
The rest of the paper is organized as follows:
In Section 2, related work for 6D pose estimation is reviewed.
Following the presentation of our method and implementation details in Section 3, the experiments and the corresponding results are discussed in Section 4 including an ablation study investigating our network architecture.
Additionally, we illustrate how PoET and its relative 6D pose estimates can be used for localization in Section 4.3.
Finally, the limitations are discussed in Section 5 and the paper is concluded in Section 6.
We refer the reader to the supplementary material for an extensive ablation study and additional results on the LM-O [11] dataset.
Related Work
Classical image-based 6D pose estimation approaches can be split into feature-based methods and template-based methods.
For the latter, object pose is determined by matching object templates against the input image [12, 13].
While template-based approaches work well on texture-less objects, they are prone to fail in scenarios where objects are occluded.
In feature-based methods, local features are extracted from the image and then matched to the 3D model to determine correspondences [14].
Based on these 2D-3D correspondences, the 6D pose of the object can be derived.
While these approaches can handle occlusion of objects, they require textures in order to perform the matching.
If RGB-D images are available, the additional information provided by the depth can be used to improve the initial pose estimate by iterative reﬁnement [15, 16] such as e.g., ICP [9].
Even though those methods can achieve state-of-the-art performance, they are computationally expensive and the availability of depth maps is not guaranteed.
In recent years, advancements in deep learning for computer vision tasks have been applied to singleview, image-based 6D pose estimation, either to replace components of classical approaches [17, 18, 19, 20], or as end-to-end learned methods, where the 6D pose is directly estimated from the input using convolutional neural networks (CNNs).
Xiang et al.
[3] proposed with PoseCNN, a CNNbased method to regress the 3D translation and rotation of each object present in the image using an object symmetry-aware loss function.
Li et al.
[8] proposed a framework that introduces prior knowledge about the object class into the network and perform pose estimation by discretizing the possible translation and rotation values to unique bins resulting in a classiﬁcation task.
Aside from depth images, using 3D object models for pose estimation yielded promising results [21, 4, 5, 6, 7].
The approaches differ in terms of how the 3D model is used.
Kehl et al. [21] and Li et al.
[4] predict the 6D pose of objects and then reﬁne the estimated pose.
With GDR-Net, Wang et al.
[22] are able to integrate PnP into an end-to-end trainable network.
Similarly, Li et al.
[6] use the 3D model of an object for iterative pose reﬁnement.
Given an initial pose estimate, a network is trained to iteratively reﬁne the pose by matching a rendered image created from the 3D model to the original image.
Labb ´ e et al.
[7] apply this approach to multiple viewpoints of the same scene resulting in improved pose estimation.
In yet another approach, Billings and Johnson-Roberson [5] provide the 3D model as an additional input to their network, dubbed SilhoNet.
Their network predicts the object silhouette and a 3D translation vector derived from the 2D bounding box position.
Based on the former, the 3D rotation of the object is estimated and corrected for silhouette symmetric objects - a signiﬁcant restriction, as the real error in rotation for symmetric objects can be very large.
Similar to our approach, Amini et al.
[23] presented a transformer-based architecture to directly regress the 6D pose for multiple objects contained in a single image.
By extending the Detection
Transformer [24] with translation and rotation heads, they are able to train the whole network in an end-to-end fashion.
However, they require the object 3D model as they use a symmetry aware loss [3].
Our approach does not require any additional information such as depth maps, 3D models or known object symmetries, but instead directly estimates the translation and rotation of objects in the camera coordinate frame from a single RGB image.
In contrast to other methods that work with regions of interest for pose estimation, we keep the complete image feature map and provide the regions of interest as an additional input to our transformer network.
Method
We present a novel transformer-based neural network for the 6D pose estimation task.
Taking a single RGB image as its input, the 6D pose of every object detected in the image is predicted simultaneously.
After generating (multi-scale) feature maps by passing the image through an object detector, they are processed by a transformer architecture.
At the end, translation and rotation are estimated in a decoupled manner.
In this section, we ﬁrst present the general structure of our network.
Afterwards, we talk about speciﬁc implementation details and data preparation.
3.1 Network Architecture
Fig.
1 shows a detailed overview of our network architecture, which consists of three steps.
First, the input image is passed through a backbone object detector network.
Both the generated (multiscale) feature maps and the predicted bounding boxes are used in subsequent processing.
PoET can be trained on top of any object detector architecture and thus extend a pre-trained object detection framework to include 6D object pose estimation.
Second, the (multi-scale) feature maps used for the object detection step are passed to the encoder of a multi-head attention-based transformer.
Our transformer architecture is a modiﬁed version of the Deformable DETR transformer module proposed by Zhu et al.
In contrast to the original DETR [24], the deformable transformer allows to process multi-scale feature maps similar to state-of-the-art object detectors.
By only attending to a limited number of feature map keypoints in the decoder, the transformer achieves a higher passthrough rate.
Additionally, a deformable transformer shows faster convergence rates than a regular transformer architecture.
The main idea behind using a transformer architecture for feature map reﬁnement is to generate features that capture the global information contained in the image.
For example, such additional information might be the image location of other objects present or general information of the overall scene extracted from the image.
While the encoder of the deformable transformer is kept unchanged, we modiﬁed the decoder to incorporate more information from the object detection step: First, the learned object query embedding is replaced by bounding box information.
For each detected object, the bounding box center coordinates ( c x , c y ) , the width w and the height h are normalized and then position-encoded [27] to generate the object query embeddings.
The embedding dimension L is chosen such that 2 · n p · L equals the hidden dimension d h of the transformer.
In our case, the number of parameters n p equals to 4.
Moreover, the inter-query attention heads ensure that information is properly propagated between the different object queries.
Second, the Deformable DETR originally only attends to a limited number of keypoints which are randomly sampled around reference points.
Instead of predicting reference points from query embeddings by a trainable fully connected layer, we directly feed the normalized center coordinates ( c x , c y ) as the reference points to the decoder.
By feeding this additional information to the decoder along with the encoder-reﬁned image feature maps and the inter-query attention heads, the decoder generates new object query embeddings which not only contain local information regarding the object but also global information extracted from the image.
Third, the object queries outputted by the transformer are passed through a translation and rotation head.
This allows us to simultaneously estimate the pose for multiple objects independent of how many objects are present and which class they belong to.
As we approach the pose estimation problem from a global image context perspective by extracting features from the whole image, the network directly estimates the translation and rotation with respect to the camera.
Our translation head is a simple multi-layer perceptron (MLP) with input dimension d h , one hidden layer and output dimension 3 .
We directly predict the translation ˜ t
= (˜ t x , ˜ t y , ˜ t z ) with respect to the camera frame.
Given the ground-truth translation t , our translation head is trained with a simple
L2-loss deﬁned as
[Equation]
The rotation prediction head is identical to the translation head besides the output dimension.
For estimating the rotation, we use the 6D rotation representation proposed by Zhou et al.
[25] as this representation does not suffer from discontinuities with respect to learning as e.g., quaternions do.
Hence, the network predicts a 6-dimensional output vector.
Afterwards, the 6D representation is used to determine the estimated rotation matrix ˜ R ∈ SO (3) as described in [25].
The rotation head is trained using a geodesic loss [28] given by
[Equation]
[Equation] where R is the ground-truth rotation and Tr ( · ) is the matrix trace operator.
To ensure numerical stability of the loss during training, the argument of the arccos is clamped between − 1+ (cid:15) and 1 − (cid:15) , where (cid:15)
= 1 e − 6 .
Our whole network is then trained with a weighted multi-task loss expressed as
[Equation]
[Equation]
where the loss is calculated for each object and then averaged across all objects present across all images in the batch.
λ t and λ rot are the weighting parameters for the translation and rotation loss respectively.
PoET can be trained either class-speciﬁc or class-agnostic.
In the class-speciﬁc case with n cls different classes, the translation and rotation output dimension are changed to 3 · n cls and 6 · n cls , respectively. For each object query, one hypothesis for each class is regressed and the ﬁnal output is chosen depending on the class predicted for the bounding box.
However, no class information is fed into the transformer.
3.2 Implementation Details & Data Preparation
While PoET can be trained on top of any object detector, we used Scaled-YOLOv4 [29] as the backbone object detector as it offers a good trade-off between speed and accuracy.
An MS-
COCO [30] pre-trained Scaled-YOLOv4 is ﬁne tuned for 10 epochs on the YCB-V dataset for the object detection task.
During the training of PoET, the weights of the object detector backbone are frozen.
We implement PoET using PyTorch [31] and train it for 50 epochs using AdamW [32] with a learning rate of 2 e − 5 and a batch size of 16.
Our best performing network has ﬁve encoder and decoder layers, d h = 256 , 16 attention heads and a positional embedding dimension of L = 32 .
The network is simultaneously trained for 3D translation and 3D rotation estimation and the weighting parameters are set to λ t = 2 and λ rot = 1 , such that both losses are in the same value range.
Given the commonly used data split for YCB-V [3, 5, 6, 7, 8, 33], we train our network onof the available 92 video sequences and reserve the remaining 12 sequences and their challenging keyframes for testing and evaluation.
Moreover, we include 80,000 synthetic images generated from 2D projections of the 3D object models provided by the original authors [3].
We refer the reader to the supplementary material, for more details regarding our implementation.
Results & Experiments
In this section, we present PoET’s performance on the YCB-V benchmark dataset for 6D pose estimation.
For evaluation, we use the AUC of ADD-S metric [3] and additionally report the average translation and rotation error in cm and degree, respectively, as done by [5].
We compare our results to state-of-the-art, single-view, RGB-based approaches and list the results of other approaches as reported in the corresponding work.
We conduct an ablation study on the network architecture and our modiﬁcations to the transformer part.
The ablation study investigates the inﬂuence of the object detector backbone, transformer modiﬁcations, data augmentation, network architecture and rotation representation on PoET’s performance.
While we present here our most important ﬁndings, we refer the reader to our supplementary material for additional ablation results.
Finally, we illustrate how
PoET’s relative pose estimates can be used for camera localization.
4.1 6D Pose Estimation
Our best performing network is class-speciﬁc, consists of 5 encoder and decoder layers withattention heads, and was trained according to Section 3.2.
Methods that rely on an object detector to predict regions of interest (ROIs) usually do not elaborate whether and, if so, how multiple predictions, bad predictions and missing predictions are treated within their approach.
Therefore, to allow for a fair comparison, we also report the results of PoET and other approaches given ground-truth bounding boxes in addition to the results on predicted bounding boxes.
We present representative qualitative results in Fig.
In Table 1 we report our results for the AUC of ADD-S metric.
Both for predicted and ground-truth bounding boxes, PoET outperforms (overall and also for most individual classes) other state-of-theart RGB-based methods that either work on the whole input image [3, 23] or that predict ROIs for pose regression [8].
We also outperform SilhoNet [5] which feeds additional information from the 3D model to its network and reduces predicted rotations by known object symmetries.
This shows that PoET and its feature maps containing global image context reduce the need for any additional inputs.
This is especially highlighted when comparing the performance of PoET to the ROI-based approaches SilhoNet and MCN [8] in the case where all networks are provided with ground-truth bounding boxes.
Only models that explicitly utilize 3D object models during inference, either through PnP during the estimation [22] or by performing iterative reﬁnement after estimating an initial pose [7, 6], achieve slightly better results but this comes at the cost of signiﬁcantly increased computational complexity and the need for accurate 3D models to be known a priori.
[Table]
The improved performance of PoET compared to other RGB-based models in terms of the ADD-S score is due to a better estimate of the 3D translation as can be seen in Table 2.
Again, groundtruth-based results are also presented.
In contrast to directly estimating the 3D translation like
PoET, PoseCNN as well as SilhoNet determine the 3D translation by estimating the depth and center pixel coordinates of an object and then reprojecting them using the known camera intrinsic parameters, which is considered the easier task to learn [3].
MCN treats the translation estimation as a classiﬁcation problem by binning the translation space.
In addition to the global image information provided by the multi-scale feature maps, our approach also learns to model the camera intrinsics, which results in a more accurate estimation of translation.
For 3D rotation estimation, we achieve the same average error as regular PoseCNN.
Not surprisingly, networks that reduce the possible rotation space based on known object symmetries achieve a better result but with limited applicability to real-world scenarios.
The mean average 3D rotation error is shown in Table 2.
The inﬂuence of reducing the rotation by geometrical symmetries ( † ) is highlighted for PoseCNN.
Since PoET makes use of the full multi-scale RGB feature maps, it outperforms SilhoNet for objects that have no symmetries in the silhouette space and only performs signiﬁcantly worse for objects with rotational symmetries around an axis in 3D space, but without requiring a priori knowledge about the 3D shape of objects or restricting the rotation space based on symmetries.
The main source for rotational errors are objects with rotational symmetries around one axis (master chef can, tomato soup can, tuna ﬁsh can).
We have investigated the axes of our rotation errors and compared them to the symmetry axes for symmetric objects.
The average tilt of rotation error axes with respect to symmetry axes across all test images and symmetric objects is only 15 degrees.
If we ignore rotational errors about symmetry axes, our average rotation error reduces to 11.24 degrees, outperforming all RGB-only competitors, see Table 2.
We refer the reader to the supplementary material for additional ablation experiments as well as PoET’s performance for the stricter BOP[33] and AUC of ADD [3] metrics and for the LM-O benchmark dataset [11]. 4.2 Ablation Study The ablation study investigates the inﬂuence of different components on the performance of our PoET framework.
By assuming a perfect object detector that provides ground truth bounding boxes to PoET, we ensure that the evaluation includes every object present in the image even for those which might be not detected by the object detector.
During the ablation study, we focus on the
AUC of ADD/ADD-S metric, the average translation error and rotation error.
All results reported in this section are for the same hyperparameter conﬁguration as described in Section 3.2, the same ﬁxed seed and trained for the same number of epochs.
The ﬁnal results are summarized in Table 3.
We compare our best performing network from Section 4.1 ( Baseline ) to a class-agnostic version ( Agnostic ) and a network with less layers ( Small ).
Finally, we investigate the inﬂuence of the integration of bounding box information into the transformer on the performance of PoET.
We train
PoET with learnable reference points ( RP ), trainable query embeddings ( Q ) or the combination of [Table] [Table] both ( RP + Q ).
In all three cases the performance is reduced in comparison to a version of PoET that uses bounding box information to generate query embeddings and reference points.
This shows that providing a transformer with bounding box information can greatly beneﬁt its training process leading to improved performance for the same training duration.
We kindly refer the reader to the supplementary material for an in-depth analysis and discussion of the ablation study.
4.3 Localization
PoET is well suited for vision-based object-relative localization.
A transformation between coordinate frames A and B expressed in coordinate frame C is fully deﬁned by the translation C t ab and the rotation R ab .
Given a set of landmarks with known ground-truth pose ( R iwo , t iwo ), a single frame that captures at least one of those landmarks can be used in combination with PoET to localize the camera by estimating the relative pose ( ˜ R ico , ˜ t ico ) to all landmarks present in the frame.
For each landmark, the estimated camera pose ( ˜ R iwc , ˜ t iwc ) can be determined by
[Equation]
The ﬁnal camera pose ( ˜ R wc , ˜ t wc ) can then be determined by taking the average over all landmarks present in the image.
YCB-V’s test sequences offer 12 different camera trajectories and each with a different constellation of multiple objects serving as landmarks.
For each individual frame we estimate the camera pose and compare it to the ground-truth.
In Fig.
3 we show an example trajectory for a single sequence.
For further examples we refer the reader to the supplementary material.
We compare three different approaches: using all detected objects ( all ), perform simple outlier rejection by taking and choosing the hypothesis the majority of objects agree on ( out ) or by incorporating the camera pose estimate from the previous frame into the outlier rejection in cases with multiple hypotheses having the same number of votes ( prev ).
For the sake of comparison, we also calculate the camera pose by choosing the estimated camera pose being closest to the current ground-truth camera pose ( best ).
Moreover, we differentiate between PoET being provided either object detections from the backbone ( bb ) or ground-truth bounding box information ( gt ).
We calculate the sample error mean and standard deviation across all frames for the position and attitude and summarize them in Table 4.
Simple outlier rejection greatly beneﬁts the localization error.
This is due to wrong relative object pose estimates having a large impact on the ﬁnal estimated camera pose when taking a simple average.
Incorporating the camera pose estimate from the previous frame only slightly improves the localization as it only helps in multi-hypothesis cases.
In comparison to the estimated trajectories using the best estimated camera pose, our outlier rejection based trajectories perform worse with around factor two.
Nonetheless, the localization results when considering only individual frames are remarkable.
The difference in performance between backbone and ground-truth detections is due to the backbone potentially missing an object or assigning a wrong class and thus throwing off the estimate.
To further motivate the use of PoET as a pose sensor in mobile robotics, we have integrated it in a state-of-the-art sensor fusion framework [34] and performed state estimation experiments with
YCB-V objects in our motion capture room using inertial data for propagation and PoET for the pose update.
For a representative trajectory, the average error in rotation was (roll, pitch, yaw) =
(4.0, 6.9, 14.8) degrees and in translation (x, y, z) = (0.084, 0.179, 0.032) meters.
The performance for our real data is only slightly worse compared to the performance on the benchmark dataset, even though we were using a completely different camera, than the one used to record the original dataset, and the objects have slightly changed in appearance in comparison to the original YCB objects.
Nonetheless, PoET is able to provide sufﬁciently accurate pose data. Summarizing, PoET achieves sufﬁcient localization accuracy such that it can be used as a pose sensor for robotics.
[Figure] [Figure]
[Table] [Table]
Limitations
In Section 4.1 it was discussed that PoET is outperformed by methods utilizing 3D object information for rotation estimation in particular for objects that have rotation-symmetric silhouettes.
Wrongly estimated rotations lead to the assumption that the camera views the objects from a different angle resulting in wrong hypotheses for the localization task as discussed in Section 4.3.
Moreover, the low resolution of the images in the YCB-V dataset means that RGB textures are not as dominant, especially when augmentation is used.
This is the main reason why PoET has difﬁculties to estimate silhouette rotation-symmetric objects that are not symmetric in RGB space, see e.g. Table 2.
Conclusion
In this work we presented a novel, transformer-based framework for multi-object 6D pose estimation.
It can be used on top of any object detector and the only input required is a single
RGB image.
The image is passed through an object detector backbone to create (multi-scale) image feature maps and to detect objects.
Bounding box information of detected objects is fed into the transformer decoder which improves the learning.
By taking the whole image into consideration during the estimation process, our framework does not rely on any additional information.
We outperform other RGB-based methods by a wide margin on the YCB-V dataset.
This is especially important for scenarios where no detailed 3D models or prior object information is available, or where computational efﬁciency is required and thus any input besides the RGB image has to be dropped.
Moreover, we highlighted how PoET’s relative 6D object pose estimation can be used as a pose sensor for robot localization tasks. Acknowledgments
This work was supported by the Federal Ministry for Climate Action, Environment, Energy,
Mobility, Innovation and Technology (BMK) under the grant agreement 881082 (MUKISANO).