On end-to-end 6DOF object pose estimation and robustness to object scale
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
This report contains a set of experiments that seek to reproduce the claims of two recent works related to keypointestimation, one speciﬁc to 6DOF object pose estimation, and the other presenting a generic architectural improvementfor keypoint estimation but demonstrated in human pose estimation.
More speciﬁcally, in the backpropagatable PnP [1],the authors claim that incorporating geometric optimization in a deep-learning pipeline and predicting an object’s posein an end-to-end manner yields improved performance.
On the other hand, HigherHRNet [2] introduces a novel heatmapaggregation method that allows for scale-aware pose estimations, offering higher keypoint localization accuracy forsmall scale objects
Methodology
We used the publicly provided code where available, adapting it to ﬁt into a model development kit to facilitate ourexperiments.
We used a dataset ﬁt for validating both claims simultaneously, and designed a set of experiments basedon the published methodologies, but also went beyond seeking to validate the higher level concepts.
Our experimentswere conducted on a Nvidia 2080 12 GB GPU with an average training time of 14 hours.
Results
We reproduce the claims of both papers by conducting several experiments in the UAVA dataset [3].
The integration ofa differentiable geometric module within an keypoint-based object pose estimation model improved its performance inmetrics.
We additionally verify that this is the case for other differentiable PnP implementations ( i.e. EPnP).
Further,our results indicate that indeed HigherHRNet improves keypoint localisation performance on small scale objects.
What was easy
Both papers provided publicly available implementations.
In addition, many different variations were also found online.
Finally, the papers themselves were very clearly written, offering insights on various important details.
What was difﬁcult
The main issue that required more effort was identifying the appropriate weights for BPnP [1] in order to balance thedifferent optimization objectives.
As expected, this varies for the context that it is applied (task, dataset) and the values presented in the paper did not work in our case.
Sub-optimal selection of weights leads to convergence issues
Communication with original authors
Introduction
Object pose estimation seeks to determines the 3D position and orientation of an object in camera-centred coordinates.
During the last years, two main directions have been emerged for data-driven 6DOF object pose estimation; directpose regression which predict pose in an end-to-end manner, and indirect that learns the surrogate task of keypointlocalisation and then solves a Perspective-n-Point (PnP) problem to estimate the resulting pose.
Even though it has beenshown [4] that the latter methods better approach the problem, there are still open challenges that need to be solved.
One issue is the splitting between the actual task at hand, and the surrogate task that they learn.
The other has to dowith the spatial nature of keypoint localisation and smaller scale objects.
Recently, two works have been presented thatseek to address these issues, BPnP [1] and HigherHRNet [2].
In this work, we seek to reproduce and verify their claimsin a task that is relevant for both of these works, drone pose estimation.
While BPnP’s relation has to do with the task athand, HigherHRNet is also relevant because commodity drones are usually small form objects, and when ﬂying aroundthe further distance themselves from the operator, effectively reducing their scale in the camera’s image domain.
Scope of reproducibility
Consequently, we opt for reproducing the claims of both of these two relevant papers addressing the aforementionedchallenges.
In more details, the authors of BPnP [1] propose a novel differentiable module which calculates thederivatives of a PnP solver through implicit differentiation, enabling the backpropagation of its gradients to thenetwork parameters, and as such allowing for end-to-end optimization and learning.
On the other hand, the authorsof HigherHRNet [2] focus on improving the 2d landmarks’ localization performance for smaller-scale humans byproposing a novel multi-scale supervision scheme for training and a heatmap aggregation module for inference.
The main claims of both papers can be summarised below:
• BPnP: An end-to-end trainable pipeline for object pose estimation, can achieve greater accuracy by combingthe reprojection losses (Table 3).
HigherHRNet: A novel method for learning scale-aware representations using high-resolution feature pyramids, eventually achieving greater results for small scale objects 1 (Table 4).
Methodology
We implemented our experiments by re-using the publicly available implementation for BPnP, and implementing
HigherHRNet after styding the paper, the original publicly available implementation, as well as other implementations.
In both cases we integrated the code base in a modular framework that facilitates reproducible experiments [5], whichgenerally required slight modiﬁcations of the original code provided by the authors to ﬁt its requirements.
The overallmethodology for our experiments is depicted in Figure 1.
On the left, a traditional monocular heatmap-based keypointlocalisation pipeline is presented, whereas on the right, the BPnP required components are illustrated.
BPnP: BPnP focuses on the Pose Retrieval stage , and following [1] we trained our model under the 3 different schemesused in the original work as well:
• heatmap loss ( l h ),
• mixture loss l m =
l h + β ∗ l proj ,• and pose loss l p =
l reg + l proj ,
where l proj =
(cid:107) π
( z | y, K ) − x ∗ (cid:107) 22 and l reg =
(cid:107) x − π
( z | y, K ) (cid:107) 22 .
Also, π is the projection function employing thepredicted pose( y ) from the PnP solver, the corresponding object’s 3D points z
and K the camera intrinsic matrix.
Apartfrom these experiments presented also in the original paper, we conducted an extra set of experiments that aimed atvalidation the concept of end-to-end 6DOF pose estimation via differentiable PnP. We used another openly availabledifferentiable PnP implementation, and additionally, also tested the faster counterpart of BPnP.
We present resultsacross many established object pose estimation metrics, as well as computational performance metrics for all theaforementioned experiments.
HigherHRNet: On the other hand, for HigherHRNet we focused on the Heatmap regression part by using differentmodels for the decoder part of the architecture, with details following in Section 3.
All the code and its documentation submitted and published this report
3.1 Model descriptions
[Figure] The following models were used as our backbone for regressing the heatmaps and the corresponding coordinate spatialdistributions (the decoder part in Figure 1):
• HRNet [6] with feature maps of width 48 and 3 stages.
The 2nd, and 3rd, stages contain 1, 4 exchange blocks,respectively, and each exchange block contains 4 residual units.• HigherHRNet [2] with feature maps of width 48 and 3 stages.
The 2nd, and 3rd, stages contain 1, 4 exchangeblocks, respectively, and each exchange block contains 4 residual units.• Stacked Hourglass [7] with depth 2 and feature maps of width
Following the heatmap predictions, we apply a decoding operation ( i.e. center of mass speciﬁcally) in order to extractthe keypoints from the heatmaps, which are later driven to the PnP algorithm for retrieving the 6D pose.
In addition, we integrated the EPnP [8] algorithm, using the available implementation in Pytorch3D [9] instead of
BPnP to assess whether other – similar in concept – implementations can verify the claim and quantify the differencesbetween these approaches.
As a side-note, it should be mentioned that our models’ conﬁguration slightly differs fromthe ones described in the original works in order to comply with the working resolution of the dataset we used.
3.2 Datasets3.2.1 UAVA
As aforementioned, our experiments were conducted on a dataset that allowed for the validation of both workssimultaneously.
This also better helps in deducing whether the claims are reproducible as the context (task or dataset)can vary.
We used the UAVA dataset 2 for object pose estimation.
UAVA targets human-robot cooperative
Unmanned
Aerial Vehicle(UAV) applications and offers two different drone models, namely DJI M2ED 3 and Ryze Tello 4 .
The
UAVA dataset provides the 3D models of both drones accompanied by ground-truth annotations such as 3D bounding boxes, 6D pose, and at the same time multi-modal data.
More importantly, the difference in size between the two drone models allows for the validation of scale-invariant pose estimation.
3.2.2 Preprocessing
We processed the original dataset in order to keep only the samples where all the 2D keypoints are within the image,given that BPnP relies on softly approximating the coordinate, and that would fail in the case of out of ﬁeld-of-viewkeypoints.
However, we should mention that we did not apply any other ﬁltering (i.e. visibility of all the keypoints,boundary cases, etc.).
3.3 Hyperparameters
We train all the models for 44 epochs and select the best performing model for testing.
We used the Adam optimizerwith a learning rate of 1 e − 4 , betas of values 0 .
9 and 0 .
999 and no weight decay, and a seed value of 1989 for ensuringreproducibility.
Albeit, we experimented with different losses ( i.e. KL, MSE) for l h , we found that L1 loss works thebest, offering the best results and faster convergence.
This could be attributed to the different resolution of the heatmapsgrid (in our case is lower) as well as the different conﬁguration of the heatmap decoder model (we used 3 stages insteadof 4).
It is worth mentioning that we also tried a bigger heatmap resolution ( e.g. 160 × 120 ) although we decided toconduct our ﬁnal experiments in the lower resolution for two main reasons.
First, most heatmaps regression decodersused in the literature make their prediction in the 1 / 4 of the original image, and second, this higher heatmap resolutionwould enforce us to further reduce the depth of the decoder model.
Speciﬁcally, for BPnP we set β value to 1 e − 5 afterconducting a greedy heuristic search, with values ranging from 0 .
001 to 1 e − 9 , as the proposed value for β coefﬁcient,did not work for our case.
The selection of a non-appropriate β coefﬁcient value can lead to stability issues as noted in
Section 5.2.
3.4 Experimental setup and code
As mentioned above, we integrated the authors’ code (BPnP) or our own re-implementations (HigherHRNet) in [5]which is a PyTorch framework for modular and reproducible workﬂows 5 .
Each model is implemented in a conﬁgurationﬁle that deﬁnes the different components (optimizer, datasets, model architecture, pre-/post-processing graphs, etc.)
andlogs all hyperparameters.
For each experiment we report the standard metrics below:
NPE
: is the magnitude (L2-norm) of difference between the ground-truth and estimated position vectors from the origin of the camera reference frame to that of the drone body frame, normalised with ground-truth vector.
AD is
the angular distance between the predicted , rotation matrix, and ground-truth,or in other
words
the magnitude of
the rotation that aligns the drone
body
frame
with
the camera reference frame.
ACC : considers an estimated pose to be correct if its rotation error is within k ◦ and the translation error is below k cm . ADD : is the average distance metric to compute the averaged distance between points transformed using the estimatedpose and the ground truth
pose.
Eventually, a pose estimation is considered to be correct if the computed average distance is within k % of the model diagonal.
Proj is the mean distance between 2D keypoints projected with the estimated pose and those projected with groundtruth pose.
An estimated pose is considered correct if this distance is within a threshold k .
3.5 Computational requirements
Table 1 showcases the total duration of each experiment (with a 24 batch size) as well as some other useful statisticssuch as the mean duration time for a forward pass, a backward pass, an optimizer step, as well as the total test durationwith batch size 1.
It is clear, that the introduction of the differentiable PnP modules in the training procedure increasesthe total training time signiﬁcantly, as the backward and step operation require more time.
We ran our experiments on amachine with the speciﬁcations presented in Table 2.
Results
Our results support the claims presented by both authors in [1] and [2] respectively.
As is demonstrated in Table 3, themodel trained with l p achieved better results in most of the metrics for both drone models.
Similarly, Table 4 indicatesthat HigherHRNet yields better results for the small-scale drone in most of the metrics, although its performance for thebigger M2ED drone is worse compared to the standard HRNet model.
[Table] [Table]
4.1 Results reproducing original papers
4.1.1 BPnP
With these experiments we show that the addition of a differentiable PnP module improves the performance in objectpose estimation task.
We provide qualitative results in Figure 3.
It is worth highlighting that training with l p does notrestrict the shape of the distribution the way that it is constrained when relying on heatmap supervision ( i [Table] e. Gaussiandistribution approximation).
Instead, the model freely localizes the keypoints, which results in more focused predictions.
This is illustrated in Figure 2 where qualitative results display the heatmaps on top of the color images.
[Table]
4.1.2 HigherHRNet
These experiments showcase that the addition of the aggregation module improves the keypoint localization performancewhen targeting smaller-scale objects.
Speciﬁcally, the HigherHRNet architecture gives better results in most of themetrics for the small form drone (Tello).
On the other hand though, this is not the case for the larger drone, where the
HigherHRNet performance is slightly worse than the standard HRNet’s one.
4.2 Results beyond the BPnP paper
Apart from the experiments conducted by the authors in [1] we provide additional to further support the main claims.
Particularly, we compared BPnP versus an alternative differentiable PnP algorithm ( i.e. EPnP) and the results are
Evidently, heatmaps by the l h model ( second and fourth columns tend to keep the 2D Gaussian shape distribution, while the l p ones ( ﬁrst and third columns ) freely approximate the ( x, y ) position enforced by the regularizer term( l reg ) of the l p loss, eventually allowing for more distinguishable 2D keypoints estimations. [Table] [Table] [Table]
[Table]
[Table]
49 M2ED
HigherHRNet
[Table]
[Table]
Hourglass
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Tello
HigherHRNet
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
demonstrated in Table 5.
We also provide extra experiments of a BPnP implementation in which the calculation of thehigher-order derivatives is ignored from the coefﬁcient’s graph as presented in Table 6.
4.2.1 BPnP vs EPnP
For this experiment we utilised the same backbone ( i [Table] e. HRNet)
but we changed the BPnP module with the EPnP. Wefollowed the exact same training procedure, hyperparameters, as well as the same loss l m .
Results are summarized in
Table 5.
It is evident that EPnP and BPnP offers comparable results in most of the metrics.
[Table]
Authors in [1] provided an alternative method for calculating the gradients through the PnP layer, which essentially isthe same method as the original, although ignoring the higher-order derivatives from the coefﬁcients graph.
Therefore,we provide results using this faster BPnP method in Table 6, comparing the two different versions, as well as theirtraining times in Table 7.
It seems that the original version outmatches the faster one, albeit there is no signiﬁcantperformance drop.
On the other hand, Table 7 indicates how the second implementation justiﬁes its name.
So, it is inusers’ ﬂuency whether they need to sacriﬁce gradient accuracy and some performance drop in exchange for efﬁcienttraining times.
[Table]
[Table]
Discussion
After conducting several experiments on the UAVA dataset
[Table]
the central claims of [1] and [2] stand true; as they bothoutperform other methods.
Particularly, for validating BPnP we conducted the same experiments as the original paper,and further
[Table]
we compare it with another differentiable PnP method ( i.e. EPnP).
The inclusion of 2D-3D geometryconstraints through differentiable geometric optimization, improves the performance.
Extending the experiments ofthe original paper, we compare another implementation of the BPnP module which ignores the high order derivativesfrom the coefﬁcient graph.
This module achieves comparable results as its counterpart apart it is much faster.
It isworth noting, that both BPnP, and EPnP are quite time-consuming as demonstrated in Table 1.
Finally, we study theperformance of the HigherHRNet [1] in a very challenging small scale object.
Indeed, the performance of the proposedheatmap aggregation module achieves better results when compared with other well-established methods.
5.1 What was easy
Implementing most of the code was straightforward as authors of both papers provide source code.
GitHub issues wereanother source of retrieving information, clarifying parts of the papers when needed.
Additionally, both of the originalpapers are quite complete, well-written making it easy to follow.
5.2 What was difﬁcult
Our major difﬁculty was related to ﬁnding the appropriate value for balancing the terms of mixture loss l m , aka the βvalue.
Even though, authors in [1] provided the value that they used for their experiments this did not work for us, asthis is a case speciﬁc parameter.
It is worth noting that a non-appropriate selection of the balancing term can lead toconvergence issues and negative results.
Even though, not related with the code of both of the papers, we feel that itwould be constitutive to mention that we faced the same difﬁculties when trying to incorporate EPnP in our workﬂow.
5.3 Communication with original authors
Authors of [1] did not specify the conﬁguration of the used network in the pose estimation task, nor the hyperparameters.
Thus, we contacted them through GitHub where they provided a detailed answer, available now to the researchcommunity.
We did not contact HigherHRNet authors [2] as the online implementations and the text and ﬁgures in theirpaper were a good enough guide to understand and implement it.