Paper: 3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation (ICLR-2024)

Reviewer comment (Experiments):
The paper claims that the method can be used for unseen object relative pose estimation, but it actually may rely on object detection to get relevant object area for pose estimation, and there is no reliable unseen object detector available.

Author response:
**Q.1 Object Detection.** In this paper, we follow the setting commonly used in previous methods [1][2][3] that focus on object rotation estimation. Nevertheless, we do consider the practical application scenario where the object bounding box is noisy. Specifically, we evaluated the robustness to this noise, and the results in Fig. 4(b) demonstrate the promising robustness of our method. Note that some of-the-shelf techniques can be used to achieve unseen object detection, such as SAM or the translation estimator presented in Gen6D. In Fig. 4 of the supplementary material, we show examples of 6D relative object pose estimates obtained by combining our method with Gen6D's translation estimator.

[1] Learning descriptors for object recognition and 3d pose estimation. Paul et al., CVPR 2015.

[2] Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects. Xiao et al., BMVC 2019.

[3] Fusing local similarities for retrieval-based 3D orientation estimation of unseen objects. Zhao et al., ECCV 2022.

Reviewer comment (Experiments):
The presentation is not clear and sometimes confusing. In the 4.5 ablation study section, the method of 'RelPose*' should be explicitly explained, and it seems that the attention module, mask module and aggregation module do not result in significant performance improvement. It would be clearer if we start from the baseline and add the proposed modules step by step to show their effectiveness.

Author response:
**Q.2 Ablation studies.** Thanks for pointing this out. RelPose* is actually the baseline suggested by Reviewer jg4A. It aims to assess the major contribution of our paper, i.e., the 3D-aware hypothesis and verification mechanism. To this end, we maintain the backbone and the 3D representation learning module in our method unchanged, solely replacing the pose-aware verification module with the energy-based module used in RelPose and RelPose++. We denote this as RelPose*. As shown in Table 4 of the main paper, the performance of RelPose* is much worse than that of our approach, confirming that pose-aware verification is pivotal in our framework.

Reviewer comment (Writing):
There are some typos. In Sec.3.1, 'as illustrated in Fig.3' should be changed to 'as illustrated in Fig.2'.

Author response:
**Q.3 Typos.** Thanks! We have corrected them.

Reviewer comment (Experiments):
The number of hypotheses during training and testing is huge, 9000 for training and 50000 for testing. How about the computational cost? Why are they different?

Author response:
**Q.4 Number of hypotheses.** We use much fewer pose samples than RelPose and RelPose++, i.e., 50,000 for us vs. 500,000 for them. The decision to use fewer samples during training than testing was driven by the computational resource requirements for enabling mini-batch training. We evaluate the computation cost as multiply-accumulate operations (MACs) and show the results on CO3D in Table 1 of the supplementary material. Our method achieves a better trade-off between efficiency and accuracy in relative object pose estimation, delivering more accurate results with only 5,000 samples than RelPose++ with 500,000 samples.

Reviewer comment (Experiments):
Although the task setting chosen by the paper is challenging for SOTA methods, the motivation of this task setting is not clearly explained. Using only one reference image can pose great difficulty for the pose estimation task. An extreme example is that the reference image is taken from frontal viewpoint and a query image is taken from rear viewpoint, which may share no common features to match at all. Meanwhile, with any modern camera device, one can actually easily capture multiple images and use them as reference. In what scenarios, and why using only one reference image is critical is not mentioned in the paper.

Author response:
**Q.1 Single-reference setting.** As discussed in Sec. 2 of the main paper, existing methods for generalizable object pose estimation depend on densely sampled reference images, limiting their applicability. For instance, in scenarios such as pose estimation for non-cooperative satellites, obtaining images with diverse object poses as references is often impractical. Furthermore, in many applications, e.g., autonomous driving, only sparse viewpoints are available as references, resulting in significant object pose differences between the query and its nearest reference. In such cases, our method remains applicable since it yields promising results with a single reference, where large object pose variations are involved. Importantly, by considering the single-reference case, our work can be thought of as tackling a more general problem in the field of generalizable object pose estimation. We therefore hope that our insights will inspire future research on this important topic.

Notably, predicting the relative object pose remains challenging when there is no overlap between the visible regions in the query and the reference. This issue has been thoroughly discussed in Sec. 2 and illustrated in Fig. 2 of the supplementary material. To enhance the fairness of the presented evaluation benchmark, we excluded image pairs with inappropriate references from both training and testing sets. Please refer to the supplementary material for the details.

Reviewer comment (Experiments):
An important component of the method is the 3D reasoning model that learns the 3D volumes from 2D feature maps of two RGB image inputs. However, this is itself a very challenging research topic about open-set 3D volume representation reconstruction with two RGB inputs. It’s difficult to understand why it is possible to obtain reliable 3D representation for an unseen object from only its two views, which may even present large variations in view angle and lighting. Also, experimental verification for this component is missing. It would be informative to show some visualizations of the intermediate results of this component.

Author response:
**Q.2 3D volume representation.** Unlike the mesh reconstruction task where the goal is to explicitly reconstruct a 3D object mesh from images, the 3D volume representation learned in our method serves the purpose of enabling 3D-aware pose verification. The spatial dimensionality of this representation aligns with that of the sampled rotation matrix, facilitating 3D transformations over the learned volume. Consequently, the subsequent verification is conditioned on a pose sample. In this context, pose-aware features are learned during training, making it challenging to visualize such deep features. To shed more light on the pose-aware features, we provide a detailed illustration of the verification scores for the sampled pose hypotheses in Fig. 5 of the supplementary material. In this figure, the x-axis represents the geodesic distance between the pose sample and the ground-truth relative object pose, while the y-axis indicates the verification scores. The plot reveals that the predicted score decreases as the geodesic distance becomes larger. This observation shows that our method is capable of capturing pose-aware features from the images.

Reviewer comment (Novelty):
In what scenarios or applications is using only one reference image necessary for the pose estimation task? Why is this a meaningful problem setting?

Author response:
**Q.1 Single-reference setting.** As discussed in Sec. 2 of the main paper, existing methods for generalizable object pose estimation depend on densely sampled reference images, limiting their applicability. For instance, in scenarios such as pose estimation for non-cooperative satellites, obtaining images with diverse object poses as references is often impractical. Furthermore, in many applications, e.g., autonomous driving, only sparse viewpoints are available as references, resulting in significant object pose differences between the query and its nearest reference. In such cases, our method remains applicable since it yields promising results with a single reference, where large object pose variations are involved. Importantly, by considering the single-reference case, our work can be thought of as tackling a more general problem in the field of generalizable object pose estimation. We therefore hope that our insights will inspire future research on this important topic.

Notably, predicting the relative object pose remains challenging when there is no overlap between the visible regions in the query and the reference. This issue has been thoroughly discussed in Sec. 2 and illustrated in Fig. 2 of the supplementary material. To enhance the fairness of the presented evaluation benchmark, we excluded image pairs with inappropriate references from both training and testing sets. Please refer to the supplementary material for the details.

Reviewer comment (Theory):
How is it possible to obtain reliable 3D volume representation for an unseen object from only its two views, which may even present large variations in view angle and lighting?

Author response:
**Q.2 3D volume representation.** Unlike the mesh reconstruction task where the goal is to explicitly reconstruct a 3D object mesh from images, the 3D volume representation learned in our method serves the purpose of enabling 3D-aware pose verification. The spatial dimensionality of this representation aligns with that of the sampled rotation matrix, facilitating 3D transformations over the learned volume. Consequently, the subsequent verification is conditioned on a pose sample. In this context, pose-aware features are learned during training, making it challenging to visualize such deep features. To shed more light on the pose-aware features, we provide a detailed illustration of the verification scores for the sampled pose hypotheses in Fig. 5 of the supplementary material. In this figure, the x-axis represents the geodesic distance between the pose sample and the ground-truth relative object pose, while the y-axis indicates the verification scores. The plot reveals that the predicted score decreases as the geodesic distance becomes larger. This observation shows that our method is capable of capturing pose-aware features from the images.

Reviewer comment (Evaluation):
What are the remaining error cases of the model for the defined task?

Author response:
**Q.3 Failure cases.** As discussed in Sec. 5 of the main paper, our method may be affected by occlusions. Specifically, when computing the verification score, we estimate the local similarities of 2D feature embeddings, which preserve local geometric information. In instances of occlusion, however, some of the local information becomes unreliable. We acknowledge this as a limitation of our method and we plan to address this issue in our future work.

Reviewer comment (Evaluation):
What is the computational cost and speed of this method?

Author response:
**Q.5 Efficiency.** During testing, our method utilizes 50,000 pose samples, while RelPose++ uses 500,000. Despite processing fewer samples, our method achieves better accuracy in relative object pose estimation. Both methods handle the samples in parallel. The computation cost, measured in multiply-accumulate operations (MACs), is reported in Table 1 of the supplementary material. Additionally, we assess the speed of the methods on an A100 GPU. On average, our method and RelPose++ process a pair of images in 35ms and 29ms, respectively. It is worth noting that some inefficiencies, such as the use of 'for' loops, exist in our current implementation, which may impact the speed.

Reviewer comment (Experiments):
The generalizability across datasets. The proposed method is generalizable and can handle unseen objects. I'm curious about the generalizability across datasets. For example, the proposed method is trained on the CO3D dataset, but how about the generalizability of the LINEMOD dataset? I think the generalizability across datasets is important for the proposed method, especially for real-world applications.

Author response:
**Q1. Generalization.** We appreciate the insightful suggestion. Since the presented benchmark includes two datasets, we did a cross-dataset experiment at the early stage of this project. This involved training a network on synthetic data from Objaverse and subsequently testing it on LINEMOD. The acc @ $30^{\circ}$ of our method and that of RelPose++ are 21.2\% and 16.7\%, respectively. Our method still achieves better performance in relative object pose estimation. However, the accuracy of both methods is lower than the results reported in our main paper. We attribute this discrepancy to factors such as the synthetic-real gap, and differences in camera intrinsic parameters and resolutions across datasets, rather than to the unseen objects. This observation aligns with discussions found in some related studies. For example, Gen6D suggests training the network on multiple datasets. In their approach, some objects from LINEMOD are chosen as training data, and all images depicting these objects are excluded during testing to ensure the scenario of unseen object pose estimation. To shed more light on this finding, we mimicked the setting in Gen6D by fine-tuning our method for only 200 steps on the same subset of LINEMOD using the model pretrained on Objaverse. This took only around two minutes. Note that, in this case, the objects from LINEMOD during testing are still unseen. The acc @ $30^{\circ}$ significantly increases from 21.2\% to 60.1\%. While these results suggest that fine-tuning effectively mitigates the domain gap, we agree with the reviewer that cross-domain generalization is an important and interesting research problem. We nonetheless leave it as future work.

Reviewer comment (Evaluation):
Running time comparisons. The paper needs to sample M=50000 poses for verification at test time. I'm concerned about the efficiency of estimating a single pose, even if the verifications are parallelized. I think the running time comparisons with other methods are necessary.

Author response:
**Q.2 Running time.** Thanks for the suggestion! Compared to RelPose++ which requires 500,000 pose samples, our method only uses 50,000 while achieving much better accuracy in relative object pose estimation. In response to the reviewer's suggestion, we assessed the runtime performance of our method and RelPose++ on an A100 GPU. Both of these two methods process the pose samples in a parallel manner. On average, our method and RelPose++ take 35ms and 29ms, respectively, to process a pair of images. Due to some inefficiencies in our current implementation such as 'for' loops, our method is slightly slower than RelPose++. However, the sacrifice in running time is deemed acceptable given the superior accuracy achieved by our method.

Reviewer comment (Experiments):
The object pose estimation setup addressed in this paper should be better motivated. If a CAD model of the object is available, multiple reference images can be rendered or other methods applied directly, e.g. [A,B,C,D,E,F]. In this paper, availability of a CAD model of the object is not assumed, but obtaining the pose of the object with respect to the camera in the query image would still require the pose of the object to be known in the reference image. It is not clear how the reference pose would be obtained in practice without assuming the CAD model is known. In addition, the method presented and evaluated in this paper does not estimate the absolute or relative 3D translation of the object between the query and reference images. Overall, the paper would be more convincing if practical applications of the presented approach were discussed.

Author response:
**Q1. Single-reference setting.** It is indeed often impractical to assume the availability of the textured 3D mesh model for a novel object, and we therefore acknowledge the contributions of the model-free approaches, including the ones listed by the reviewer. The missing references have been added to the main paper. To reasonably define the pose of an unseen object, these methods rely on multi-view calibrated reference images. Let us take Gen6D as an example since methods such as Pose from Shape [B] and OSOP [E] still require the 3D object model to generate the references. Gen6D exploits real reference images that are calibrated using a structure-from-motion technique. The goal then is to estimate the relative object pose between the query and the most similar reference selected from all multi-view reference images. Our method addresses a more general and challenging scenario where only a single reference is available. In this case, the potentially large object pose difference between the query and the reference makes methods such as Gen6D inapplicable, whereas our method can still predict the relative object pose. Moreover, in our experiments, we show that, when the reference object pose is known, such as in the CO3D, Objaverse, and LINEMOD datasets, we can exploit this information to derive the query object pose. Nevertheless, our method can still be applied to scenarios where the reference is not calibrated. For example, RelPose and RelPose++ introduce a scenario of sparse-view 3D reconstruction for unseen objects, where our method can serve as a stepping stone. In this context, one view act as reference and its rotation is set to the identity matrix. The pairwise relative transformations obtained by our approach can be leveraged to reconstruct the object's 3D structure.

**Q2. Translation estimation.** Rotation estimation is acknowledged to be more challenging than translation estimation because it involves computing an element $\mathbf{R}\in SO(3)$, and as such, it has received more attention [1][2][3]. 
As discussed in Gen6D, object translation estimation is facilitated by 2D object detection, and zero-shot object detection can be achieved using Gen6D's detector or other techniques such as SAM. In our paper, we conducted an experiment to evaluate the robustness of our approach to the object bounding box. As shown in Fig. 4 (b), our method achieves better robustness than other evaluated approaches, thus showing its potential to be integrated with such object detectors. In Fig. 4 of the supplementary material, we illustrated some 6D object pose estimation results, combining our method and the translation estimator introduced in Gen6D.

[1] Learning descriptors for object recognition and 3d pose estimation. Paul et al., CVPR 2015.

[2] Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects. Xiao et al., BMVC 2019.

[3] Fusing local similarities for retrieval-based 3D orientation estimation of unseen objects. Zhao et al., ECCV 2022.

Reviewer comment (Experiments):
It is not clear if the RelPose, RelPose++ and the presented approach were all trained using the same images for the comparison on CO3D presented in Table 1. The supplementary material mentions that the approach is trained using objects from the objaverse dataset. It is unclear if this is also the case for the experiments in Table 1.

Author response:
**Q.4 Comparison on CO3D.** All evaluated methods were compared fairly. We strictly followed the setting in RelPose++, training and testing our method on the same data as RelPose and RelPose++. The results reported in Table 1 of our main paper are comparable to the ones shown in Table 1 of RelPose++. On Objaverse and LINEMOD, we also conducted a fair comparison, using the same training and testing data for all methods.