Paper: Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation (ICLR-2024)

Reviewer comment (Presentation):
The majority of the figures in this paper such as Fig.2, Fig.2, and Fig.10, exhibit low quality. It would be better if the authors could consider revising them to enhance the clarity and resolution.

Author response:
This point has also been raised by other reviewers. We have tried our best to address the presentation issues with our initial draft by improving the quality and clarity of the figures. We are happy to take any more suggestions to improve them even further. Please refer to the updated draft for updated and improved figures.

Reviewer comment (Experiments):
To my understanding, in the experiments, the baseline models are evaluated in the target domain without domain adaptation. In this context, it is reasonable that those methods cannot generalize well in the target domain. I was wondering if there are some existing domain adaption approaches that can be applied to the baseline models. The evaluation would be more convincing, comparing NeMo + 3DUDA with a competitor such as NeMo + another domain adaption method.

Author response:
To the best of our knowledge, our method appears to be the only current source-free image-only UDA method for object pose estimation. It is unclear what kind of methodology we could combine with NeMo for UDA since that would be nontrivial and could be an interesting new research avenue. We do have baselines (P3D [a] and DMNT [b]), which are improvements over the NeMo model and claim to be robust to OOD as well. P3D uses synthetic and real data combinations and different training methods for robust 3D pose estimation. DMNT uses deformable meshes and neural textures for robust pose estimation. 

[a] Yang et. al. - Robust category-level 3d pose estimation from synthetic data

[b] Wang et. al. - Neural textured deformable meshes for robust analysis-by-synthesis

Reviewer comment (Experiments):
The pre-render feature maps are generated from the source mesh. As the vertex features are not updated here, how to make sure those feature maps are reliable in the target domain? Are there situations in which the method might be stuck in local optima or even diverge? Conducting an ablation study on the pre-rendered feature maps would be valuable.

Author response:
We discuss this exact problem in Section 3.2 (Multi-Pose Initialization) and provide an ablative example in Figure 7. The source model does sometimes get stuck in local optima when evaluated on the target domain data, and therefore the estimated result may depend on the pose initialization. Therefore, we do multi-pose initializations for our method, and use the final estimates to update only robust, local vertex features irrespective of the global object pose estimated. We have added an ablation for multi-pose initializations in our updated draft (Table 4).

Reviewer comment (Evaluation):
It seems the method is time-consuming due to the iterations and pre-rendering. However, the actual time consumption during testing is unclear.

Author response:
Thanks for pointing out that this should have been discussed in the paper. Evaluation time for our model (if not doing batched inference) is ~0.33 seconds/sample on a Nvidia RTX 2060 GPU. We have added it in the paper.

Reviewer comment (Experiments):
It is recommended to evaluate the proposed method on commonly used datasets for category-level pose estimation, such as REAL275 [1] or Wild6D [2].

Author response:
**W1. It is recommended to evaluate the proposed method on commonly used datasets for category-level pose estimation, such as REAL275 or Wild6D**.

- Our focus in this work is largely on outdoor setups (e.g. OOD-CV) due to relatively more diverse OOD conditions (weather, lighting, context, etc.) available in such datasets. 

- Both WILD6D+ and REAL275 datasets are indoor RGBD datasets in NOCS [1] format, which are used primarily for robotic applications and have 5-6 categories. Historically, works have either focused on datasets like Pascal3D+, ObjectNet3D+ or REAL275, WILD6D+ and we follow the same convention. This is likely due to incompatibility between the camera parameters and data formats of the two kinds of datasets. It is not trivial to convert one to another (or convert models of one kind to another) and is beyond the scope of this review period due to technical and time-related limitations. We believe that combining these two baseline branches would strengthen the research field, but it remains a future work.

- Most models [3,4,5] using these datasets rely on the depth channel to run PnP or combine depth with NOCS to solve pose so they cannot be trivially modified as an RGB model (to compare with our method) and require extensive work.

- We therefore follow the convention of datasets used in previous works related to our pose estimation method.

As an alternative for the Synthetic to Real evaluation done in REAL275 and WILD6D+ benchmarks, here are the results for synthetic to real dataset using the SyntheticP3D source domain dataset[2], which consists of 6 synthetic objects. The real dataset is Pascal3D+. 

| Methods           | $\pi/6$ Accuracy | $\pi/18$ Accuracy |
|-------------------|:----------------:|:-----------------:|
| Resnet-50 General |       53.5       |        13.2       |
| P3D               |       76.5       |        41.3       |
| NeMo              |       71.8       |        39.5       |
| Ours              |       88.9       |        66.7       |


[1] Wang et al. - Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.

[2] Yang et al. - Robust Category-Level 3D Pose Estimation from Synthetic Data.

[3] Lee et al. - TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation.

[4] Zhang et al. - Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild.

[5] Liu et al. - IST-Net: Prior-free Category-level Pose Estimation with Implicit Space Transformation.

Reviewer comment (Novelty):
It would be beneficial to include relevant works [3,4,5,6] in the paper to provide a comprehensive overview of the existing literature in the field.

Author response:
**W2. It would be beneficial to include relevant works [3,4,5,6] in the paper to provide a comprehensive overview of the existing literature in the field.**

Thanks for pointing out the works. We have added the mentioned references in the updated draft.

Reviewer comment (Theory):
How does 3DUDA address the issue of minimizing the impact of translation and object size on neural feature rendering?

Author response:
**Q1. How does 3DUDA address the issue of minimizing the impact of translation and object size on neural feature rendering?**

The problem that we are focusing on in this work is UDA for 3D pose estimation, therefore, we are not evaluating our models for translation. Many works utilize object detectors for localizing objects and estimating translation/scale, and our method can be extended in a similar fashion. We do not evaluate our model for scale robustness but we can optimize the distance from camera parameters and therefore are able to deal with the object scale changes in our current datasets.

Reviewer comment (Experiments):
What values are set for the number $R$ of vertices of the mesh and the hyperparameter of $N$ of the clutter model? How do they impact the performance of the method?

Author response:
**Q2. What values are set for the number of vertices of the mesh and the hyperparameter of $N$ of the clutter model? How do they impact the performance of the method?**

The number of vertices used is approximately 1000 for all categories. We take 1000 as the number of vertices, since the distance between nearest vertices after projection is roughly the same as the stride of the backbone network. Halving the number of vertices does not have a significant effect on the results. However, doubling the number of vertices leads to significant deterioration. This is likely due to a large number of vertices corresponding to the same feature vector, which complicates the feature learning process. Here is an ablation analysis - 

| Number of vertices | Median Error | $\pi/18$ Accuracy | $\pi/6$ Accuracy |
|:------------------:|:------------:|:-----------------:|:----------------:|
|         500        |     3.56     |        90.1       |       93.9       |
|        1000        |     3.03     |        91.0       |       94.4       |
|        2000        |     20.63    |        50.9       |       66.3       |

Our method is not sensitive to the change in the number of clutter models. This could be attributed to the fact that our similarity threshold can be construed as a clutter model with uniform distribution. We do see small improvements as the number of clutter models is drastically increased, but that comes at a high computational cost. Here is an ablation analysis for the same (Table 5 in the updated draft).

| Clutter ($N$) | Median Error | $\pi/18$ Accuracy | $pi/6$ Accuracy |
|:-------------:|:------------:|:-----------------:|:---------------:|
|       1       |     3.01     |        90.1       |       95.1      |
|       5       |     3.03     |        91.1       |       94.4      |
|       20      |     2.99     |        92.3       |       95.5      |
|       35      |     2.79     |        91.7       |       94.5      |

Reviewer comment (Presentation):
Poor presentation — While the paper holds good insights and corresponding technological contributions, the presentation of the whole paper is relatively poor, making it hard to follow the overall message.

Author response:
We regret that the presentation of our initial draft was deemed poor by the reviewer. According to their suggestions, we have updated our paper. We have reframed the main content to make it easier to follow, improved the quality and placement of figures and made sure that all figures are properly referenced in the text. We request the reviewer to take a look at the newer version to see if the presentation issues are addressed.

Reviewer comment (Presentation):
Figure placement is not well-aligned with the text context. Figure 3, which seems to be a very important description explaining one of the main contributions (matching sub-vertices), is not mentioned in the main manuscript.

Author response:
We regret that the presentation of our initial draft was deemed poor by the reviewer. According to their suggestions, we have updated our paper. We have reframed the main content to make it easier to follow, improved the quality and placement of figures and made sure that all figures are properly referenced in the text. We request the reviewer to take a look at the newer version to see if the presentation issues are addressed.

Reviewer comment (Presentation):
The main manuscript refers to figures in the appendix very often, which is inconvenient to read, while some of these figures (e.g., Figure 5) seem important enough to be contained in the main paper for effective description.

Author response:
With our restructuring of the paper, we have moved some important details up from the appendix to the main text as we agree with the reviewer that some of them are too important to be deferred to the appendix. We have updated old Figure 3 and moved old Figure 5 to the main draft (refer to Figures 1 and 3 in the updated draft). Unfortunately, numerous experimental analysis performed in this work can not all fit in the space constraints of the main text. We hope that the current restructuring in the new draft bolsters the contributions that we made with this work. We are happy to take more suggestions regarding the same and address any more issues that the reviewer thinks still exist.

Reviewer comment (Experiments):
Ablation studies regarding several design choices of the proposed method should be contained in the main manuscript as well, since they can effectively validate the authors’ claim.

Author response:
With our restructuring of the paper, we have moved some important details up from the appendix to the main text as we agree with the reviewer that some of them are too important to be deferred to the appendix. We have updated old Figure 3 and moved old Figure 5 to the main draft (refer to Figures 1 and 3 in the updated draft). Unfortunately, numerous experimental analysis performed in this work can not all fit in the space constraints of the main text. We hope that the current restructuring in the new draft bolsters the contributions that we made with this work. We are happy to take more suggestions regarding the same and address any more issues that the reviewer thinks still exist.

Reviewer comment (Experiments):
Except for the OOD-CV results, the only previous baseline is NeMo. The authors should explain why there can't be other methods in this comparison, as being better than only one baseline is not enough to claim the superiority of the proposed method. Additionally, while the paper mainly focuses on data corruption scenarios, it would strengthen the motivation if the approach could also be compared in conventional category-level object pose benchmarks like CAMERA and REAL275 datasets provided by NOCS, to show it can be generally applied to conventional syn-to-real UDA scenarios.

Author response:
We thank the reviewer for drawing attention to this point. We have added results for other baselines' extended results tables in our updated draft (Table 6 and Table 7 (page 22 and 23)) which includes four more comparative methods for every experiment. As can be gleaned from these baselines, like OOD-CV, our method performs exceedingly well even when dealing with extreme UDA setups.

Thank you for drawing our attention to CAMERA and REAL275 datasets. 

 - Our focus in this work is largely on real, outdoor setups (e.g. OOD-CV) due to relatively more diverse OOD conditions (weather, lighting, context, etc.) available in such datasets. 

- Conventionally, 3D pose estimation works have either focused on setups and datasets like PASCAL3D+, ObjectNet3D or on more robotic application based synthetic to real datasets like REAL275, CAMERA and WILD6D+. This is due to some distinct technical and usage difference between these dataset setups. It’s non-trivial to convert the camera calibrations, annotations and dataformats from one to another and would require a significant amount of time and is likely out of scope for the review period. To the best of our knowledge, there is no recent work which has used both of these kinds of datasets together, and we believe this would be a good future work to combine these two types of datasets into one format.

- Both CAMERA and REAL275 datasets are indoor RGBD datasets, which are used primarily for robotic applications and have 5-6 categories. Most models[3,4,5] using these datasets[1] rely on the depth channel to run PnP or combine depth with NOCS[1] to solve pose so they cannot be trivially modified as an RGB model (to be compared with our method) and require extensive work. 

- As recommended by the reviewer - for a conventional synthetic to real analysis, here are results for image only unsupervised domain adaptation from SyntheticP3D source domain dataset[2] (which has 6 synthetic object classes) to real Pascal3D+ target domain dataset.


| Methods           | $\pi/6$ Accuracy | $\pi/18$ Accuracy |
|-------------------|:----------------:|:-----------------:|
| Resnet-50 General |       53.5       |        13.2       |
| P3D               |       76.5       |        41.3       |
| NeMo              |       71.8       |        39.5       |
| Ours              |       88.9       |        66.7       |

[1] Wang et al. - Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.

[2] Yang et al. - Robust Category-Level 3D Pose Estimation from Synthetic Data.

[3] Lee et al. - TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation.

[4] Zhang et al. - Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild.

[5] Liu et al. - IST-Net: Prior-free Category-level Pose Estimation with Implicit Space Transformation.

Reviewer comment (Experiments):
Ground truth pose illustrated in Figures 1 and 5 seems to be not perfectly aligned with the image. The authors should explain how these GTs are obtained, and how they are utilized — are they only used for evaluation?

Author response:
We want to clarify that the CAD objects in old Figure 1 and 5 (updated Figure 3) are for visualisation only. We use Shapenet3D models and rotate them according to the dataset’s annotations (GT) and model’s (Ours, NeMo) pose estimation results. There might be some noise in GT pose annotation as well the visualization setup which might be the reason for non-perfect alignment. The images as well as their ground truth annotations (GTs) in the figures are obtained from the OOD-CV dataset.

Reviewer comment (Experiments):
Ablation is not enough, (e.g., Ablation on top-n

Author response:
It is unclear what the reviewer means by missing ablation studies **(top-n)** in our 3-D pose estimation problem. We believe that we have ablated all major design choices and method motivations in Section A.5. In summary, our ablation (Section A5) includes ablation of	

a. Our motivation and observations regarding local part robustness (Figure 5, 9, 10),

b. The efficacy of the Selective Vertex Feature Update,

c. False Positives in vertex-Feature Similarity,

d. Effect of the learnable concentration parameter,

e. Multi-Pose Initialization,

f. Quality of vertex Features after Selective Adaptation.

In addition, we have added the following more ablations in the revised version: 

g. Effect of number of clutter features hyperparameter,

h. Additional ablation figures to prove the local part robustness observation mentioned in the previous response paragraph. (Figures 9 and 10)

If the reviewer is referring to the top-n multipose initializations, we do observe faster convergence with increasing number of initial poses. However, during multipose initialization, we only consider estimated poses whose rotational matrix distance (refer to Section 4 (Metrics paragraph)) from one another is at least $\pi/5$. If the estimated poses are less than the threshold for all, we only consider the pose with the highest similarity to the image feature vector. We did not see any significant improvement in accuracy using more than 5 initial poses for our experiments. Here are ablation results for the car category for different numbers of pose initializations when adapted on OOD-CV (Combined dataset).

| Number of initialized poses | Epochs range for convergence | $\pi/18$ Accuracy |
|-----------------------------|------------------------------|-------------------|
|              1              |            130-170           |        95.3       |
|              3              |            100-120           |        97.1       |
|              5              |             50-90            |        97.1       |

If there are specific ablation studies that the reviewer feels are missing, we would be happy to run them.

Reviewer comment (Experiments):
I think this method is similar to the iterative optimization often used in instance-level post-processing. It is unfair to compare this method with other single forward methods.

Author response:
It will be helpful if the reviewer can clarify which methods or provide citations of the works that use iterative optimization and are similar to our UDA method. To the best of our knowledge, our method is the first method to do source-free, image-only unsupervised domain adaptation for 3D object pose estimation. If the reviewer refers to the render-and-compare optimization used in our model, we would like to clarify that this is indeed a common optimization method used for a number of pose estimation methods such as [a,b,c] (this is not our contribution) and it is standard practice to compare such methods with feedforward methods as can be seen in these works. 

[a] Li et al. - DeepIM: Deep Iterative Matching for 6D Pose Estimation

[b] Wang et al. - NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation

[c] Chen et al. - Category level object pose estimation via neural analysis-by-synthesis

Reviewer comment (Writing):
I'm not sure if it's right to say "normalized real-valued feature activations"? (The fourth line from the bottom of the third page)

Author response:
The phrase "normalized real-valued feature activations"  refers to feature activations of a neural model which are real numbers and are normalized to have unit norm. We request the reviewer to be more specific as to why they believe the usage is incorrect.

Reviewer comment (Presentation):
The figures in this article are very rough and difficult to understand.

Author response:
We have updated the visual clarity of our figures in the main draft (refer to Figures 1, 2 and 3) and modified them to be easier to understand. We hope that the revised figures address this concern of the reviewer.