Paper: 3D Object Representation Learning for Robust Classification and Pose estimation (ICLR-2024)

Reviewer comment (Novelty):
Wang et al. achieve very comparable results, and in some cases superior performance (e.g. for L3) on Occluded PASCAL3D+ (Compare Table 1 in Wang et al. to Table 1 in the submission). I would have expected a much more detailed comparison to the most relevant papers (in particular "Neural Textured Deformable Meshes for Robust Analysis-by-Syntheses" by Wang et al.), with clear articulation of what the differences are, and pros and cons of the proposed method with respect to these alternatives.

Author response:
We are sorry for the confusion with the related paper [G]. We cited the paper for the sake of completeness, and should have better pointed out that it is contemporaneous concurrent work that was **not published and only available on ArXiv** at the time of submission.  We note that according to ICLR reviewer guidelines we should therefore not be required to compare it as a baseline in our experiments (see last answer in the FAQ in [F]). Nevertheless, when comparing our method with [G] the reviewer must have misread the results of the Tables in [G] since our method outperforms [G] significantly at classification and pose estimation in all IID and OOD scenarios. In addition, as the reviewer requested, we contacted the authors and received their code to test on the corrupted-P3D+ dataset. We report the results in Table 4, and can observe that our model outperforms all types of corruption with a large margin, and we are around 20 percent higher overall.

We further want to emphasize that, our proposed method performs classification efficiently in real-time, which takes on average 0.02 seconds per image, whereas [G] simply runs a brute force search during inference which takes on average 0.5 seconds per image on the Pascal3D+ dataset under the same setup, suffering from a 25 times slower inference in addition to the lower accuracy and robustness, and hence making it practically infeasible.

The way how we formulate the contrastive loss is also different, which can be illustrated by the loss function in two papers. We use a vMF-based contrastive loss that encourages features to be distinct through a dot product operation whereas [G] uses a Euclidean-distance based loss estimating the distance between feature vectors. 
The main focus of [G] is different from this paper. Its goal is to obtain deformable (exact) geometry of the object in its forwarding.
In summary, these two papers have major differences in both low-level implementation and high-level goals, and performance [G] in terms of both accuracy and efficiency for the classification task.

Reviewer comment (Novelty):
Could you please provide a detailed explanation of the differences with respect to the most similar approaches (all work by Wang et al., in particular Wang et al. 2023)?

Author response:
We are sorry for the confusion with the related paper [G]. We cited the paper for the sake of completeness, and should have better pointed out that it is contemporaneous concurrent work that was **not published and only available on ArXiv** at the time of submission.  We note that according to ICLR reviewer guidelines we should therefore not be required to compare it as a baseline in our experiments (see last answer in the FAQ in [F]). Nevertheless, when comparing our method with [G] the reviewer must have misread the results of the Tables in [G] since our method outperforms [G] significantly at classification and pose estimation in all IID and OOD scenarios. In addition, as the reviewer requested, we contacted the authors and received their code to test on the corrupted-P3D+ dataset. We report the results in Table 4, and can observe that our model outperforms all types of corruption with a large margin, and we are around 20 percent higher overall.

We further want to emphasize that, our proposed method performs classification efficiently in real-time, which takes on average 0.02 seconds per image, whereas [G] simply runs a brute force search during inference which takes on average 0.5 seconds per image on the Pascal3D+ dataset under the same setup, suffering from a 25 times slower inference in addition to the lower accuracy and robustness, and hence making it practically infeasible.

The way how we formulate the contrastive loss is also different, which can be illustrated by the loss function in two papers. We use a vMF-based contrastive loss that encourages features to be distinct through a dot product operation whereas [G] uses a Euclidean-distance based loss estimating the distance between feature vectors. 
The main focus of [G] is different from this paper. Its goal is to obtain deformable (exact) geometry of the object in its forwarding.
In summary, these two papers have major differences in both low-level implementation and high-level goals, and performance [G] in terms of both accuracy and efficiency for the classification task.

Reviewer comment (Novelty):
More detailed discussions and comparisons with NeMo. NeMo is highly related to the proposed paper, but the discussion is missing in the introduction and related works. As far as I can see, the 3D object representation in the paper is already proposed by NeMo. NeMo already finds that using a cubic for an object class can achieve good performance in 3D pose estimation. They also use contrastive loss for training and render-and-compare strategy for pose estimation as in this paper. I think the difference is that the paper extends this existing 3D representation to training on multiple categories and uses it for classification. Based on this point, I think a huge part of the contributions of this paper belong to NeMo, and the real contributions are limited.

Author response:
Our method indeed builds on several previous works while substantially extending them.
Compared to NeMo, we make several key contributions: (1) An architecture with a class-agnostic backbone, while NeMo uses a separate backbone for each object. This enables us to introduce a class-contrastive loss that enables object classification, while NeMo is not capable of classifying images. As illustrated in the t-SNE plots in reviewer-wnbh’s answer. (2) A principled way of speeding up the inference without having to perform render-and-compare, which advances the brute-force testing and optimization of other concurrent methods [G] (see answer to reviewer-lehm for more details). (3) A comprehensive mathematical formulation that derives a vMF-based contrastive training loss that is different from Euclidean-distance based contrastive loss of NeMo. (4) We demonstrate the possibility to exploit individual vertices during inference rather than considering vertices collectively as a mesh in NeMo, opening potentials tackling segmentation or object part detection tasks. Finally, all our advances lead to (5) substantial improvements in terms of OOD robustness over all baselines at image classification, while at the same time performing on par with models that were specifically designed for robust pose estimation, such as NeMo.

We appreciate the reviewer's helpful feedback, as well as the time and effort the reviewer has put into reviewing our work. We are happy that the concerns of the reviewer have been addressed. We also toned down the claim about introducing a new representation in the paper (**see changes in purple** in the newest revision of the manuscript). 


While the wording of our contribution in the abstract was not optimal, we want to further emphasize that our **new class-agnostic backbone** and the **novel class-contrastive loss** lead to a representation that goes beyond the original representation in NeMo, as it leads to a much better disentanglement of the class information, while at the same time disentangling the 3D pose (see T-SNE plots provided to reviewer wnbh). Additionally, our representation used for classification is significantly different from NeMo, as we do not rely on the 3D mesh during classification and hence do not require the expensive render-and-compare optimization for classification.

Reviewer comment (Experiments):
For the evaluation of classification, I think the paper should compare with the state-of-the-art 2D object detectors for classification to show the advantages. For example, I think the YOLO can be easily trained using the same training data as the proposed method, i.e., the projected 2D bounding boxes from the 3D cubics.

Author response:
Thank you for the suggestion. We trained YOLO using the official repository [D] on the Pascal3D+ data with standard data augmentation in terms of scale, rotation and mirroring data augmentation. From the table below, we can observe that YOLO obtains similar performances for IID data, but suffers even more than the other classification baselines under out-of-distribution shifts in terms of occlusion, whereas our proposed model achieves much stronger robustness. While refining our YOLO results through additional investigation might be possible, we speculate that utilizing the official training code for YOLO might lead to potential overfitting, given its original focus on detection rather than classification. We anticipate that more advanced data augmentation (such as CutOut or CutPaste) techniques could enhance generalization in YOLO's performance. Notably, it is crucial to emphasize that our method, in contrast, does not necessitate data augmentation and demonstrates effective generalization.

**Table 3:** Classification results on P3D+ and occluded-P3D+ comparing Yolo and our approach. 

| Nuisance  | L0   | L1                | L2                | L3                | Mean              |
|-----------|------|-------------------|-------------------|-------------------|-------------------|
| YOLO      | 99.3 | 62.4              | 44.9              | 25.0              | 44.1              |
| Ours      | **99.5** | **97.2**        | **88.3**          | **59.2**          | **81.6**          |

[D]: https://docs.ultralytics.com/

Reviewer comment (Evaluation):
There is a lack of an efficiency comparison with existing methods. It seems that the high efficiency in classification is a critical contribution of this work. A comparison of the inference speed and the number of parameters between the proposed framework and other methods can further support this claim.

Author response:
**There is a lack of an efficiency comparison with existing methods. It seems that the high efficiency in classification is a critical contribution of this work. A comparison of the inference speed and the number of parameters between the proposed framework and other methods can further support this claim.**

In terms of classification, our method achieves similar real-time performance as the other baselines, processing over 50 images per second. Among the baselines there is a large variation in terms of the number of parameters (Resnet: 84M, Swin-T: 28M, ViT-b-16: 86M), but we do not observe any correlation with OOD robustness. Compared to those, our model contains 83M which is comparable. However, concerning the pose estimation baseline Nemo, we have a much lower parameter count (83M vs 996M) and faster inference speed. We will make sure to emphasize this more in the revised paper.

Reviewer comment (Novelty):
Authors start the paper with the statement: “we pioneer a framework for 3D object representation..”, the statement is in contrast with various papers that investigate object representation using techniques on how to map image features to 3D models of faces and vertices. For example M1 or M2.

Author response:
**Authors start the paper with the statement: “we pioneer a framework for 3D object representation..”, the statement is in contrast with various papers that investigate object representation using techniques on how to map image features to 3D models of faces and vertices. For example M1 or M2.**

We acknowledge the relevance of these papers and their techniques in mapping image features to 3D models. We will revise our claim in a more nuanced manner to better align with the state of the art. Additionally, we will incorporate references to M1 and M2 in our related work section, recognizing their significance in the exploration of object representation techniques.

Reviewer comment (Experiments):
Do baselines use 3D? (It looks that authors use 3D but baselines don't).

Author response:
**Do the baselines use 3D? (It looks that authors use 3D but baselines don't).**

To the best of our knowledge, our method is the first in the literature to use a 3D representation for image classification. Moreover, it is not clear how to provide existing architectures with additional 3D annotation. Based on the feedback of reviewer-wnbh, we conducted an experiment where we extended several of the classification baselines with an additional pose estimation head. In particular, we follow the popular approach of casting the pose estimation problem as bin classification by discretizing the pose space [C]. We train the models with both the image classification and pose estimation loss until convergence. As can be observed from the table below, the additional 3D information does not improve the classification performance. Moreover, this claim is supported by the t-SNE plots shown in [Figure 2](https://drive.google.com/file/d/1QYy-ExWpWtytRur3oK53yxTSEg951wud/view?usp=share_link). Simply adding 3D-pose information via an additional branch does not induce the network to learn any 3D-aware structure in the representation, whereas the 3D-aware structure is directly built-in in our approach, thus leading to large benefits in terms of robustness and interpretability.

If the reviewer has any alternative baselines to suggest that leverage the advantages of 3D, we are open to incorporating and testing them in our evaluation.


**Table 2:** Classification results on P3D+ and occluded-P3D+ comparing resnet baseline performances by training a network per task and a single network for both. We don’t observe any synergy in the unique model trained on both tasks. 

| Nuisance | L0    | L1  | L2  | L3 | Mean  | 
|------------------|------|----------|---------|-----|----|
| Resnet (cls-only)| **99.3** | 93.8  | **77.8**| **45.2** | **72.3** |
| Resnet (cls&pose)| 99.2 | **93.9** | 77.6  | 45.0 | 72.2  |

Reviewer comment (Experiments):
Confusion in dataset creation: Originally, Pascal authors claim that 10812 testing images from Pascal3D+ are used to create Occluded PASCAL3D (from official github page). Authors use 10812 Pascal3D+ images as a validation set. Is the authors’ validation imageset the same as the testset in Occluded Pascal? If so, then it does look like a bug in dataset creation. This question and the question above are crucial for deciding between acceptance or rejection.

Author response:
**Im puzzled in the dataset creation. Originally, pascal authors claim that 10812 testing images from Pascal3D+ are used to create Occluded PASCAL3D (from official github page). Authors use 10812 Pascal3D+ images as a validation set. Is the authors validation imageset same as the testset in Occluded Pascal? If so, then it does look like a bug in dataset creation. This question and the question above are look for me as crucial for deciding between acceptance or rejection.**

We are sorry for the confusion, this is a mistake in the manuscript. The word “validation” should be replaced with “testing”. We download and use the dataset as released by [C] where the test data from Pascal3D+ was corrupted with random occluders to generate the occluded-Pascal3D dataset.

Reviewer comment (Experiments):
Although the conference is called Conf. on Learning Representation, the paper looks to be structured for a computer vision conference. For example, I would be more interested in how the proposed representation (and its modification) affects the result of how the method expresses the objects rather than many classification results that are provided (I'm not saying skip them all) and are better suited for CVPR-like venues. I would like to see some examples where the representation is beneficial to other methods and where it is in contrast worse.

Author response:
**I would like to see some examples where the representation is beneficial to other methods and where it is in contrast worse.**

Based on the request, we present the T-SNE in [Figure 2](https://drive.google.com/file/d/1QYy-ExWpWtytRur3oK53yxTSEg951wud/view?usp=share_link) and [Figure 3](https://drive.google.com/file/d/10oM2QT8RpnxfmyIsOLBU4EL9PfzPs5H-/view?usp=share_link), illustrating the feature representations of ResNet, NeMo, and our approach.

In the context of classification, by looking at [Figure 2](https://drive.google.com/file/d/1QYy-ExWpWtytRur3oK53yxTSEg951wud/view?usp=share_link), it becomes evident that our method, much like any classification baseline, results in well-defined clusters for the classes. Conversely, with NeMo, there is significantly less clear clustering.

Regarding pose information in the representation, we observe in [Figure 3](https://drive.google.com/file/d/10oM2QT8RpnxfmyIsOLBU4EL9PfzPs5H-/view?usp=share_link) (only car samples from the test dataset are shown) that we can easily extract information about the 3D pose of the object (we only show the azimuth angle for illustration purposes) while a ResNet model that has been trained on both classification and pose estimation task does not have such information explicitly encoded. 

Therefore, the plots confirm that our approach exhibits class-aware features, enabling high classification performances while simultaneously incorporating the 3D pose awareness into the representation.