Paper: Depth-Guided Self-Supervised Learning: Seeing the World in 3D (ICLR-2024)

Reviewer comment (Presentation):
Figure 1 lacks details. For instance, it is unclear what "2D augmentation" is being performed in PixDepth. It would also be good to visually represent your SSL objective.

Author response:
We thank the reviewer for these suggestions. 2D Augmentation refers to the standard set of augmentations applied on the 2D image. For instance, random crop, flip, color jitter etc. We don’t show the SSL objective here explicitly because we are using several standard methods for SSL and the SSL algorithm itself is not the focus of our work. Our focus is on extending input with a depth map and using augmentations that correspond to different (inferred) 3D views.

Reviewer comment (Novelty):
Most improvements come from adding the depth channel, which I could not consider an important contribution.

Author response:
We note that 3D Views and the ensemble of Depth + 3D Views also improve performance over state-of-the-art baseline SSL methods, allowing us to argue for the broader claim that depth signals are valuable early in the vision pipeline. Perhaps the reviewer considered the contribution of a depth channel as unimportant because it is such a minor change to the architecture. We agree it is a minor architectural tweak, but wish to emphasize that it is a significant conceptual change: very little work in the field considers computations to extract depth from 2D images to precede object recognition.

Reviewer comment (Experiments):
What is the reason behind the statement of improvements of SwAV in Table 1 when no results for SwAV are provided?

Author response:
We do not show results with SwAV + 3D views because the multi-crop setting used in SwAV makes it non-trivial to be adapted to 3D Views. We have mentioned this in the paper but we will update the paper to clarify.

Reviewer comment (Experiments):
For the results on imagenet-100 no improvement is achieved, it is actually the opposite. This is not clearly reflected in the text in a tricky way. Why are there no results with the combination method (depth + 3D)?

Author response:
We are not sure if we understand the question correctly. We report results with depth + 3D in Table 2 on ImageNet-100. We do not report results Depth + 3D on ImageNet-1k due to compute constraints. Apologies, we should have mentioned this fact in the article text.

Reviewer comment (Experiments):
I am afraid that the added robustness to corruptions in ImageNet-C and ImageNet-3DCC comes from the robustness of the depth estimation network (trained with depth GTs over a considerable amount of data).

Author response:
We begin by emphasizing that the depth-estimation network need not be used during inference. As Section 4.5 (Table 6) indicates, when depth is unavailable at inference, the proposed method is still better than the corresponding baseline SSL method. 
Regarding the reviewer’s point that the depth-estimation network confers some robustness benefit to the learned embeddings, we certainly agree, although the benefit is indirect through the SSL method. Our investigation was focused on the empirical question of whether such an indirect benefit would indeed be obtained.

Reviewer comment (Novelty):
Using depth signals as augmentation is not new in SSL. Previous works, e.g., DepthContrast, have explored it thouroughly.

Author response:
We thank the reviewer for pointing us to DepthContrast [1]. As we understand it, DepthContrast requires ground truth depth information in the form of point-cloud data and relies on format-specific encoders (like PointNet++ for point clouds and UNet for Voxels). In contrast, our method takes a standard 2D RGB image input and assumes only that depth maps can be estimated from a separate module, allowing our method to be  seamlessly integrated into standard SSL pipelines. More importantly, the motivation behind the DepthContrast and our method is very different.  

We have discussed comparisons with other related work in the Appendix A.

Reviewer comment (Experiments):
The proposed method lacks generalizbility. Though it can be adopted to any SSL frameworks, the adopted depth estimation model is supervised trained on several datasets. The performance of depth estimation can not gurantee in scenarios that have a huge domain gap compared to the trained datasets.

Author response:
Depth-Prediction Transformer (DPT) [2] is robust to various natural image datasets as shown in the paper via zero-shot cross dataset transfer. It is due to these reasons that various works in 3D reconstruction [3, 4] and view synthesis [5, 6] use DPT to estimate depth maps.

Reviewer comment (Novelty):
Due to the utilization of the supervised depth-estimation model, it is questionable to claim the proposed method as a SSL framework.

Author response:
While the reviewer is technically correct, we urge the reviewer to appreciate our underlying premise. We argue that biological agents obtain depth information for free by continuous-time interaction with their environments, and so should a complete, continual learning AI agent. But rather than developing a complex multi-module AI system, we start with the premise that depth maps are available and then ask how they can benefit self-supervised learning. The means by which depth is obtained is not central to our investigation; however, the fact that depth is obtained via extraction from a 2D image (as the human brain does) is key. This work provides insights into and benefits of how to effectively utilize this depth information.

Reviewer comment (Experiments):
Experiments are all conducted on the subset of Imagenet or the modification of Imagenet. Results on more datasets are expected.

Author response:
We have reported results on the ImageNet-1k dataset (Table 3) consisting of 1.2 million real-world images. It is common in the SSL literature to report the pretraining results only on ImageNet-1k [7, 8, 9, 10]. We also emphasize that SSL in general is computationally expensive.

Reviewer comment (Experiments):
The proposed method is evaluated only in small-scale datasets (e.g., ImageNet-100, ImageNet-1k), which is insufficient to claim the effectiveness of the method for large-scale scenarios.

Author response:
We have reported results on ImageNet-1k which consists of 1.2 million real-world images. It is a bit shocking to see the reviewer classify this dataset as a small-scale dataset. Even in recent SSL papers [1, 2], Imagenet-1k is the only dataset used for evaluation with a single backbone (ResNet-50)

Reviewer comment (Experiments):
The proposed method is evaluated only in small-scale models (e.g., ResNet-18, ResNet-50), limiting the demonstration of its scalability to larger or more complex architectures.

Author response:
We have reported results on ImageNet-1k which consists of 1.2 million real-world images. It is a bit shocking to see the reviewer classify this dataset as a small-scale dataset. Even in recent SSL papers [1, 2], Imagenet-1k is the only dataset used for evaluation with a single backbone (ResNet-50)

Reviewer comment (Experiments):
The proposed method is evaluated only in a classification task, which does not sufficiently demonstrate its applicability to other types of downstream tasks.

Author response:
We respectfully disagree with the reviewer here. As we pointed out to other reviewers, the goal of SSL is to learn representations that are useful for various downstream tasks. It is common in SSL literature to primarily report linear classification accuracy. In our paper, we focused more on understanding the various components of our method (including depth channel dropout, quality of 3d views) rather than evaluating on a range of downstream tasks.

Reviewer comment (Novelty):
The authors should narrow down the scope of the proposed method from general SSL to a specific SSL method to make the claims more precise and supportable.

Author response:
We respectfully disagree with the reviewer here. As we pointed out to other reviewers, the goal of SSL is to learn representations that are useful for various downstream tasks. It is common in SSL literature to primarily report linear classification accuracy. In our paper, we focused more on understanding the various components of our method (including depth channel dropout, quality of 3d views) rather than evaluating on a range of downstream tasks.

Reviewer comment (Experiments):
Since the current backbone takes RGB-Depth images as inputs, targeting downstream tasks for RGB-D inputs (e.g., RGB-D segmentation, 3D reconstruction, 3D view synthesis, human pose estimation) would be a more reasonable choice to effectively claim the method's usefulness.

Author response:
We respectfully disagree with the reviewer here. As we pointed out to other reviewers, the goal of SSL is to learn representations that are useful for various downstream tasks. It is common in SSL literature to primarily report linear classification accuracy. In our paper, we focused more on understanding the various components of our method (including depth channel dropout, quality of 3d views) rather than evaluating on a range of downstream tasks.

Reviewer comment (Evaluation):
The current claim is too general and insufficiently supported by the presented evidence.

Author response:
We respectfully disagree with the reviewer here. As we pointed out to other reviewers, the goal of SSL is to learn representations that are useful for various downstream tasks. It is common in SSL literature to primarily report linear classification accuracy. In our paper, we focused more on understanding the various components of our method (including depth channel dropout, quality of 3d views) rather than evaluating on a range of downstream tasks.