Paper: On the Viability of Monocular Depth Pre-training for Semantic Segmentation (ICLR-2024)

Reviewer comment (Experiments):
It would be interesting to see the segmentation performance on PASCAL VOC and COCO with the depth pretraining.

Author response:
We cannot report results on PascalVOC and COCO due to a lack of camera calibration on both datasets, unlike KITTI, CityScapes, and NYU-V2. Consequently, we cannot perform depth pre-training accordingly. Although it is possible to use a more generic depth model (e.g., DPT trained on a mixed dataset) for fine-tuning on both datasets, we attempted to conduct the experiment as suggested by the reviewer but encountered restrictions in compute resources (batch size restricted by GPU memory).

Reviewer comment (Writing):
The writing is poor, sometimes the notations are not well defined or clarified. For instance, in EQ.1, since we are doing pretraining for depth estimation, why in the loss function, the depth term is missing? Instead, there is the wrapped image? Similarly, the input of the loss function in EQ.2 is not clarified as well.

Author response:
Eq. 1 and 2 refer to the standard structure-from-motion formulation, which is a convention for inferring depth from multi-view/video data (please see Appendix B.1 for a discussion regarding this). Inferring depth does not require depth measurements as supervision. One only needs images with sufficiently exciting textures that are co-visible between them. We will clarify this in the paper revision.

Reviewer comment (Evaluation):
Some results are not sufficiently analyzed. For example, in the left figure of Fig.3, it seems that with larger training size, the bigger gain could be obtained. However, this is counter-intuition, which needs proper explanations.

Author response:
For Fig. 3-left, a larger training set size for fine-tuning semantic segmentation improves results. This shows that pre-training with depth not only improves sample complexity but also scales well with respect to it, which is not the case for pre-training with ImageNet, where it saturates after 64 samples.

Reviewer comment (Experiments):
We see the the main comparisons are between ImageNet classification-based pre-training vs Depth pre-training. One thing to consider is that depth pre-training happens in-domain with the same data to be fine-tuned. My question is whether the superiority of depth pre-training over ImageNet pre-training mainly relies on in-domain knowledge or data distribution. What if the depth pre-training is conducted on a different dataset rather than the one to be fine-tuned. Will it still has such significant improvement over classification pre-training?

Author response:
The superiority of depth pretraining over ImageNet is not contingent on in-domain pretraining. The last row of Table 2 shows initializing with weights from DPT, which does not include KITTI data for pretraining. Despite being out-of-domain, depth pretraining still improves over ImageNet pretraining. Please also see *'A general note to the reviewers and AC'* for a detailed discussion.  

We hope this response addresses the reviewer's concerns. If there are further concerns, we are more than willing to provide additional explanations.

Reviewer comment (Experiments):
The paper studies how depth-pretraining helps for downstreaming tasks. Since the main downstreaming task is still depth prediction, I feel that pre-training on depth prediction shows some improvements that are not that impressive to the community. Though segmentation is also studied on CityScape, maybe testing on more datasets would be more convincing.

Author response:
There may be some misunderstanding but the downstream task in this manuscript is always semantic segmentation. We show that pre-training on monocular depth estimation can in general improve downstream semantic segmentation.

Reviewer comment (Evaluation):
In Figure 2, I wonder why Depth-rand is on par with ImageNet pre-trained for ResNet-18, but the observation is very different for ResNet-50. I wonder if the authors have any thoughts on this?

Author response:
One possibility is ResNet50 has bigger capacity than ResNet18, so there can be more local minimum and the model is subject to overfitting. We will include the discussion in the paper as suggested.

Reviewer comment (Experiments):
It would be helpful to clarify the experiment settings in Table 1 at the beginning of the experiment section. I wonder if the authors may further clarify the difference between ‘Depth-Rand’ and ‘Depth’?

Author response:
We will clarify. Depth-rand refers to depth model trained from scratch (randomly initialized weights), while depth refers to the best depth model, trained from standard ImageNet initialization. We show that despite not using any annotation, only training for monocular depth on 3 hours of driving video, can leads to similar results to training on ImageNet, which is collected through crowd-sourcing with the assistance of around 15,000 paid annotators. Nevertheless, since ImageNet is there, one can make best use of it to train a better depth model, which further improving the performance yielding the best result.

Reviewer comment (Novelty):
The paper lacks innovation, as many existing works have already explored ways to combine depth and segmentation, while this paper mainly conducts more extensive experiments to summarize a more helpful pipeline, with no other contribution beyond that.

Author response:
Regarding the innovation of the paper, please refer to *A general note to the reviewers and AC*.

Reviewer comment (Experiments):
Almost all experiments in this paper assume the depth ground truths are reliable and calibrated with the RGB images. What if the depth ground truths are not reliable (contain noise or are not calibrated with RGB images)? How does this impact the performance of the proposed pipeline?

Author response:
This question is very interesting. As suggested by the reviewer, we introduce miscalibration to the depth model by allowing camera intrinsics to vary by up to 10%. Surprisingly, we achieve on-par improvement compared with using the ground-truth calibration on CityScapes. We posit that the connection between depth and object semantics is influenced by the object scale. A miscalibrated camera affects the object's scale, but it does not significantly change the object's semantics. We are more than happy to include this discussion in the paper.

We hope this response addresses the reviewer's concerns. If there are further concerns, we are more than willing to provide additional explanations.

Reviewer comment (Presentation):
Where is Table 4.1 (Page 6)?

Author response:
**R:** It’s a typo. We are referring to Table 2. Will correct.

Reviewer comment (Experiments):
There is potential unfairness in comparison. In Table 2, "Depth" used KITTI as pre-training set while testing on the same dataset. This comparison to the baseline (ImageNet pre-training) is unfair, because the training distribution is much closer to the testing distribution for "Depth". In contrast, ImageNet is characterized as object-centric and thus has a pretty large domain gap to KITTI, which is collected by autonomous cars. What about performance comparison on COCO between ImageNet pre-train and Depth pre-train? In Table 3, the "Supervised Segmentation" performance should be results that are trained on KITTI instead of MS-COCO if the authors used KITTI as both pre-training and testing data.

Author response:
**R:** Please see general note for in-domain training. Nonetheless, this is an advantage of pretraining for depth. One does not need to restrict the pretraining data to a particular dataset, but can leverage any of structure-from-motion (video), binocular stereo, or measurements of depth sensor (time-of-flight) to do pretraining.  

*In Table 3, the "Supervised Segmentation" performance should be results that trained on KITTI instead of MS-COCO if the authors used KITTI as pre-training and testing data.*

**R:** The reviewer suggests performing supervised pre-training, and apply supervised fine-tuning on the same task and same dataset, which breaks the purpose of pre-training as one can simply used the supervised “pretrained” network for inference without finetuning.

Reviewer comment (Experiments):
There is a lack of analysis on representation transferability. For example, how is the performance of Depth Pre-train over other segmentation datasets such as COCO and ADE20K? How about transferring to object detection? Transferability is one of the major functions of pre-training. The authors should present relevant experimental results.

Author response:
**R:** Thanks to the reviewer for the suggestion. We are indeed investigating the performance of incorporating depth in pre-training in our follow-up work, especially on different semantic-related tasks. But domain adaptation is beyond the scope of the current paper.