Paper: DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation (ICLR-2024)

Reviewer comment (Reproducibility):
Will the pretraining weights be released? Would the ImageNet depth data be released? This could be discussed.

Author response:
Q2. About the open source of DFormer.

In the supplementary materials, we have provided the source code for the finetuning.
Moreover, all estimated depth maps, pretrained weights, and source code will be made publicly available.

Reviewer comment (Evaluation):
In the introduction, it was argued: "the interactions are densely performed between the RGB branch and depth branch during finetuning, which may destroy the representation distribution". Do you have any observations to support this argument? E.g., some destroyed distributions or feature maps could be analyzed.

Author response:
Q3. Observation to support the argument within the introduction.

As stated in [1], the mean and variance of the BN layer can reflect the overall distribution center of input data and the distribution breadth of input data, respectively.
Fusing RGBD features within the RGB-pretrained models would bring in drastic changes in the feature distribution, which makes the previous statistic of batch normalization incompatible with the input features.
To verify whether this phenomenon exists, we visualize the statistics of batch normalization in a random layer of the DFormer in Fig. 11 in the new revision.
For the RGBD-pretrained DFormer, the running means and the variance of the BN layer only have slight changes after finetuning, illustrating the learned RGBD representation is transferable for the RGBD segmentation tasks.
While, for the RGB-pretrained DFormer, the statistics of the BN layer are indeed drastically changed after finetuning, indicating that the encoding is mismatched.
The situation causes difficulties in optimizing the RGB-pretrained weights for the RGBD scenes.
Following your suggestion, we also visualize some features of the DFormer that use the RGB and RGBD pretraining in Fig. 12 in the new revision. 

[1] An empirical study of adder neural networks for object detection. In NeurIPS, 2021.

Reviewer comment (Presentation):
ACNet (ICIP 2019) should be added to Table 1.

Author response:
Q4. Comparison to ACNet.

Thanks for your suggestion. We add this method in Tab.1 of the new revision.

Reviewer comment (Experiments):
The proposed model is highly efficient and it has large gains thanks to the RGB-D pertaining. If the RGB-D pertaining strategy is applied to heavier state-of-the-art RGBD segmentation models like CMX and CMNeXt, how much gain can be achieved? If it is possible, this could be assessed and would help provide a fairer comparison.

Author response:
Q6. Apply the RGB-D pretraining to other methods.

Following your suggestion, to make the comparison more fair, we pretrain the CMX (MiT-B2) on RGB-D ImageNet and it obtain about 1.4% improvement on the mIoU.
We also provide the RGB pretrained DFormers to provide more insights.
Under the RGB-only pretraining, DFormer-L still outperforms the CMX (MiT-B2) by about 1.1% with less computation cost.
For the RGB-D pretraining, compared to the CMX (MiT-B2), the improvement of DFormer-L is further enlarged from 1.1% to 1.5%, which we attribute to that the pretrained fusion weight within DFormer can achieve better and more efficient fusion between RGB-D data. 

| Methods | Pretrained weight | Param | Flops | NYU DepthV2 | SUNRGBD |
|  - | -|  :-:  |:-:  | :-:  | :-: |
| CMX (MiT-B2) | RGB-only | 66.6M | 67.6G | 54.4 | 49.7 |
| DFormer-B | RGB-only|29.5M | 41.9G | 53.3 | 49.5 |
| DFormer-L | RGB-only | 39.0M | 65.7G | 55.4 | 50.6 |
| CMX (MiT-B2) | RGB-D |66.6M | 67.6G | 55.8 | 51.1 |
| DFormer-B | RGB-D |29.5M | 41.9G |55.6 | 51.2 |
| DFormer-L | RGB-D | 39.0M | 65.7G | 57.2 | 52.5 |

Reviewer comment (Experiments):
Fig. 11 shows that the proposed module is sophisticated. Would it be nice to provide more detailed ablations to study other design choices based on this module architecture?

Author response:
Q7. More detailed ablation towards the block.

In Tab.5~8, we have provided the ablation experiments about the components of our RGB-D block, the pooling size of our GAA, as well as the fusion manners in GAA and LEA.
We notice that the modules that only encode RGB features may be not well ablated.
Here we provide more results for the modules that encode RGB features.
Due to limited time and computation resources, we use a short pretraining duration of 100 epochs on the DFormer-S.
In the future, we will add this ablation experiment of full pretraining duration to the main paper.
Note that the results in the below table only ablates the structure within the modules (gray part in Fig.11) that only process the RGB features. 

|DWConv Setting|Attention Operation|Param|Flops|NYUDepthV2|
|-|-| :-:  |:-: | :-:  | 
|DWConv $7\times7$|Hadamard Product|18.7M|25.6G|51.9| 
|DWConv $5\times5$|Hadamard Product|18.7M|23.9G|51.6| 
|DWConv $9\times9$|Hadamard Product|18.7M|27.1G|51.9| 
|DWConv $7\times7$|Addition|18.7M|25.0G|51.3|
|DWConv $7\times7$|Concatanation|19.3M|26.9G|51.7|

Reviewer comment (Experiments):
What is the effect of using different depth estimation models? How effective is the accuracy of depth estimation for RGBD model pre-training, and will there be accumulation of errors?

Author response:
Q1. Discussion about different depth estimation methods.

In the Sec. A of our appendix, we have conducted experiments to explore the effect of the depth maps generated by different methods.
Specifically, we adopt the Adabins[1] (trained on 26K pairs on KITTI [2]) and more advanced Omnidata[3] (trained on 4M RGB-D pairs, adopted in MultiMAE), where the Omnidata uses more training data and obtained more advanced zero-shot capacity.
The estimated depth maps of the two methods are shown in the Fig.10.
After finetuning, the DFormer-B that pretrained with different depth maps achieves the same performance, i.e. 55.6.
This indicates the quality of depth maps generated by recent depth estimation methods has little effect on the performance of DFormer.
Besides, we will also explore replacing the depth with other modalities and analyze the influence of different estimation methods on our model in the future.

[1] Adabins: Depth estimation using adaptive bins. In CVPR, 2021.

[2] Vision meets robotics: The kitti dataset. IROS 2013.

[3] Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In ICCV 2021.

Reviewer comment (Experiments):
How is the comparison between the fusion building block and the fusion module proposed in previous methods, such as cmx? Also, do the authors try to perform RGB-D pretraining for other methods, so as to perform a more comparable setting?

Author response:
Q3. Pretraining the CMX.

Thanks for your suggestion. To make the experiments more comprehensive, we use the pretraining manner of DFormer to pretrain the CMX (MiT-B2) and then finetune the model via their official code.
As shown in below, the pretraining manner in this paper brings about a 1.4% improvement for the CMX (MiT-B2), but our DFormer-L still outperforms CMX (MiT-B2) that adopts the same pretraining manner by a large margin.
We attribute this to the well-designed building block and the efficient encoding design of our DFormer.



| Methods | Pretraining | Param | Flops | NYU DepthV2 | SUNRGBD |
| - |-|:-: | :-: | :-: | :-: |
| CMX (MiT-B2) | Only RGB | 66.6M | 67.6G | 54.4 | 49.7 |
| CMX (MiT-B2) | RGB-D |66.6M | 67.6G | 55.8 | 51.1 |
| DFormer-B | RGB-D | 29.5M | 41.9G |55.6 | 51.2 |
| DFormer-L | RGB-D | 39.0M | 65.7G | 57.2 | 52.5 |

Reviewer comment (Experiments):
How does the DFormer perform if only RGB pretrain + D initialization for finetuning?

Author response:
Q4. DFormer with RGB pretrain + D initialized

To provide more insights, we pretrain the RGB part via RGB-only pretraining and randomly initialize the depth part of the DFormer, and the results are shown in the below table. 
Under the RGB-only pretraining, DFormer-L still outperforms the CMX (MiT-B2) by about 1.1% with less computation cost.
For the RGB-D pretraining, compared to the CMX (MiT-B2), the improvement of DFormer-L is further enlarged from 1.1% to 1.5%, which we attribute to that the pretrained fusion weight within DFormer can achieve better fusion between RGB-D data. 

| Methods | Pretrained weight | Param | Flops | NYU DepthV2 | SUNRGBD |
|  -| :-: |:-:  |  :-:  | :-:  |   :-:  |
| CMX (MiT-B2) | RGB-only | 66.6M | 67.6G | 54.4 | 49.7 |
| DFormer-B | RGB-only|29.5M | 41.9G | 53.3 | 49.5 |
| DFormer-L | RGB-only | 39.0M | 65.7G | 55.4 | 50.6 |

Reviewer comment (Experiments):
How is the effect and how is improvement from the light hamburger decoder in the proposed model? Whether the authors try to use other decoders?

Author response:
Q5. DFormer with different decoders.

Considering that the RGB-D interaction has been achieved within each block of our DFormer encoder, there is no need to specifically design extra fusion modules within the decoder.
Thus, we follow an RGB segmentation method SegNeXt (NeurIPS, 2022) to use the Ham head as the decoder.
To better understand the effect of our DFormer, we further provide the results of DFormer-B with different decoder heads in the below table.

| Decoder heads  | Param| Flops | NYU DepthV2 | SUNRGBD |
|  - | :-: | :-:  | :-: | :-: |   
| Ham head  | 29.5M | 41.9G | 55.6 | 51.2 |
| MLPs head |  29.1M | 43.5G | 55.3 | 50.9 |
| Non-local head |  32.8M | 49.4G | 56.0 | 51.5 |
| CMX decoder  | 35.0M | 48.3G | 55.4 | 51.0 |

Reviewer comment (Novelty):
The novelty and contributions are too limited. First, the proposed RGB-D block slightly modifies popular techniques, i.e., self-attention mechanism (Vaswani et al., 2017), depth-wise convolution, and attention weights (Hou et al., 2022), and combines them to fuse RGB and depth features. Second, the design of the RGB-D block follows the widely used idea in SOD, i.e., global and local information fusion. Third, the decoder directly uses the existing head from SegNext (Guo et al., 2022a) without any novel design. Thus, the contribution only comes from the pretraining idea, which is limited.

Author response:
Q1.About the architectural design

The main goal of this paper is to provide a new scheme for the acquisition of transferable representations for RGB-D segmentation tasks. 
Thus, the focus of our architectural design is how to achieve such RGB-D pretraining in an efficient way, instead of building complicated modules.
To this end, we take the interaction between RGB-D into the attention mechanism to achieve efficient fusion within each building block.
Besides, we observe that the depth information only needs a small portion of channels to encode, rather than a whole RGB pretrained backbone as done in previous works. 
Based on these designs, even with a simple decoder head, i.e., Ham head, MLPs heads, our DFormer can achieve state-of-the-art performance, as shown in Q3.

Reviewer comment (Experiments):
The authors missed some related methods [1-4] for comparison.  
[1] Visual Saliency Transformer. ICCV 2021.  
[2] 3-d convolutional neural networks for rgb-d salient object detection and beyond. TNNLS 2022.  
[3] Bi-Directional Progressive Guidance Network for RGB-D Salient Object Detection. TCSVT 2022.  
[4] UCTNet: Uncertainty-aware cross-modal transformer network for indoor RGB-D semantic segmentation. ECCV 2022.

Author response:
Q2.Comparison with more methods

In the main paper, we have conducted comparative experiments with 13 RGB-D semantic segmentation methods and 11 RGB-D salient object detection methods to verify the effectiveness of our RGB-D pre-training manner.
Thanks to your suggestions, we add the mentioned three RGB-D SOD methods, namely VST, RD3D, and BPGNet in Tab.2 of the new revision.
Our DFormer still outperforms these methods by a large margin.
Note that we fail to compare with UCTNet (ECCV,2022) due to its unopened source code, which is also pointed out by reviewer KRMX.

Reviewer comment (Experiments):
Some widely used RGB-D SOD benchmark datasets [7-9] are also ignored.  
[7] Depth-induced multi-scale recurrent attention network for saliency detection. ICCV 2019.  
[8] Learning selective mutual attention and contrast for rgb-d saliency detection. TPAMI 2021.  
[9] Saliency detection on light field. CVPR 2014.

Author response:
Q5. More RGB-D SOD benchmarks

This paper focuses on the acquirement of transferable representations for RGB-D segmentation tasks, rather than refreshing state-of-the-art performance on all the RGB-D benchmarks.
To this end, we conduct experiments on several popular RGB-D segmentation benchmarks and RGB-D SOD benchmarks following the settings of recent works like HiDANet(TIP,23)[1], SPSN(ECCV,22)[2],and SPNet (ICCV,21)[3].

Thanks to your suggestions, we further conduct experiments on the mentioned RGB-D SOD datasets in Tab.19 of the revised appendix.
We report the brief results in the following table, and our DFormer still outperforms other methods significantly.

'-' means the results are not provided in the paper. We will reproduce these missing results in the next revision to make the comparison more comprehensive.
|Dataset|Param|Flops||DUTLF|-Depth|[4]||ReDW|eb-S|[5]||LF|SD|[6]|
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
|Metric|M|G|M|F|S|E|M|F|S|E|M|F|S|E
DSNet|172.4|141.2|-|-|-|-|.133|.676|.716|.736|.068|.849|.867|.890
BIANet|49.6|59.9|-|-|-|-|-|-|-|-|-|-|-|-
SPNet|150.3|68.1|-|-|-|-|-|-|-|-|-|-|-|-
VST|83.3|31.0|.024|.948|.943|.969|.113|.763|.759|.826|.061|.889|.882|.921
RD3D+|28.9|43.3|.030|.945|.936|.964|.130|.697|.718|.786|-|-|-|-
BPGNet|84.3|138.6|.031|.938|.930|.958|-|-|-|-|.066|.875|.874|.908
C2DFNet|47.5|21.0|.025|.934|.931|.958|-|-|-|-|.065|.863|.880|.883
MVSalNet|-|-|-|-|-|-|-|-|-|-|.072|.880|.856|.906
HiDANet|130.6|71.5|-|-|-|-|-|-|-|-|-|-|-|-
DFormer-L|38.8|26.2|.023|.952|.945|.970|.106|.813|.760|.826|.056|.903|.885|.925

[1]HiDANet: Rgb-d salient object detection via hierarchical depth awareness. TIP,2023.

[2]Spsn: Superpixel prototype sampling network for rgb-d salient object detection. ECCV,2022.

[3]Specificity-preserving rgb-d saliency detection. ICCV,2021.

[4]Depth-induced multi-scale recurrent attention network for saliency detection. ICCV,2019.

[5]Learning selective mutual attention and contrast for rgb-d saliency detection. TPAMI,2021.

[6]Saliency detection on light field. CVPR, 2014.