Paper: TransCues: Boundary and Reflection-empowered Pyramid Vision Transformer for Semantic Transparent Object Segmentation (ICLR-2024)

Reviewer comment (Evaluation):
In Fig. 5, does it show that the proposed method is consistently effective for different backbones? This should be better discussed.

Author response:
> **A1** The relationship between the complexity, size, and performance of the backbone is shown in Figure 5. It is evident that there is a clear trade-off between the three factors. The performance improves as the backbone size increases. This relationship is crucial for users as they need to balance the computational demands with the performance requirements of their specific applications. Our research findings suggest that choosing a larger backbone leads to better performance, which provides a strategic advantage for those aiming to enhance the effectiveness of their models.

Reviewer comment (Evaluation):
In Table 6, it would be nice to show the computation complexity of the two designed modules for analysis.

Author response:
> **A2** The below is the updated version for Table 6. We added to the revised paper.
 > | **Backbone** | **GFLOPs&#8595;** | **MParams&#8595;** | **BFE** | **RFE** | **Stanford2D3D (mIoU) &#8593;** | **Trans10K-v2 (mIoU) &#8593;**  |
 > |--------------|:------------:|:-------------:|:------------:|:------------:|------------------------------------------------------|------------------------------------------------------|
 > | PVTv1-T      | 10.16 | 13.11 | -          | -          | 45.19                                                | 69.44                                                |
 > | PVTv2-B1     | 11.48 | 13.89 | -          | -          | 46.79 +1.6          | 70.49 +1.05          |
 > | PVTv2-B1     | 13.22 | 14.37 | -          | &check; | 48.12 +2.93          | 72.65 +3.21          |
 > | PVTv2-B1     | 19.55 | 14.39 | &check; | -          | 50.22 +5.03         | 74.89 +5.45 |
 > | PVTv2-B1     | 21.29 | 14.87 | &check; | &check; | 51.55 +6.36| 77.05 +7.61 |

Reviewer comment (Theory):
How to theoretically verify that the proposed method did really make use of reflection cues? This could be better discussed.

Author response:
> **A3** As depicted in Figure 2 and quantified in Table 6, our method effectively extracts regions of reflection, substantially improving overall performance. A potential enhancement to this approach could involve incorporating a binary classifier responsible for identifying the lack of reflections before activating the RFE module. This preliminary classification step has the potential to conserve computational resources by determining when no reflections are present and deciding to bypass the RFE process, thus optimizing the method's efficiency.

> To provide a theoretical verification of the effectiveness of the RFE module, we conducted the following additional experiments:
> * We take a model with the RFE module that has already been trained. To prove that RFE is effective, we compare the feature maps before and after passing through the RFE module. Please see **Figure 16** of the revised paper. In our example, we can see that after passing through the RFE module, we can get a stronger reflection signal, such as the transparent glass area or the specular reflection appearing at the base of the wine glass.
> 
> * Using the same model, we try disabling the RFE module at inference by passing the feature map before RFE directly to the next step. Note that at training, the RFE module is well-trained as usual. **Figure 17** of the revised paper shows that bypassing RFE results in a noisy feature map and wrong mask prediction. This means that our learning of RFE does not yield a trivial function, e.g., identity, and RFE does play an important role in processing the feature maps and output reflection masks.

Reviewer comment (Novelty):
It is hard to find any novel operations in the proposed reflection feature enhancement module as it simply combines existing mechanisms. It would be nice to clarify the technical novelty and theoretical contributions of the proposed modules.

Author response:
> **A4** The RFE module utilizes an extensive convolution-deconvolution architecture to process input data and generate an improved feature map. The system takes in input features and carefully processes them to generate a sophisticated feature map. The utilization of a layered architecture is of utmost importance, as it effectively handles visual data at various levels of abstraction, effectively navigating the complexities associated with diverse visual occurrences, including reflections. In contrast to models designed for global reflection scenarios, which assume that reflections are present throughout the entire image, our RFE module demonstrates superior performance in detecting local reflections. Implementing a localized technique is of utmost importance in real-world scenarios characterized by irregular and area-specific reflections rather than uniform ones. The RFE module demonstrates high accuracy in detecting glass surfaces within various environmental conditions, primarily by prioritizing the analysis of localized reflections. Hence, this approach offers a more accurate and contextually nuanced method for discerning reflection within the multifaceted fabric of real-world environments.

Reviewer comment (Experiments):
There are extensive segmentation methods that introduce boundary-relevant loss designs or other designs. Please consider incorporating some existing boundary-relevant designs for a comparison. This can better show the superiority of your proposed boundary feature enhancement module.

Author response:
> **A5** Let's consider some recent methods that incorporate boundary-relevant designs, such as **[Tian et al., 2022]** and **[Lin et al., 2020]**. These papers incorporate the Canny filter into their methodologies, albeit for different purposes. **[Tian et al., 2022]** employs the Canny operator to enhance image boundary delineation. In contrast, **[Lin et al., 2020]** use it to extract precise mirror edges from ground truth masks to create supervisory edge maps. The Canny filter is celebrated for its ability to generate edges that are not only thin and smooth but also notably accurate. This precision stems from using Non-maxima suppression and finely adjustable parameters to cater to varying image characteristics. However, these advantages come at the cost of increased complexity and computational demand compared to simpler alternatives like the Sobel filter. Furthermore, the Canny filter's reliance on manual parameter adjustment necessitates extensive experimentation to ascertain optimal settings, as thoroughly examined in a dedicated section of **[Tian et al., 2022]**. In contrast, our decision to use the Sobel filter is informed by our commitment to efficiency, particularly in applications that require real-time processing, such as robotic navigation. With its relative simplicity and lower computational load, the Sobel filter aligns with our objectives to develop a framework that balances performance with practicality. This choice underscores our framework's suitability for scenarios where speed and efficiency are paramount without significantly compromising the edge detection quality needed for effective navigation.
 > 
 > **[Tian et al., 2022]** Xin Tian, Ke Xu, Xin Yang, Baocai Yin and Rynson W. H. Lau. Learning to Detect Instance-Level Salient Objects Using Complementary Image Labels. In IJCV, 2022.
 > 
 > **[Lin et al., 2020]** Lin Jiaying, Wang Guodong and Lau Rynson W.H. Progressive Mirror Detection. In CVPR, 2020.

> * Currently, we cited [Tian et al., 2022] following the suggestion of Reviewer 3ba2. Unfortunately, their source code is not publicly available, and their experiments did not cover the same datasets as ours due to their emphasis on a distinct salient instance detection task. Hence, a direct comparison is not feasible.
>  
> * In Table 16, a comparison was made between the PMD method [Lin et al., 2020] and our approach, revealing a significant performance advantage on our part, such as **81.54%** vs **91.04%** mIoU and **66.05%** vs **69.61%** mIoU on MSD and PMD datasets, respectively.

Reviewer comment (Novelty):
The related work follows that of Trans4Trans. It would be nice to add more recent related state-of-the-art works.

Author response:
> **A6** We have revised Section 2 as suggested, please check the revised version (with highlighted changed text)

Reviewer comment (Experiments):
Would it be possible to incorporate your model with the RGB-D modalities for an experiment? This could be discussed.

Author response:
> **A7** Indeed, it is feasible to integrate our model with RGB-D modalities. One approach is to incorporate an additional encoder to extract Depth features. These Depth features are combined with RGB features before being passed to the FPM module. This methodology aligns with the PDNet architecture **[Mei et al., 2021]** described in Table 17 and the TROSNet architecture **[Sun et al., 2023]** outlined in Table 19. It is worth noting that including depth information in these approaches resulted in a notable performance improvement. Specifically, PDNet exhibited a $4\%$ increase, while TROSNet demonstrated an $8\%$ increase. We also added this discussion to the Section D.4 of the revised paper.
 > 
 > **[Mei et al., 2021]** Haiyang Mei, Bo Dong, Wen Dong, Pieter Peers, Xin Yang, Qiang Zhang, and Xiaopeng Wei. Depth-aware mirror segmentation. In CVPR, 2021.
 > 
 > **[Sun et al., 2023]** Tianyu Sun, Guodong Zhang, Wenming Yang, Jing-Hao Xue, and Guijin Wang. Trosd: A new rgb-d dataset for transparent and reflective object segmentation in practice. IEEE TCSVT, 2023.