Paper: Weakly Supervised Virus Capsid Detection with Image-Level Annotations in Electron Microscopy Images (ICLR-2024)

Reviewer comment (Experiments):
Limited Model Exploration. The paper primarily focuses on using Faster-RCNN with a ResNet-101 backbone as the detection model. It would be beneficial for the authors to consider exploring other models, such as DETR, to evaluate their effectiveness in the proposed approach.

Author response:
We agree, that our model exploration is limited. We motivate our decision to use CNN-based architectures by the limited data set sizes for virus detection in EM. However, we agree that a more thorough model exploitation can give additional insights, specifically as ViTs have shown great success in weakly supervised setups. We hence include the comparison of Ours(Opt) based on the ResNet-101 and Ours(Opt) based on a ViT backbone in the updated version of our paper (appendix A.2). 
Nevertheless, we refrained from delving deeper into the exploration of alternative detection models such as DETR. Our ablation study, employing ViT backbones for Ours(Opt), suggests that DETR might face challenges and likely underperform, particularly given the constraints posed by the relatively small dataset sizes. However, we acknowledge that leveraging different detection models has the potential to enhance the performance of all methods, including our proposed approach (Ours(OD)).

Reviewer comment (Experiments):
It is crucial to include testing with DETR to evaluate the effectiveness of transformer-based architectures.

Author response:
We agree, that our model exploration is limited. We motivate our decision to use CNN-based architectures by the limited data set sizes for virus detection in EM. However, we agree that a more thorough model exploitation can give additional insights, specifically as ViTs have shown great success in weakly supervised setups. We hence include the comparison of Ours(Opt) based on the ResNet-101 and Ours(Opt) based on a ViT backbone in the updated version of our paper (appendix A.2). 
Nevertheless, we refrained from delving deeper into the exploration of alternative detection models such as DETR. Our ablation study, employing ViT backbones for Ours(Opt), suggests that DETR might face challenges and likely underperform, particularly given the constraints posed by the relatively small dataset sizes. However, we acknowledge that leveraging different detection models has the potential to enhance the performance of all methods, including our proposed approach (Ours(OD)).

Reviewer comment (Experiments):
In the experiments, the proposed method is not compared with other state-of-the-art weakly supervised learning methods, such as Zeng et al. 2019, Wei et al. 2022, and Lu et al. 2020. In addition, it is not compared with other CAM-based weakly supervised object detection methods in the experiments, such as Xu et al. 2022. Without a comparison with recent state of the art, it is difficult to determine the superiority of the proposed methods over other approaches.

Author response:
We acknowledge this concern and regret that certain methods, such as [1] and [2], lacked publicly available training code. Other methods like GC-Net [3] lack the ability to detect more than one object, which is why we chose not to include it in the comparisons. We opted to include alternative state-of-the-art methods — specifically, LayerCAM [4], TS-CAM [5], and Reattention [6]. We further included GradCAM [7], as we make use of this method as an initialization schema. As ViTs have shown great success in current state of the art WSOL methods, we include the ViT as well as ResNet backbones for the application of GradCAM and LayerCAM.
In the interest of a more equitable comparison, we opted to integrate the known virus size into the existing methods. Through an extensive ablation study (appendix A.5), we systematically determined the most suitable approach for informing these methods (section 4.4). 

| Method                        | Herpes                | Adeno                   | Noro                 | Papilloma                      | Rota                 |
|-------------------------------|--------------------------|--------------------------|--------------------------|--------------------------|--------------------------|
| $\mathrm{GradCAM}$ ResNet      | 78.79 $\pm$ 2.04        | 19.17 $\pm$ 0.78        | 05.54 $\pm$ 2.99        | 11.57 $\pm$ 4.17        | 31.78 $\pm$ 21.58       |
| $\mathrm{LayerCAM}$ ResNet     | 78.44 $\pm$ 2.73        | 16.48 $\pm$ 9.34        | 05.04 $\pm$ 1.91        | 10.87 $\pm$ 5.33        | 31.22 $\pm$ 20.07       |
| $\mathrm{GradCAM}$ ViT        | 61.87 $\pm$ 11.87       | 08.00 $\pm$ 2.12        | 19.31 $\pm$ 13.64       | 04.03 $\pm$ 4.52        | 13.12 $\pm$ 7.37        |
| $\mathrm{LayerCAM}$ ViT       | 68.33 $\pm$ 6.59        | 09.18 $\pm$ 5.64        | 10.82 $\pm$ 11.78       | 17.41 $\pm$ 11.33       | 09.74 $\pm$ 2.42        |
| $\mathrm{TS-CAM}$             | 32.06 $\pm$ 1.02        | 39.25 $\pm$ 4.13        | 14.64 $\pm$ 4.66        | 07.11 $\pm$ 3.85        | 43.53 $\pm$ 3.93        |
| $\mathrm{Reattention}$        | 68.85 $\pm$ 0.62        | **58.49** $\pm$ 2.22        | 55.09 $\pm$ 8.92        | 35.60 $\pm$ 13.01       | 59.05 $\pm$ 11.40       |
| **$\mathrm{Ours (Opt)}$** | 86.98 $\pm$ 1.92  | 47.85 $\pm$ 11.82 | 54.65 $\pm$ 4.94  | 70.02 $\pm$ 2.85  | 71.73 $\pm$ 3.51 |
| **$\mathrm{Ours (OD)}$**  | **91.20** $\pm$ 0.24 | **58.28** $\pm$ 5.91 | **74.32** $\pm$ 1.18 | **78.33** $\pm$ 2.40 | **78.34** $\pm$ 2.15 |

As shown in the table above, we found that other weakly supervised methods fail to reliably detect virus particles in EM images, eventhough we also incorperate the virus size in existing methods. Our methods usually outperforms all others by a large margin, except in the adeno virus, where we are en par with Reattention. Please find a detailed discussion in the paper (section 4.4, appendix A.5).

Reviewer comment (Experiments):
The method requires the object size to be known in advance. This needs additional effort to estimate the size of target objects before applying the method. It would be helpful to provide an in-depth discussion on this design (probably also including the effects of using different estimated object sizes).

Author response:
We regret we were not able to communicate it properly in the paper. Usually, the virus size can be found in literature. Moreover, as the standard deviation of virus particles is small, the size can easily be approximated by measuring only a few instances. We will include a discussion in the updated version of our paper in appendix A.4.
We also agree, that the correct approximation of the objects size is important for our method in order to work well. To communicate this, we added an additional experiment in appendix A.4 about the influence of approximation error in the updated version of our paper. For a quick overview please see the provided table:

| Error               | $0\%$                   | $10\%$                  | $20\%$                  | $30\%$                  |
|---------------------|-------------------------|-------------------------|-------------------------|-------------------------|
| $\mathrm{Ours (Opt)}$ | **86.98** $\pm$ 1.92   | 82.38 $\pm$ 02.34       | 79.14 $\pm$ 03.10       | 67.31 $\pm$ 02.03       |


As expected, we found that the performance reduces with increased error in the size approximation.

Reviewer comment (Evaluation):
It seems that the proposed method needs to repeat the optimization process (i.e., solving Equation (2)) for each virus. The time cost may be high if there is a large number of viruses in the input image.

Author response:
We agree that the computation time increases with the amount of visible virus particles in the image.
This is however a specific design choice for dealing with the non-object centered nature of EM images.
Our evaluation shows that current state of the art methods lack to reliably detect multiple instances of the same object as they only have been trained on image level labels, and hence are able to only focus on the most prominent virus in the image (see appendix C.2). Our approach is able to side step this limitation to be applicable in the context of EM.

Reviewer comment (Experiments):
The method is evaluated on only virus detection in electron microscopy images, where viruses do not overlap. Thus, the method may not generalize to object detection (e.g., cell or nuclei detection) in other microscopy imaging modalities, such as hematoxylin and eosin (H&E) or immunohistochemistry (IHC) stained brightfield microscopy images, and fluorescence mages, which often have touching or overlapping cells or nuclei. In addition, the repeated optimization for each object would be expensive for H&E or IHC images that typically have thousands of or even more cells/nuclei.

Author response:
In general, the detection of overlapping instances of the same object in a weakly supervised setup is no trivial task and remains a major challenge. We would like to emphasize that we specifically designed an approach for the detection of virus particles in EM images. In our extended evaluation we can show that current state of the art approaches fail to reliably detect virus particles in EM, which highlights the need of a specific method. 
Addressing the broader challenge of detecting overlapping instances of the same object within a weakly supervised setup is inherently complex and represents a substantial hurdle in general weakly supervised setups. Nonetheless, we maintain our confidence in the adaptability of our approach to diverse microscopy modalities. Qualitative results presented in the appendix (section C) illustrate instances where virus particles form clusters and exhibit touching behavior. Notably, our current method focuses exclusively on non-overlapping viruses, employing non-maximum suppression with a strict threshold to mitigate overlap.
We believe that relaxing this threshold would likely enable our method to detect overlapping instances. However, a thorough evaluation and extension of our method in this regard are reserved for future research.

Reviewer comment (Experiments):
The proposed method is based on the GradCAM method. What if the GradCAM does not provide good initializations or even wrong saliency maps? What are the effects of inaccurate saliency map creation on the quality of the pseudo-labels generated by the proposed method?

Author response:
In the Appendix A.1.1 we investigate different methods for initialization and the power of the optimization process. The experiment shows, that even with non-optimal initialization (like random initialization) the optimization process is able to converge to an optimum, when enough iterations are used. In our extended evaluation, we found that GradCAM based WSOL failed to reliably detect the virus capsids in negative stain images (see appendix C.1), as the classifier was focusing on the boarder of the virus. However, the inclusion of our optimization procedure enables a more robust detection.

Reviewer comment (Reproducibility):
During the optimization process of the proposed method, i.e., solving Equation (2), is the classifier C fixed and not updated? If so, is the optimization of Equation (2) simply to find the position that has the highest value in the prediction map C(I * M(p_t)) at each time step? But if not, what algorithm is used to optimize Equation (2)?

Author response:
Yes, the classifier is fixed during the optimization. We will emphasize this more in the updated version of our paper.

Reviewer comment (Reproducibility):
During the postprocessing, the method uses non-maximum suppression to eliminate virus particles that have low detection scores. Are the detection scores of virus particles obtained from the initial CAM map or the prediction map from the Gaussian-filtered input, C(I * M(p_t))?

Author response:
We kindly refer to Section 3.5 in our paper. Here we explain, that the detection score is computed individually for each detected virus in a postprocessing step. To do so, we mask all other detected virus particles by a circular disk with radius of the known virus size and derive the score of the remaining virus by forwarding the masked image to the pretrained classifier.

Reviewer comment (Experiments):
In addition to lack of comparisons, one of the main limitations of this work that may prevent broader interest in the ICLR community is that the proposed method is too specific to EM and is not evaluated on diverse tasks. Though EM is unique compared to natural images which are generally more object-centric, other modalities such as histopathology and multiplexed imaging share similar characteristics (as noted in [5]), with the image scale is objective with units per pixel being fixed. Specifically, the contributions of this work would be strengthened if shown that a simpler heuristic can also be created for other imaging domains.

Author response:
Our proposed method does not aim to reach a large range of applications and object types, but it is designed for the detection of particles in electron microscopy images. In our extensive evaluation, we are able to show that the proposed method is able to outperform current state-of-the-art methods, even though we also include the known virus size in existing approaches. We believe that this is most likely due to the fact, that current state-of-the-art methods highly benefit from large dataset sizes, object-centric datasets, and high SNR. These large amounts of annotated data are not available for virus detection in EM. Additionally, EM images aren't object-centric but usually contain multiple instances of the same object in one micrograph. Finally, low SNR can hinder the performance of these methods in EM. 
This finding underlines the need for methods that are specifically designed to work well for virus detection in EM. 
Despite this, we believe that the proposed method holds significance for the broader ICLR community. Firstly, its simplicity renders it accessible and intuitive, making it potentially valuable for a wide audience. Secondly, its adaptability to a broader spectrum of applications is noteworthy. By replacing the circle with ellipses and incorporating size optimization, as suggested by [3], the approach demonstrates versatility.
While a comprehensive evaluation of these adaptations is beyond the scope of this paper, we recognize their potential impact and plan to delve into a detailed assessment in future work.

Reviewer comment (Theory):
The current setup with Gaussian as a prior assumes that the object to detect is in a round shape. How easily it can be extended to different objects and how accurately it will work?

Author response:
## Round shape as a prior and discussion on braoder impact
We believe that the extension of the proposed method to a wider range of applications is possible. In [1] the authors use a generator to predict ellipses which are then used to mask the input image for training of a detector with image-level labels only. Substituting the proposed circle by ellipses will open the possibility to detect objects with different aspect ratios. They use an additional size penalty as a loss to additionally optimize for the size. 
Combining the proposed method with [1] will most likely achieve a broader application of the approach. 
However, as discussed above, our evaluation shows the need for more specific approaches in the field of virus detection in EM. We hence leave the extension of the approach to a wider range of applications for future work.


[1] Lu, Weizeng, et al. "Geometry constrained weakly supervised object localization." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16. Springer International Publishing, 2020.