Paper: TriSAM: Tri-Plane SAM for zero-shot cortical blood vessel segmentation in VEM images (ICLR-2024)

Reviewer comment (Experiments):
There are aspects of the paper (see questions) that are not well justified. The experimental results do not consider state of the art methods on vessel segmentation (e.g. [1-3] to illustrate just a few examples), including some that reduce the annotation effort (see [2]).

Author response:
**[Reply]**  We have conducted more experiments, such as nnUNET and MAESTER. Please refer to the answer to Q1 of General Response. We have cited and discussed the provided references [1-3] in Section 5. 

We also would like to highlight that the main focus of this paper is **zero-shot** vessel segmentation. As annotating 3D volumes for blood vessels is not only labor-intensive but also requires well-trained domain experts, efficient deep-learning algorithms are urgently needed. End-to-end training methods require annotating data for training in any new scenario. We deeply believe that in the era of foundational models, the zero-shot setting is the correct path to solving this problem. The emergence of the SAM model and the success of our proposed TriSAM have demonstrated this point. Comparing our approach to supervised learning methods is meaningless because supervised learning methods fall far short of the generalization capacity of our approach, and the annotation-training paradigm of supervised learning is far from what we desire. Therefore, the proposed BvEM dataset is also designed for evaluating zero-shot model performances, not only for training deep learning models. We reported the performance of 3D UNet solely for the completeness of the paper.

Reviewer comment (Writing):
What do the authors mean by this sentence "Moreover, imaging the whole mouse brain using VEM technology is under planning"?

Author response:
**[Reply]**  In response to the BRAIN Initiative, NIH has proposed a series of transformative projects. A major goal is scaling up the VEM technology to the whole mouse brain. A group of leading scientists calls for mapping a whole mouse brain in the next ten years. In response, a project mapping the connectome in a whole hippocampus has been granted as the first step. Considering the deep neuroscience background that is not familiar to computer scientists, we have deleted this sentence to avoid confusion.

Reviewer comment (Reproducibility):
How are turning points detected?

Author response:
**[Reply]**  When tracking along the z-axis, we apply SAM to segment blood vessels in the zx and zy planes. Potential turning points are identified as points in these segments with the smallest and largest z-values. The process is detailed at the end of Section 3.

Reviewer comment (Reproducibility):
How is the tracking approach integrated with SAM?

Author response:
**[Reply]**  SAM-based tracking: For the next slice, we use prompts derived from the enlarged bounding box and the segment's center in the current slice. Specifically, we pad the bounding box of the segment to create the bounding box prompt and calculate the segment's center for the point prompt. These prompts are used to segment blood vessels in the subsequent slice, as detailed in Section 3.2 "SAM-based Tracking."

Reviewer comment (Reproducibility):
How is the model trained? The zero-shot aspect does not come across clear.

Author response:
**[Reply]**  Model training: Our proposed zero-shot method **eliminates** the need for model training. We utilize a pre-trained SAM for segmentation, leveraging its ability to segment objects from a point or bounding box. The initial points for segmentation are generated using an unsupervised color thresholding method. While this thresholding may miss significant portions of the blood vessel, these areas are effectively detected and segmented through SAM-based tracking.

Reviewer comment (Theory):
The paper states that: "By choosing the best plane during tracking, the shape and size will not change dramatically". This seems like a flawed argument. Across neighboring slides, the vessels should not dramatically change in size but progressively. However, as the brain vessels are tortuous, the change of shape can always occur.

Author response:
**[Reply]** Blood vessels gradually change their shape and size in 3D, but the variations can significantly differ across different 2D planes, as illustrated in Figure 10 in the revised paper.
We aim to highlight that along the optimally selected plane, changes in shape and size of the blood vessel are more stable **compared to other planes**. This statement has been updated in the revised version. We also compare the scale variation for different methods. Please refer to the answer to Q2 of the General Response.

Reviewer comment (Experiments):
There are many learning-based blood vessel segmentation methods, but only 3D UNet is used as a comparative method. The other Color Thresholding and SAM+IoU Tracking methods are not deep learning-based methods.

Author response:
**[Reply]** We have conducted more experiments, such as nnUNET and MAESTER. Please refer to the answer to Q1 of General Response.

We also would like to highlight that the main focus of this paper is **zero-shot** vessel segmentation. As annotating 3D volumes for blood vessels is not only labor-intensive but also requires well-trained domain experts, efficient deep-learning algorithms are urgently needed. End-to-end training methods require annotating data for training in any new scenario. We deeply believe that in the era of foundational models, the zero-shot setting is the correct path to solving this problem. The emergence of the SAM model and the success of our proposed TriSAM have demonstrated this point. Comparing our approach to supervised learning methods is meaningless because supervised learning methods fall far short of the generalization capacity of our approach, and the annotation-training paradigm of supervised learning is far from what we desire. Therefore, the proposed BvEM dataset is also designed for evaluating zero-shot model performances, not only for training deep learning models. We reported the performance of 3D UNet solely for the completeness of the paper.

Reviewer comment (Experiments):
How the threshold for each initial seed selection and plane selection is determined? It seems difficult to find the optical threshold manually. Also, how are segmentation results different according to the threshold?

Author response:
**[Reply]** The ablation study on the threshold for initial seed selection and triplane selection is detailed in Section 4.1, with results in Figure 7. We observed consistent method performance across various thresholds.

As shown in Section 4.1,  the performance of the proposed method is largely insensitive to threshold variations, except at extremely high values, indicating our method's robustness to threshold settings.

Reviewer comment (Experiments):
Is the proposed TriSAM SOTA even when compared to the supervised image segmentation methods such as nnUNET?

Author response:
**[Reply]** We have conducted more experiments, such as nnUNET and MAESTER. Please refer to the answer to Q1 of General Response.

We also would like to highlight that the main focus of this paper is **zero-shot** vessel segmentation. As annotating 3D volumes for blood vessels is not only labor-intensive but also requires well-trained domain experts, efficient deep-learning algorithms are urgently needed. End-to-end training methods require annotating data for training in any new scenario. We deeply believe that in the era of foundational models, the zero-shot setting is the correct path to solving this problem. The emergence of the SAM model and the success of our proposed TriSAM have demonstrated this point. Comparing our approach to supervised learning methods is meaningless because supervised learning methods fall far short of the generalization capacity of our approach, and the annotation-training paradigm of supervised learning is far from what we desire. Therefore, the proposed BvEM dataset is also designed for evaluating zero-shot model performances, not only for training deep learning models. We reported the performance of 3D UNet solely for the completeness of the paper.

Reviewer comment (Novelty):
The proposed method is more like an engineering implementation than a scientific research. It consists of four steps: Initial seed generation, Tri-plane selection, SAM-based tracking, and recursive redirection. Even though the rationale is simple, it works effectively.

Author response:
**[Reply]** ***(a) Zero-shot approach***: One major challenge in 3D blood vessel segmentation is the limited labeled data to train effective deep learning models, as labeling 3D volumes is extremely time-consuming. Although existing unsupervised learning approaches eliminate the need for manual labeling, their 3D segmentation performance is not adequate. In contrast, we convert the 3D segmentation task into the tracking task and propose to integrate the existing foundation model, SAM, into our zero-shot pipeline achieving SOTA performance without any manual annotation.
***(b) SAM Adaptation***: Directly applying SAM to track blood vessel segments over slices does not achieve optimal performance (Table 2). Popular SAM-adaption methods need labeled data for finetuning. In contrast, the proposed SAM-based tracking pipeline is both effective and data efficient.

Reviewer comment (Experiments):
How to determine the threshold in Tri-Plane selection and SAM-based tracking? Do we need to change the value of threshold when applied to other data sets?

Author response:
**[Reply]** The ablation study concerning the threshold is presented in Figure 7 and detailed in Section 4.1. We applied consistent hyperparameters across all species, which demonstrated effective generalization.

Reviewer comment (Novelty):
The existing VEM segmentation methods might be discussed in detail. What is the difference between the proposed method and existing works?

Author response:
**[Reply]** Most existing VEM segmentation methods are supervised and need extensive labeled data, often limiting generalization across datasets. We summarized the existing VEM segmentation works in Section 5. Our method, requiring no labeled data, shows superior generalization to different datasets.

Reviewer comment (Experiments):
The threshold plays a vital role in the SAM-based tracking and recursive redirection. An ablation study of threshold could be provided to determine the influence of different values of threshold.

Author response:
**[Reply]** The ablation study concerning the threshold is presented in Figure 7 and detailed in Section 4.1. We applied consistent hyperparameters across all species, which demonstrated effective generalization.