Paper: How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks? (ICLR-2024)

Reviewer comment (Experiments):
The weakness could relate to the details of the pretraining strategy. Typically, image-wise [1, 2] or pixel-wise [3] pre-training relies on the InfoNCE loss for clustering embedding samples in the latent space, rather than directly applying penalties based on labels via cross-entropy loss. It would be more interesting to observe results achieved through category-guided InfoNCE loss [4, 5] pre-training using this dataset.

Author response:
Thanks for your suggestion. We must admit that category-guided InfoNCE loss was not our first try by default because it is not a standard approach for medical segmentation tasks as of now. [[Ma et al., MEDIA 2021](https://www.sciencedirect.com/science/article/pii/S1361841521000815)] have listed all the popular losses for medical segmentation tasks, each of these loss functions may have their own strengths to specific scenarios, such as dealing with unbalanced class. Therefore, we initially selected a combination of Dice loss and binary cross-entropy loss, the most commonly used segmentation loss, as recommended by the introduction and Figure 1 in [[Ma et al., MEDIA 2021](https://www.sciencedirect.com/science/article/pii/S1361841521000815)].

But we are very much interested in exploring more loss options. Category-guided InfoNCE loss, as you suggested, is potentially good because it can cluster embedding samples in the latent space and has proven effective for supervised learning in natural image classification [4] and segmentation tasks [5]. Thanks for sharing the references. The references [1, 2, 3], however, discussed self-supervised learning, so their applicability and effectiveness in the context of supervised learning for segmentation tasks might not have been thoroughly explored. Following your suggestion, a comprehensive comparison between InfoNCE loss and our dice + bce loss will be included in our final version.

Reviewer comment (Experiments):
There exist many promising domain transfer methods, yet the paper appears to lack exploration in this area. The current approach appears to be straightforward fine-tuning on other datasets, such as TotalSegmentator and JHH.

Author response:
The current approach is pretty robust (evidenced in Table 3) because our dataset covers a variety of domains (i.e., 68 hospitals with different scanners and protocols). Therefore, models pre-trained on this dataset are expected to be generalizable for novel domains, e.g., TotalSegmentator, FLARE’23, and JHH. These three datasets are completely unseen during the pre-training and represent a variety of diversity. Specifically, the TotalSegmentator dataset represents for the Central European population from Switzerland, the FLARE’23 dataset represents for East Asian population from China, and the JHH dataset represents for another population (anonymous for peer review). Our models achieve comparable or even superior performance to the IID counterparts (Table 3). Therefore, domain transfer becomes less important if the model is pre-trained on large and diverse datasets (elaborated in the next two points).

1. The domain transfer problem could be solved by methodology innovation, as you suggested, and also by training AI models on enormous datasets. This point has been more clear recently demonstrated by large language models (GPT) and vision foundation models (SAM), which show incredible performance in “new domain”. However, this achievement may not be directly attributed to method-driven solutions for domain transfer, but simply because the AI might have been trained on similar sentences or images. This was also pointed out by Yann Lecun—*“[beware of testing on the training set](https://twitter.com/ylecun/status/1723752958037315874)”*—in response to the incredible results achieved by GPT.

2. In some sense, our paper explores dataset-driven solutions for domain transfer. The robust performance of our models when direct inference on multiple domains could also be attributed to our large-scale, fully-annotated medical dataset—as one of our major contributions. The release of this dataset can foster AI models that are more robust than the majority of existing models that are only trained on a few hundred CT volumes from limited domains. We completely agree that existing domain transfer methods could be supplemented with direct inference and fine-tuning to further improve AI performance.

Reviewer comment (Reproducibility):
I would like to confirm whether the dataset and the pre-trained weights will be made accessible to the public.

Author response:
We confirm that the dataset, the pre-trained weights, and the source code will be made accessible to the public upon the acceptance of the paper. Pre-trained weights have already been pre-leased for peer review and public audience through the links in the general responses. Feel free to test it out. And we have already attached our source code to the supplementary material. The dataset, however, cannot be pre-leased because we are in the process of licensing our ImageNetCT-9K.

Reviewer comment (Novelty):
UniverSeg [1] is another paper that trains networks on a very large dataset, 22K scans, which is even larger than this paper. Although Arxiv version of UniverSeg is available since April 2023, I see this work and UniverSeg as concurrent works since UniverSeg is recently presented in ICCV. However, I still think that mentioning UniverSeg and discussing the similarities/differences in the final version would be useful.

Author response:
Thank you for sharing UniverSeg with us—its relevance to our work is significant and appreciated. We have now included a discussion of UniverSeg, along with other concurrent and related works, in the revised Section 2. These works share a level of **similarity** with ours, aiming to develop medical AI models that can segment multiple anatomical structures and generalize to out-of-distribution datasets.

**Opinions on UniverSeg.** UniverSeg demonstrates an impressive ability to generalize across various unseen tasks and modalities without the need for fine-tuning, offering computational efficiency. However, its current performance falls short of the benchmarks set by fine-tuned models (as their upper bound reference). **In contrast**, our research aims to advance the fine-tuning performance by pre-training models on larger, per-voxel annotated datasets. We believe that our ImageNetCT-9K dataset could also further enhance UniverSeg's generalization ability.

**Perspectives on additional related and future work** The forthcoming public release of ImageNetCT-9K and SuPreM is anticipated to positively influence Segment Anything Models (SAM) within the medical field. We hold the same belief with [[Kirillov et al. ICCV 2023](https://openaccess.thecvf.com/content/ICCV2023/papers/Kirillov_Segment_Anything_ICCV_2023_paper.pdf)] that the foundation models for image segmentation require supervised training on board data at scale, and we both contribute to a large-scale annotated dataset in the respective field. In addition, we already have some proof of concept that the model tends to understand **objectness** in a broader sense through full supervision in Table 4.

Reviewer comment (Novelty):
The paper explores the transferability of supervised learning by combining multiple 3D CT segmentation datasets. Similar work has been done in various fields, such as training various tasks and data in computer vision, which has shown improved results across tasks. This paper incorporates 3D CT data and tasks, with the only difference being the collection of more publicly available and private data, without bringing new insights or technological innovations to the community.

Author response:
Creating large-scale (9K) per-voxel annotated medical datasets of over 25 classes and making this resource publicly available takes a village. Your acknowledgment of our dataset as *“the largest dataset for multi-organ segmentation currently available”* is greatly appreciated. While it is common in computer vision to improve performance through joint training on a combination of existing datasets, our study overcomes technical barriers and brings new insights as follows.

1. ImageNetCT-9K is NOT a simple combination of existing datasets. We have now included Appendix B.1 Figure 5, in conjunction with Table 5, to better illustrate the evolution from public datasets to our ImageNetCT-9K. The 9K CT volumes in the combination of public datasets only contain a total of **39K** annotated organ masks, while our ImageNetCT-9K provides **296K** annotated organ/tumor masks for these CT volumes, substantially increasing the number of masks by **7.6** times.

2. Creating **296K** high-quality organ/tumor masks for 9K CT volumes requires extensive medical knowledge and annotation cost (much more difficult than annotating natural images). Based on our experience and those reported in [[Park et al., Diagnostic and interventional imaging 2020](https://pubmed.ncbi.nlm.nih.gov/31358460/)], trained radiologists annotate abdominal organs at a rate of 30–60 minutes per organ per three-dimensional CT volume. This translates to **247K** human hours for completing ImageNetCT-9K. We employed a highly efficient annotation method, combining AI with the expertise of three radiologists using active learning (details in **Q6**), to overcome this challenge and produce the largest annotated dataset to date.

3. ImageNetCT-9K can be used as a training resource and a testbed for AI algorithms. As acknowledged by Reviewer fMfJ, it can *“have a good impact on the whole community.”* Our study introduces **new insights** on the use of ImageNetCT-9K in transfer learning. We delve into a key debate within general computer vision: the comparative effectiveness of representation learning via human-annotated data (supervised pre-training) versus raw data (self-supervised pre-training). As Reviewer fMfJ acknowledged, *”this paper concluded on the debate of whether self-supervised or supervised pre-training lead to better performance and data efficiency. This debate **had not be resolved** without the invention of a fully-annotation dataset of such scale.”* We are committed to making ImageNetCT-9K publicly available, thereby enabling further research to derive insights from various angles and foster technological innovations. Reviewer mSzX recognized that *”publicly available pretrained models on such large datasets are huge assets for medical imaging.”*


---

**Reference**

- Park, S., L. C. Chu, E. K. Fishman, A. L. Yuille, B. Vogelstein, K. W. Kinzler, K. M. Horton et al. "Annotated normal CT data of the abdomen for deep learning: Challenges and strategies for implementation." *Diagnostic and interventional imaging* (2020).

Reviewer comment (Experiments):
The conclusion that supervised pre-training has an advantage over other pre-training methods in 3D medical imaging is generalized to general 3D vision tasks in an inconsistent manner. The introduction discusses pre-training strategies for 3D vision tasks, but the experiments are all conducted on medical images. It is well known that there are significant differences between medical images and natural images, and whether the experimental conclusions on 3D CT data can be extended to other 3D vision tasks is not explored in the paper.

Author response:
We completely agree with your comments and have now revised our title, abstract, introduction, and conclusion to narrow down our current research scope to 3D medical imaging. 

The potential applications and implications of our dataset have now been included in the future work section. Given that our dataset includes detailed per-voxel annotations for 25 organs and tumors, it enables the automatic generation of 3D shape representations. These representations can be formatted as point clouds, voxel occupancy grids, meshes, and implicit surface models (e.g., signed distance functions), each catering to different algorithmic needs. We anticipate our dataset could be useful for a variety of other 3D medical vision tasks [[Li et al., 2023](https://arxiv.org/abs/2308.16139)], such as pose estimation, surface reconstruction, depth estimation, etc.

Reviewer comment (Experiments):
Unfair comparison in the experimental section is a fatal flaw. Although the authors claim that collecting 9000 data is their contribution, the same 9000 data were not used for pre-training when comparing with other self-supervised/supervised pre-training methods. Therefore, it is not rigorous to conclude that SPT is superior to other supervised/self-supervised methods.

Author response:
Thank you for your valuable feedback. Ensuring a fair and rigorous comparison is our top priority. Therefore, we must clarify that while we have developed and released a suite of models pre-trained on 9K data, these models were NOT used for comparisons against other self-supervised/supervised pre-training methods within this paper. The only experiment that used 9K-models was the direct inference on external datasets (Table 3), where we benchmarked against methods trained specifically on those datasets, representing an upper bound in performance. This benchmark is to ensure that the released models are the most effective and robust ones (as we could provide) for the research community to directly use.

For the other experiments, our aim was to evaluate the efficacy of supervised pre-training relative to other pre-training methods, so we designed them within a controlled setting. 

1. For supervised pre-training, the largest study to date was by [[Liu et al., ICCV 2023](https://openaccess.thecvf.com/content/ICCV2023/papers/Liu_CLIP-Driven_Universal_Model_for_Organ_Segmentation_and_Tumor_Detection_ICCV_2023_paper.pdf)], which was developed on 3,410 (2,100 for training and 1,310 for validation) annotated CT volumes. For self-supervised pre-training, the largest one was by [[Tang et al., CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf)], which was trained on 5,050 unannotated CT volumes. To do a rigorous comparison, we **benchmarked** with these advanced pre-training methods by pre-training our model using 2,100 CT volumes (*same as Liu et al. and fewer than Tang et al.*) in the Table 2, Figure 1 and Appendix C.2 Figure 8.
2. We further **scaled down** the number of CT volumes to 21 to explore the edge of our supervised pre-training method. Surprisingly, Figure 2a shows these experiments and demonstrates that the model trained with 21 CT volumes, 672 masks, and 40 GPU hours shows a transfer learning ability similar to [[Tang et al., CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf)] trained with 5,050 CT volumes and 1,152 GPU hours.
3. Lastly, we **scaled up** the SOTA self-supervised method [[Tang et al., CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf)] by pre-training it on the same 9K CT volumes. Under this similar setting, our supervised pre-training 9K model substantially outperforms the SOTA self-supervised 9K-model (see Appendix C2 Table 8). 

---

**Reference**

- Liu, Jie, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. "Clip-driven universal model for organ segmentation and tumor detection." *ICCV* (2023).
- Tang, Yucheng, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. "Self-supervised pre-training of swin transformers for 3d medical image analysis." *CVPR* (2022).

Reviewer comment (Experiments):
The results of fine-tuning SPT on 63 novel classes are not impressive. Although there is no comparison with totalsegmentator, the performance of totalsegmentator trained on 1000 data seems to surpass what is reported in Table 4. For example, totalsegmentator can achieve over 95% Dice on iliopsoas, while it is less than 90% in the paper.

Author response:
In TotalSegmentator, the labels were largely generated by a **single** nnU-Net re-trained continually (see Figure 1b in [Wasserthal et al.](https://pubs.rsna.org/doi/full/10.1148/ryai.230024)). Depending solely on nnU-Net could introduce a potential label bias favoring the nnU-Net architecture. This means two points. 

1. Their ground truth (revised pseudo labels) could be biased to the nnU-Net architecture. nnU-Net trained and tested on this dataset will achieve an **unreachable** performance as shown in Totalsegmentator [github](https://github.com/wasserth/TotalSegmentator/blob/master/resources/evaluate_results.txt). For example, they report the DSC score of 0.894 for pancreas segmentation, this number has never been reached literature. The MSD top 1 result (**[0.828](https://decathlon-10.grand-challenge.org/evaluation/challenge/leaderboard/)**); TCIA Pancreas-CT Dataset top 1 result (**[0.845](https://paperswithcode.com/sota/pancreas-segmentation-on-tcia-pancreas-ct)**); even the FELIX Project (producing the largest pancreas dataset in USA) only achieves **[0.87](https://www.medrxiv.org/content/10.1101/2022.09.24.22280071v1)** DSC score. None of these advanced benchmarks have achieved 0.894 reported in Totalsegmentator. Therefore, it is sensible to assume the ground truth is biased to the nnU-Net architecture (including iliopsoas that you mentioned and many other classses). This is the reason why we do not make comparisons with Totalsegmentator. 

2. Due to the potential label biases, whenever TotalSegmentator is employed for benchmarking, nnU-Net and models building upon nnU-Net would always outperform other segmentation architectures (e.g., UNETR, TransUNet, SwinUNet, etc.). This observation has also been made in several publications that used the TotalSegmentator dataset. For example, Table 3 in [[Huang et al., 2023](https://arxiv.org/abs/2304.06716)] showed that nnFormer, UNETR, and Swin UNETR were all outperformed by nnU-Net and models building upon nnU-Net in TotalSegmentator. More importantly, the average DSC achieved by our model on TotalSegmentator is also much higher than [[Huang et al., 2023](https://arxiv.org/abs/2304.06716)] as compared in the following table.

| method &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  | organ &nbsp; &nbsp; &nbsp; | vertebrae &nbsp; &nbsp; &nbsp; | cardiac &nbsp; &nbsp; &nbsp; | muscle &nbsp; &nbsp; &nbsp; | 
|  ----  | ----  |  ----  |  ----  |  ----  | 
| [Huang et al., 2023](https://arxiv.org/abs/2304.06716) | 89.82 | 90.43 | 90.89 | 88.83 |
| Ours | **92.09** | **91.29** | **92.21** | **95.40** | 

Therefore, we believe that the segmentation results in TotalSegmentator reported in our paper are **compelling** and should be considered a **faithful** benchmark.

---

**Reference**

- Wasserthal, Jakob, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye et al. "Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images." *Radiology: Artificial Intelligence* (2023).
- Huang, Ziyan, Haoyu Wang, Zhongying Deng, Jin Ye, Yanzhou Su, Hui Sun, Junjun He et al. "STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training." *arXiv* (2023).

Reviewer comment (Experiments):
The paper does not clearly explain how the three expert radiologists collaborated to clean the data, including how they worked together, established uniform standards, and how they corrected tumor masks in the pseudo labels. What was the time cost involved, and so on?

Author response:
Thank you very much for the suggestion. We have now enclosed this information in the revised Section 3.1 and Appendix B.3.

**Automated organ annotations.** Our annotation pipeline involved an interactive segmentation approach, an integration of AI algorithms and human expertise, which premises to improve the efficiency while upholding high-quality annotations. *One senior radiologist* revised the annotations predicted by our AI models, and in turn, the AI models improved their predictions by learning from these revised annotations. This interactive process continued to enhance the quality of annotations until no major revision is needed. Subsequently, *five junior radiologists* examine the final visualizations for accuracy (examples of the rendered images are illustrated in Appendix B.3 Figure 7). The junior radiologists were responsible for reviewing the correctness of the annotations and marking the patient ID for any major discrepancies. Such cases are then reviewed by the senior radiologist. Our uniform annotation standards, largely overlapping with those in [[Ma et al., FLARE 2022](https://arxiv.org/abs/2308.05862)], require trained radiologists to spend approximately 30–60 minutes annotating each organ in a three-dimensional CT volume. 

**Automated (pseudo) tumor annotations.** We have established uniform annotation standards for tumors, with both senior and junior radiologists actively refining and adhering to these guidelines.

- Liver tumors: Liver tumors include primary tumor lesions and metastases in the liver. Annotations should encompass the entire tumor, including any invasive parts, necrosis, hemorrhage, fibrous scars, and calcifications. Healthy areas or unrelated lesions are not included.

- Kidney tumors: Kidney tumors include both benign and malignant tumor lesions growing in the kidneys. The entire tumor and its invasive parts to surrounding areas, plus internal changes like necrosis and calcification, should be annotated. Exclude healthy structures.

- Pancreatic tumors: Pancreatic tumors include all benign and malignant tumor lesions growing in the pancreas. Annotations cover the whole tumor and its invasive growth into adjacent areas, including changes like cysts, necrosis, and calcification. Exclude healthy structures.

- Colon tumors: Colon tumors include all benign and malignant tumor lesions developing from the colon wall. The entire tumor and its invasion into nearby structures, along with internal changes like necrosis, should be annotated, excluding healthy areas.

- Hepatic vessel tumors: Hepatic vessel tumors include all primary tumor lesions developing from the intrahepatic vessel wall and tumor thrombus in intrahepatic vessels. Annotations should include the tumor within the vessels, excluding external parts and unrelated lesions.

Overall, our ImageNetCT-9K dataset offers **51.8K** pseudotumor masks visually inspected by radiologists, though without biopsy confirmation. While these masks lack pathological validation, we anticipate they will serve as a valuable foundation for expanding precise tumor annotations in future research.

Reviewer comment (Writing):
The title may be a little misleading. If I understood it correctly, the word "transfer" in the title is not referring to transfer learning in this case, but simply asking whether supervision is also good for 3D data as it has been for 2D data. When first reading the title, I thought you were exploring transferring 2D supervised models to 3D tasks. Others may also make that incorrect assumption.

Author response:
The word “transfer” in the title refers to transfer learning, a procedure that involves two stages. **Firstly**, a model is pre-trained on a pretext task, specifically organ segmentation in our paper. **Secondly**, the model is transferred (fine-tuned) to multiple target tasks, including new datasets, classes, and tasks. The target task performance can indicate whether the model pre-training is helpful compared with (1) learning the model from scratch and (2) other publicly available pre-trained models. In this paper, we have assessed the transfer learning ability of models under **five distinct settings**. 
1. Transfer to different datasets (domains) to segment the same organs—classes that were used for pre-training (Appendix E.2 Table 10).
2. Transfer to segmentation tasks of organs, muscles, vertebrae, and cardiac structures—classes that were not used for pre-training (revised Table 4; Appendix E.3 Table 11).
3. Transfer to segmentation tasks of pancreatic tumor segmentation—more challenging classes that were not used for pre-training (revised Table 4; Appendix E.3 Table 11).
4. Transfer to few-shot segmentation tasks using only a limited number of annotated CT volumes—classes that were not used for pre-training (Figure 1; Figure 2b; Appendix D.1 Figure 9).
5. *New:* Transfer to classification tasks that identify fine-grained tumors, including PDAC, Cyst, and PanNet in JHH (Figure 3).

Thank you for your valuable feedback regarding our title. We agree the previous title can be confusing and de-appreciate our contribution. We have now revised it to **“How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?”** and are actively considering further refinements for greater clarity. One of the contributions of our study, as described in Section 3.2, is the systematic benchmarking of transfer learning performance, particularly from 3D segmentation tasks to a wider range of 3D imaging tasks. Another contribution is the creation of large-scale, per-voxel annotated ImageNetCT-9K, which in turn, makes the transfer learning benchmarking possible.

Reviewer comment (Writing):
The use of the word "ImageNet" in the dataset name may want to be reconsidered. Besides being a large dataset with a variety of anatomy, the link to ImageNet is a bit weak and may also suggest properties that the dataset does not have (e.g., per-image classification labels).

Author response:
Our objective in developing ImageNetCT-9K is to drive algorithmic advancements and set new benchmarks in the field of 3D medical imaging. In many ways, our dataset echoes the early days of ImageNet, as both datasets emerged at times when large-scale data, diverse classes, and detailed labels were sparse in their respective fields. The limitations of publicly available datasets have been summarized with statistics in Appendix B.1 Table 5 and Figure 5.

Segmentation is often conceptualized as per-voxel classification. In the medical domain, segmentation holds the same fundamental importance as classification does in general computer vision [[Ma & Wang, Nature Methods 2023](https://www.nature.com/articles/s41592-023-01885-0)]. We bet that ImageNet-like datasets in the medical domain should be formed as per-voxel segmentation labels. Our dataset aligns with this vision by providing per-voxel labels, offering a level of detail far surpassing ImageNet's per-image labels. Concretely, the per-voxel labels in our dataset (**272.7B annotated voxels**) are much more extensive than the per-image labels in ImageNet (**14M annotated images**).

We really appreciate your suggestions and are actively looking for better names for our dataset to deliver our vision more precisely for the medical imaging field and to inspire further research endeavors towards this end.

Reviewer comment (Experiments):
CTs are very specific types of 3D data -- it's difficult to make a claim for all 3D data from these experiments alone. I think the paper could be improved by focusing the message more narrowly, perhaps on medical imaging segmentation.

Author response:
We completely agree with your comments and have now revised our title, abstract, introduction, and conclusion to narrow down our current scope to 3D medical imaging. 

Given that our dataset includes detailed per-voxel annotations for 25 organs and tumors, it enables the automatic generation of 3D shape representations. These representations can be formatted as point clouds, voxel occupancy grids, meshes, and implicit surface models (e.g., signed distance functions), each catering to different algorithmic needs. We anticipate our dataset could be useful for a variety of other 3D medical vision tasks [[Li et al., 2023](https://arxiv.org/abs/2308.16139)], such as pose estimation, surface reconstruction, depth estimation, etc. Since these studies go far beyond the scope of the current manuscript and our expertise, we would like to leave the investigation as an independent work in the future. The potential applications and implications of our dataset have now been included in the future work section and Appendix F.4.

Reviewer comment (Novelty):
To me, the results are not surprising -- if you have a such a large dataset with rich segmentation labels, then it should do better than self-supervision. This is in contrast to classification, where this may not be true. I think the main point of the paper should be that this is a new dataset that will be valuable to the community, not that rich supervision is useful in segmentation.

Author response:
Thank you for recognizing the value of our dataset. As detailed in **Q1**, we have systematically benchmarked the transfer learning ability of public models and are preparing to release a suite of models that are pre-trained on our extensive, annotated dataset. The download links are available in our common responses. These two contributions align closely with the conference of *Learning Representation* (IC*LR*) and our submission is categorized into the *“datasets and benchmarks”* primary area.

The debate between the effectiveness of representation learning using human-annotated data (supervised pre-training) versus raw data (self-supervised pre-training) has been a longstanding topic in general computer vision, as we review in Section 2. As acknowledged by Reviewer fMfJ, *this paper concluded on the debate of whether self-supervised or supervised pre-training leads to better performance and data efficiency. This debate had not be resolved without the invention of a fully-annotation dataset of such scale.* Our constructed dataset enables this crucial comparison and promises to be a valuable asset for future algorithm benchmarking in the field.

Based on our comparison, two observations are particularly surprising. Please note that in this comparison, the model is transferred to the classes and datasets that differ from those used for pre-training.

1. Efficiency in pretext task: Supervised pre-training requires **99.6%** fewer data and **96.5%** less computation than self-supervised pre-training. The model trained with 21 CT volumes, 672 masks, and 40 GPU hours shows a transfer learning ability similar to that trained with 5,050 CT volumes and 1,152 GPU hours. See details in Figure 2a.

2. Efficiency in target task: Supervised pre-training requires **50%** fewer manual annotations for fine-grained tumor classification than self-supervised pre-training. This is particularly critical for tumor imaging tasks because annotating tumors requires much more effort and often relies on the availability of pathology reports. See details in Figure 2b and Appendix D.1 Figure 9.

Reviewer comment (Evaluation):
Is there anything surprising from the results? I think it is clear that if you have rich segmentation labels you can learn a better segmentation model than via self-supervised learning.

Author response:
Thank you for recognizing the value of our dataset. As detailed in **Q1**, we have systematically benchmarked the transfer learning ability of public models and are preparing to release a suite of models that are pre-trained on our extensive, annotated dataset. The download links are available in our common responses. These two contributions align closely with the conference of *Learning Representation* (IC*LR*) and our submission is categorized into the *“datasets and benchmarks”* primary area.

The debate between the effectiveness of representation learning using human-annotated data (supervised pre-training) versus raw data (self-supervised pre-training) has been a longstanding topic in general computer vision, as we review in Section 2. As acknowledged by Reviewer fMfJ, *this paper concluded on the debate of whether self-supervised or supervised pre-training leads to better performance and data efficiency. This debate had not be resolved without the invention of a fully-annotation dataset of such scale.* Our constructed dataset enables this crucial comparison and promises to be a valuable asset for future algorithm benchmarking in the field.

Based on our comparison, two observations are particularly surprising. Please note that in this comparison, the model is transferred to the classes and datasets that differ from those used for pre-training.

1. Efficiency in pretext task: Supervised pre-training requires **99.6%** fewer data and **96.5%** less computation than self-supervised pre-training. The model trained with 21 CT volumes, 672 masks, and 40 GPU hours shows a transfer learning ability similar to that trained with 5,050 CT volumes and 1,152 GPU hours. See details in Figure 2a.

2. Efficiency in target task: Supervised pre-training requires **50%** fewer manual annotations for fine-grained tumor classification than self-supervised pre-training. This is particularly critical for tumor imaging tasks because annotating tumors requires much more effort and often relies on the availability of pathology reports. See details in Figure 2b and Appendix D.1 Figure 9.

Reviewer comment (Evaluation):
Could the authors explain why the performance "scratch" out-performed most of the pre-training method in Tab. 2?

Author response:
The goal of Table 2 is to provide a practical benchmark for the transfer learning ability of readily available pre-trained models. Our intent is not to compare the specific pre-training methodologies of each model for two primary reasons. **Firstly**, the majority of researchers tend to fine-tune pre-existing models rather than retrain them from scratch due to convenience and accessibility. **Secondly**, reproducing these models would require specialized hyper-parameter tuning and varied computational resources. For example, models like Swin UNETR [[Tang et al., CVPR 2023](https://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf)] are pre-trained using large-scale GPU clusters at NVIDIA, making them challenging for us to faithfully retrain. Considering both practical user scenarios and computational constraints, we decided to directly use their released models and fine-tune them with consistent settings on the same datasets.  

Using existing pre-trained models can inevitably lead to certain problems. For example, the U-Net family has seen numerous variations over the years [[Siddique et al., 2021](https://ieeexplore.ieee.org/abstract/document/9446143)]. Pre-trained models released before 2021 typically employed a basic version of U-Net (e.g., [[Zhou et al., 2019](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7405596/)] and [[Chen et al., 2019](https://arxiv.org/abs/1904.00625)] in Table 2). On the other hand, our U-Net benefits from a more advanced code base, thanks to the [MONAI](https://monai.io/) platform at NVIDIA, which includes enhanced architectures and advanced training optimization strategies. Consequently, our U-Net, even trained from scratch, is capable of surpassing the performance of these older baseline models.