Abstract-Accurate segmentation of lung and infection in COVID-19 CT scans plays an important role in the quantitative management of patients. Most of the existing studies are based on large and private annotated datasets that are impractical to obtain from a single institution, especially when radiologists are busy fighting the coronavirus disease. Furthermore, it is hard to compare current COVID-19 CT segmentation methods as they are developed on different datasets, trained in different settings, and evaluated with different metrics. In this paper, we created a COVID-19 3D CT dataset with 20 cases that contains 1800+ annotated slices and made it publicly available. To promote the development of annotation-efficient deep learning methods, we built three benchmarks for lung and infection segmentation that contain current main research interests, e.g., few-shot learning, domain generalization, and knowledge transfer. For a fair comparison among different segmentation methods, we also provide unified training, validation and testing dataset splits, and evaluation metrics and corresponding code. In addition, we provided more than 40 pre-trained baseline models for the benchmarks, which not only serve as out-ofthe-box segmentation tools but also save computational time for researchers who are interested in COVID-19 lung and infection segmentation. To the best of our knowledge, this work presents the largest public annotated COVID-19 CT volume dataset, the first segmentation benchmark, and the most pre-trained models up to now. We hope these resources (https://gitee.com/junma11/ COVID-19-CT-Seg-Benchmark) could advance the development of deep learning methods for COVID-19 CT segmentation with limited data. [4] , [5] , [6] . CT is shown to be more sensitive in the early diagnosis of COIVD-19 infection compared to Reverse Transcription-Polymerase Chain Reaction (RT-PCR) tests [7] . Wang et al. trained a deep learning model on 325 COVID-19 CT images and 740 typical pneumonia images. Their model identified 46 COVID-19 cases that were previously missed by the RT-PCR test. Further, quantitative information from CT images, such as lung burden, percentage of high opacity and lung severity score, can be used to monitor disease progression and help us understand the course of COVID-19 [8] , [9] .
Artificial Intelligence (AI) methods, especially deep learning-based methods, have been widely applied in medical image analysis to combat COVID-19 [10] . For example, AI can be used for building a contactless imaging workflow to prevent transmission from patients to health care providers [11] . In addition, most screening and segmentation algorithms for COVID-19 are developed with deep learning models [7] , [12] , [13] , and most automatic diagnosis and infection quantification systems in COVID-19 studies rely on segmentation generated by deep neural networks [12] , [14] , [15] , [16] .
Although tremendous studies show that deep learning methods have potential for providing accurate and quantitative assessment of COVID-19 infection in CT images [10] , the solutions mainly rely on large and private datasets. Due to the patient privacy and intellectual property issues, the datasets and solutions may not be publicly available. However, researchers may hope that the datasets, entire source code and trained models could be provided by authors [17] .
Existing studies demonstrate that the classical U-Net (or V-Net) can achieve promising segmentation performance if hundreds of well-labelled training cases are available [15] , [?] . 83.1% to 91.6% of segmentation performance in Dice score coefficient was reported in various U-Net-based approaches. Shan et al. developed a neural network based on V-Net and the bottle-neck structure to segment and quantify infection regions [?] , [18] . In their human-in-the-loop strategy, they achieved 85.1%, 91.0% and 91.6% dice with 36, 114, and 249 labelled CT scans, respectively. Huang et al. [15] employed U-Net [19] for lung and infection using an annotated dataset of 774 cases, and demonstrated that trained model could be used for quantifying the disease burden and monitoring disease progression or treatment response. In general, we observed a trend that training deep learning models with more annotations will decrease the time needed for contouring the infection, and increase segmentation accuracy, which is consistent with the Shan et al.'s findings [?] . However, annotations for 3D CT volume data are expensive to acquire because it not only relies on professional diagnosis knowledge of radiologists, but also takes much time and labor, especially in current situation. Thus, a critical question would be:
how can we annotate COVID-19 CT scans efficiently?
Towards this question, three basic but important problems remain unsolved:
• There is no publicly well-labelled COVID-19 CT 3D dataset; Data collection is the first and essential step to develop deep learning methods for COVID-19 segmentation. Most of the existing studies rely on private and large dataset with hundreds of annotated CT scans. • There are no public benchmarks to evaluate different deep learning-based solutions and different studies use various evaluation metrics. Similarly, datasets for training, validating and testing are split diversely, which also makes it hard for readers to compare those methods. For example, although MedSeg (http://medicalsegmentation.com/covid19/) dataset with 100 annotated slices were used in [20] , [21] , [22] , [23] to develop COVID-19 segmentation methods, it was split in different ways and the developed methods were evaluated by different metrics. • There are no publicly available baseline models trained for COVID-19. U-Net, a well-known segmentation architecture, is commonly used as a baseline network in many studies. However, due to different implementations, the reported performance variesin different studies [21] , [24] , even though the experiments are conducted on the same dataset.
In this paper, we focus on annotation-efficient deep learning solutions and aim to alleviate the above problems by providing a well-labelled COVID-19 CT dataset and a benchmark. In particular, we first annotated left lung, right lung and infections of 20 COVID-19 3D CT scans, and then established three tasks to benchmark different deep learning strategies with limited training cases. Finally, we built baselines for each task based on U-Net.
Our tasks target on three popular research fields in medical imaging community:
• Few-shot learning: building high performance models from very few samples. Although most existing approaches focus on natural images, it has received growing attention in medical imaging because generating labels for medical images is much more difficult [25] , [26] . • Domain generalization: learning from known domains and applying to an unknown target domain that has different data distribution [27] . Adversarial training is usually adopted to achieve it in recent studies. [28] , [29] . • Knowledge transfer: reusing existing annotations to boost the training/fine-tuning on a new dataset. Transferring knowledge from large public datasets is a well-established practice in medical imaging analysis [30] , [31] . Using generative adversarial network to transfer knowledge from publicly available annotated datasets to new datasets for segmentation also receives significant attention [32] , [33] . Our contributions can be summarized as following:
• 20 well-labelled COVID-19 CT volume data are publicly released to the community. To the best of our knowledge, this is the largest public COVID-19 3D CT segmentation datasets. • 3 benchmark tasks are set up to promote studies on annotation-efficient deep learning segmentation for COVID-19 CT scans. Specifically, we focus on few-shot learning, domain generalization, and knowledge transfer. • 40+ trained models are publicly available that can serve as baselines and out-of-the-box tools for COVID-19 CT lung and infection segmentation.
Annotations of COVID-19 CT scans are scarce, but several lung CT annotations with other diseases are publicly available. Thus, one of the main goals for our benchmark is to explore these existing annotations to assist COVID-19 CT segmentation. This section introduces the public datasets used in our segmentation benchmarks. Figure 1 presents some examples from each dataset.
A. Existing lung CT segmentation datasets 1) StructSeg lung organ segmentation: 50 lung cancer patient CT scans are accessible, and all the cases are from one medical center. This dataset served as a segmentation challenge 1 during MICCAI 2019. Six organs are annotated, including left lung, right lung, spinal cord, esophagus, heart, and trachea. In this paper, we only use the left lung and right lung annotations.
2) NSCLC left and right lung segmentation: This dataset consists of left and right thoracic volume segmentations delineated on 402 CT scans from The Cancer Imaging Archive NSCLC Radiomics [34] , [35] , [36] .
B. Existing lung lesion CT segmentation datasets 1) MSD Lung tumor segmentation: This dataset is comprised of patients with non-small cell lung cancer from Stanford University (Palo Alto, CA, USA) publicly available through TCIA. The dataset served as a segmentation challenge 2 during MICCAI 2018. The tumor is annotated by an expert thoracic radiologist, and 63 labelled CT scans are available.
2) StructSeg Gross Target Volume segmentation of lung cancer: The same 50 lung cancer patient CT scans as the above StructSeg lung organ segmentation dataset are provided, and gross target volume of tumors are annotated in each case.
3) NSCLC Pleural Effusion (PE) segmentation: The CT scans in this dataset are the same as those in NSCLC left and right lung segmentation dataset, while pleural effusion is delineated for 78 cases [34] , [35] , [36] . 
We collected 20 public COVID-19 CT scans from the Coronacases Initiative and Radiopaedia, which can be freely downloaded 3 with CC BY-NC-SA license. The left lung, right lung, and infection were firstly delineated by junior annotators, then refined by two radiologists with 5 years experience, and finally all the annotations were verified and refined by a senior radiologist with more than 10 years experience in chest radiology. On average, it takes about 400 ± 45 minutes to delineate one CT scan with 250 slices. We have made all the annotations publicly available [37] at https://zenodo.org/record/3757476 with CC BY-NC-SA license.
As mentioned in Section 1, there is a need for innovative strategies that enable annotation-efficient methods for COVID-CT segmentation. Thus, we set up three tasks to evaluate potential annotation-efficient strategies. In particular, we focus on learning to segment left lung, right lung and infection in COVID-19 CT scans using (Table I) . are held-out for validation. All labelled COVID-CT-Seg CT scans are kept for testing (Table II) .
Task 3 is designed to address the problem of knowledge transfer with heterogeneous annotations, where out-of-domain data are used to boost the annotation-efficient training on limited in-domain data. Specifically, in both subtasks (lung segmentation and lung infection segmentation), 80% and 20% of out-of-domain data are used for training and validation, while 20% and 80% of in-domain data are used for training and testing, respectively (Table III) .
Motivated by the evaluation methods of the well-known medical image segmentation decathlon 4 , we also employ two complementary metrics to evaluate the segmentation performance. Dice similarity coefficient, a region-based measure, is used to evaluate the region overlap. Normalized surface Dice [38] , a boundary-based measure is used to evaluate how close the segmentation and ground truth surfaces are to each other at a specified tolerance τ . For both two metrics, higher scores admit better segmentation performance, and 100% means perfect segmentation. Let G, S denote ground truth and segmentation, respectively. We formulate the definitions of the two measures as follows:
1) Region-based measure:
2) Boundary-based measure:
∂S ⊂ R 3 denote the border region of ground truth and segmentation surface at tolerance τ , which are defined as B (τ )
We set tolerance τ as 1mm and 3mm for lung segmentation and infection segmentation, respectively. The tolerance is computed by measuring the inter-rater segmentation variation between two different radiologists, which is also in accordance with another independent study [38] .
The main benefit of introducing Surface Dice is that it ignores small boundary deviations because small inter-observer errors are also unavoidable and often not clinically relevant when segmenting the objects by radiologists.
U-Net ( [19] , [39] ) has been proposed for 5 years, and many variants have been proposed to improve it. However, recent study [40] demonstrates that a basic U-Net is still difficult to surpass if the corresponding pipeline is designed adequately. 4 http://medicaldecathlon.com/files/MSD-Ranking-scheme.pdf In particular, nnU-Net (no-new-U-Net) [40] was proposed to automatically adapt preprocessing strategies and network architectures (i.e., the number of pooling, convolutional kernel size, and stride size) to a given 3D medical dataset. Without manual tuning, nnU-Net achieved better performance than most specialised deep learning pipelines in 19 public international segmentation competitions and set a new state-of-theart in the majority of 49 tasks. The source code is publicly available at https://github.com/MIC-DKFZ/nnUNet. U-Net is often used as a baseline model in existing COVID-19 CT segmentation studies. However, reported results vary a lot even in the same dataset, which make it hard to compare different studies. To standardize the U-Net performance, we build our baselines on nnU-Net, which is the most powerful U-Net implementation to the best of our knowledge. To make it comparable between different tasks, we manually adjust the patch size and network architectures in task 2 and task 3 to be the same as task 1. Figure 2 shows details of the U-Net architecture.
A. Results of Task 1: Learning with limited annotations Table IV presents average DSC and NSD results of lung and infection of each subtask in Task 1. It can be found that • the average DSC and NSD values among different folds vary greatly. It demonstrates that reporting 5-fold cross validation results is necessary to obtain an object evaluation because 1-fold results may be biased. • promising results for left and right lung segmentation in COVID-19 CT scans can be achieved with as few as four training cases. Models trained for segmenting lung obtained significantly better results compared with those trained for segmenting lung and infection simultaneously. • There is still large room for improving infection segmentation with limited annotations. Statistical tests are needed to justify whether there is an improvement from training the model for segmenting lung and infection at the same time.
This task is quite challenging as the model does not see any cases from target domain during training. In other words, the trained models are expected to generalize to the unseen domain (COVID-19 CT). Table V shows left lung and right lung segmentation results in terms of average and standard deviation values of DSC and NSD. It can be found that • 3D U-Net achieves excellent performance in terms of DSC on the validation set that is from the same source domain. Average NSD values are lower than DSC values, implying that most of the errors come from the boundary. • the performance on the testing set drops significantly on both subtasks. The performance of the model trained on NSCLC Lung dataset is worse than the model trained on StuctSeg lung. We suppose it is because the difference in the distribution lung appearance is smaller between StructSeg and COVID-19-CT. • on the validation set, the performance of lesion segmentation is not as good as the performance of lung segmentation (Table V) , which means that tumor segmentation remains a challenging problem. This observation is in line with recent results in MICCAI tumor segmentation challenge, i.e. liver tumor segmentation [41] and kidney tumor segmentation [42] . • The models almost fail to predict COVID-19 infections on testing set, which highlights that the lesion appearances differ significantly among lung cancer, pleural effusion, and COVID-19 infections in CT scans.
One possible limitation would be the number of cases in our dataset is relatively small. On one hand, we focus on how to learn from a few training cases(task 1) and how to efficiently use existing annotations. Thus, the number of training cases is acceptable for our benchmark tasks. On the other hand, we have 16 or 20 testing cases in each task. The number of testing cases is comparable with the setting of recent MICCAI 2020 segmentation challenges. For example, MyoPS 2020 (Myocardial pathology segmentation combining multi-sequence CMR) has 15 testing cases [43] . StructSeg (Automatic Structure Segmentation for Radiotherapy Planning Challenge 2020) has 10 testing cases [44] and ASOCA (Automated Segmentation Of Coronary Arteries) has 20 testing cases [45] . Another possible limitation is that only quantitative results of 3D U-Net are reported. We will also provide 2D U-Net baselines for all tasks in the near future.
In this paper, we created a COVID-19 CT dataset, established three segmentation benchmark tasks towards annotation-efficient methods, and provided 40+ baselines models based on state-of-the-art segmentation architectures. All the related results are publicly available at https://gitee.com/ junma11/COVID-19-CT-Seg-Benchmark. We hope this work can help community to accelerate designing and developing AI-based COVID-19 CT analysis tools with limited data.
ACKNOWLEDGMENT This project is supported by the National Natural Science Foundation of China (No. 91630311, No. 11971229). We would like to thank all the organizers of MICCAI 2018 Medical Segmentation Decathlon, MICCAI 2019 Automatic Structure Segmentation for Radiotherapy Planning Challenge, the Coronacases Initiative and Radiopaedia for the publicly available lung CT dataset. We also thank Joseph Paul Cohen for providing convenient download link of 20 COVID-19 CT scans. We also thank all the contributor of NSCLC and COVID-19-Seg-CT dataset for providing the annotations of lung, pleural effusion and COVID-19 infection. We also thank the organizers of TMI Special Issue on Annotation-Efficient Deep Learning for Medical Imaging because we get lots of insights from the call for papers when designing these segmentation tasks. We also thank the contributors of the two great COVID-19 related resources: COVID19 imaging AI paper list 5 and MedSeg 6 for their timely update about COVID-19 publications and datasets. Last but not least, we thank Chen Chen, Xin Yang, and Yao Zhang for their important and valuable feedback on this benchmark.