Paper: Thin-Thick Adapter: Segmenting Thin Scans Using Thick Annotations (ICLR-2024)

Reviewer comment (Theory):
The work introduces terminology that does not exist in the medical imaging community ("thick" and "thin" slices) and provides context around it that is highly inaccurate. There is no such thing as generating thick slices from thin slices. Voxel spacing, and particularly the resolution within the z axis of a scan, strongly depends on the image acquisition and reconstruction process, including the properties of the scan used.

Author response:
**Q1.1. The main criticism to this work is that it introduces a terminology that does not exist in the medical imaging community (thick and thin slices) and provides some context around it that is highly inaccurate.**

**A1.1.:** Thanks for the review! The terminology of thin slices and thick slices [1,2,3,15,17], also refers to thin sections and thick sections [4,5,6,16,18], is a widely used terminology in the radiology community and in medical imaging analysis. We believe there is a large body in the clinical and computer vision community that do utilise the terminology of thick and thin slice CT imaging. For this community it is an appreciated problem that pixel level annotations of thin slice CT data is extremely time consuming. The TTA provides a practical means of addressing this issue. We have adapted a model trained on thick slice data to accurately segment on thin slice data. These thin slice segmentations can then be used in various research based applications. Reliance on manual segmentations to produce such annotations, delays research advancements and we believe provides strong impetus for reviewers to consider the TTA as a novel and citable research tool.

Reviewer comment (Theory):
In reality, both in clinical practice and research, it is much more desirable to have high resolution (what here is denoted as thin slices). This is not often possible due to constraints in the acquisition process, but it is always desired. Hence, there is no such thing as generating "thick slices" from "thin" ones.

Author response:
**Q1.2. There is not such thing as generating thick slices from thin slices. Voxel spacing and in particular the resolution within the z axis of a scan strongly depends on the image acquisition and reconstruction process, including the properties of the scan used.**

**A1.2.:** Many thanks for bring out the question about generate thick slices from thin slices. There are several methods to obtain thick slices as mentioned in Section 1:

- One is directly from the primary reconstruction of projection data (also refers to sinogram or raw data [7]) during image acquisition [4]. 

- The other method involved during post-processing, axial thick slices can be resliced from axial thin slices which were obtained from primary reconstruction, without having access to the projection data [8,9]. 

  - Here is a concrete example [8]: “The DICOM files were imported into the software package Mimics 18.0 (Materialise). The initial file created from the 0.625 mm scan was then resliced to 1 mm, 1.25 mm, 2 mm, 2.5 mm and 5 mm for each head. The FOV and pixel size remained consistent through this process.” 
  - Here is another example [9]: “The range of the distal femur through the proximal tibia was covered. Images were obtained with use of 0.625 mm of slice increment, 110 kV 230 mAs, and in-plane resolution of 512 × 512. … The original CT and MRI data were imported into Mimics (version 14.1, Materialise, Leuven, Belgium). The imported images were first reconstructed into volume and then resliced to create a new axial image set with the slice increment of 1.5 mm.”

- Axial thin slices allow for obtaining not only axial thick slices but also non-axial (sagittal, coronal) thick slices through multiplanar reformation (MPR, also known as multiplanar reconstruction) [4]. There are various algorithms used to thicken MPRs, including Average Intensity Projection (AIP), Maximum Intensity Projection (MIP), Minimum Intensity Projection (MinIP), and others [4]. These algorithms are applied in different scenarios, for instance, MIP is frequently utilized for visualizing bones and vascular structures [10].

- Although MPR is commonly utilized to generate non-axial images from axial thin data [4], axial thick data can also be obtained through axial multiplanar reformation [11,12,13,19,20,21,22].

  - Here is a concrete example [11]: “Twenty-six images of a 7.5-mm slice thickness were reconstructed from the data sets of 2.5-mm-thick and 1.25-mm-thick interval images (159 images for each phase) by a radiology technologist on a workstation (Advantage Windows 3.1; General Electric Medical Systems, Milwaukee, WI) using the axial multiplanar reformation technique.”

  - Here is another example [14]: ”The resulting thin slices were then postprocessed to obtain 1-cm thick slices by using three different algorithms. The first algorithm, maximum intensity projection (MIP), generates a thick slice from a set of thin slices in which the resulting pixels represent the brightest voxels of each stack of slices.”

Furthermore, there would be a notable advantage in exclusively using annotations on axial thick slices regardless they are generated either from primary reconstruction or thin slices. The TTA we proposed would make establishing ground truth annotations less resource-intensive, given that accurate segmentation of thin slice scans could be achieved using the annotations from thick slices.

Reviewer comment (Novelty):
On the methods side, the contributions are marginal. The proposed approach consists of resampling thick images to make them thin, then training a network using a loss (cross pseudo supervision) that has previously been proposed in the literature. This work extends it to 3D, which seems straightforward, and uses it to train the model.

Author response:
**Q3. I would have liked to see more comments regarding the novelty of the work. For instance, there are open-source tools, like 3D Slicer, that offer tools to interpolate slices, i.e. Thick Slab reconstruction. In this case, one obtains the best possible slices in terms of thickness, uses slicer to get thick slides and annotates there. In which way is it better to use your approach than that one offered by 3D slicer?**

**A3.:** Thank you for initiating this discussion comparing Thin-Thick Adapter (TTA) to 3D slicer and interpolation techniques. Before delving into this question, it's crucial to reiterate our primary objective, as mentioned in the paper, which is to use annotations on thick slices to perform segmentation on thin slices **without requiring any thin-thick pair**. This means the **thick annotation can be sourced from any public thick annotated dataset**, and **it doesn't have to be the exact thick slices generated from your thin slices**. And the mention of thick slice acquisition in Section 1 is just simply to introduce how thick slices are obtained in clinical practice.

As you pointed out, there are open-source tools like 3D Slicer [1] that enable users to generate thick slices from thin slices through **Thick Slab Reconstruction (TRS)** [2]. Additionally, 3D Slicer can interpolate thick slices to obtain upsampled thin slices using built-in interpolation algorithms in the **Resample Image** [3] of 3D Slicer. Specifically, Thick Slab Reconstruction (TRS) employs the exact method mentioned in **A1.2**, which is utilizing multiplanar reformation (MPR), such as Maximum Intensity Projection (MIP), to obtain thick slices from thin slices [2]. And the Resample Image function in 3D slicer incorporates multiple interpolation algorithms, such as linear interpolation, to upsample thick slices into interpolated thin slices [3].

Let's consider the scenario you mentioned where users obtain thick slices from thin slices using Thick Slab Reconstruction, and annotate these thick slices, then interpolate the annotated thick slices to obtain upsampled annotated thin slices using interpolation algorithms such as linear interpolation in the Resample Image function. However, this approach has two significant drawbacks:

- The method still requires users to pixel-level annotate the thick scans, which is not necessary in our TTA. The thick annotations used in TTA can come from any publicly annotated thick dataset, such as BHSD [4] for brain hemorrhage, HECKTOR [5] for head and neck tumors, HaN-Seg [6] for organs-at-risk, and numerous others. With the development of the medical imaging community, obtaining annotated thick data is possible from public datasets in most cases, without the need for manual annotation.

- Since the upsampled thin scans are obtained by interpolating thick scans, there is a substantial loss of information in details compared to the original thin slices, resulting in inaccurate annotations. In contrast, our Thin-Thick Adapter operates directly on thin slices, avoiding information loss and enabling more accurate segmentation.

You may also would like to mentioned that there are extensions in 3D Slicer [1], such as TotalSegmentator [7], MONAI Label [8], and Nvidia AI-assisted annotation (AIAA) [9], which can assist users in automatically annotating thick scans. However, it's important to acknowledge two significant drawbacks in this approach:

- The extensions used for automatic segmentation in 3D Slicer typically serve as basic assistance tools for segmentation. Although some extensions, like MONAI Label [8], employ deep learning techniques, they are not specifically fine-tuned for a particular segmentation task and lack tunability. Consequently, they perform segmentation directly on scans without any adaptation, which falls short compared to a network like nnUNet [10] that can be trained for a specific task, not to mention when compared to our TTA.

- This approach shares the same issue as the one in the first scenario. The upsampled annotated thin slices, obtained through interpolation from annotated thick slices, experience substantial information loss compared to the original thin slices, resulting in inaccurate segmentation.

You may also consider initially interpolating annotated thick scans into upsampled annotated thin slices using 3D Slicer, then utilize these interpolated scans to train a model (e.g., nnUNet [10]) for segmentation on your thin slices. To compare with this approach, we conducted various experiments in Section 6.1, including comparisons with interpolation methods like trilinear interpolation [11] and UniRes [12]. The results presented in Table 1 demonstrate that our method significantly outperforms these interpolation methods, and the visualization is shown in Figure 3.

Reviewer comment (Writing):
Figure 3 refers to some acronyms that are not introduced in the text.

Author response:
**Q2. Figure 3 refers to some acronyms that are not introduced in the text.**

**A2.:** My appreciation for your review, and the we will describe acronyms mentioned in Figure 3 as the following three points

- In Figure 3, the left side displays various orientation views of CT scans, including axial, sagittal, and coronal, as mentioned in Section 1 Introduction, with reference.

- At the bottom, there are legends depicting different semantics corresponding to various categories in brain hemorrhage, including Epidural Hemorrhage (EDH), Intraparenchymal Hemorrhage (IPH), Intraventricular Hemorrhage (IVH), Subarachnoid Hemorrhage (SAH), and Subdural Hemorrhage (SDH), as mentioned in Section 4 Dataset.

- At the top, there are images, ground truth (GT), and various established models, as discussed in Section 6.1 Experiment Setup.

Reviewer comment (Experiments):
Annotation robustness appears especially relevant with fine-grained (and limited) data such as thin slices. Might any indication of inter-grader reliability be known, for the labelled thin slices?

Author response:
**Q1. Annotation robustness appears especially relevant with fine-grained (and limited) data such as thin slices. Might any indication of inter-grader reliability be known, for the labeled thin slices?**

**A1.:** My deep appreciation for your helpful suggestion of inter-grader reliability/inter-rater reliability. Combined efforts between three annotators ADH, AB and CB were used to produce the annotations.

Author and board certified neuroradiologist CW initially conducted three 150 minute training sessions to teach author ADH intracranial hemorrhage identification, segmentation and classification.  

ADH then produced segmentations independently for 25 thin slice CT image stacks. These segmentations were then reviewed by author AB and finally CW. Following review of the segmentations feedback was provided and necessary modifications made to the segmentations. This process occurs in several iterative rounds until all authors had reached consensus that robust annotations capturing intracranial hemorrhage were produced.

We believe this iterative approach involving multiple authors creating the annotations addresses the concerns related to the robustness of the thin slice annotations used as our ground truth measure.

While we thank the reviewer for the suggestion of using a measure of inter-grader reliability, since the annotations were not produced independently we believe this is not appropriate in our circumstance. We believe a more robust marker of the reliability of the segmentations will be offered when the model is externally validated

Reviewer comment (Experiments):
In Section 5.1, the generation of thin slices from thick slices via duplicating and depth spacing adjustment (i.e. thinning) does not appear to take neighbouring slices into account, which seems natural at the thin slice boundaries. Might some interpolation/additional processing have been considered for the output thin slice at the original thick slice boundaries?

Author response:
**Q2. In Section 5.1, the generation of thin slices from thick slices via duplicating and depth spacing adjustment (i.e. thinning) does not appear to take neighbouring slices into account, which seems natural at the thin slice boundaries. Might some interpolation/additional processing have been considered for the output thin slice at the original thick slice boundaries?**

**A2.:** Thanks for the advice for taking neighbouring slices into account in data alignment. In Section 6, we conducted a comparison of interpolation methods widely used in medical imaging, such as trilinear interpolation and UniRes. The interpolated images are shown in Figure 2, and, as you noted, interpolation considers neighboring slices, resulting in a more natural appearance on sagittal and coronal views. However, two major issues limit interpolation performance. First, the interpolated mask lacks accuracy and precision, hindering overall performance. Second, interpolated images are notably blurrier than the original thick slices, diminishing the contrast of small structures and brain tissue details, consequently reducing performance.

It's important to highlight that our goal is not to generate pseudo-thin slices from thick slices. Instead, we utilize thick annotations to enhance the segmentation performance of thin slices. The alignment of data with thick slices only serves as an intermediate step to support thin slice segmentation. The results in Tables 1 and 2 demonstrate that our TTA is currently the most effective solution for addressing the thin slice segmentation problem without any annotation on thin slices.

Reviewer comment (Theory):
In Section 5.2, the methodology involving gradual linear increase of λ as the weight of the cross-supervision loss, is not justified in detail.

Author response:
**Q3. In Section 5.2, the methodology involving gradual linear increase of λ as the weight of the cross-supervision loss, is not justified in detail.**

**A3.:** Thanks for pointing it out. We set λ to increase linearly from 0 to 0.5, and it remains fixed after 500 epochs, which has been updated in Section 6.1 in revised paper. Moreover, we have conducted an ablation of investigating different ranges of $\lambda$ and which was fixed after various epochs $\epsilon$ in Section 6.4. of the updated revised paper.

|                          | CQ500-Thin |           |           |           | ROTEM-Thin |           |           |           | Both |           |           |           |
|--------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|---------------------------|-----------|-----------|-----------|
|                          | mDSC | aDSC | mIoU | aIoU | mDSC | aDSC | mIoU | aIoU | mDSC | aDSC | mIoU | aIoU |
|--------------------------|------|------|------|------|------|------|------|------|------|------|------|------|------|
| $\epsilon = 0, \lambda = 0.5$ | 76.93 | 83.01 | 63.98 | 71.26 | 58.83 | 75.12 | 46.91 | 60.03 | 75.70 | 80.31 | 62.83 | 67.58 |
| $\epsilon = 300, \lambda = 0 \rightarrow 0.5$ | 77.48 | 83.49 | 64.09 | 71.73 | 59.31 | 75.48 | 47.53 | 60.67 | 76.62 | 80.94 | 63.21 | 68.34 |
| $\epsilon = 700, \lambda = 0 \rightarrow 0.5$ | 77.51 | 83.37 | 64.12 | 71.79 | 59.33 | 75.36 | 47.48 | 60.55 | 76.58 | 80.91 | 63.17 | 68.29 |
|--------------------------|------|------|------|------|------|------|------|------|------|------|------|------|------|
| $\epsilon = 500, \lambda = 0 \rightarrow 0.3$ | 76.50 | 81.76 | 62.99 | 69.37 | 58.45 | 74.84 | 46.32 | 59.78 | 74.81 | 80.12 | 61.13 | 66.74 |
| $\epsilon = 500, \lambda = 0 \rightarrow 0.7$ | 77.13 | 82.66 | 63.41 | 70.08 | 59.51 |75.46 | 47.01 | 60.38 | 75.95 | 81.11 | 63.39 | 67.84 |
|--------------------------|------|------|------|------|------|------|------|------|------|------|------|------|------|
| **$\epsilon = 500, \lambda = 0 \rightarrow 0.5$** | **77.54** | **83.62** | **64.33** | **71.86** | **59.69** | **75.64** | **47.68** | **60.82** | **76.94** | **81.36** | **63.45** | **68.57** |

The table above displays the ablations of various ranges of hyper-parameters ($\lambda$ and $\epsilon$) in the CPS loss. The results indicate that the configuration $\lambda = 0 \rightarrow 0.5$ and $\epsilon = 500$, used in our experiments, represents the optimal hyper-parameters. Furthermore, our method demonstrates consistently good performance across different sets of hyper-parameters, with only minor fluctuations, suggesting the robustness of our approach.

Reviewer comment (Experiments):
Could the authors provide a discussion on using CPS over EMA for the semi-supervised component in their TTA method?

Author response:
**A1.1.:** Thanks for the suggestion. As mentioned in Section 6.1, we selected CPS [1,13] as our unsupervised domain adaptation (UDA) technique due to its superior performance compared to other semi-supervised techniques [2], including Entropy Minimization [3], Mean Teacher [4], and Interpolation Consistency [5] in a semi-supervised setting on the BHSD dataset. The BHSD paper [2] extensively investigated various semi-supervised methods in Section 3.4, employing 96 labeled data and 500 unlabeled data from the BHSD dataset. The results demonstrated that CPS outperformed other methods, such as Mean Teacher (which typically uses exponential moving average from the student model to the teacher model), by nearly **5%** in DSC. This significant difference in performance led us to explore the potential of CPS in our proposed TTA method.

Reviewer comment (Experiments):
The authors could consider showing an ablation study for different ranges of $\lambda$ to show the robustness of the method.

Author response:
**A1.3.:** Thanks for the suggestions, we conducted the third ablation which investigated the different ranges of $\lambda$ and which was fixed after various epochs $\epsilon$ in Section 6.4. of the updated revised paper.

|                          | CQ500-Thin |           |           |           | ROTEM-Thin |           |           |           | Both |           |           |           |
|--------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|---------------------------|-----------|-----------|-----------|
|                          | mDSC | aDSC | mIoU | aIoU | mDSC | aDSC | mIoU | aIoU | mDSC | aDSC | mIoU | aIoU |
|--------------------------|------|------|------|------|------|------|------|------|------|------|------|------|
| $\epsilon = 0, \lambda = 0.5$ | 76.93 | 83.01 | 63.98 | 71.26 | 58.83 | 75.12 | 46.91 | 60.03 | 75.70 | 80.31 | 62.83 | 67.58 |
| $\epsilon = 300, \lambda = 0 \rightarrow 0.5$ | 77.48 | 83.49 | 64.09 | 71.73 | 59.31 | 75.48 | 47.53 | 60.67 | 76.62 | 80.94 | 63.21 | 68.34 |
| $\epsilon = 700, \lambda = 0 \rightarrow 0.5$ | 77.51 | 83.37 | 64.12 | 71.79 | 59.33 | 75.36 | 47.48 | 60.55 | 76.58 | 80.91 | 63.17 | 68.29 |
|--------------------------|------|------|------|------|------|------|------|------|------|------|------|------|
| $\epsilon = 500, \lambda = 0 \rightarrow 0.3$ | 76.50 | 81.76 | 62.99 | 69.37 | 58.45 | 74.84 | 46.32 | 59.78 | 74.81 | 80.12 | 61.13 | 66.74 |
| $\epsilon = 500, \lambda = 0 \rightarrow 0.7$ | 77.13 | 82.66 | 63.41 | 70.08 | 59.51 |75.46 | 47.01 | 60.38 | 75.95 | 81.11 | 63.39 | 67.84 |
|--------------------------|------|------|------|------|------|------|------|------|------|------|------|------|
| **$\epsilon = 500, \lambda = 0 \rightarrow 0.5$** | **77.54** | **83.62** | **64.33** | **71.86** | **59.69** | **75.64** | **47.68** | **60.82** | **76.94** | **81.36** | **63.45** | **68.57** |

The table above displays the ablations of various ranges of hyper-parameters ($\lambda$ and $\epsilon$) in the CPS loss. The results indicate that the configuration $\lambda = 0 \rightarrow 0.5$ and $\epsilon = 500$, used in our experiments, represents the optimal hyper-parameters. Furthermore, our method demonstrates consistently good performance across different sets of hyper-parameters, with only minor fluctuations, suggesting the robustness of our approach.

Reviewer comment (Evaluation):
While the proposed method achieves good performance, could the authors discuss what is the norm in CT acquisition --- do most acquisitions generate thick datasets, or, are thin datasets more common? The significance of the authors' contribution depends on which of the two - thick or thin - is more prevalent in practice.

Author response:
**A1.4:** This is a very meaningful question and thanks for bringing it out. To answer the question, we need to differentiate between **CT acquisition in clinical practice** and the **release of a dataset** for the medical imaging community.

In clinical practice, axial thin slices are typically collected during primary reconstruction [6], with exceptions for specific occasions like fast scans or low-resolution CT. It is because a radiologist often needs to scroll through a stack of axial thin slices to localize lesions [7], especially for small lesions, which can only be captured on thin slices [8]. Additionally, thin slices enable the generation of thick slices, not only in the axial but also in non-axial (sagittal, coronal) thick slices through multiplanar reformation (MPR) without the needs of projection data [6]. This capability is elaborated further in the response to Reviewer oEQ1 question 1.2. Moreover, thin slices have broader applications, including accurate brain hemorrhage volume estimation, which is beyond the capabilities of thick slices. In contrast, thick slices are not frequently required during reconstruction and are primarily used for detecting low-contrast lesions due to their higher signal-to-noise ratio (SNR) [9]. Hence, in clinical practice, thin slices are more common, and since projection data are not directly viewed, storing thin slices is more prevalent.

Regarding current public datasets in medical imaging and computer vision, there is a temporal aspect to consider. Older CT scans typically produce thick slices, and thin scans are more storage-intensive compared to thick scans. Consequently, public datasets such as RSNA Pneumonia Detection Challenge [10] predominantly consist of thick datasets. However, as CT and storage technology advances, more recent datasets like CQ500 [11] include thin slices.  However, obtaining large-scale pixel-level annotations for these thin scans remains impractical. This challenge motivates our paper.

Reviewer comment (Writing):
In Eqn (2) and (4), $l_{sup}$ and $l_{cps}$ correspond to which loss? Dice and Cross-Entropy or something else?

Author response:
**A2.:** Thanks for pointing it out, both $l_{sup}$ and $l_{cps}$  are the combination of dice and cross-entropy losses, which are aligned with the default configured of nnU-Net's joint loss [12]. This has been updated in Section 5.2 of revised paper.

Reviewer comment (Writing):
In Eqn (3), $L^l_{cps}$ is not defined in later equations.

Author response:
**A3.:** Thanks for the suggestions, as we mentioned in Section 5.2, The cross-pseudo supervision loss, $L_{cps}$, comprises two parts: $L_{cps}^l$ and $L_{cps}^u$, incorporating the CPS loss on labeled and unlabeled datasets. This means that the $L_{cps}^l$ is calculated in the same fashion as $L_{cps}^u$, and the only modification is to switch unlabeled data to labeled data. In order to avoid ambiguity, we have updated the concrete loss formula in equation (4) and (5) of the updated revised paper.

Reviewer comment (Experiments):
The authors could consider doing a study to show how their method can reduce annotation burden significantly. For example, they could consider the existing thick datasets as “thin”, and generate a corresponding “thick” dataset by reducing the resolution by 2 (so an available volume 512 x 512 x 300 would have a thick counterpart of 512 x 512 x 150). This new “thick” dataset (image + GT) could be obtained by dropping intermediate frames, or, by asking clinicians to annotate on these really coarse volumes. Results in this setting would demonstrate how it can ease the burden on annotators.

Author response:
**A4.:** Many thanks for reviewer providing the inspiring suggestion!  While this experiment could be considered to provide objective evidence of the reduction in annotation burden. We believe radiologists who have performed manual segmentation would accept that annotation of thick data is significantly easier than thin slices. Instead of the 1 in 2 reduction mentioned in the question, manual annotation is reduced by 5 or 10 times when comparing thick to thin segmentation.(thick slices are up to 5mm thickness and 30-50 slices, and thin slices are less than 1mm and 300-500 slices). 

Based on our own annotating process, we also speculate that

- most of the annotation time is spent creating/adjusting the segmentation edge, not the inner (which can be filled)
- thus, interpolation techniques do not help significantly for intermediate frames because the edges still need the name level of refinement

Taking into account these factors, annotation time is expected to be approximately proportional to number of frames, whether they are derived from a thin dataset, or thick dataset with reduced resolution.