Paper: Reconstruction of Cortical Surfaces with Spherical Topology from Infant Brain MRI via Recurrent Deformation Learning (ICLR-2024)

Reviewer comment (Reproducibility):
No methodological or implementation details are provided, which makes the paper unreproducible and hard to evaluate. The paper lacks network architecture, code, dataset preprocessing details (crucial to CSR), data augmentation details, and information about how baselines were tuned. The short implementation subsection only lists learning rates, number of iterations, and loss weights, and spends the other half of its space describing instance normalization. Although the paper is under the ICLR page limit and there is unbounded supplementary space, the framework and its details should be fully described, and code should be shared if possible.

Author response:
**Response:** We refer the reviewer to Figure S1 in the supplementary material for details on the common network architecture for the white and pial models. We added descriptions on data processing and more details on reproducing other baseline methods in Sections 4.1 and 4.2. Please let us know if more details are needed.

Reviewer comment (Novelty):
The ODE-based deformation contribution is presented as novel in Section 3.2, but it appears to be the same formulation as used in most previous CSR work (e.g., CortexODE, CortexFlow), with the only distinction being the use of a recurrent UNet instead of an unrolled UNet. The paper should clarify how this contribution is non-trivially different from previous work and describe any specific innovations.

Author response:
**Response:** We want to address a potential misunderstanding regarding the contribution of our work. Our primary innovation lies in the integration of CSR and spherical mapping into a unified framework. Unlike existing CSR methods, our approach takes advantage of one-to-one vertex correspondence between the spherical template and cortical surfaces, eliminating the need for additional time-consuming spherical mapping and enhancing overall efficiency.

While the ODE-based problem formulation and recurrent deformation learning strategy are not novel when considered individually, we want to highlight that we are the first to apply a recurrent learning strategy for CSR. This decision serves a dual purpose. Firstly, it optimizes parameter usage and GPU memory, resulting in a streamlined design that enables a single model to predict both white and pial surfaces. This contrasts with many existing methods, such as CorticalFlow++ and SurfFlow, which employ distinct models for each surface. Secondly, the recurrent learning strategy allows for the incremental learning of large deformations through multiple smaller deformations. This is crucial for our method, given the challenges posed by using a spherical template, where the initial template is farther from the target surfaces compared to segmentation-based initial surfaces or convex hulls, commonly used in existing deep learning-based methods.

We apologize for any confusion caused by our initial description and have revised our paper to present our contributions more clearly.

Reviewer comment (Experiments):
Baseline performance results reported in the experiments do not match trends seen in previous work. While the proposed method is shown to widely outperform well-established baselines (e.g., Vox2Cortex, CorticalFlow++), prior studies on infant data report much smaller differences (around 0.01 mm) than the large differences (1 mm or more) reported here (e.g., Table 1 in CoTAN, MICCAI’23). This raises questions about how these baselines were implemented. Were their hyperparameters tuned on validation data, and are there supplementary plots supporting this? Additionally, several baseline methods start from an initial template cortical surface, not a sphere as in this paper — were proper initial templates used? If not, published hyperparameters may not transfer and would require significant reworking for fair comparison. The paper should clarify these discrepancies and provide detailed descriptions of baseline implementation and tuning.

Author response:
**Response:** We attribute the observed challenges faced by baseline methods in our study to the unique characteristics of the dataset employed. The infant brain MRI images pose significant difficulties due to poor image contrast and dynamic variations in both contrast and brain volume. These challenges have not been adequately addressed by existing methods, as these methods are typically designed for adult brain MRI images where such issues are either absent or less pronounced.

1. It is thought by the reviewer that it is straightforward to use a spherical template in other baseline methods. However, this is not true for two reasons. Firstly, some methods impose stringent requirements on the initial surface. For instance, CortexODE requires the segmentation of the ribbon as a preliminary step, utilizing the segmentation mask to construct the initial surface. Altering this surface would fundamentally disrupt their pipelines, making it a nontrivial task. Secondly, opting for a spherical template amplifies the deformation required to align with the target surface. Current methods struggle with substantial deformations, hence their reliance on templates closer to the target. The recurrent deformation learning strategy we adopted is designed to tackle this issue, enabling our model to effectively handle such large deformations.

2. While we acknowledge there may still be room for improvement in the performance of competing methods, it's essential to underscore that we have exerted our utmost effort within the constraints of time and resources available to us.

Reviewer comment (Experiments):
The paper evaluates the method on only a single dataset (infant brain surface extraction) despite the proposed approach not being specific to infants. The method could be applied to other age groups and datasets (e.g., HCP, ADNI for adults; dHCP for neonates). To assess generalizability, experiments should be conducted on at least one additional dataset, as is common in the field. If this is not possible, the authors should explain why the method is specific to infants aged 0–8 months.

Author response:
**Response:** Our primary goal is to solve an important but less-explored problem, which is to do CSR from infant brain MRI images. A vast majority of existing methods are developed for adult brain MRI images. Compared to adult brain MRI images, which exhibit good white matter vs. gray matter contrast, infant brain MRI images show much worse and dynamically changing image contrast and thus are significantly more challenging for CSR (see Figure S2 for some examples). In addition, the brain sizes are relatively stable for adult brains. However, the infant brain is fast growing. For example, the brain volume for a 12-month infant is approximately 2-3 times larger than a newborn (i.e., 0-month-old). Therefore, this issue is not well considered by existing methods.

3. As previously stated, our primary objective is to address a significant yet less-explored issue. Consequently, we directed our attention to CSR within what is arguably the most challenging age group—infants aged 0-12 months. It is noteworthy that, although neonatal brains undergo substantial changes in volume similar to infant brains, neonatal images from dHCP do not display the pronounced contrast changes observed in infant brains (refer to Figure 2 in the supplementary material).

Reviewer comment (Novelty):
The paper makes overstretched claims that are not well supported. For example: (a) The introduction (para. 3) claims that graph convolution-based methods such as Hoopes et al. and Bongratz et al. do not generate topologically valid surfaces and require topology correction. This is untrue, as both methods use the same topologically-valid template deformation approach and do not require correction. (b) The paper claims that infant brain myelination (“tissue contrast changes”) makes cortical surface reconstruction more challenging due to poor segmentation. This is only true in the unsupervised setting (e.g., EM-style methods). Since the paper’s segmentor is trained fully supervised, segmentation is not more challenging than in adult brains, as shown by work using the iSeg dataset. These claims should be tempered.

Author response:
**Response:** 

  1) We have adjusted the corresponding section in our paper to moderate this assertion, as recommended. While the authors of graph convolutional network-based works have noted that their methods do not require topology correction, it is important to clarify that these methods cannot ensure the topological correctness of the generated surfaces, even if the initial template surface is topologically correct. This is evidenced by our updated results in Table 1, which shows the self-intersection rates of the predicted surfaces by Vox2Cortex are significantly higher than flow-based methods, including ours.

  2) In contrast to certain existing methods, such as CortexODE, which require the prior segmentation of the ribbon (i.e., subcortical-filled white matter), our approach directly reconstructs cortical surfaces from the image. Consequently, our model faces substantial challenges due to the poor image contrast.

  3) We want to clarify that our method uses the segmentation as an auxiliary task mainly for additional regularization. During inference, the model does not require a segmentation map, thus the MR image quality will have a big impact on the results. In spite of this, our method showed decent performance even when the segmentation module is disabled, as we reported in the ablation study (see Table 2 in our paper).

Reviewer comment (Novelty):
The recurrent framework is not novel, as it has been applied to medical image registration for a long time. Even though it might be the first time introduced into the CSR, it is not novel enough for ICLR.

Author response:
**Response:** We want to emphasize again that we see the overarching concept of integrating CSR and spherical mapping as our main contribution, not the specific strategies, e.g., the recurrent deformation learning.
 
Additionally, we want to mention that although the recurrent learning strategy is not new, we are the first to use it for CSR. Recurrent learning benefits both efficiency and effectiveness, allowing our method to handle both the white and pial surface reconstruction within a unified framework.

Reviewer comment (Novelty):
The proposed framework: template -> white surface -> pial surface, is pretty similar to CortexODE, with the change of using a sphere template and multiple-step surface reconstruction.

Author response:
**Response:** The multi-step strategy, progressing from the template to the white surface and culminating at the pial surface, has its roots in traditional CSR methods, such as FreeSurfer. This sequential approach is widely embraced across all CSR methods, except for those specifically tailored for exclusive white or pial surface reconstruction. Our method adopts this proven strategy in a deep learning framework.

Reviewer comment (Novelty):
The proposed method contains limited innovation and is mainly incremental work. Even though the author wants to emphasize the benefit of spherical mapping, from the CSR point of view, the method is not novel enough for ICLR.

Author response:
**Response:** Our research builds upon established CSR methods, particularly those based on diffeomorphic flows. While we recognize the significant achievements of previous methods in reducing processing time from hours to mere seconds, our work seeks to contribute by addressing challenges that remain unresolved or insufficiently considered in existing approaches.
 
Our primary contribution lies in the integration of CSR and spherical mapping. To achieve this goal, we introduce a recurrent deformation learning strategy to deform a spherical template onto the white and pial surfaces based on images. Unlike existing CSR methods, our approach leverages the one-to-one vertex correspondence between the spherical template and the cortical surfaces, requiring no extra time to achieve spherical mapping.
 
Moreover, our method significantly enhances efficiency compared to all existing methods by eliminating the time-consuming spherical mapping step. However, it is essential to acknowledge that employing an initial template that is more distant from target surfaces, as opposed to using a convex hull or an initial surface based on cortical ribbon segmentation, significantly increases the level of difficulty. To tackle this challenge, we introduced a recurrent learning strategy to allow a large deformation to be learned incrementally with multiple smaller deformations. Notably, using ribbon segmentation to obtain an initial surface for deformation can lead to topological defects and thus requires topology correction, as no segmentation methods are perfect. In contrast, our method circumvents the error-prone segmentation by reconstructing the surfaces directly from images and requires no time-consuming topology correction as the flow-based approach can theoretically guarantee topological correctness.
 
In summary, we believe our method signifies a significant stride forward in the field, serving as a pioneering solution in seamlessly integrating CSR and spherical mapping.

Reviewer comment (Writing):
The details of the proposed mehthod are not clearly.

Author response:
**Response:** We refer the reviewer to Figure S1 in the supplementary material for details on the common network architecture of the white and pial models. Please let us know if more details are needed.

Reviewer comment (Evaluation):
The time consuming should be given.

Author response:
**Response:** The inference time of our method is 1.2 seconds for predicting the white and pial surfaces in the same hemisphere. It is noteworthy that our method saves a substantial amount of time for downstream tasks because no extra time is needed for spherical mapping. CorticalFlow++ and SurfFlow inference time are at the same level (~1 seconds) for CSR. However, they typically require an extra 30 minutes for spherical mapping.

The CortexODE inference process involves multiple steps, including ribbon segmentation, topology correction, SDF computation, initial surface reconstruction via marching cubes, and surface deformation. The total computing time for all key steps is 5.5 seconds, covering the prediction of white and pial surfaces in the same hemisphere. 

The Vox2Cortex inference time is 17 seconds, encompassing all four surfaces in both hemispheres.

Reviewer comment (Evaluation):
The segmentation performance also should be given.

Author response:
**Response:** We reported the segmentation performance and a visual comparison with ground truth of Figure 2 in the supplementary material. The average dice coefficient values of graymatter, white matter, and cerebrospinal fluid are 0.883土0.014, 0.885土0.014 and 0.826土0.028, respectively.

Reviewer comment (Writing):
The details of the proposed mehthod are not clearly.

Author response:
**Response:** We refer the reviewer to Figure S1 in the supplementary material for details on the common network architecture of the white and pial models. Please let us know if more details are needed.

Reviewer comment (Evaluation):
The segmentation performance also should be given.

Author response:
**Response:** We reported the segmentation performance and a visual comparison with ground truth of Figure 2 in the supplementary material. The average dice coefficient values of graymatter, white matter, and cerebrospinal fluid are 0.883土0.014, 0.885土0.014 and 0.826土0.028, respectively.