Paper: Point2SSM: Learning Morphological Variations of Anatomies from Point Clouds (ICLR-2024)

Reviewer comment (Experiments):
The (compared) models have very many different aspects making a pin-pointing of the decisive difference harder. E.g. why did the authors not compare always the same geometric convolutional backbone / encoder for both correspondence and auto encoder-based solutions?

Author response:
We recognize Point2SSM and the comparison models have many different aspects and it is difficult to differentiate. In the revision, we will make the differences more clear. When comparing against existing methods, we implemented the models exactly as they were proposed for fair comparison. For this reason, we did not alter the backbones of the models. We did include a version of the autoencoder model with the dynamic graph convolution (denoted DG-AE) to aid in understanding the impact of the encoder architecture. Furthermore, we provide an ablation experiment in the supplementary material (Appendix A) to demonstrate the impact of each aspect of Point2SSM. This comparison includes encoder architecture (PointNet vs DGCNN), attention module architecture (self-attention vs MLP), and mapping error loss regularization (weighted by alpha). This illustrates how every design choice works together to provide the best accuracy.

Reviewer comment (Theory):
I wonder whether DPC is the optimal choice for a pairwise/groupwise correspondence model, as it does not consider instance optimisation which was shown to improve point registration nor does it create an unbiased mean for the shape model (but simply selects one reference cloud).

Author response:
Finally, we’d like to address your question about the optimality of DPC for groupwise correspondence. As there is no notion of ground truth correspondence points in these datasets, the creation of an unbiased mean isn’t possible. Instead, we select the point cloud with minimum Chamfer distance to all other point clouds in the training dataset to be the reference. This can be considered the medoid of the dataset. We found experimentally that the choice of the reference does not greatly impact accuracy, but the medoid is the least biased choice. We are unfamiliar with the notion of instance optimization in this context and would appreciate it if you could point us to some references.

Reviewer comment (Novelty):
Although the paper addresses a clinically relevant application, the technical novelty of this paper is limited since it is based on existing architecture and loss without any new insights or analysis. This remains a major weakness and is hard to address during rebuttal.

Author response:
We acknowledge your concern regarding the technical novelty of our paper, particularly in terms of not introducing a novel loss function or network layers. While it's true that our work draws inspiration from various tasks and fields (such as keypoint detection, self-attention, and pairwise point correspondence), we want to emphasize the uniqueness of our contribution lies in the synthesis of these ideas for the specific task of anatomical SSM. Point2SSM is the first deep learning method designed to directly generate 3D anatomical statistical shape models from point clouds in an unsupervised manner.  Although individual components have precedent in previous work, the novel combination and resulting overall architecture, as illustrated in Figure 2, is a distinctive contribution. Our results demonstrate the advantages this innovative approach offers. Our work provides significant value as it enhances the feasibility of SSM generation and broadens its possible applications, potentially accelerating its adoption as a clinical research tool.

Reviewer comment (Experiments):
Since the corresponding promoting loss acts between the minibatch samples, what is the effect of minibatch size on the model performance?

Author response:
Regarding your inquiry about the impact of minibatch size on model performance, we have conducted an ablation experiment on the pancreas data in the revised paper, see Appendix J. This ablation demonstrates that Point2SSM accuracy is not sensitive to batch size.

Reviewer comment (Experiments):
Only three anatomical structures have been studied. The inclusion of more anatomical structures is needed to support the generalizability of the method.

Author response:
While we initially focused on three highly variable anatomical structures due to space constraints, we understand your suggestion for including more structures to convey generalizability. To address this, we have introduced an additional experiment on a vertebrae dataset in Appendix K, featuring a complex bone with thin structures that serves as a challenging test case. This supplementary experiment further supports the improved accuracy and generalizability demonstrated by Point2SSM.

Reviewer comment (Experiments):
The methods describe the objective as reconstructing a sparse set of point reconstruction but, in practice, uses N=M=1024, which raises concern about the initial objective.

Author response:
Finally, regarding your inquiry about sparse reconstruction, we’d like to clarify the objective of Point2SSM is to predict $M$ correspondence points from $N$ unordered, randomly selected input points. The $M$ correspondence points should be well-spread and dense enough to capture the details of the shape. While related work in computer vision has sought to learn sparse keypoints or structure points, sparsity is not a goal in SSM. In our main experiments, we elect to use $N=M=1024$ as this ensures the output points are dense enough to capture the complexity of the organs. We also performed an input sparsity ablation experiment with varying input sizes (from $N=128$ to $N=4096$), and the results are depicted in the "Sparse Input" row of Figure 6. This analysis affirms that Point2SSM outperforms comparison models across different input sizes. In our revision, we will provide additional justification for this choice and clarify any potentially misleading statements about sparse reconstruction.

Reviewer comment (Experiments):
Single Anatomy Modeling: Point2SSM's current limitation to model only one anatomical structure at a time could be expanded upon. How might the method be adapted to accommodate the study of multiple anatomical structures simultaneously, and could this expansion improve the model's understanding of the interrelationships between various anatomical features?

Author response:
Single Anatomy Modeling: Point2SSM is designed to replace traditional SSM construction methods, such as PSM, which model only a single anatomy. For this reason, experiments were performed on a single anatomy, allowing Point2SSM to learn anatomy-specific features. However, Point2SSM is not inherently restricted to a single anatomy.  Our revised paper (see Appendix L) presents an experiment showcasing Point2SSM's performance when trained on multiple anatomies, yielding comparable results to a single anatomy model. In future work, Point2SSM could be modified to utilize anatomy labels in multi-anatomy training to potentially leverage both anatomy-specific and common properties. This would require a more sophisticated approach and is out of the scope of this work.

Reviewer comment (Experiments):
Pre-alignment Requirement: The necessity for pre-aligned input point clouds could restrict Point2SSM's use in scenarios with non-aligned data sources. What methods could be integrated into Point2SSM to automate the alignment process, or to make the model robust to variations in data orientation due to patient movement or diverse scanning protocols?

Author response:
Pre-alignment Requirement: Existing approaches for building SSMs require shapes to be pre-aligned, and there are available tools for doing so, including center of mass and iterative closest point alignment. Using aligned input in Point2SSM allowed for direct comparison with the optimization-based PSM approach. This was important in the evaluation as it allowed us to compare the correspondence accuracy of Point2SSM and the other point cloud deep learning techniques to a traditional optimization-based method. Future work could explore extending Point2SSM to allow for misaligned input. However, this would require performing post-alignment on the predicted correspondence points so that anatomy orientation and location are not captured as modes of variation in the resulting SSM.

Reviewer comment (Evaluation):
Uncertainty Quantification: The lack of uncertainty quantification in the model's outputs is a notable concern for clinical decision-making. Are there potential strategies for integrating uncertainty estimation into Point2SSM to make it more suitable for clinical environments where risk assessment and error margins are vital?

Author response:
Uncertainty Quantification: Point2SSM, like all established methods of its kind, is a deterministic neural network that does not quantify uncertainty. You are correct that this is an important future direction for this work, as it would improve the trustworthiness of output in a clinical scenario. Many existing methods of uncertainty quantification in neural networks could be directly applied to Point2SSM, such as Monte Carlo variational dropout [1], concrete dropout [2], ensembling [3], etc. Progress has been made in uncertainty quantification in supervised deep learning methods for predicting SSM from images [4]. These methods could be extended to Point2SSM; however, it is out of the scope of this work.

Reviewer comment (Experiments):
Performance with Sparse Data: While Point2SSM has demonstrated promising results with limited datasets, its performance with sparse data is not fully explored. Investigating this could provide insights into how the model might be optimized to better capture the variability and complexity of anatomical shapes, leading to more accurate and robust SSMs.

Author response:
Performance with Sparse Data: We have performed an experiment evaluating performance given sparse data, the results of which are in the “Sparse Input” row of Figure 6. Here, we tested the ability of Point2SSM and comparison models to predict 1024 output correspondence points given sets of input points of size 128, 256, 512, 1024, 2048, and 4096. Point2SSM performs the best overall, given all of these input sizes. To aid in understanding this comprehensive evaluation, we’ve added a visualization that showcases examples of sparse input points at each level in Appendix I.

Reviewer comment (Novelty):
Model Expansion for Multiple Anatomies: Enhancing Point2SSM to handle misaligned inputs and model multiple anatomies could greatly increase its clinical relevance. How could the model be developed to not only process various anatomical structures within a single framework but also address the variations and interconnections between them?

Author response:
Model Expansion for Multiple Anatomies: We agree that enhancing Point2SSM to handle misaligned inputs and model multiple anatomies would increase its usability. We have described this as a future direction to explore. Even without these features, Point2SSM greatly enhances the feasibility of SSM generation and broadens its possible applications. The abundant future directions of Point2SSM indicate that this contribution will be of value and interest to the community.

Reviewer comment (Novelty):
The method seems to have significant overlap with the ISR method from Chen et al., 2020. Specifically, it seems to me that replacing PointNet++ with DGCNN + SFA in ISR would almost be the same method. This is not an issue for me, but: if true, I think the inspiration/similarities should be mentioned and Chen et al., 2020 cited in sec. 3.1. If false, I would like the authors to clarify the differences.

Author response:
Overlap with ISR Method: Our method is similar to, and partially inspired by, the ISR method from Chen et al., 2020. However, in addition to the use of DGCNN and SA instead of PointNet++, the overall architecture is different. In ISR, a PointNet++ encoder learns a subset of points, called sample points, and associated features. The MLP-based “Point Integration Module” then learns to transform these sample points to output correspondence points. Where Point2SSM learns a weighted combination of input points, ISR outputs a weighted combination of learned sample points. Additionally, Point2SSM employs the mapping error loss regularization term, whereas ISR utilizes Chamfer distance alone. We acknowledge these methods are similar and appreciate your suggestion to clarify this in Section 3.1. In revision, we will provide a detailed comparison, and acknowledge the similarities.

Reviewer comment (Theory):
It is stated that enforcing the CPs to be a convex combination of the input points "increases surface sampling accuracy". It is not fully clear to me why this must be true. From the attention maps in App. B, Fig. 7, right, CPs could be placed very inaccurately even if they are convex combinations of the attended points. I would like clarification on this point. If it is an empirical observation, I would like it clearly stated (ideally, with an ablation experiment but this is not crucial to me).

Author response:
Convex Combination of Correspondence Points (CPs): The statement that requiring the CPs to be a convex combination of the input points increases surface sampling accuracy could be made more clear. While this factor does not constrain CPs to the surface directly, constraining them to be within the convex hull prevents them from being very far away. Empirically, we see that the methods with this constraint (Point2SSM, ISR, and DPC) perform better on the point accuracy metrics (Figure 3). We will make this statement more clear in the revision.

Reviewer comment (Reproducibility):
It is not clear to me why Point2SSM has such a benefit when it comes to adding new samples. How would this work? Would one "simply" continue training from previous weights, or does the architecture enable something smarter?

Author response:
Benefit of Adding New Samples: In the paper we state: “Point2SSM can also enable sequential or online learning, as well as incremental model updating, making it highly adaptable to real-world clinical scenarios where shape data accumulates over time.” This is an advantage of the deep learning approach in general, as compared to the traditional optimization-based SSM construction methods. We will rephrase this to make sure it does not suggest that Point2SSM has a benefit over other deep learning methods in this regard.

Reviewer comment (Experiments):
Point2SSM has many more parameters than other models (App. E, Table 6.). Is comparing it with the much smaller models fair? Would "upscaling" the other models narrow the performance gap? I would like the authors to clarify why/why not.

Author response:
You raise an interesting question about the model scale. Point2SSM has many more parameters than the other models due to the attention module.  However, the ablation experiment in Appendix A helps demonstrate that the accuracy improvement is not simply a product of the model scale. The third row of Table 1 reports results when the attention module is replaced with a simple MLP, reducing the parameters from 22,098,560 to 2,707,328. The results, while worse than the full Point2SSM model, are still better than the comparison models performance on the pancreas dataset, as reported in Figure 3. This indicates that upscaling the other models would not close the performance gap. We will revise Appendix E to include this discussion.

Reviewer comment (Writing):
Top of page 5: please specify what "Euclidean neighbors" mean. As I can tell, it is the K-nearest neighbors - if so I suggest using that. While Lang et al., 2021 also used "Euclidean neighborhood", I think it would make the text in this paper easier to read with another term.

Author response:
“Euclidean neighbors” at the top of page 5 does mean K-nearest neighbors. Here, the term Euclidean neighbors was used to clarify that the nearest neighbors are found using Euclidean distance rather than geodesic distance, which would be preferred but requires a full mesh representation of shape. We will rephrase this to make it clear that it is K-nearest neighbors computed using Euclidean distance.

Reviewer comment (Presentation):
The text in the figures is very small. Also, blue text on blue background in Fig. 5 is hard to read. Improving this would make the results much easier to interpret.

Author response:
In revision, we will improve the legibility of the text in figures and correct the blue-on-blue text in Figure 5.

Reviewer comment (Presentation):
For the robustness experiments, it would be nice to illustrate how the perturbations look like. This can just be in the supplementary.

Author response:
We agree that visualizations of the input perturbations would be helpful in understanding the robustness evaluation. We have already added this to the revised version, see Appendix I.

Reviewer comment (Experiments):
For the sparse input experiment, how are the points sampled? Random, farthest point sampling or other?

Author response:
In the sparse input experiment, the points are sampled randomly. This task is more challenging than if points were sampled via farthest point sampling. We will clarify this in the revision.

Reviewer comment (Presentation):
When illustrating modes of variation, I find it more clear to deform the mean shape instead of showing deformation vectors. However, this is minor for me and the authors may choose to change it or not.

Author response:
When illustrating modes of variation, we are both deforming the mean shape and showing the deformation vectors to provide the clearest visualization of the differences.

Reviewer comment (Writing):
Section 3.1, first line: second use of "unordered" seems redundant.

Author response:
We agree the second use of "unordered" is redundant in Section 3.1, and we have removed it.