Paper: SITReg: Multi-resolution architecture for symmetric, inverse consistent, and topology preserving image registration using deformation inversion layers (ICLR-2024)

Reviewer comment (Novelty):
The symmetric formulation proposed in section 3.1 seems not entirely new. Indeed, the authors acknowledge this by stating that a similar approach has been used in recent registration methods (Estienne et al., 2021; Young et al., 2022).

Author response:
**The symmetric formulation proposed in section 3.1 seems not entirely new. Indeed, the authors acknowledge this by stating that a similar approach has been used in recent registration methods (Estienne et al., 2021; Young et al., 2022).**

It is true that Estienne et al. (2021) propose a by construct symmetric network. However,  their formulation is clearly different from the ideas in Section 3.1, and does not guarantee inverse or cycle consistency by construct. The method by Young et al. (2022) does not employ any approach close to our symmetric formulation, and we rather referred to it due to its similar multi-resolution feature extraction approach.

Reviewer comment (Novelty):
The multi-resolution formulation proposed in section 3.2 seems rather natural (also in the symmetric setting), so it is unclear how this is different from standard multi-resolution formulations, which are ubiquitous in image registration settings.

Author response:
**What is the key novelty in section 3.2? How is this different from the traditional multi-resolution (except the symmetric formulation)?**

We agree that the principle itself behind the multi-resolution approach is rather standard and we have not claimed otherwise in our article.

The main challenge with the multi-resolution approach was to make it computationally feasible for large volumetric data (as in image registration). For that, we developed the new implicit deformation inversion layer (which can have impact outside this subfield) which uses 5 times less memory than implementing the same idea in the SVF framework. Also, designing the deformation prediction networks $u^{(k)}$ to output deformations invertible by the deformation inversion layer required non-trivial mathematical work of obtaining the optimal invertibility bounds (Appendix E).

The idea of first extracting features independently in multiple resolutions using a ResNet encoder and then using those features to gradually update the deformation is also partially new. While Young et al. (2022) proposed a similar architecture, in our architecture the information flows back from the lower to the higher resolution levels only through the deformations, as opposed to their heavier U-net style approach.

Reviewer comment (Novelty):
What is the key novelty in section 3.2? How is this different from the traditional multi-resolution (except the symmetric formulation)?

Author response:
**What is the key novelty in section 3.2? How is this different from the traditional multi-resolution (except the symmetric formulation)?**

We agree that the principle itself behind the multi-resolution approach is rather standard and we have not claimed otherwise in our article.

The main challenge with the multi-resolution approach was to make it computationally feasible for large volumetric data (as in image registration). For that, we developed the new implicit deformation inversion layer (which can have impact outside this subfield) which uses 5 times less memory than implementing the same idea in the SVF framework. Also, designing the deformation prediction networks $u^{(k)}$ to output deformations invertible by the deformation inversion layer required non-trivial mathematical work of obtaining the optimal invertibility bounds (Appendix E).

The idea of first extracting features independently in multiple resolutions using a ResNet encoder and then using those features to gradually update the deformation is also partially new. While Young et al. (2022) proposed a similar architecture, in our architecture the information flows back from the lower to the higher resolution levels only through the deformations, as opposed to their heavier U-net style approach.

Reviewer comment (Evaluation):
Can you comment on the significance of the improvement of the registration results?

Author response:
**Can you comment on the significance of the improvement of the registration results?**

Please refer to the global response for a detailed clarification on why we believe there is a clear improvement compared to the baselines.

Regarding statistical significance, this is indicated in Tables 1 and 2 with an asterisk, and it was calculated using a permutation test (explained in Appendix K). This obviously does not yet guarantee ultimate clinical significance, but showing that would be clearly beyond the scope of our paper (or a single ICLR paper in more general).

Reviewer comment (Novelty):
The advantage of symmetry and inverse consistency over loss-based approaches is not clear, as all methods seem to provide symmetric and inverse consistent registrations apart from numerical accuracy. Can you comment on this?

Author response:
**The advantage of symmetry and inverse consistency over loss-based approaches is not clear, as all methods seem to provide symmetric and inverse consistent registrations apart from numerical accuracy. Can you comment on this?**

While it may seem that cycle consistency loss is small for all of the methods, the improvement of two magnitudes by our method compared to the baselines is actually visually very significant. This is now well demonstrated in Figure 6 of Appendix H in the updated manuscript.

As for inverse consistency, the performance of SYMNet is indeed very good, and we only claim to achieve similar performance in this metric. That is visually demonstrated in the new Figure 7 of Appendix H.

Reviewer comment (Novelty):
Insufficient study on related work. Unlike what is stated in the paper, the LDDMM method has been used in deep learning, such as DeepFlash[1], NODEO[2], R2Net[3], etc. Also, the inverse consistency by construction using multistep deep registration [4], which is quite close to this paper, is missing in the related work and experimental comparison.

Author response:
**Insufficient study on related work. Unlike what is stated in the paper, the LDDMM method has been used in deep learning, such as DeepFlash[1], NODEO[2], R2Net[3], etc.**

We do not claim in the paper that LDDMM has not been used in deep learning at all but instead that it has not been used much due to its computational cost. The methods you proposed, although interesting, do not qualify as examples of applying the diffeomorphic LDDMM in deep learning in our setup. In more detail:

- DeepFlash does not involve optimization over diffeomorphisms. Instead, it uses initial velocity fields generated by a traditional LDDMM method as the optimization targets. Hence, it is not applicable to our unsupervised setting.
- NODEO is a very interersting method but to mitigate computational cost it has to do significant modifications to the default LDDMM framework and as a result it actually loses the by construct diffeomorphic properties. To achieve invertibility they have to use a separate loss function on the Jacobian determinants (Equation 10).
- R2Net, similarily to NODEO, has to make simplications to the theoretical LDDMM framework. As a result, it does not guarantee diffeomorphisms by construct, but instead applies a loss function on the Jacobian determinants to enforce that (Equation 10).

Reviewer comment (Theory):
Unclear presentation with unexplained statements. Such as, 1) "However, SYMNet does not guarantee symmetricity by construct", why? How to draw this conclusion? 2) Denoting the feature extraction network by h, how is this feature extraction network designed? Do we need to pretrain it or train together with the following network? 3) Squaring and scaling are not enough to guarantee the diffeomorphic property of deformations, we need an additional loss term or strategies to enforce the smoothness of the initial velocity fields first. However, this paper uses the same loss term on the smoothness of deformations in non-diff VoxelMorph, is it a reasonable choice?

Author response:
**1) "However, SYMNet does not guarantee symmetricity by construct", why? How to draw this conclusion?**

Our statement simply reflects the fact that, while its optimization target is symmetric, there is nothing in the SYMNet architecture that would directly enforce the predictions to be symmetric. That is also experimentally shown by the cycle consistency metric results, and in the new Figure 6 (Appendix H). We added a reference to the figure in the text.

Reviewer comment (Experiments):
Insufficient experimental results. This paper should compare the two most related works, i.e., [Iglesias 2023] that is mentioned in the introduction and the above [Greer 2023] that is missing in the paper, and some traditional methods, like SyN in ANTs and Symmetric LDDMM. Also, can you show more qualitative results to demonstrate the improvement of the proposed method and analyze the contribution of each design accordingly?

Author response:
**This paper should compare the two most related works, i.e., [Iglesias 2023] that is mentioned in the introduction and the above [Greer 2023] that is missing in the paper, and some traditional methods, like SyN in ANTs and Symmetric LDDMM.**

Both of these works are very recent, parallel with, and independent of our work. According to the ICLR reviewer guide, an empirical comparison with such methods is not expected. We have nevertheless included our updated paper to include references to these (Iglesias was included already before). SyN, on the other hand, is extremely slow (ANTs implementation ~30 min per one registration), and it has already been compared against our strong baselines with in general worse performance (Balakrishnan et al., 2019; Mok & Chung,
2020a;b). Hence, including it here would be both challenging and not expected to bring additional value.

Reviewer comment (Novelty):
All the strategies used in this paper are what were used before, so, what are the challenges of this work? Is it necessary to have such a complicated network in practice?

Author response:
**All the strategies used in this paper are what were used before, so, what are the challenges of this work? Is it necessary to have such a complicated network in practice?**

Multiple works in the past have tried to enforce one or all of the properties (symmetricity, inverse consistency, topology preservation), either by a loss function or by construct. However, only classical iterative registration algorithms have enforced all of them by construct, see e.g. Ashburner (2007) or Avants et al. (2008). There papers are very highly cited (~8000 and ~5000 citations), and their main point is exactly proposing a method which fulfills the properties by construct. The challenge we set out to tackle was to develop the first such method in the deep learning setting.

To summarize, our novel methodological innovations were the trick to make the method symmetric (Section 3.1) and the new implicit deformation inversion layer (with potential impact outside this subfield), which uses 5 times less memory, making it feasible to apply the multi-resolution architecture for the large volumetric data in image registration. The work also involves non-trivial mathematical work of obtaining the optimal invertibility bounds (Appendix E). We showed how to integrate these innovations with well-established architectural building blocks for a method whose overall performance exceeded that of the existing methods. (To clarify, our innovations could be integrated with simpler networks to ensure the desirable properties, but for sota performance the best available building blocks need to be adopted, hence the "complexity".)

Reviewer comment (Evaluation):
Although the new framework is interesting, as listed in Table 2, the accuracy improvement is not significant.

Author response:
**Although the new framework is interesting, as listed in Table 2, the accuracy improvement is not significant.**

While it is true that the accuracy improvement based on the tissue overlap metrics alone is not very large (although statistically significant) compared to one of the baselines (cLapIRN), the significance lies in the fact that our method achieves the accuracy with significantly more regular deformations. We added a new figure to the appendix to demonstate that visually (Figure 5, Appendix H).

For a more detailed response on our results, which we actually believe showcase a significant improvement compared to the baselines, please see the global response.

Reviewer comment (Presentation):
In Figure 1, the images x_1 and x_2 are not defined previously and are unclear to me.

Author response:
**In Figure 1, the images x_1 and x_2 are not defined previously and are unclear to me.**

Thanks for spotting this typo. We fixed it in the updated version.

Reviewer comment (Presentation):
Cannot find Figure 3.2 in Section 3.2.

Author response:
**Cannot find Figure 3.2 in Section 3.2**

The intended reference was Figure 2 but it was shown incorrectly due to a mistake in latex syntax. It has been fixed in the updated version.

Reviewer comment (Theory):
Is the image registration framework diffeomorphic?

Author response:
**Is the image registration framework diffeomorphic?**

Yes, Theorem 3.3 proves that the architecture is topology preserving (invertible), and smoothness is guaranteed by the B-spline representation (Appendix I).