Paper: DeepSPF: Spherical SO(3)-Equivariant Patches for Scan-to-CAD Estimation (ICLR-2024)

Reviewer comment (Presentation):
SO(3)-equivariance is at the heart of the paper, as the title, Abstract, and Introduction imply. However, the method section is missing the crucial part of the equivariance discussion, which is instead placed in the Appendix.

Author response:
**2) SO(3)-equivariance - Paper structure**

We thank the reviewer for suggestions on the structure of the paper.
We hoped it would improve the readability of the overall methodology, but as the reviewer correctly noticed, we have to add this to the main part of the work. 
We have changed the structure and replaced part of the evaluation (qualitative results of point cloud completion) with the SO(3) equivariance analysis.

Reviewer comment (Theory):
After equation 7 in Section 3: Since we are in 3D, why is the sphere \( \mathcal{S}^3 \) and not \( \mathcal{S}^2 \)?

Author response:
**4) Q1 - Typo**

This is a typo, and we have now fixed both to $\boldsymbol{\nu}\in \mathcal{S}^2$.
We apologize for the simple mistake and thank the reviewer for noticing it.

Reviewer comment (Writing):
Section 3.2: Which “previous work” is meant in the second line in the first paragraph and the first line of page 5?

Author response:
**5) Q2 - Unclearity with previous work**

We acknowledge that this might have been unclear and needs to be explicitly addressed this point in the manuscript.
The prior work we refer to [2].

Reviewer comment (Theory):
Is the proposed method SPF also equivariant to reflections? I.e., is the entire O(3) group covered?

Author response:
**6) Q3 - Equivariance to Reflections - O(3) Group**

In the context of scan-to-CAD, point cloud registration, and retrieval, we typically steer clear of the O(3) group. 
This avoidance is primarily because the ground truth labels do not represent reflections.

We evaluated our method and Vector Neurons [3] on the O(3)-group and found that both of them are actually O(3)-equivariant.
This can be explained by the fact that reflections are essentially rotation matrices, and therefore, the theories that demonstrate equivariance are the same.
However, due to the ground truth labels mentioned above, we deliberately restrict the estimation process for the scan-to-CAD task and, therefore, do not consider reflections.
The restriction is made in the singular value decomposition. 
For more information, we refer to the work of Umeyama [4] or the supplementary material of DeepGMR [5].

Reviewer comment (Experiments):
Can the proposed method generalize to dimensions higher than 3?

Author response:
**7) Q4 - Generalization to higher dimensions**

We assume that this is the case, but the benefits would probably be reduced as the SPF essentially depends on E$(\boldsymbol{\nu}, \boldsymbol{p})$, and if we consider higher dimensions, this distance graph needs to be adjusted to work sufficiently.

Reviewer comment (Reproducibility):
It would be useful if the authors summarized all the learnable parameters of SPF.

Author response:
**9) Q6 - Learnable parameters**

The learnable parameters of SPF are:
- From E$(\boldsymbol{\nu}, \boldsymbol{p})$ we have $\theta$ and $\phi$.
- From $\text{U}(P(\boldsymbol{\nu}, \boldsymbol{p}))$ we have the fully connected layer $f_\psi$.
- From $\text{V}(r)$ we have the radius $r$.

Reviewer comment (Evaluation):
What is the computational complexity of the DeepSPF compared to the baselines? Could the authors provide a speed comparison?

Author response:
**11) Q8 - Could the authors provide a speed comparison?**

As some of our models run on different servers with varying loads (utilizing the same GPU but different CPUs), we are currently consolidating all computations onto a single independent device. This is to ensure a fair speed comparison across the board.
We aim to update the manuscript by the end of this week.

Reviewer comment (Writing):
Equation 4: Should \( l \) be \( i \) in the {} under the max?

Author response:
**12) Minor Issues**

We thank the reviewer for finding all minor issues. We have addressed them all in the manuscript.

Reviewer comment (Writing):
Punctuation between the equations is missing, e.g., 9 and 10.

Author response:
**12) Minor Issues**

We thank the reviewer for finding all minor issues. We have addressed them all in the manuscript.

Reviewer comment (Presentation):
Equation 12 is a part of equation 11.

Author response:
**12) Minor Issues**

We thank the reviewer for finding all minor issues. We have addressed them all in the manuscript.

Reviewer comment (Experiments):
More recent related works should be compared, such as Section 4.4, which does not compare with the latest rotation-invariant method: Rotation-invariant transformer for point cloud matching [1]. Section 4.6 does not compare the state-of-the-art point cloud completion methods: Diverse point cloud completion with Geometry-Aware Transformers [2].

Author response:
**1) Related Work - Rotation-Invariant Methods [1]**

As suggested by the reviewer, we have appended the evaluation of [1] for point cloud registration on ModelNet40 using zero-intersection noise.
We show the final results here:

| Model                       | RRMSE | RMSE(t)   |
|-----------------------------|-------|-----------|
| Hao et al. [1]              | 81.14 | 0.1353    |
| Ours (*E*)                  | 7.45  | 0.0031    |
| Ours (*E*+*U*)              | 6.49  | 0.0029    |
| Ours (*E*+*U*+*V*)          | **5.17** | **0.0022** |

To obtain rotation and translation values from [1], we followed the official code to train and obtain features and correspondences.
We then used the evaluation script of [1] to subsequently obtain the rotation matrix and translation vector.
Noticeably, the result is significantly worse. 
Thus, using [1] on 3D instance point clouds with no point-to-point correspondences leads to a large reduction in alignment quality.


**2) Related Work - Point Cloud Completion [2]**

Our main contribution is the proposed learning of SO(3)-equivariant representation by considering local and global structures. 
We use the example of point cloud completion to evaluate between one non-equivariant encoder (PointNet [3]) and one SO(3)-equivariant encoder (VN-based PointNet [4]).
As such, we wanted to have a structure that is as simple as possible (such as PCN [5]) so that we and the reader can have an easier time understanding the differences/improvements that are possible from our encoder.

[2] considered a more extensive pipeline for the point cloud completion process with a high number of processing steps.
In contrast, we proposed an encoder that can be used in many different architectures to improve the point cloud completion pipeline.
As such, we decided not to compare with [2].
To illustrate this, we have added [2] as an outlook for future work that could be improved by a comprehensive evaluation of the joint combination of our method with [2].
If the reviewer insists, we can add the evaluation.

Reviewer comment (Evaluation):
In Table 3, the proposed method does not perform well in some classes and reasons should be analyzed and discussed.

Author response:
**3) Table 3 - Performance of certain classes**

We thank the reviewer for the valuable criticism and correct assessment that this experiment needs more analysis. 
We conducted two additional studies to enhance the analysis of this experiment, and the details have been included in the appendix.
Due to the character limit in the response, we refer to the appendix for the explanation.

Reviewer comment (Writing):
Some sentences are not clear. For example, the red arrow in Figure 1 is ambiguous, and the lower-right circle in the SPF part lacks an arrow. In Section 5, typo: ``resizeable'' ⇒ ``resizable''.

Author response:
**4) Unclear Figure 1 and Typos**

We thank the reviewer for the comment. As multiple reviewers mentioned the issue with Figure 1. we have improved the structure and hopefully reduced ambiguity.
We also thank the reviewer for finding the typo.

Reviewer comment (Writing):
The main weaknesses of the paper are related to its exposition and readability qualities, with a need to reduce the level of technical detail in the main text and aim for a clearer presentation of the key concepts.

Author response:
We thank the reviewer for the open criticism of our manuscript.
We understand that we included a lot of equations and technical details.
We simplified and improved the readability of the main section.
We hope this extended rebuttal has clarified our intention to enhance our work, and we would appreciate any suggestions or improvements you may have.

Reviewer comment (Writing):
The statement “conventional SO(3)-equivariant methods do not sufficiently investigate the correlation between global and local structures” is unclear; it is not specified whether this is specifically true for equivariant methods or for SO(3) symmetries.

Author response:
Thank you for your feedback. We have rephrased the sentence to provide a more precise explanation of the issue:
"These methods often treat the issues of local and global SO(3)-equivariance separately, leading to an insufficient investigation of SO(3)-equivariance between global and local structures."
These methods refer to the SO(3)-equivariant methods we name and cite directly beforehand. 
Following Vector Neurons [1], we avoid a more extensive explanation of how SO(3) is defined.
We would gladly provide a more detailed explanation in the appendix if needed.

Reviewer comment (Presentation):
Figure 1 is unclear, and it is not evident how the caption relates to elements in the figure itself.

Author response:
We improved the figure and caption and hope the caption now relates better to the figure itself.
Additional details regarding these changes can be found in the general response section of our reply, as another reviewer highlighted concerns related to this figure.

Reviewer comment (Theory):
There is no clear analysis provided (proposition, theorem, etc.) that supports the claim that the proposed network is equivariant.

Author response:
We realize that having the proposition and theorem in the appendix might not have been the correct decision.
As such, we have replaced parts of the evaluation with the analysis of equivariance to improve the manuscript.