Paper: Efficient Point Cloud Matching for 3D Geometric Shape Assembly (ICLR-2024)

Reviewer comment (Novelty):
The whole pipeline is developed upon GeoTransformer (CVPR 2022) and uses a majority of the previous design. The differences are the use of PMT to replace the Self- and Cross-Attention mechanisms in the coare level, as well as the use of PMT after each stage in decoder. I think this weakens the contribution and novelty of this paper.

Author response:
We recognize that the presentation of our method might raise questions about the uniqueness of our technical contribution. It's important to clarify that our approach is built upon the coarse-to-fine matching framework, which serves as a flexible and shared platform for efficient matching across diverse domains [1,2,3,4]. Each method tailors its approach to different matching tasks within this shared platform by designing specific matching layers and model-fitting algorithms. In this context, we underscore our contributions as follows:  

Firstly, we introduce Proxy Match Transform, a low-complexity matching layer that effectively refines the matching of the feature pair. Combined with the prevalent coarse-to-fine matching framework, our method outperforms the state-of-the-art methods on the popular object assembly benchmarks while exhibiting order-of-magnitude smaller computational costs.  

Secondly, with its substantial reduction in computational cost, the Proxy Match Transform enables its adoption as a fine-level matcher. To the best of our knowledge, this is the first attempt of its kind, refining matching at a granular level. This approach yields state-of-the-art results on recent 3D object assembly benchmarks, where intricate matching is paramount.  

Lastly, we theoretically analyze that the Proxy Match Transform layer can effectively approximate the high-order convolutional layers and introduce two sufficient conditions that need to hold.
 

**References.**  
[1] Zhou, Qunjie, Torsten Sattler, and Laura Leal-Taixe. "Patch2pix: Epipolar-guided pixel-level correspondences." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.  
[2] Sun, Jiaming, et al. "LoFTR: Detector-free local feature matching with transformers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.  
[3] Yu, Hao, et al. "Cofinet: Reliable coarse-to-fine correspondences for robust pointcloud registration." Advances in Neural Information Processing Systems 34 (2021): 23872-23884.  
[4] Qin, Zheng, et al. "GeoTransformer: Fast and Robust Point Cloud Registration With Geometric Transformer." IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).

Reviewer comment (Presentation):
The methodology part is overly complex, and I do not think it is organized well and easy to follow.

Author response:
We acknowledge our method section was not easy to follow, and the concerns raised about its complexity and organization. In response, we have revised Section 3 and added an overview in the early part of the section.

This revision is primarily focused on improving the clarity of notations and presenting the methods in a more structured and unambiguous manner. Additionally, we have revised the section to avoid some omissions in the equations and clarify some confused notations, ensuring that each step is clearly explained and logically follows from the previous one. The revised section also includes additional explanatory text to guide the reader through the methodology, aiming to make it more accessible and easy to follow.  

Notably, we've adjusted the notations as follows: $\mathbf{P}$ represents the proxy, same as before. The spatial resolution of the proxy is now denoted by $D_{\text{proxy}}$ rather than $P$. Additionally, we've introduced $\mathcal{X}$ as the notation for the source point cloud, and $\mathcal{Y}$ for the target point cloud.  

In the final iteration, we will conduct a comprehensive revision, including any additional comments given during the author-reviewer discussion period.

Reviewer comment (Writing):
In Eq. (2) and the following equations, the calculation of attention matrix $\mathbf{A}$ is unclear.

Author response:
We would like to thank the reviewer for the careful reading and the constructive feedback.
To enhance readability and organizational clarity, we have included the absent descriptions of the attention matrix in this section. Additionally, we have made revisions to the methodology section of the manuscript. Kindly review the updated version of the manuscript.  

We adopt the relative-position encoding strategy of PerViT [1] to compute the attention matrix $\mathbf{A}\_{\mathcal{X}}^{(h)}$.
Specifically, given a query and key positions $\mathbf{q}, \mathbf{k} \in \mathbb{R}^{3}$, the Euclidean distances is defined as follows: $\mathbf{R}\_{\mathbf{q},\mathbf{k}}=||\mathbf{q}-\mathbf{k}||\_{2}$.
An MLP takes this distance to produce an attention score:

$$(\mathbf{A}\_{\mathcal{X}}^{(h)})\_{\mathbf{q},\mathbf{k}} \coloneqq \text{Linear}(\text{ReLU}(\text{Linear}(\mathbf{R}\_{\mathbf{q}, \mathbf{k}};\mathbf{W}\_{\text{p1}}))\mathbf{W}\_{\text{p2}}^{(h)})$$,
where $\mathbf{W}\_{\text{p1}} \in \mathbb{R}^{1\times N\_{\text{h}}}$ and $\mathbf{W}\_{\text{p2}} \in \mathbb{R}^{N\_{\text{h}} \times 1}$ are the linear projection parameters, and ReLU gives non-linearity to the function.

**References.**  

[1] Juhong Min, Yucheng Zhao, Chong Luo, and Minsu Cho. Peripheral vision transformer. Advances in Neural Information Processing Systems (NeurIPS), 2022.

Reviewer comment (Presentation):
It is misleading that the proposed PMT is used to replace the attention mechanisms used in GeoTransformer, while in Fig. 1 it is compared to the convolutions. Also, in many other places convolutions are introduced, but all the computation of PMT seems like attention-based.

Author response:
We apologize for any confusion in our explanation. The relation between convolution and attention is described in the Lemma 1 of the appendix.
To clarify it in the main paper, we added the related theorem in the preliminary of Section 3, saying that multi-head self-attention (MHSA) can express convolution, as theoretically proven by the work of Cordonnier et al. (2020) [1].
The theorem from their study shows that:

**Theorem (Cordonnier et al., 2020).** _A multi-head self-attention layer with_ $N_h$ _heads of dimension_ $D_h$_, output dimension_ $D_{out}$ _and a relative positional encoding of dimension_ $D_p \geq 3$ _can express any convolutional layer of kernel size_ $\sqrt{N_{h}} \times \sqrt{N_{h}}$ _and_ $\min(D_{h},D_{out})$ _output channels._

**High-order Convolution as MHSA.** Inspired by this finding, we express high-order convolution at position $(\mathbf{x}, \mathbf{y})$ in an **attention-based** form:

$$\text{Conv}(\mathbf{F}\_\mathcal{X}, \mathbf{F}\_\mathcal{Y}; \mathbf{K})\_{(\mathbf{x}, \mathbf{y})} \coloneqq \sum\_{(\mathbf{n}, \mathbf{m}) \in \mathcal{N}(\mathbf{x}) \times \mathcal{N}(\mathbf{y})} \mathbf{C}_{(\mathbf{n}, \mathbf{m})} {K}((\mathbf{n}, \mathbf{m}) - (\mathbf{x}, \mathbf{y})) \\\\
= \sum\_{h\in[N\_{h}]}\mathbf{A}\_{((\mathbf{x},\mathbf{y}),:)}^{(h)}\mathbf{C} w^{(h)}$$

where $\mathbf{A}$ is an attention matrix, $\mathbf{C}=\mathbf{F}\_{\mathcal{X}}\mathbf{F}\_{\mathcal{Y}}^\top$ is a correlation, $w^{(h)}$ is a learnable weight scalar, and $K: \mathbb{R}^{6} \xrightarrow{} \mathbb{R}$ is a convolutional kernel. This approach is formally supported by the proof provided in Appendix (Sec. A.1, Lemma 1) of our paper, to which we encourage the reviewer to refer for a detailed understanding.
In this context, Fig. 1 also illustrates that PMT can significantly reduce computational overhead compared to conventional high-dimensional convolution. 

**References.**  
[1] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. In Eighth International Conference on Learning Representations-ICLR 2020, number CONF, 2020.

Reviewer comment (Theory):
In Eq. (2), it seems the output of PMT is the enhanced features, while in Eq. (4), the output is some correlation scores. Do I make a mistake in understanding this?

Author response:
$$\text{PMT}(\mathbf{F}\_{\mathcal{X}}) \coloneqq \sum\_{h \in [{N_h}]} \mathbf{A}\_{\mathcal{X}}^{(h)} \\ \mathbf{F}\_{\mathcal{X}} \\ \mathbf{P}^{(h)\top} {w}\_{\mathcal{X}}^{(h)}.$$

In Eq. 2 (above), $\text{PMT}(\mathbf{F}_{\mathcal{X}})$ is indeed intended to represent the enhanced feature, rather than any form of correlation score. 
The PMT layer computes the output features **based on the correlation** between the input feature and the proxy. 

$$\displaylines{\begin{align} {\text{PMT}(\mathbf{F}\_{\mathcal{X}})^{(h)}}\_{(\mathbf{n}, \mathbf{m})}  &= {\mathbf{A}\_{\mathcal{X}}^{(h)}}\_{(\mathbf{n}, :)}\mathbf{F}\_\mathcal{X} \\ {\mathbf{P}^{(h)\top}} \_{(:,\mathbf{m})}\{w}_{\mathcal{X}}^{(h)} \\\ &= {\mathbf{A}\_{\mathcal{X}}^{(h)}}\_{(\mathbf{n}, :)} {\mathbf{C}\_{\mathcal{X}}^{(h)}}\_{(:, \mathbf{m})} {w}\_{\mathcal{X}}^{(h)} \\\ &= \sum\_{\mathbf{n}' \in \mathcal{N}(\mathbf{n})} {\mathbf{A}\_{\mathcal{X}}^{(h)}}\_{(\mathbf{n}, \mathbf{n}')} {\mathbf{C}\_{\mathcal{X}}^{(h)}}\_{(\mathbf{n}', \mathbf{m})} {w}\_{\mathcal{X}}^{(h)}.\end{align}}$$

The above equation specifies the output value at head $h$ for a given input feature $\mathbf{F}\_{\mathcal{X}}$ at position $(\mathbf{n},\mathbf{m})$. 
It is critical to understand that this does not represent correlation scores, but rather the computed enhanced feature after the application of the attention matrix $\mathbf{A}^{(h)}$ and learnable weight $w^{(h)}$. 

The term $\mathbf{C}_{\mathcal{X}}^{(h)} \coloneqq \mathbf{F}\_{\mathcal{X}} \mathbf{P}^{(h)\top}$ is introduced in this equation as an intermediate computation within each PMT's head. This term does not constitute the final output of PMT; rather, it represents the correlation between the input feature $\mathbf{F}\_{\mathcal{X}}$ and the proxy $\mathbf{P}^{(h)}$.
This intermediate correlation is further processed by PMT's mechanisms to yield the final, enhanced feature representation.

Reviewer comment (Experiments):
The experiments are only conducted on synthetic dataset, which makes me doubt its value in real applications. Therefore, it is better to include some real data. If there is no real data in this task, as this method is strongly based on GeoTransformer, simply running on GeoTransformer's benchmark also makes sense.

Author response:
We have extended our experimental evaluation to include the Fantastic Breaks dataset (CVPR 2023) [1], which consists of real data samples for shape re-assembling. We acknowledge that this dataset is relatively small, containing only 150 samples. Still, we believe it provides a valuable preliminary indication of our model since it is trained on synthetic data, but it performs in real-world scenarios.  

**Table 1: Experimental results on real-world dataset, `everyday` $\rightarrow$ Fanstastic Breaks [1]**
|Method|CRD $\downarrow$|CD $\downarrow$|RMSE (R) $\downarrow$|RMSE (T) $\downarrow$| 
|:----|:---:|:---:|:---:|:---:|
||($10^{-2}$)|($10^{-3}$)|($^{\circ}$)|($10^{-2}$)|
|Global|26.41|16.37|88.92|22.99|
|LSTM|26.48|18.53| 85.26|25.00|
|DGL|26.92|15.22|86.66|22.76|
|NSM|25.05|18.62|81.88|22.54|
|GeoTransformer|7.79|6.54|3.79|**10.17**|
|**PMT (Ours)**|**7.30**|**6.52**|**39.38**|11.35|

**Table 2: Experimental results on real-world dataset, `artifact` $\rightarrow$ Fanstastic Breaks [1]**
|Method|CRD $\downarrow$|CD $\downarrow$|RMSE (R) $\downarrow$|RMSE (T) $\downarrow$| 
|:----|:---:|:---:|:---:|:---:|
||($10^{-2}$)|($10^{-3}$)|($^{\circ}$)|($10^{-2}$)|
|Global|26.50|17.23| 87.97|24.19|
|LSTM|25.85|18.25|85.18|23.29|
|DGL|26.23|16.98|87.96|23.58|
|NSM|26.09|17.28|86.69|23.36|
|GeoTransformer|13.69|15.14|70.08|20.96|
|**PMT (Ours)**|**12.24**|**13.32**|**66.88**|**19.03**|

Furthermore, to emphasize the generalizability of our approach, we have conducted transferability experiments between  BreakingBad `everyday`, and `artifact` subsets. The results of these experiments confirm that PMT retains its efficacy when applied to data distributions that differ from the training set, indicating robust transfer learning capabilities.  

**Table 3: Transferability experimental results on BreakingBad `everyday` $\rightarrow$ `artifact`**

|Method|CRD$\downarrow$|CD$\downarrow$|RMSE(R)$\downarrow$|RMSE(T)$\downarrow$|
|:----|:---:|:---:|:---:|:---:|
||($10^{-2}$)|($10^{-3}$)|($^{\circ}$)|($10^{-2}$)|
|Global|24.81|11.82|86.55|27.50|
|LSTM|24.81|12.44|84.21|27.53|
|DGL|25.02|11.69|86.69|27.87|
|NSM|24.76|11.33|84.60|26.24|
|GeoTransformer|**3.78**|3.05|61.35|**15.41**|
|**PMT(Ours)**|3.96|**2.97**|**59.02**|16.31|

**Table 4: Transferability experimental results on BreakingBad `artifact` $\rightarrow$ `everyday`**

|Method|CRD$\downarrow$|CD$\downarrow$|RMSE(R)$\downarrow$|RMSE(T)$\downarrow$|
|:----|:---:|:---:|:---:|:---:|
||($10^{-2}$)|($10^{-3}$)|($^{\circ}$)|($10^{-2}$)|
|Global|27.34|15.24|85.67|29.09|
|LSTM|26.84|14.90|84.81|28.87|
|DGL|26.85|14.40|86.22|29.00|
|NSM|25.68|14.58|85.68|27.55|
|GeoTransformer|4.38|3.61|61.95|**14.95**|
|**PMT (Ours)**|**4.20**|**3.33**|**61.25**|16.06|

**References.**  
[1] Nikolas Lamb, Cameron Palmer, Benjamin Molloy, Sean Banerjee, and Natasha Kholgade Banerjee. Fantastic breaks: A dataset of paired 3d scans of real-world broken objects and their complete counterparts. In Proc. IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2023.

Reviewer comment (Presentation):
The methodology part should be re-organized and the symbols simplified. Fig. 1 does not help understand the main contributions.

Author response:
We appreciate pointing this out again. In response to the concerns, we have revised the method section in our manuscript to improve its readability. We have also made efforts to simplify the symbols and re-organize the content to make it more accessible to readers. Please refer to our revised manuscript for the changes we have implemented so far, and we will continue to work on further improvements to ensure that the final iteration of our paper is as clear and comprehensible as possible. If you have any specific suggestions for this, we will do our best to reflect them as well.

Reviewer comment (Writing):
The paper is written in a way that is rather difficult to follow -- I would suggest doing another pass (maybe with feedback from some external readers) to make the language more simple and streamlined.

Author response:
We acknowledge our method section was not easy to follow, and the concerns raised about its complexity and organization. In response, we have revised Section 3 and added an overview in the early part of the section.

This revision is primarily focused on improving the clarity of notations and presenting the methods in a more structured and unambiguous manner. Additionally, we have revised the section to avoid some omissions in the equations and clarify some confused notations, ensuring that each step is clearly explained and logically follows from the previous one. The revised section also includes additional explanatory text to guide the reader through the methodology, aiming to make it more accessible and easy to follow.  

Notably, we've adjusted the notations as follows: $\mathbf{P}$ represents the proxy, same as before. The spatial resolution of the proxy is now denoted by $D_{\text{proxy}}$ rather than $P$. Additionally, we've introduced $\mathcal{X}$ as the notation for the source point cloud, and $\mathcal{Y}$ for the target point cloud.  

In the final iteration, we will conduct a comprehensive revision, including any additional comments given during the author-reviewer discussion period.

Reviewer comment (Novelty):
I am also not entirely sure of the magnitude of the contribution. The proxy match transform is a clever trick that significantly enhances efficiency and accuracy in real-world training scenarios. But it sits inside a large pipeline that draws heavily upon previous work, and it is difficult to gauge its conceptual contribution. Is it a small tweak to make a big system better, or something so critical its impact goes beyond said big system?

Author response:
We recognize that the presentation of our method might raise questions about the uniqueness of our technical contribution. It's important to clarify that our approach is built upon the coarse-to-fine matching framework, which serves as a flexible and shared platform for efficient matching across diverse domains [1,2,3,4]. Each method tailors its approach to different matching tasks within this shared platform by designing specific matching layers and model-fitting algorithms. In this context, we underscore our contributions as follows:  

Firstly, we introduce Proxy Match Transform, a low-complexity matching layer that effectively refines the matching of the feature pair. Combined with the prevalent coarse-to-fine matching framework, our method outperforms the state-of-the-art methods on the popular object assembly benchmarks while exhibiting order-of-magnitude smaller computational costs.  

Secondly, with its substantial reduction in computational cost, the Proxy Match Transform enables its adoption as a fine-level matcher. To the best of our knowledge, this is the first attempt of its kind, refining matching at a granular level. This approach yields state-of-the-art results on recent 3D object assembly benchmarks, where intricate matching is paramount.  

Lastly, we theoretically analyze that the Proxy Match Transform layer can effectively approximate the high-order convolutional layers and introduce two sufficient conditions that need to hold.  


**References.**  
[1] Zhou, Qunjie, Torsten Sattler, and Laura Leal-Taixe. "Patch2pix: Epipolar-guided pixel-level correspondences." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.  
[2] Sun, Jiaming, et al. "LoFTR: Detector-free local feature matching with transformers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.  
[3] Yu, Hao, et al. "Cofinet: Reliable coarse-to-fine correspondences for robust pointcloud registration." Advances in Neural Information Processing Systems 34 (2021): 23872-23884.  
[4] Qin, Zheng, et al. "GeoTransformer: Fast and Robust Point Cloud Registration With Geometric Transformer." IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).

Reviewer comment (Writing):
Since "shape assembly" is more commonly used to refer to assembling shapes from parts (e.g. a chair from seat, back and legs), it might be clearer to use "shape re-assembly" instead, or even "fractured shape re-assembly".

Author response:
Thank you for your suggestion to use "shape re-assembly" or "fractured shape re-assembly" in place of "shape assembly." We understand the reasoning behind this suggestion.  

However, our decision to use the term "geometric shape assembly" was guided by the precedent set in pioneering research by Sell ́an et al. (2022) [1]. In their study, which introduced BreakingBad dataset simulating physically broken objects resulting from external forces, the term "geometric shape assembly" was specifically chosen to describe this area of study.  

By aligning with the terminology used in Sell ́an et al. (2022) [1], we aim to maintain consistency within the field and ensure clarity for readers who are familiar with this line of research.   

We hope this explanation clarifies our choice of terminology and its alignment with established research in this area.

**References.**  
[1] Silvia Sell ́an, Yun-Chun Chen, Ziyi Wu, Animesh Garg, and Alec Jacobson. Breaking bad: A dataset for geometric fracture and reassembly. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2022.

Reviewer comment (Writing):
"... two sets of features, $\mathcal{F}_P$ and $\mathcal{F}_Q$, associated with each point cloud" --> this reads as: each point cloud has two sets of features. You might want to rephrase as "... two sets of features $\mathcal{F}_P$ and $\mathcal{F}_Q$ associated with the two point clouds respectively" or something like that.

Author response:
Appreciate the constructive feedback. We've refined the wording to eliminate ambiguity. Kindly consult the updated version of our manuscript for reference.

Reviewer comment (Writing):
Please don't use $P$, $\mathbf{P}$ and $\mathcal{P}$ to denote totally different things (near Eq. 3). It's super-confusing.

Author response:
Thanks for the suggestions. In response, we've revised the notations within the equations. Notably, we've adjusted the notations as follows: $\mathbf{P}$ represents the proxy, same as before. The spatial resolution of the proxy is now denoted by $D_{proxy}$ rather than $P$. Additionally, we've introduced $\mathcal{X}$ as the notation for the source point cloud (not using $\mathcal{P}$), and $\mathcal{Y}$ for the target point cloud.

Reviewer comment (Theory):
Limited theoretical insight: the paper feels a bit ad hoc, in the sense of "I have done this and it works." It is not clear how the architecture was derived.

Author response:
**[W1&2]. Lack of motivation & theoretical insight**

**Answer:**

**Motivation and derivation:** 
As outlined in our paper's introduction, high-order feature transform methods have been known to excel in addressing matching problems by capturing structural patterns of correlations in high-dimensional spaces. It is more critical for geometric shape assembly, which requires a meticulous search for pairwise correlations to capture the fine local details of each shape fragment. Despite their efficacy, these methods encounter challenges due to quadratic complexity, limiting their applicability to coarse-grained matching with a restricted number of points. In response to this constraint and with the goal of leveraging a computationally efficient high-order feature transform, we introduce a novel layer termed Proxy Match Transform. We demonstrate its effectiveness and efficiency in the context of shape assembly tasks.

**Role of proxy tensor $\mathbf{P}$:**
As discussed in Section 3.1 of our paper, the proxy plays a crucial role in achieving the previously mentioned goal of designing an efficient high-order feature transform layer. Essentially, the Proxy Match Transform employs the proxy to reduce the quadratic computational cost of high-order convolution to a sub-quadratic level while effectively expressing the high-order convolution. We have dedicated efforts to analyze the role of the proxy both theoretically and empirically; In Section 3.1, we have discussed the derivation demonstrating how the Proxy Match Transform can express high-order convolution with only sub-quadratic complexity. In Section 3.2, we established sufficient conditions for its approximation. In Section A.1, we have put the extended theoretical analysis of the Proxy Match Transform. In Section A.2, we added the empirical analysis on the proxy and showed how the proxy helps in facilitating critical information exchange between two input point clouds.

**Role of weight $w$:**
Weight value w is a learnable parameter in Proxy Match Transform layer, serving to accumulate the computation results for each head of the Proxy Match Transform with an appropriate weight.