Paper: Motion PointNet: Solving Dynamic Capture in Point Cloud Video Human Action (ICLR-2024)

Reviewer comment (Theory):
**Understanding reconstruction objective** The loss function in (15) is not fully make sense. Given the model is trained from scratch, the GT $F_s$ used as supervision is also random features at the beginning, how would the contrastive objective leads the model towards the correct direction as it is the only loss used in pre-training stage?

Author response:
**Response to [Comment 1]:** Thanks for this insightful comment. The contrastive objective is indeed the sole loss during pre-training, but it does not rely on random features. Instead, it utilizes the inherent spatial and temporal coherence within the point cloud video data itself. Our contrastive loss encourages the model to learn a representation that captures the underlying structure of the point cloud sequence, which, while initially random, quickly becomes structured as the model begins to learn the spatial-temporal dynamics inherent in the data. This aligns with the principles of self-supervised learning, where the data provides its own supervision.

Reviewer comment (Experiments):
**Why 2 stage training**. Could we jointly train a classification head along with the contrastive objective? What will the performance like.

Author response:
**Response to [Comment 3 & 4]:** Thanks for these two insightful comments. The two-stage training process is designed to first allow the network to learn robust spatio-temporal features from the encoder and then to refine this representation using the classification head in the second stage. In this way, we can keep our encoder as lightweight as possible while still enforcing its strong learning ability. However, we acknowledge the potential benefits of exploring different training settings and concur with the reviewer's point. We have reported the results of training on the encoder only in Section 4, Tab. 7. We also add the two baseline comparisons below. This will demonstrate the effectiveness of our proposed method beyond the complexity of the architecture and training procedure. The experiments are conducted on the MSR-Action3D Dataset with 24 frames. Due to page limitations in the main text, we will augment the experimental results and discuss more in the supplementary material.

| Training Setting | Accuracy(%) |
|---:|---:|
| pretrain + finetune | 97.52 |
| setting 1 | 95.75 (-1.77) |
| setting 2 | 96.11 (-1.41) |

Here the **pretrain + finetune** is the setting used in our paper, **setting 1** indicates the suggested baseline 1, and **setting 2** indicates the suggested baseline 2.

Reviewer comment (Theory):
The formulation and explanation of the PDEs-solving could be more rigorous mathematically. Some key equations lack details on the formulations.

Author response:
**Response to [Comment 1 & 7]:** We acknowledge the importance of a rigorous mathematical foundation and a better interpretability of the proposed methods. We reorganize Section 3.2 in the revision paper to include a more thorough derivation of the PDEs and the reasoning behind our chosen formulations. Please check the revision paper for more details.

Reviewer comment (Experiments):
The design space of the PDEs-solving module could be explored more thoroughly. For example, how are the basis operators and reconstruction loss function chosen?

Author response:
**Response to [Comment 2]:** We appreciate this insightful comment. 

Firstly, we designed our PDEs-solving module based on the combination of the attention mechanism and the classic spectral methods for PDE. We explain the motivation for this design in our response to **[Reviewer RMAg 's Comment 1]**. Please check for more detail.

Secondly, we use a contrastive-based InfoNCE loss in our pretrain stage to refine the focus of dynamic capture to a distinct objective. By treating the $Feature \in \mathbb{R}^{T\times M}$ (getting from the encoder) before applying spatial/temporal pooling as the negative sample, we force the model to learn the implicit mapping between the spatial and temporal space, instead of reconstruct the de-pooling feature by closer both Ft and Fs to $Feature$ (which against the learning objectives). We have conducted several ablation studies in Section 4.3 (Tab. 8) to underscore the superiority of our loss design. Additional discussions will be included in the Section 3.2 to justify our choices.

Reviewer comment (Experiments):
The comparisons to some recent works like PointMapNet are missing. This could help better demonstrate advantages over other lightweight models.

Author response:
**Response to [Comment 3]:** We addressed this by including a comparison with recent works like PointMapNet, showcasing the advancements our model brings to the field. These comparisons are added to Section 4.2 to strengthen our argument of the proposed method's superiority.

Reviewer comment (Experiments):
The evaluations are limited to action recognition. It remains unclear how the dynamic modeling capability would transfer to other tasks like segmentation or detection.

Author response:
**Response to [Comment 4]:** We agree that evaluating our approach on a diverse set of tasks could demonstrate its generalizability. But, limited time for rebuttal does not allow us to conduct extant experiments to further evaluate its variegation. Therefore, we plan to extend our evaluation to tasks such as segmentation and detection for our future research.

Reviewer comment (Experiments):
There lacks ablation and analysis on different encoder architectures. Can other lightweight encoders also benefit from the PDEs-solving?

Author response:
**Response to [Comment 5]:** Yes, our PDEs-solving module can also improve the performance of other encoders. We have conducted ablation studies the PST-Transformer encoder to demonstrate the broad applicability of our PDEs-solving approach in Section 4.3 (Tab. 7).

Reviewer comment (Evaluation):
The model interpretability is limited. Visualizations or analyses connecting the PDEs-solving to improved dynamics are lacking.

Author response:
**Response to [Comment 1 & 7]:** We acknowledge the importance of a rigorous mathematical foundation and a better interpretability of the proposed methods. We reorganize Section 3.2 in the revision paper to include a more thorough derivation of the PDEs and the reasoning behind our chosen formulations. Please check the revision paper for more details.

Reviewer comment (Experiments):
It is not that clear what the most important part in the PDEs-solving module. To my understanding, it is basically a variant of Transformer. More comparision with vanilla Transformer is encouraged.

Author response:
**Response to [Comment 1]:** The core component in our PDEs-solving module is the spectral method which we detailed in Section 3.2 (PDEs-solving Module, Eq.10-12). We would like to emphasize several points to respond to the reviewer's question more clearly.

1. **Superiority of spectral method:** In implementing our idea, which formulates the dynamic modeling as a PDEs-solving problem, there are several options for the design of our PDEs-solving module. For instance, a pure Transformer method [1]. As discussed in previous work [2, 3], such transformer-based design attempts to learn the operator as a whole to approximate input-output mappings. However, dealing with input-output mappings in high-dimensional space can be challenging for a simplistic deep model to cover adequately while still maintaining network efficiency. To tackle the complex mappings problem, the spectral methods that we use decompose complex nonlinear mappings into multiple basis operators (Eq.10), which also holds the universal approximation capacity with theoretical guarantees.

2. **Ablation results:** Nevertheless, the strong perceptivity inherent in the attention mechanism cannot be overlooked. We design our PDEs-solving module base on the combination of both the Transformer model and the spectral method. We conducted ablation experiments in Section 4.3 (Tab. 8) to make a comparison with the regular Transformer. After removing the spectral method, our PDEs-solving module can be viewed as a pure Transformer model with an encoder-decoder structure. Results in Tab. 8 highlight a substantial decline in performance upon removing the spectral method, indicating its critical importance in our model.

Due to page limitations in the main text, we will augment our discussion in the supplementary material.

Reviewer comment (Writing):
It cloud be better to provide more details of PDEs and explain more the reason to use the PDE method.

Author response:
**Response to [Comment 2]:** We appreciate the reviewer's suggestion for a more comprehensive elaboration on the PDEs. 

Firstly, traditional methods relying solely on global supervision from the action labels lack temporal guidance, potentially leading to an inadequate representation of dynamics in the point cloud video action recognition. To address this limitation, we decouple the spatial-temporal feature and build a PDEs problem by a directive temporal-to-spatial mapping. Together with the contrastive loss (Eq. 15) in the pretrain stage, our PDEs-solving module refines the focus of dynamic capture to a distinct objective. By doing so, we expect to provide temporal guidance. 

Another insight is that PDEs can directly incorporate the feature variations over time as a variable in the modeling process. This process is visualized as Eq. 10-12 in our proposal, where we use multiple basis operators to optimize the complex temporal-to-spatial mapping. This allows a more accurate and nuanced representation of how point cloud features evolve over time compared to previous temporal modeling approaches, which is crucial for recognizing and interpreting human actions accurately.

Due to page limitations in the main text, we will expand on our methodology in the supplementary material to offer a clearer explanation.

Reviewer comment (Experiments):
The PDEs-solving module seems independent of point clouds. I wonder whether the proposed module can be used for traditional video understanding.

Author response:
**Response to [Questions 1]:** We appreciate the reviewer's suggestion regarding the potential use of our PDEs-solving module in traditional video understanding. While theoretically feasible, given that both point cloud videos and traditional videos share spatial-temporal characteristics, adapting this module for structured pixel-based video data poses significant challenges. Unlike point clouds, traditional videos represent data in a highly structured, grid-like pixel format, which poses different demands on data processing and feature extraction. Adapting the PDEs-solving approach, initially tailored for the unstructured nature of point clouds, to this structured format would require significant modifications. These might include redefining the way spatial relationships are perceived and processed within the PDE framework. Additionally, the encoding of temporal information in pixel-based videos is substantially different from point clouds, necessitating a reevaluation of how temporal dynamics are captured and integrated using PDEs in this new context. Given these challenges, exploring the application of our PDEs-solving module to traditional video understanding represents an intriguing and valuable direction for our future research.