Paper: 3D Point Cloud Sequences as 2D Videos (ICLR-2024)

Reviewer comment (Novelty):
The concept of this "point geometry video" is not entirely new. In fact, it is very similar to two international patents published in 2022: (a) "Learning-based point cloud compression via tearing transform," with International Publication Number: WO2022232547A1 (available via https://patentimages.storage.googleapis.com/51/19/af/ee3e55ed75618b/WO2022232547A1.pdf), and (b) "Learning-based point cloud compression via unfolding of 3D point clouds" with International Publication Number: WO2022271602A1 (available via https://patentimages.storage.googleapis.com/71/49/ab/6e2ea50f6af4a1/WO2022271602A1.pdf). Idea-wise, this PGV work is especially similar to (b), the "unfolding" work, because the "unfolding" work exactly proposes to view the 3D coordinates of point clouds as pixel values on images, so that they can be processed by downstream 2D pipelines (e.g., see Fig. 3 of the "unfolding" work). Additionally, the core idea to create such images in the PGV work (Fig. 3 in PGV paper) and that of the "unfolding" work is basically the same (Fig. 3 in "unfolding" work) — both intend to "deform" a predefined regular 2D grid to match with the original point cloud. While the detailed designs differ (e.g., architecture differences, overfitting in PGV vs. latent codeword representation in "unfolding" work), and the PGV work adds an extra step for promoting temporal consistency (Fig. 4 of the PGV paper), the idea of the PGV proposal still has many overlaps with that of the "unfolding" work, weakening the novelty of the PGV work.

Author response:
Thanks for pointing out the two patents we were not previously aware of. However, after carefully examining the links provided by the reviewer, we found that both patents propose the method for single point cloud compression, which is not similar to our goal, i.e., structuring the irregular 3D point cloud sequences into regular point geometry video. 
Besides, the concept of representing a 3D geometric model with a 2D RGB image was first demonstrated by the paper 'Geometry Images' in 2002. 
The challenge lies in designing effective deep learning methods to implement this unfolding process, which varies across different papers. It is crucial to emphasize that our method diverges from this concept by not focusing on individual point clouds. Instead, it processes entire 3D point cloud sequences, aiming to maintain both intra-frame smoothness and inter-frame temporal consistency. The **most significant novelty** of our PGV representation lies in being the first to structure irregular 3D point cloud sequences using only their coordinates.

Reviewer comment (Novelty):
The method used in Fig. 3 (FWSS) is highly related to a thread of point cloud decoders based on deforming a pre-defined pattern like a 2D grid. Representative works in this thread include AtlasNet, FoldingNet, PointCapsuleNet, etc., which may be informative to also discuss.

Author response:
First of all, the FWSS module is **technically different** from the mentioned works, AtlasNet, FoldingNet, and PointCapsuleNet, because they encode the input point cloud into a latent codeword, while FWSS directly takes the pre-defined grid and overfits the ground truth point cloud. We acknowledge that the reviewer has already identified this, as mentioned in the comments 'overfitting in PGV vs. latent codeword'. Besides, although the mentioned works can achieve a certain degree of generalization ability, they still suffer from relatively low fidelity between the generated point clouds and the ground truth ones. In our PGV representation, we choose the overfitting manner implemented with a multi-scale structure because the PGV demands **high reconstruction fidelity** to 're-organize' or 'sort' the ground truth point cloud to generate its structured format. 
We have updated the related work section of the manuscript to discuss these works. However, these approaches are confined to **static** geometric representations and thus **unsuitable** for dynamic point cloud sequences, which require spatio-temporal structurization. To the best of our knowledge, our representation is the **first attempt** at structuralizing dynamic point cloud sequences. Transitioning from static point clouds to dynamic sequences introduces additional challenges due to the unstructured nature in both spatial and temporal domains. We address these challenges by proposing the PGV representation, which fills the gaps involved in cross-modal learning between 3D point cloud sequences and 2D videos.

Furthermore, the FWSS module alone does not perform well. As shown in Table 2 of our paper, the absence of the STJS module, i.e., when relying solely on FWSS, significantly degrades the learned correspondence. As visualized in Fig. 11, while the FWSS module can generate 2D color images with spatial smoothness, it lacks temporal consistency (see Fig. 11a), especially when dealing with large inter-frame motion. However, by adding the STJS module, we achieve PGV with both satisfactory spatial smoothness and temporal consistency (see Fig. 11b). More details are provided in Appendix Section A.1.

Reviewer comment (Evaluation):
The overfitting scheme in this work can limit the practical usage of the proposal due to high computational complexity, as the network needs to be retrained for every new point cloud sequence.

Author response:
We have mentioned the issue of computational complexity in the discussion section of our paper (Section 6). Although our method is still limited by the efficiency of the EMD loss, we believe there is potential for improvement with the advancement of loss functions. Taking the popular NeRF method as an example, it initially faced high computational complexity upon its proposal, but subsequent NeRF-based works have improved its efficiency.

From a different perspective, successfully generating PGV itself implies time savings, as the spatio-temporally regularized PGV representation described in our paper lacks a real-world ground truth and is even more challenging to achieve through human annotation. In other words, there are no readily available ground truth sequences that exhibit both spatial smoothness and temporal consistency. Even if it were possible to annotate them, creating such sequences through human annotation would require tedious efforts and consume a significant amount of time. 
 Additionally, the PGV representation can be used in an **offline** manner to **automatically** process irregular 3D point cloud sequences, and the resulting PGVs will be **permanently** stored, on which the conducted downstream tasks can greatly benefit in terms of efficiency, performance, and implementation.

Last but not least, being an early attempt to apply spatio-temporal regularization to 3D point cloud sequences, we argue that solving the fundamental challenge of creating these regularized sequences from scratch is more crucial than addressing efficiency issues.

Reviewer comment (Experiments):
It is unclear how the proposed method behaves if the point clouds to be processed contain more complicated topologies, such as more holes or multiple objects. The 8i sequences used in the experimentation may be somewhat limited due to their simplicity.

Author response:
We add the resulting PGV for a sequences of rotating *torus* (Figure A.10) and *multiple objects* (Figure A.11) in our uploaded revised manuscript. These examples feature either holes or multiple objects. 
In Fig. A.10, it is evident that the generated PGV sequence maintains spatial smoothness within local areas of the PGV frames and temporal consistency among different frames of point clouds, as indicated by the colors on the points for the challenging shape *torus*.
In Fig. A.11, for the generated PGV of the sequence with multiple objects, it is observable that each object is automatically mapped to a patch region on the PGV frames. Moreover, the correspondence of any one object across different frames of the point clouds is still preserved, as demonstrated by the colors on the points of different objects.

As we discuss in **Section 6** of our manuscript regarding representation quality, completely eliminating spatial discontinuity and temporal inconsistency in the generated PGV, especially for highly complex shape topologies, is an extremely challenging task due to its non-trivial nature in parameterization. We believe this can be improved by introducing explicit geometric constraints, such as local area- or shape-preserving regularization in the spatial domain and Laplacian smoothing in the temporal domain. In our experiments, we observed that even with non-perfect smoothness and correspondence, the PGV representation still significantly enhances performance in downstream tasks (refer to results in **Sections 5.2** and **5.3**).

Reviewer comment (Presentation):
I would recommend the authors put the symbols with their corresponding items on the figures. For instance, in section 3.2 we see the $\mathbf{S}^0$ is the pre-defined 3D grid corresponding to the cube in Fig. 4. But what are $\mathbf{S}_f$, $\mathbf{S}_d$, and $\mathbf{S}'$ referring to? Readers would be happy to see them clearly noted in the figure.

Author response:
Thank you for your recommendation.
In our figures, $\mathbf{S}_f$ represents the learned 'Flow Maps' resulting from the 'Conv3D Blocks', as depicted in Fig. 4. $\mathbf{S}_d$ denotes the stack of copies of the first frame $\mathbf{I}_0$ emanating from the FWSS module, positioned to the right of the FWSS module and the pre-defined 3D Grid in Fig. 4. Meanwhile, $\mathbf{S}'$ corresponds to the stack of the remaining frames of the final PGV, located in the black box beneath the summation (`+') operation in Fig. 4.
We acknowledge the importance of clarity in our figures and will incorporate these symbols in Fig. 4 in our revised draft.

Reviewer comment (Theory):
For me, the convolutional manner in the STJS module might not be proper. Since the FWSS results are without temporal consistencies, it's surprising that local convolution kernels are able to handle irrelevant information across different timesteps.

Author response:
We have investigated the effects of FWSS and STJS by visualizing their outputs, as depicted in Fig. 11 of the paper. It can be seen that the coarse PGV, produced solely by FWSS, already shows similar patterns in consecutive frames, indicating the presence of coarse temporal consistency along the time dimension. Although this level of temporal consistency may not be as precise as that achieved with the STJS module, the intra-frame spatial smoothness and the inter-frame temporal consistency within the small time windows can aid in spatial-temporal feature extraction, particularly when the Conv3D blocks slide along both the spatial and temporal dimensions.

Reviewer comment (Reproducibility):
Can the authors explain in detail how the flow maps in the STJS module are learned?

Author response:
As illustrated in Fig. 4, the flow maps in the STJS module are generated by a decoder composed of Conv3D blocks, denoted as $h\_{{\phi}}\^d(\cdot)$. 
This decoder deforms a predefined 3D grid, $\mathbf{S}\^0$, which is concatenated with the encoded residual features from the ResConv3DBlocks encoder, $h\_{{\phi}}\^e(\cdot)$.
The three-layer residual-style encoder processes the remaining frames from the FWSS module and embeds both intra- and inter-frame information into the feature embedding. This is subsequently followed by a decoder with three layers of Conv3D blocks to generate the flow maps.
As detailed in Section 4.2, this process can be formulated as $\mathbf{S}\_f = h\_{{\phi}}\^d(\texttt{CAT}(h_{{\phi}}\^e(\mathbf{S}_r),\mathbf{S}\^0))$, where $\texttt{CAT}(\cdot)$ denotes the concatenation operation, and $\mathbf{S}_r$ represents the remaining frames from the FWSS module.

Reviewer comment (Experiments):
Can the method deal with partial point clouds, incomplete data, or irregular data? Say if we have a point cloud sequence unprojected from a video, we may want to take as inputs partial point clouds due to occlusions, and some parts in one frame might have no correspondence in some other frames.

Author response:
Converting irregular 3D point cloud sequences into regular ones entails an approximation trade-off. As the number of points and frames increases, enhancing spatial and temporal information respectively, our PGV more closely approximates regular data with accurate correspondence. This increased precision results from more sampling points along the time dimension, enabling the network to align better with the actual trajectories. However, fewer samples in partial and incomplete data lead to challenges in achieving temporal consistency close to real trajectories. Despite this, our method still facilitates a structured format to approximate correspondences.

The mixed sequence in Fig. 5(a) (bottom) and the Lidar sequence of Fig. 8 demonstrate our method's performance in irregular scenarios. 
In Fig. 5(a) (bottom), while the basketball in the second frame is absent in others, the basketball and hand areas across different frames display a form of correspondence. 
Such correspondence, though not exactly semantically correct, can still serve to enhance the structuredness of the resulting sequences.
Fig. 8 showcases a scenario where a car-mounted Lidar scans point clouds while moving along a street. The resulting PGV (Fig. 8(b)) and the point cloud sequence format of the PGV (Fig. 8(c)) reveal that, although some points are missing as frames evolve, the most related parts of the point cloud still correspond, as indicated by similar patterns in the PGV.

PGV offers a compact, structured format for partial and incomplete sequences, enhancing feature extraction in downstream tasks. Nonetheless, the accuracy of correspondence improves with more complete and coherent input sequences, as evidenced by the ablation study in Figure 10.

Reviewer comment (Reproducibility):
The paper is a little hard to read, the method is not clarified clearly. How to optimize the network with EMD loss? How to obtain the point cloud from 'I'.

Author response:
**Response**: We refer the reviewer to our **overall response** for a comprehensive description of our PGV framework. Here, we provide a more detailed clarification of the content mentioned in the technical motivation in Section 3.

Generally, we construct a network composed of 2D convolution (Conv2D) $f\_{{\psi}}(\cdot)$, with ${\psi}$ being the network parameters.
The network $f\_{{\psi}}(\cdot)$ takes a pre-defined 2D regular grid $\mathbf{C} \in \mathbb{R}^{H \times W \times 3}$ as input. 
And we optimize $f\_{{\psi}}(\cdot)$ to over-fit each of $\\{\mathbf{P}_t\\}\_{t=0}^{T-1}$ independently as a 2D color image $\mathbf{I}_t = f\_{{\psi}_t}(\mathbf{C})$ by minimizing the EMD loss between $\mathbf{P}_t$ and the point cloud re-organized from $\mathbf{I}_t$, which means treating image pixels as point coordinates to form a point cloud $\mathbf{P}_t' \in \mathbb{R}^{(HW) \times 3}$, i.e., $\mathbf{P}_t'=\\{ \mathbf{I}_t[u,v,:] | 1 \leq u \leq H, 1 \leq v \leq W \\}=\\{f\_{{\psi}_t}(\mathbf{C})[u,v,:] | 1 \leq u \leq H, 1 \leq v \leq W \\}$. Here, $\mathbf{I}_t[u,v,:]$ represents the pixel values at position $(u, v)$ across all three color channels in $\mathbf{I}_t$.
Then, the optimization using EMD loss can be formulated as 
$$
\min\_{{\psi}_t} \sum\_{t=0}^{T-1} \mathcal{L}\_{\text{EMD}}\left(\mathbf{P}_t', \mathbf{P}_t \right)=\min\_{{\psi}_t} \sum\_{t=0}^{T-1} \mathcal{L}\_{\text{EMD}}\left( \\{ f\_{{\psi}_t}(\mathbf{C})[u,v,:]  | 1 \leq u \leq H, 1 \leq v \leq W \\}, \mathbf{P}_t \right)
$$

Reviewer comment (Experiments):
How it will perform when deal with unseen data (o.o.d. data), since the author claimed they adopt a over-fitting manner to perform pc2pixel transforms.

Author response:
**Response**: We refer the reviewer to our **overall response** for a comprehensive description of our PGV framework. As addressed in our response to this reviewer's first question, during the representation stage (as seen in the middle of Fig. 2), the network is optimized with EMD loss.

This optimization, which does not require any human-annotated labels, can operate on any 3D point cloud sequences with dimensions $T\times N\times 3$ and can be considered self-supervised. Therefore, the performance of the PGV representation stage remains unaffected, regardless of whether the data is unseen. We have applied our framework to various input sequences to generate their PGVs, as demonstrated in  Fig. 5(a), Fig.8 (b)-(c), Fig. A.7 and Fig. A.8.

Reviewer comment (Writing):
what's pre-defined 2d grids?

Author response:
**Response**: We have provided the details of the pre-defined 2D grid in the APPENDIX Section A.1.
The pre-defined regular 2D grids are created by regularly sampling coordinates from the first slice of the 3D space $[0,~1]^3$, respectively. For a pre-defined 2D grid $\mathbf{C} \in \mathbb{R}^{H \times W \times 3}$, the values of each pixel coordinate $(u, v)$ are defined as $\mathbf{C}(u,v,1)=\frac{u-1}{H-1}$, $\mathbf{C}(u,v,2)=\frac{v-1}{W-1}$, and $\mathbf{C}(u,v,3) = 0$, where $1\leq u \leq H$ and $1\leq v \leq W$. 

Deforming a pre-defined 2D regular grid into a target 3D point cloud means utilizing the grid's three-channel values as 3D points coordinates, and minimizing the point set similarity between these coordinates and those of the target 3D point cloud. We implement this process as a reconstruction task, optimizing a network to input the grid's coordinates and supervising it by minimizing the Earth Mover's Distance (EMD) loss between the output coordinates and the target point cloud's coordinates.

Reviewer comment (Writing):
what does "deforms a pre-defined 2D regular grid" mean.

Author response:
**Response**: We have provided the details of the pre-defined 2D grid in the APPENDIX Section A.1.
The pre-defined regular 2D grids are created by regularly sampling coordinates from the first slice of the 3D space $[0,~1]^3$, respectively. For a pre-defined 2D grid $\mathbf{C} \in \mathbb{R}^{H \times W \times 3}$, the values of each pixel coordinate $(u, v)$ are defined as $\mathbf{C}(u,v,1)=\frac{u-1}{H-1}$, $\mathbf{C}(u,v,2)=\frac{v-1}{W-1}$, and $\mathbf{C}(u,v,3) = 0$, where $1\leq u \leq H$ and $1\leq v \leq W$. 

Deforming a pre-defined 2D regular grid into a target 3D point cloud means utilizing the grid's three-channel values as 3D points coordinates, and minimizing the point set similarity between these coordinates and those of the target 3D point cloud. We implement this process as a reconstruction task, optimizing a network to input the grid's coordinates and supervising it by minimizing the Earth Mover's Distance (EMD) loss between the output coordinates and the target point cloud's coordinates.

Reviewer comment (Reproducibility):
Need clarification in the rebuttal on the detailed pipeline, such as how to represent the point cloud as image.

Author response:
**Response**: We refer the reviewer to our **overall response** for a comprehensive description of our PGV framework.
To comprehend the transformation between the 3D point cloud and 2D image format, it is important to grasp the fundamental characteristics of these two formats.
RGB images are essentially 5-dimensional signals, denoted as $(r, g, b, u, v)$, where the color information $(r, g, b)$ is indexed by regular grid coordinates $(u, v)$. In contrast, point cloud data is represented by spatial coordinates $(x, y, z)$, which provide both the geometric information and act as indirect cues for determining inter-point relationships, such as proximity and correspondence.
For each frame of PGV, the transformation process between the point cloud and image format is executed by filling the spatial coordinates $(x,y,z)$ to the color channels $(r,g,b)$ at corresponding pixel locations $(u,v)$ on the 2D grid, and vice versa.