TPU-GAN: L EARNING TEMPORAL COHERENCE FROM DYNAMIC POINT CLOUD SEQUENCES
Anonymous authors
Paper under double-blind review
A BSTRACT
Point cloud sequence is an important data representation that provides ﬂexible shape and motion information.
Prior work demonstrates that incorporating scene ﬂow information into loss can make model learn temporally coherent feature spaces.
However, it is prohibitively expensive to acquire point correspondence information across frames in real-world environments.
In this work, we propose a super-resolution generative adversarial network (GAN) for dynamic point cloud sequences without requiring point correspondence annotation.
Our model, Temporal
Point cloud Upsampling GAN (TPU-GAN), can implicitly learn the underlying temporal coherence from point cloud sequence, which in turn guides the generator to produce temporally coherent output.
In addition, we propose a learnable masking module to adapt upsampling ratio according to the point distribution.
We conduct extensive experiments on point cloud sequences from two different domains: particles in the ﬂuid dynamical system and human action scanned data.
The quantitative and qualitative evaluation demonstrates the effectiveness of our method on upsampling task as well as learning temporal coherence from irregular point cloud sequences.
I NTRODUCTION
Temporally evolving point clouds are ubiquitous in various ﬁelds, e.g. particles in the N -body system, or real-world moving objects captured by LiDAR sensor or RGB-D camera.
Understanding the spatiotemporal pattern lies in these sequential data are of great importance to engineering and scientiﬁc disciplines.
However, high-resolution point cloud sequences can be very expensive to obtain.
In practice, high-resolution physics simulation demands substantial computational resources and 3D scanned data are often sparse and noisy due to hardware constraints.
Impressive progress has been made by applying learning-based algorithms (Yu et al., 2018b; Li et al., 2019;
Wu et al., 2019; Yifan et al., 2019; Qian et al., 2021) to reconstruct a dense and clean point cloud given the observed sparse ones.
While these works provide powerful framework for upsampling static point clouds, the change of points with respect to time is not taken into consideration.
Prantl et al.
(2020) demonstrates that supervising the scene ﬂow (displacement of points) of generated points during the training process can greatly stabilize the learned latent space for point cloud super-resolution task.
Yet this requires knowing the point correspondence across the frames to derive the scene ﬂow , which is expensive and sometimes impossible to acquire in real-world scenarios.
For 3D scanned data, the point clouds in two frames do not necessarily have correspondences and the number of points may change.
In this case, generative adversarial networks (GANs) (Goodfellow et al., 2014) provides an attractive alternative for learning useful generative and discriminative representation without label and heuristic loss function (e.g. L2 loss that penalizes the scene ﬂow difference).
Under the context of point cloud sequence’s super-resolution, we can use the discriminator to learn both the spatial and temporal patterns from data implicitly.
With these learned spatiotemporal discriminative representation, we can further use them to guide the generator to produce output that are spatially and temporally coherent.
More formally, in this work we present a GAN framework for super-resolution on the dynamic point cloud sequences.
By experimenting on point cloud sequences from two very different domains - ﬂuid particles and 3D scanned human action, we show that the proposed upsampling model can produce output that are spatially and temporally consistent with target data without explicit supervision on the point displacements across frames.
We also show that the discriminative representation learned by temporal discriminator can be used for classiﬁcation task and provides competitive performance.
Furthermore, we propose a learnable masking module to adaptively upsample points that have a irregular density distribution.
This allows model to produce non-uniform point distribution that matches the target domain.
R ELATED WORKS
Deep learning on static point clouds The pioneering work PointNet (Qi et al., 2017a) carries the success of deep neural networks on 2D structured grid data to the ﬁeld of unstructured 3D point cloud processing.
PointNet proposes a neural network architecture with multi-layer perceptron (MLP) shared acrossed points and aggregation mechanism that respects permutation invariance in the input.
Following PointNet, a wide array of works had contributed to improve and extend the neural representation learning paradigm on point cloud data (Qi et al., 2017b;
Li et al., 2018; Wang et al., 2019b; Wu et al., 2020; Zhao et al., 2019;
Shen et al., 2018)
These methods have shown to learn good representation for static point cloud analysis task such as classiﬁcation, semantic segmentation and object detection.
Learning-based point cloud upsampling With the success in learning-based image superresolution (Dong et al., 2015; Ledig et al., 2017), a number of works transfer the learning-based super-resolution concept to the 3D point cloud.
PU-Net (Yu et al., 2018b) proposes the ﬁrst learningbased method that upsamples point cloud features with multi-level expansion units.
Yu et al.
(2018a) extends and improves PU-Net by adding an edge-aware loss to consolidate the upsampled point clouds.
Yifan et al.
(2019) proposes a progressive point cloud upsampling network which supervises the upsampled point clouds at different levels.
Li et al.
; Wu et al.
(2019) leverage GANs to learn high-quality upsampling result, which demonstrate adversarial training can beneﬁt point cloud upsampling task.
Very recent work Qian et al.
(2021) proposes Inception DenseGCN and NodeShufﬂe, which signiﬁcantly improves the upsampling quality.
Another concurrent work PUGeo-Net (Qian et al., 2020) proposes a geometric-centric deep learning framework for learning the local fundamental forms.
Spatiotemporal representation learning on point clouds Different deep learning methodologies for learning discriminative spatiotemporal representation for video classiﬁcation have been proposed (Ng et al., 2015; Donahue et al., 2016; Feichtenhofer et al., 2016; Carreira & Zisserman, 2018; Tran et al., 2015), and been further explored on the generative tasks including video generation (Tulyakov et al., 2017; Clark et al., 2019;
Saito et al., 2020) and super-resolution (Xie et al., 2018; Wang et al., 2019a; Haris et al., 2019; Chu et al., 2020).
Following previous approaches that treat video data as volumetric data, Choy et al.
; Luo et al.
(2020) transform point cloud sequences into volumetric voxel-based representation and apply convolution neural networks on it.
Liu et al.
(2019b; 2020) extends the set convolution from PointNet++ (Qi et al., 2017b) by establishing neighborhood across different frames.
PSTNet (Fan et al., 2021b) proposes a spatiotemporal decoupled point convolution for point cloud sequence processing. Recent works Fan et al. (2021a) ; Wang et al.
(2021a) introduce attention mechanism to better capture data correlation across frames.
In terms of generative tasks, Tranquil Clouds (Prantl et al., 2020) proposes a temporal loss that supervises the scene ﬂow across frames and stabilize the generation of point clouds.
Rempe et al.
(2020) proposes CaSPR, a deep learning framework for learning continuous object-centric spatiotemporal representation from point clouds.
Lu et al.
(2021) proposes interpolation network to enhance the temporal resolution of point cloud sequence. Very recent work Wang et al. (2021b) proposes a learning-based recurrent temporal alignment module for assimilating temporal information when upsampling point cloud sequences. In general, our work is closely related to PointNet++ (Qi et al., 2017b) and MeteorNet (Liu et al., 2020) which use set abstraction with different grouping strategies to learn discriminative representation on point clouds.
In the generator, we adopt a similar multi-scale feature extractor architecture as the one proposed in PU-GCN (Qian et al., 2021).
Compared to PU-GAN (Li et al., 2019), our method takes the change of points in time dimension into account, and uses hierarchical structures in the discriminator instead of operating on full resolution point clouds.
Under the hood of sequential point cloud data generative modelling, Tranquil Clouds (Prantl et al., 2020), CaSPR (Rempe et al., 2020), and SPU (Wang et al., 2021b) share similar target with our work.
CaSPR focuses on objectcentric tasks and thus is not available for processing multi-object collections or scenes.
Tranquil Clouds requires point correspondence across frames to evaluate the ﬁrst-order and second-order ﬁnite difference loss, which our method does not require during the training phase.
SPU augments upsampling process with temporal information extracted by recurrent network, while in this work we use temporal discriminator instead of recurrent architectures to learn the temporal coherence.
(c) (b) (a)
Input
N x
Feature Extractor
Encoding
N x C
Upsampling Module
Masking Module
Mask
N x
Residuals
N x 3r
Masked Residuals rN x
Rearrange
Output r’N xrN x
Duplicate x r times
Generator
Temporal Discriminator
Input 3 x rN x
Set Abstraction
Set Abstraction Set Abstraction
Flow Module 3 x ( rN / )
x rN / x
Set
Abstraction x
L layers
Confidence 1 x
Remove duplicate points
Spatial Discriminator
Input rN x
Set
Abstraction x
L layers
MLP 1 x
MLP 1 x C
Confidence 1 xtt t +
(Detailed schematic of each sub-module can be found in Appendix A.1)
M ETHODS 3. H IERARCHICAL SPATIAL FEATURE LEARNING
Set abstraction Point clouds are irregular and unordered data where traditional convolution operations usually do not ﬁt.
To cope with this irregularity, we adopt the set abstraction layer from Qi et al.
(2017a;b) to extract features from point clouds for discriminative models in our proposed framework.
Inside a set abstraction layer, point cloud is divided into different groups, and then a universal set function approximator is applied to extract the local features of each group
[Equation]
where F denotes features of points, r denotes the positions of points, N ( i ) denotes the neighborhood of point i , γ and h are learnable functions which are usually implemented as multi-layer perceptrons (MLPs).
Multi-scale graph convolution networks Recent work PU-GCN (Qian et al., 2021) proposes
Inception DenseGCN, and demonstrates the superiority of this multi-scale GCN-based feature extractor on the point cloud upsampling tasks.
Similar to the set abstraction layer from PointNet, the
GCN layer in PU-GCN divides the point cloud into different groups and applies learnable functions to these groups to extract local features.
The major differences are that ﬁrst, the GCN layer in PU-GCN constructs the neighborhood following DGCNN’s
(Wang et al., 2019b) formulation by dynamically querying k -nearest neighbors in the latent space instead of Euclidean space, and second, in PU-GCN, each layer aggregates features from different scales of neighborhood at each layer.
These differences make the Inception DenseGCN layer in PU-GCN more effective at extracting features from different scales and improve the overall performance of model.
However, since it is more computationally intensive than the standard set abstraction, we only apply Inception DenseGCN to the generator, where it is most effective.
3. T EMPORAL FEATURE LEARNING
To capture the underlying patterns of point cloud sequences over the time dimension, we construct a module that operates on the temporal neighborhood.
We adapt the ﬂow embedding layer from Liu et al.
(2019b; 2020) to operate on neighborhood across frames.
Given two consecutive frames of point clouds { X t , X t +1 } and their corresponding features { F t , F t +1 } , we use the ﬂow embedding layer to mix points across frames and extract temporal features.
The mixture of features { F t , F t +1 } across two consecutive frames via ﬂow embedding layer is deﬁned as:
[Equation]
where j denotes the index of points from frame t +
and i denotes the index of points from frame t .
3. E NFORCING TEMPORAL COHERENCE VIA TEMPORAL DISCRIMINATOR
When the target data are point cloud sequences that come from physical processes like particle dynamics or camera-scanned human action, these data are highly correlated in the time dimension.
One way to capture this temporal coherence would be directly using the scene ﬂow information in the ground truth data (i.e. displacements of points) to supervise the training process, i.e. adding an additional loss term that evaluates the scene ﬂow difference between ground truth and prediction.
However, for many real-world scenarios, there are often no clear correspondences between point clouds from two frames.
Therefore, the displacement of points between different frames are not annotated and cannot be used for training.
To enforce the temporal coherence without manually encoding scene ﬂow information, we propose a discriminator to learn the temporal pattern from ground truth data implicitly and use the discriminator to guide generator predicting temporally coherent output.
Our proposed temporal discriminator is built upon set abstraction layer and ﬂow embedding layer discussed in the previous section.
The temporal discriminator takes three consecutive frames of point clouds as input.
These point clouds are ﬁrst downsampled and transformed by several layers of set abstraction.
Then we apply the ﬂow module, which consists of two ﬂow embedding layers, to assimilate temporal information from the point cloud sequences.
Finally, we use set abstraction layer and a MLP to aggregate global information and predict whether the input sequence is valid (temporally coherent) or not.
3. A DAPTIVE UPSAMPLING
In most of the learning-based point cloud upsampling methods, all the points in the input will be upsampled and the ratio between the number of input points and output points is ﬁxed.
Given an upsampling network with a upsampling ratio of r , for the point x i in the input point cloud, there will be r points in the upsampled point cloud that is directly upsampled from the input point x i .
Since these networks are trained and operates on local patches, the locality will drive the upsampled points to be very close to the original input point.
Every input point will have r neighbors very close to them in the upsampled point cloud.
However, not all point clouds are distributed in a uniform way such that every point has similar amount of points in their neighborhood.
For instance, in particle-based ﬂuid dynamics, near-surface or ﬂying away particles will have very different density compared to particles in the center of ﬂuid ﬁeld.
To suppress unnecessary upsampling, we propose a learnable masking module, which predicts a mask M that tells the network if it needs to upsample a point.
Concretely, for each point in the input point cloud, the masking module will predict a conﬁdence value p i , which forms the soft mask vector
M p =
[ p 1 , p 2 , . .
. , p N ] T .
The soft mask is then quantized and transformed into a binary mask M b based on predeﬁned threshold (cid:15)
[Equation]
In practice we set this threshold value to a small number (e.g. (cid:15) = 0 .
01 1 ) so that the upsampling of a point will be suppressed only when the conﬁdence level is low.
Finally we use the binary mask to modulate the predicted residuals by doing an element-wise multiplication, so that the residuals of unnecessary upsampling will be masked to zero.
Note that here we use the binary mask to modulate the output residuals instead of the input encoding of the upsampling module.
The reason is that if we mask the features of some points to zero, then the upsampling module cannot aggregate features from these points when predicting the residuals.
During training, we generate the label M t for masking module by counting the number of neighbors of input points in the target point cloud Q
[Equation]
where the neighbor set: N ( i ) =
{ j | ||
r j −
r
i || 2 < r 0 , j ∈ Q } is derived by querying the neighbor of point i in the target point cloud Q .
The search radius r 0 is set to be slightly larger than the average nearest distance between points in the target point cloud.
Based on the generated labels, we use L norm to evaluate the loss between soft mask M p and ground truth M t :
[Equation]
3. T RAINING
Model architectures We follow the design of PU-GCN to build the feature extractor in the generator, where we ﬁrst use three layers of GCN to extract features from the input points.
Inside the upsampling and the masking module, we use two layers of GCN to aggregate features from neighborhood and use MLP to project the latent encoding into prediction.
As for discriminators, we adopt the hierarchical structure similar to the PointNet++, where we use set abstraction layer/ﬂow embedding layer to extract features at different scales and continue to downsample the point cloud through network layers.
(Model details can be found in Appendix A.1)
Loss function We adopt the least-squared GAN (Mao et al., 2017) setting to train our model.
Based on LS-GAN’s formulation, we can derive the adversarial loss for temporal discriminator D t and spatial discriminator D s ,
[Equation]
where X denotes the low-resolution input, Y denotes the high-resolution target, Y t , (cid:101)
Y t denotes three consecutive frames of point clouds: Y t =
{ Y t − 1 , Y t , Y t +1 } , (cid:101)
[Equation] X t [Equation]
The corresponding adversarial losses for generator are:
[Equation]
To encourage the output of generator lies close to the ground truth point clouds, we use Chamfer
Distance (CD) to evaluate and penalize the distance between generator output ˜ Y and ground truth Y
[Equation]
where y denotes point in Y ,
˜ y denotes point in ˜ Y , and | · | denotes the number of points in the corresponding point cloud.
Compared to other advanced distance metrics, Chamfer Distance is simpler and thus much faster to evaluate.
Combining with the masking loss discussed in the previous section, the total loss for generator is deﬁned as:
[Equation]
In practice we choose λ 1 , λ 2 such that all loss terms have similar magnitude (roughly 0 ∼ 1).
For example, we use λ 1 = 1 × 10 4 , λ 2 = 1 × 10 2 for ﬂuid dataset and λ 1 = 1 × 10 2 , λ 2 = 1 × 10 3 for action dataset.
Stabilizing GAN’s Training We implement two techniques to stabilize the training process.
The ﬁrst one is spectral normalization proposed by Miyato et al.
(2018), which constrains the Lipschitz constant of the discriminator by setting the spectral norm of weights to 1.
We found this technique is crucial especially when training on the ﬂuid dataset.
We hypothesize the main reason is that the irregular distribution and complex dynamics of ﬂuid particles can make discriminator prone to produce large gradients and thus make the training unstable.
The second technique we adopt is label smoothing from Salimans et al.
(2016), where we soften the target label in adversarial loss from 1 to a random variable uniformly chosen from [0 . 8 , 1 . 2] .
E XPERIMENT 4. I MPLEMENTATION DETAILS
Dataset We train and evaluate our model on two different datasets.
The ﬁrst dataset is generated from particle-based ﬂuid dynamics simulation, where particles distribute and move under the physics governing equation.
We use DFSPH (Bender & Koschier, 2015) solver from SPlisHSPlasH 3 to generate the training data.
We set the upsampling ratio to be r = 8 on the ﬂuid dataset.
The second dataset is from scanned human action - the MSR-Action3D dataset (Li et al., 2010), which consists of 567 Kinect depth videos of 20 action categories.
In this dataset, points from two different frames do not necessarily have correspondence and number of points varies in different frames.
We adopt the same preprocessing and splitting strategy from previous works (Liu et al., 2020; Fan et al., 2021b).
The upsampling ratio is set to be r = 16 on the action dataset.
Evaluation metrics
In addition to the Chamfer distance discussed in the prvious section, we introduce three other spatial distance metrics to further evaluate the model performance.
The ﬁrst metric is Earth Mover’s Distance (EMD), which has been reported by previous work are more reliable and discriminative than Chamfer Distance in dense point clouds’ evaluation (Liu et al., 2019a).
EMD evaluates the distance between two clouds by solving a linear assignment problem.
Other than EMD, we use Maximum Mean Discrepancy (MMD) to evaluate the spatial distribution difference between prediction and ground truth.
MMD discriminates the point clouds by measuring their high-level similarity instead of difference within pairs of matched points.
In practice we found MMD is most discriminative on dense point clouds, thus we only report MMD on the ﬂuid dataset.
We also report the Hausdorff distance (HD) following previous works in static point cloud upsampling task.
The evaluation of the temporal coherence are depending on the speciﬁc tasks, so we will discuss them in the corresponding sub-sections.
Baseline To demonstrate the effectiveness of our proposed methods, we compare our model against two recent state-of-the-art works, PU-GCN (Qian et al., 2021) and Tranquil Clouds (Prantl et al., 2020).
Tranquil clouds proposes the ﬁrst learning-based method for upsampling point clouds in temporally coherent manner.
Note that Tranquil Clouds uses mingling loss to encourage the points to spread and expand, this results in its much higher Chamfer Distance during evaluation, since
Chamfer Distance favors upsampled points lying close to their original input points.
The second baseline PU-GCN is the state-of-the-art model on static point cloud upsampling.
The generator in our model shares a similar architecture with it (with some differences on hyperparamter choice), while our generator contains an additional masking module to upsample input in an adaptive manner.
(More details on metrics, baselines’ implementation, and dataset generation can be found in Appendix
A.2, A.3, A.4)
4. D ENSE AND NON - UNIFORM POINT CLOUDS FROM PHYSICAL SIMULATION
Advection consistency In ﬂuids, particles’ motion are subject to the underlying governing equation, such that given the current position x t and v t , there exists an continuous advection function: Ψ : x t +1 = Ψ
( x t , v t ) that advects particles from current state to next time step.
Therefore we can evaluate the temporal coherence of generated sequences by checking if they are consistent with the underlying advection function.
Given the ground truth data, we can use the kernel model in
Smooth Particle Hydrodynamics (SPH) (Gingold & Monaghan, 1977) to approximate the value of
[Figure]
advection function at arbitrary locations (for more details on SPH and kernel model, please refer to the Appendix A.5).
Based on the approximated advection function Ψ and the generator G , we deﬁne the advection consistency for generator as:
[Equation]
(12) where X t , V t denotes the positions and velocities of particles at time step t , and D ( P , Q ) denotes the distance between two point clouds P , Q , which can be evaluated using the aforementioned spatial distance metrics. Model Static Advection Consistency
CD ↓ × 10 − EMD ↓ × 10 − MMD ↓ × 10 − HD ↓ × 10 −
CD ↓ × 10 −
EMD ↓ × 10 −
MMD ↓ × 10 −
HD ↓ × 10 −
TPU-GAN 2.
TPU-GAN + vel 2.
PU-GCN 2.
Tranquil Clouds 5.
Static column reports the per-frame difference between upsampled point clouds and ground truth.
The Advection
Consistency column reports the temporal coherence of two consecutive frames of upsampled point clouds.
Comparison with other models We implemented two versions of our model.
In the ﬁrst version, the temporal discriminator only takes position as input, while in the second version the temporal discriminator takes velocities of particles as additional features.
The quantitative evaluation results are shown in the Table 1 and the visualization of sequence upsampled from real coarse ﬂuid simulation are shown in the left column of Figure 2.
Compared to PU-GCN which does not take temporal coherence into account, TPU-GAN improves the advection consistency without the information of displacements across frames, which demonstrates the effectiveness of temporal discriminator.
TPU-GAN also outperforms PU-GCN on high-level distance metrics (EMD, MMD), which highlights the superiority of our proposed method in terms of reconstruction quality on irregular point clouds.
Furthermore, we showcase that with velocity features, TPU-GAN is on par with Tranquil Clouds in terms of temporal coherence, while improves the static upsampling quality on all the metrics signiﬁcantly.
Note that
Tranquil Clouds is strongest when using EMD to evaluate advection consistency, in part because they train the model with a temporal loss that is essentially equivalent to evaluating the advection consistency using EMD, with advection function approximated by EMD assignment instead of SPH’s kernel interpolation.
4. R EAL - WORLD SCANNED POINT CLOUDS
[Table]
[Table]
Perceptual loss for point cloud sequences For real-world scanned data, we often don’t know the corresponding points in the next frame given the current frame.
Therefore it is difﬁcult to directly compare the motion of upsampled sequence and target sequence.
Inspired from previous video generation works which use pretrained I3D network (Carreira & Zisserman, 2018; Unterthiner et al., 2018) to evaluate the spatio-temporal quality of generated videos at sequence level, we use a pretrained point cloud classiﬁer to extract sequence-level spatio-temporal features.
The pretrained classiﬁer we used here is PSTNet (Fan et al., 2021b), the state-of-the-art model on point cloud sequence classiﬁcation.
We use the feature map from the middle of the network as the extracted features (more details on sequence perceptual loss can be found in Appendix A.3), and based the on the extracted features we deﬁne the perceptual loss (PL) for point cloud sequences as: [Equation]
[Equation]
where Y =
{ Y 1 , Y 2 , . . .
, Y T } denotes the reference sequence, ˜ Y =
{ ˜ Y 1 , ˜ Y 2 , . . .
, ˜ Y T } denotes the predicted sequence and φ denotes the feature extractor.
Comparison with other models The quantitative evaluation result on MSR-Action 3D dataset is shown in the Table 2 and visualization of upsampled point clouds is shown on the right column of the
Figure 2.
As indicated in the quantitative result, TPU-GAN outperforms other methods on frame-level
EMD and sequence-level PL, with a minor gap between PU-GCN on CD.
This demonstrates that
TPU-GAN can improve the upsampling quality in both frame-level and sequence-level under a scenario that no point correspondence annotation is available.
Self-supervised temporal feature learning To further evaluate the representation learned by temporal discriminator, we apply it as a feature extractor for classiﬁcation task.
We freeze all the layers before the last set abstraction layer (SA) and MLP (the schematic of temporal discriminator is shown in Figure 1), then test the network’s performance on action classiﬁcation.
In Table 3, we report the video-level accuracy for three supervised methods and classiﬁer based on feature extracted by temporal discriminator.
By leveraging spatiotemporal feature extracted by the temporal discriminator, the classiﬁer outperforms PointNet++ that only utilizes single-frame features.
It also signiﬁcantly outperforms classiﬁcation model which uses only the set abstraction layer and MLP.
PSTNet still leads the performance, mainly because it is trained in a fully-supervised way (i.e. all parameters are trained on the classiﬁcation task) and it contains much more parameters than temporal discriminator (PSTNet: 7.4M, TempoDis: 0.80M).
4. A BLATION STUDY
Architectural choice We investigate the inﬂuence of our architectural choice to the model performance on the MSR-Action 3D dataset.
The result is shown in the Table 4.
The result indicates
[Figure]
[Table]
[Figure]
that: ﬁrst, increasing the frame number of point clouds exposed to the temporal discriminator (" w/o
TempoDis" vs "Input two frames" vs "Full model" (input three frames)) can improve the the sequence level accuracy.
However too long input sequence ("Input ﬁve frames") will deteriorate the frame-level upsampled quality.
As it will make points cluster around input points and exhibit halo artifacts.
(See
Figure 10 in the Appendix for visualization)
[Figure]
Moreover, we ﬁnd that spatial discriminator is crucial for the nonuniform dense point cloud upsampling.
The result of network will shrink towards the center of the point cloud without spatial discriminator.
We calculates the top 0 .
1% furthest distances between every pair of points in the upsampled patches from ﬂuid dataset, and we normalize all the distances using the furthest distance in ground truth.
As shown in the Figure 3, without spatial discriminator, the upsampled point clouds’ furthest distance distribution will shift to the left.
Adaptive upsampling We analyze the point distribution in the upsampled results from different methods by comparing their density to the ground truth.
The analysis was conducted on the ﬂuid data, where particle density is a crucial physical quantity that dominates the dynamics of incompressible ﬂuids.
We evaluate the particle number density following the deﬁnition in the SPH model, and the result is shown in the
Figure 4.
For ground truth data, the density distribution has two sharp peaks, which corresponds to particles near surface area and particles in the central part respectively.
For upsampled point clouds, all the distribution has a smoothed bell shape, while our method produce a sharper density distribution with a mode η = 7 . 4 more close to the ground truth ( η 1 = 6 . 3 , η 2 = 8 . 4 ).
This demonstrates the adaptive masking module in the generator effectively learns to suppress unnecessary upsampling and approximates the target density distribution, while other methods have the density distribution shifted to the right.
Model size In Table 5, we report the number of trainable parameters for concurrent learning-based upsampling models.
Despite the dual discriminator setting, our model is still much smaller than PU-Net and
Tranquil Clouds which in turn makes it more efﬁcient during inference.
C ONCLUSION
In this work, we propose TPU-GAN, a GAN framework for upsampling dynamic point cloud sequences.
Our proposed method is able to learn useful spatiotemporal representation for both generative task and discriminative task from raw point cloud sequences.
Through experiments on different data sources, we demonstrate the effectiveness of our method on the generative modelling of point cloud sequences.
[Table]
A PPENDIX A.
M ODEL DETAILS
We implement our model in PyTorch 1.7.1.
All the training and experiments are run on a platform equiped with a single GTX-1080Ti.
We train all our model for 150k gradient updates using Adam optimizer (Kingma & Ba, 2017), which took approximately 45 hours on the Fluid dataset andhours on the MSR-Action 3D dataset.
The detail of our model architecture are listed below:
• For all the GCN, set abstraction and ﬂow embedding layer, we use max pooling as aggregation function.
• The feature extractor in the Generator comprises one layer of GCN and two layers of Inception DenseGCN.
At the GCN layer, we construct neighborhood by searching each points’ 20 nearest neighbors based on the point position and use a MLP with hidden units: [3 , 32 , 128] to encode the point features.
For Inception DenseGCN we follow the design and hyperparamter choice in the original PU-GCN paper (Qian et al., 2021).
• The masking and upsampling module in the Generator has two layer of GCN and a [128 , 64 , c ] MLP, where c = 3 r for upsampling module, c
= 1 for masking module and r is the upsampling ratio.
The ﬁrst layer of GCN operates on the 12 nearest neighbors and the second layer operates on the 4 nearest neighbors.
Each GCN layer contains a [128 , 32] bottleneck linear layer and a [32 , 64 , 128] MLP to update point features.
• We follow the design of PointNet++ to build the Spatial Dsicriminator .
Concretely,
we use three set abstraction layers:(0 .
25 N, 32 , [3 , 32 , 64]) , (0 . 1 N, 32 , [64 , 128 , 256]) , (0 .
06 N, 16 , [256 , 384 , 512]) and a [512 , 256 , 64 , 1] MLP (with N the number of input points).
• The Temporal Discriminator also adopts a hierarchical structure that contains two set
abstraction layers: (0 .
25 N, 32 , [3 , 32 , 64]) , (0 .
125 N, 32 , [64 , 128 , 256]) , followed by two ﬂow embedding layers: (0 .
125 N, 64 , [256 , 256 , 256]) , (0 .
125 N, 64 , [256 , 256 , 256]) , and ﬁnally a set abstraction layer: (0 .
125 N, 16 , [256 , 384 , 512]) and a MLP: [512 , 256 , 64 , 1] .
[Figure] [Figure]
A. D ATASET
Fluid We simulate 24 ﬂuid sequences with random initialization of ﬂuid block position and scene conﬁguration, each sequence contains 200 snapshots with a time interval of 0.005s and particle radius of 0 . 0125 .
Each snapshot contains roughly 100k ∼ 200k particles.
We use 20 sequences as training data, and test on the rest 4 sequences.
We adopt the patch training strategy, where we sample patches containing 10k particles from the ﬂuid ﬁeld and uniformly downsample them to create input-target pairs for training and evaluation.
MSR-Action 3D
We follow the description from Liu et al.
(2020) 6 to process the raw data into point clouds.
We uniformly downsample the processed data to create low-resolution input similar to ﬂuid data, but here we input the whole frame instead of sampled patches to the network.
A. M ETRIC IMPLEMENTATION
Earth mover distance We use Liu et al.
(2019a)
’s
EMD implementation 7 to evaluate the loss.
This implementation is based on auction algorithm and thus is consistent with the EMD code used in the original Tranquil Clouds paper.
In fact, our evaluation of Tranquil Clouds on the ﬂuid dataset also roughly matches the results reported in their original paper ( 3 .
× 10 − 2 in our paper, 3 .
× 10 −in original paper, the difference is likely caused by our ﬁne-tuning on the pretrained weights and different implementation of EMD).
Maximum Mean Discrepancy We use the GeomLoss library 8 to evaluate the MMD, with a
Gaussian kernel and bandwidth σ = 0 .
Perceptual loss for point cloud sequence We evaluate the perceptual loss for action dataset based on features from the third convolution layer of PSTNet.
As shown in the original paper (Fan et al., 2021b), the convolution layer output higher activation on the salient motion.
This motivates us to compare the activation of reconstructed point clouds against the ground truth, which provides a measurement for the similarity of two sequences in terms of motion.
In practice we found that using activation from different middle layers (layer 2 , 3 , 4 ) produce similar results.
To make the extracted feature permutation invariant to the input points, we conduct a global top-k pooling (with k = 128 ) before calculating the loss.
A. B ASELINE IMPLEMENTATION
Tranquil Clouds We use the publicly available source code 9 released by the authors to carry out experiment.
On the ﬂuid dataset, we ﬁnd it unnecessary to retrain the model since the ﬂuid data in both papers are essentially generated from similar volume-conserving SPH simulator, so we just ﬁne-tune the pretrained weights provided by the authors on our own ﬂuid dataset for 5 epochs.
For action dataset, we explore two strategies to train the model.
The ﬁrst is to train the model from scratch without temporal loss, since there is no point correspondence information in the dataset.
However this makes Tranquil Clouds perform much worse in terms of temporal coherence.
The second is directly using the pretrained weights which preserves temporally coherent latent space.
In general we found the latter provides much better temporal coherence thus we stick with pretrained weights to conduct experiment.
PU-GCN We implement PU-GCN from scratch in PyTorch and compare our implemented version with the ofﬁcial TensorFlow version released by the authors 10 .
We found they have similar performance.
As our training and evaluation pipeline was mostly implemented in PyTorch, so for convenience we use PyTorch version of PU-GCN throughout the experiment.
A. K ERNEL MODEL IN SPH
A common method to solve the Navier-Stokes equation and discretize ﬂuids under the Lagrangian framework is Smooth Particle Hydrodynamics (SPH) (Gingold & Monaghan, 1977).
In SPH, physical quantities at an arbitrary point in the space are approximated by the quantities of nearby particles.
Conretely, an arbitrary scalar (or vector) ﬁeld A ( r ) at location r can be represented by a convolution:
[Equation] [Equation]
and
[Equation]
i
[Equation]
where W is weighting function or smooth kernel as deﬁned in SPH, h is the cutoff, which deﬁnes the range of particles to be considered, V ( r (cid:48) ) denotes the volume at r , N ( i ) denotes the neighbor particles of particle i that are within the cutoff, ρ denotes the particle number density and m denotes the mass of each ﬂuid particle.
Using this model, we can approximate the advection Ψ( r ) at arbitrary location r by:
[Equation]
with,
[Equation]
where r ( t ) j denotes the position of particle j at time step t .
Throughout the experiment we use cubic spline kernel function as the smooth kernel W ( · ) .
A. V ISUALIZATION AND STUDY ON THE MASKING MODULE
In this section we present visualization on the learnable masking module from different perspectives.
Figure 7 shows the effect of masking module on the very same network.
Figure 8 visualizes the prediction of masking module on randomly selected ﬂuid patches. Figure 9 shows the superiority of using proposed masking loss (equation 4) to train the masking module. Without the supervision of masking loss, the mask module will suppress the upsampling too much, which results in clustered points and halo structures. Reference
Masking on
Masking off
[Figure]
[Figure]
No
[Figure]
loss
[Figure]
V ISUALIZATION OF ABLATION
[Figure]
V ISUALIZATION OF POINT CLOUDS GENERATED DURING TRAINING PROCESS
[Figure]
[Figure]
A. S CENE - FLOW ESTIMATION
We apply our proposed method to a self-driving dataset (Kitti-scene ﬂow) to demonstrate the effectiveness of our model in real-world self-driving application.
We use the Kitti point cloud scene ﬂow dataset proposed by Liu et al. (2019b).
To mimic real world scenario where data are very sparse, we downsample the original point clouds (around 30000 - 40000 points per frame) to obtain sparse point clouds (2048 points per frame).
We apply different upsampling networks to upsample the sparse point clouds (visualization is shown in Figure 13).
Then we use upsampled point clouds as input data for a scene ﬂow prediction network (pre-trained FlowNet3D (Liu et al., 2019b)).
To quantitatively measure model’s performance under different conditions, we adopt two metrics to evaluate the accuracy of predicted scene ﬂow - end point error (EPE 3D), which evaluates the L distance between ground truth and prediction, and 3D outliers, which evaluates how many predictions are extremely deviated from ground truth (>0.3m or >5%).
Since data is very limited here (onlypairs of two consecutive frames are available), we follow previous works’ evaluation protocol (Li et al., 2019; Qian et al., 2021; Liu et al., 2019b), all the upsampling networks are pretrained (on the ﬂuid dataset) instead of training from scratch on Kitti scene data.
As shown in the Table 6, the performance of scene ﬂow prediction network drops signiﬁcantly when input points are sparse, while upsampling networks can help mitigate the performance degradation.
Particularly, TPU-GAN outperforms other networks on both metrics.
This demonstrates the superiority of our proposed method and its potential to self-driving application.
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]