Original: T RANQUIL C LOUDS : N EURAL N ETWORKS 
Revised: T RANQUIL C LOUDS : N EURAL N ETWORKS FOR L EARNING T EMPORALLY C OHERENT F EATURES IN P OINT C LOUDS

Original: FOR

Original: L EARNING

Original: T EMPORALLY

Original: C OHERENT

Original: F EATURES

Original: IN

Original: P OINT C LOUDS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Point clouds, as a form of Lagrangian representation, allow for powerful and ﬂexible applications in a large number of computational disciplines.
Revised: Point clouds, as a form of Lagrangian representation, allow for powerful and ﬂexible applications in a large number of computational disciplines.

Original: We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time.
Revised: We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time.

Original: We identify a set of inherent problems with these approaches: without knowledge of the time dimension, the inferred solutions can exhibit strong ﬂickering, and easy solutions to suppress this ﬂickering can result in undesirable local minima that manifest themselves as halo structures.
Revised: We identify a set of inherent problems with these approaches: without knowledge of the time dimension, the inferred solutions can exhibit strong ﬂickering, and easy solutions to suppress this ﬂickering can result in undesirable local minima that manifest themselves as halo structures.

Original: We propose a novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos.
Revised: We propose a novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos.

Original: We combine these techniques in a super-resolution method with a truncation approach to ﬂexibly adapt the size of the generated positions.
Revised: We combine these techniques in a super-resolution method with a truncation approach to ﬂexibly adapt the size of the generated positions.

Original: We show that our method works for large, deforming point sets from different sources to demonstrate the ﬂexibility of our approach.
Revised: We show that our method works for large, deforming point sets from different sources to demonstrate the ﬂexibility of our approach.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning methods have proven themselves as powerful computational tools in many disciplines, and within it a topic of strongly growing interest is deep learning for point-based data sets.
Revised: Deep learning methods have proven themselves as powerful computational tools in many disciplines, and within it a topic of strongly growing interest is deep learning for point-based data sets.

Original: These
Revised: These

Original: Lagrangian representations are challenging for learning methods due to their unordered nature, but are highly useful in a variety of settings from geometry processing and 3D scanning to physical simulations, and since the seminal work of Qi Charles et al.
Revised: Lagrangian representations are challenging for learning methods due to their unordered nature, but are highly useful in a variety of settings from geometry processing and 3D scanning to physical simulations, and since the seminal work of Qi Charles et al.

Original: (2017), a range of powerful inference tasks can be achieved based on point sets.
Revised: (2017), a range of powerful inference tasks can be achieved based on point sets.

Original: Despite their success, interestingly, no works so far have taken into account time.
Revised: Despite their success, interestingly, no works so far have taken into account time.

Original: Our world, and the objects within it, naturally move and change over time, and as such it is crucial for ﬂexible point-based inference to take the time dimension into account.
Revised: Our world, and the objects within it, naturally move and change over time, and as such it is crucial for ﬂexible point-based inference to take the time dimension into account.

Original: In this context, we propose a method to learn temporally stable representations for point-based data sets, and demonstrate its usefulness in the context of super-resolution.
Revised: In this context, we propose a method to learn temporally stable representations for point-based data sets, and demonstrate its usefulness in the context of super-resolution.

Original: An inherent difﬁculty of point-based data is their lack of ordering, which makes operations such as convolutions, which are easy to perform for Eulerian data, unexpectedly difﬁcult.
Revised: An inherent difﬁculty of point-based data is their lack of ordering, which makes operations such as convolutions, which are easy to perform for Eulerian data, unexpectedly difﬁcult.

Original: Several powerful approaches for point-based convolutions have been proposed (Qi et al., 2017; Hermosilla et al., 2018; Hua et al., 2018), and we leverage similar neural network architectures in conjunction with the permutation-invariant Earth Mover’s Distance (EMD) to propose a ﬁrst formulation of a loss for temporal coherence.
Revised: Several powerful approaches for point-based convolutions have been proposed (Qi et al., 2017; Hermosilla et al., 2018; Hua et al., 2018), and we leverage similar neural network architectures in conjunction with the permutation-invariant Earth Mover’s Distance (EMD) to propose a ﬁrst formulation of a loss for temporal coherence.

Original: In addition, several works have recognized the importance of training point networks for localized patches, in order to avoid having the network to rely on a full view of the whole data-set for tasks that are inherently local, such as normal estimation (Qi Charles et al., 2017), and super-resolution (Yu et al., 2018a).
Revised: In addition, several works have recognized the importance of training point networks for localized patches, in order to avoid having the network to rely on a full view of the whole data-set for tasks that are inherently local, such as normal estimation (Qi Charles et al., 2017), and super-resolution (Yu et al., 2018a).

Original: This also makes it possible to ﬂexibly process inputs of any size.
Revised: This also makes it possible to ﬂexibly process inputs of any size.

Original: Later on we will demonstrate the importance of such a patch-based approach with sets of changing cardinality in our setting.
Revised: Later on we will demonstrate the importance of such a patch-based approach with sets of changing cardinality in our setting.

Original: A general challenge here is to deal with varying input sizes, and for super-resolution tasks, also varying output sizes.
Revised: A general challenge here is to deal with varying input sizes, and for super-resolution tasks, also varying output sizes.

Original: Thus, in summary we target an extremely challenging learning problem: we are facing permutation-invariant inputs and targets of varying size, that dynamically move and deform over time.
Revised: Thus, in summary we target an extremely challenging learning problem: we are facing permutation-invariant inputs and targets of varying size, that dynamically move and deform over time.

Original: In order to enable deep learning approaches in this context, we make the following key contributions: Permutation invariant loss terms for temporally coherent point set generation
Revised: In order to enable deep learning approaches in this context, we make the following key contributions: Permutation invariant loss terms for temporally coherent point set generation;

Original: ; A Siamese training setup and generator architecture for point-based super-resolution with neural networks; Enabling improved output variance by allowing for dynamic adjustments of the output size; The identiﬁcation of a specialized form of mode collapse for temporal point networks, together with a loss term to remove them.
Revised: A Siamese training setup and generator architecture for point-based super-resolution with neural networks; Enabling improved output variance by allowing for dynamic adjustments of the output size; The identiﬁcation of a specialized form of mode collapse for temporal point networks, together with a loss term to remove them.

Original: We demonstrate that these contributions together make it possible to infer stable solutions for dynamically moving point clouds with millions of points.
Revised: We demonstrate that these contributions together make it possible to infer stable solutions for dynamically moving point clouds with millions of points.

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: More formally, we show that our learning approach can be used for generating a point set with an increased resolution from a given set of input points.
Revised: More formally, we show that our learning approach can be used for generating a point set with an increased resolution from a given set of input points.

Original: The generated points should provide an improved discretization of the underlying ground truth shape represented by the initial set of points.
Revised: The generated points should provide an improved discretization of the underlying ground truth shape represented by the initial set of points.

Original: For the increase, we will target a factor of two to three per spatial dimension.
Revised: For the increase, we will target a factor of two to three per spatial dimension.

Original: Thus, the network has the task to estimate the underlying shape, and to generate suitable sampling positions as output.
Revised: Thus, the network has the task to estimate the underlying shape, and to generate suitable sampling positions as output.

Original: This is generally difﬁcult due to the lack of connectivity and ordering, and in our case, positions that move over time in combination with a changing number of input points.
Revised: This is generally difﬁcult due to the lack of connectivity and ordering, and in our case, positions that move over time in combination with a changing number of input points.

Original: Hence it is crucial that the network is able to establish a temporally stable latent space representation.
Revised: Hence it is crucial that the network is able to establish a temporally stable latent space representation.

Original: Although we assume that we know correspondences over time, i.e., we know which point at time t moved to a new location at time t + ∆ t , the points can arbitrarily change their relative position and density over the course of their movement, leading to a substantially more difﬁcult inference problem than for the static case.
Revised: Although we assume that we know correspondences over time, i.e., we know which point at time t moved to a new location at time t + ∆ t , the points can arbitrarily change their relative position and density over the course of their movement, leading to a substantially more difﬁcult inference problem than for the static case.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Deep learning with static point sets was ﬁrst targeted in PointNet (Qi Charles et al., 2017) via order-invariant networks, while PointNet++ (Qi et al., 2017) extended this concept to generate features for localized groups similar to a convolution operation for grid-based data.
Revised: Deep learning with static point sets was ﬁrst targeted in PointNet (Qi Charles et al., 2017) via order-invariant networks, while PointNet++ (Qi et al., 2017) extended this concept to generate features for localized groups similar to a convolution operation for grid-based data.

Original: This concept can be hierarchically applied to the generated groups, in order to extract increasingly abstract and global features.
Revised: This concept can be hierarchically applied to the generated groups, in order to extract increasingly abstract and global features.

Original: Afterwards, the extracted features can be interpolated back to the original point cloud.
Revised: Afterwards, the extracted features can be interpolated back to the original point cloud.

Original: The goal to deﬁne point convolutions has been explored and extended in several works.
Revised: The goal to deﬁne point convolutions has been explored and extended in several works.

Original: The
Revised: The

Original: MCNN approach (Hermosilla et al., 2018) phrased convolution in terms of a Monte Carlo integration.
Revised: MCNN approach (Hermosilla et al., 2018) phrased convolution in terms of a Monte Carlo integration.

Original: PointCNN (Hua et al., 2018) deﬁned a pointwise convolution operator using nearest neighbors, while extension-restriction operators for mapping between a point cloud function and a volumetric function were used in Atzmon et al.
Revised: PointCNN (Hua et al., 2018) deﬁned a pointwise convolution operator using nearest neighbors, while extension-restriction operators for mapping between a point cloud function and a volumetric function were used in Atzmon et al.

Original: The PU-Net (Yu et al., 2018a) proposed a network for upsampling point clouds, and proposed a similar hierarchical network structure of PointNets along the lines of PointNet++ to deﬁne convolutions.
Revised: The PU-Net (Yu et al., 2018a) proposed a network for upsampling point clouds, and proposed a similar hierarchical network structure of PointNets along the lines of PointNet++ to deﬁne convolutions.

Original: Being closer to our goals, we employ this approach for convolutional operations in our networks below.
Revised: Being closer to our goals, we employ this approach for convolutional operations in our networks below.

Original: We do not employ the edge-aware variant of the
Revised: We do not employ the edge-aware variant of the

Original: PU-Net (Yu et al., 2018b) here, as we focus on temporal changes in our work.
Revised: PU-Net (Yu et al., 2018b) here, as we focus on temporal changes in our work.

Original: Permutation invariance is a central topic for point data, and was likewise targeted in other works (Ravanbakhsh et al., 2016; Zaheer et al., 2017).
Revised: Permutation invariance is a central topic for point data, and was likewise targeted in other works (Ravanbakhsh et al., 2016; Zaheer et al., 2017).

Original: The Deep Kd-network (Klokov and Lempitsky, 2017) deﬁned a hierarchical convolution on point clouds via kd-trees.
Revised: The Deep Kd-network (Klokov and Lempitsky, 2017) deﬁned a hierarchical convolution on point clouds via kd-trees.

Original: PointProNets (Roveri et al., 2018) employed deep learning to generate dense sets of points from sparse and noisy input points for 3D reconstruction applications.
Revised: PointProNets (Roveri et al., 2018) employed deep learning to generate dense sets of points from sparse and noisy input points for 3D reconstruction applications.

Original: PCPNet (Guerrero et al., 2018), as another multi-scale variant of
Revised: PCPNet (Guerrero et al., 2018), as another multi-scale variant of

Original: PointNet, has demonstrated high accuracy for estimating local shape properties such as normal or curvature. P2PNet
Revised: PointNet, has demonstrated high accuracy for estimating local shape properties such as normal or curvature

Original: (Yin et al., 2018) used abidirectional network and extends PointNet++ to learn a transformation between two point clouds with the same cardinality.
Revised: 2018) used a network and ++ to learn a transformation between two point clouds with the same cardinality.

Original: Recently, the area of point-based learning has seen a huge rise in interest.
Revised: Recently, the area of point-based learning has seen a huge rise in interest.

Original: One focus here are 3D segmentation problems, where numerous improvements were proposed, e.g., by SPLATNet (Su et al., 2018), SGPN (Wang et al., 2018a), SpiderCNN (Xu et al., 2018), PointConv (Wu et al., 2018), SONEt(Li et al., 2018a) and 3DRNN (Ye et al., 2018).
Revised: One focus here are 3D segmentation problems, where numerous improvements were proposed, e.g., by SPLATNet (Su et al., 2018), SGPN (Wang et al., 2018a), SpiderCNN (Xu et al., 2018), PointConv (Wu et al., 2018), SONEt(Li et al., 2018a) and 3DRNN (Ye et al., 2018).

Original: Other networks such as Flex Convolution (Groh et al., 2018), the SuperPoint Graph (Landrieu and Simonovsky, 2018), and the fully convolutional network (Rethage et al., 2018) focused on large scale segmentation.
Revised: Other networks such as Flex Convolution (Groh et al., 2018), the SuperPoint Graph (Landrieu and Simonovsky, 2018), and the fully convolutional network (Rethage et al., 2018) focused on large scale segmentation.

Original: Additional areas of interest are shape classiﬁcation (Wang et al., 2018b;
Revised: Additional areas of interest are shape classiﬁcation (Wang et al., 2018b;

Original: Lei et al., 2018; Zhang and Rabbat, 2018; Skouson, 2018) and object detection (Simon et al., 2018; Zhou and Tuzel, 2018), and hand pose tracking (Ge et al., 2018).
Revised: Lei et al., 2018; Zhang and Rabbat, 2018; Skouson, 2018) and object detection (Simon et al., 2018; Zhou and Tuzel, 2018), and hand pose tracking (Ge et al., 2018).

Original: Additional works have targeted rotation and translation invariant inference (Thomas et al.,  and point cloud autoencoders (Yang et al., 2018; Deng et al., 2018).
Revised: Additional works have targeted rotation and translation invariant inference (Thomas et al., 2018), and point cloud autoencoders (Yang et al., 2018; Deng et al., 2018).

Original: A few works have also targeted generative models based on points, e.g., for point cloud generation (Sun et al., 2018), and with adversarial approaches (Li et al., 2018b).
Revised: A few works have also targeted generative models based on points, e.g., for point cloud generation (Sun et al., 2018), and with adversarial approaches (Li et al., 2018b).

Original: It is worth noting here that despite the huge interest, the works above do not take into account temporally changing data, which is the focus of our work.
Revised: It is worth noting here that despite the huge interest, the works above do not take into account temporally changing data, which is the focus of our work.

Original: A notable exception is an approach for scene ﬂow (Liu et al., 2018), in order to estimate 3D motion directly on the basis of point clouds.
Revised: A notable exception is an approach for scene ﬂow (Liu et al., 2018), in order to estimate 3D motion directly on the basis of point clouds.

Original: This work is largely orthogonal to ours, as it does not target generative point-based models.
Revised: This work is largely orthogonal to ours, as it does not target generative point-based models.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We assume an input point cloud X
Revised: We assume an input point cloud X

Original: = { x 1 , x 2 , ..., x k } of size k ∈ [1 , k max ] .
Revised: = { x 1 , x 2 , ..., x k } of size k ∈ [1 , k max ] .

Original: It consists of points x i ∈ R d , where d includes 3 spatial coordinates and optionally additional features.
Revised: It consists of points x i ∈ R d , where d includes 3 spatial coordinates and optionally additional features.

Original: Our goal is to let the network f s ( X ) infer a function ˜
Revised: Our goal is to let the network f s ( X ) infer a function ˜

Original: Y which approximates a desired super-resolution output point cloud Y =
Revised: Y which approximates a desired super-resolution output point cloud Y =

Original: { y 1 , y 2 , ..., y n } of size n ∈
Revised: { y 1 , y 2 , ..., y n } of size n ∈

Original: [1 , n max ] with y i ∈ R 3 , i.e. f s ( X ) =
Revised: [1 , n max ] with y i ∈ R 3 , i.e. f s ( X ) =

Original: ˜ Y ≈ Y .
Revised: ˜ Y ≈ Y .

Original: For now we assume that the number of output points n is deﬁned by multiplying k with a user-deﬁned upsampling factor r , i.e. n = rk .
Revised: For now we assume that the number of output points n is deﬁned by multiplying k with a user-deﬁned upsampling factor r , i.e. n = rk .

Original: Figure 2a)
Revised: Figure 2a)

Original: illustrates the data ﬂow in our super-resolution network schematically.
Revised: illustrates the data ﬂow in our super-resolution network schematically.

Original: We treat the upsampling problem as local one, i.e., we assume that the inference problem can be solved based on a spatially constrained neighborhood.
Revised: We treat the upsampling problem as local one, i.e., we assume that the inference problem can be solved based on a spatially constrained neighborhood.

Original: This allows us to work with individual patches extracted from input point clouds.
Revised: This allows us to work with individual patches extracted from input point clouds.

Original: At the same time, it makes it possible to upsample adaptively, for example, by limiting the inference to relevant areas, such as complex surface structures.
Revised: At the same time, it makes it possible to upsample adaptively, for example, by limiting the inference to relevant areas, such as complex surface structures.

Original: For the patch extraction we use a ﬁxed spatial radius and normalize point coordinates within each patch to lie in the range of [ − 1 , 1] .
Revised: For the patch extraction we use a ﬁxed spatial radius and normalize point coordinates within each patch to lie in the range of [ − 1 , 1] .

Original: Our ﬁrst building block is a measure for how well two point clouds represent the same object or scene by formulating an adequate spatial loss function.
Revised: Our ﬁrst building block is a measure for how well two point clouds represent the same object or scene by formulating an adequate spatial loss function.

Original: Following Achlioptas et al.
Revised: Following Achlioptas et al.

Original: (2017), we base our spatial loss L S on the Earth Mover’s Distance (EMD), which solves an assignment problem to obtain a differentiable bijective mapping φ : ˜ y
Revised: (2017), we base our spatial loss L S on the Earth Mover’s Distance (EMD), which solves an assignment problem to obtain a differentiable bijective mapping φ : ˜ y

Original: → y .
Revised: → y .

Original: With φ we can minimize differences in position for arbitrary orderings of the points clouds via:
Revised: With φ we can minimize differences in position for arbitrary orderings of the points clouds via:

Original: [Equation]
Revised: [Equation]

Original: 3. T EMPORAL C OHERENCE
Revised: 3. T EMPORAL C OHERENCE

Revised: L + v · ∆ t f s

Revised: [Equation]

Revised: x )

Revised: X t

Original: [Figure]
Revised: [Equation]

Revised: X t

Revised: X t + - v · ∆ t

Revised: X

Revised: Y

Revised: X

Revised: Y k n ~

Original:  When not taking temporal coherence explicitly into account, the highly nonlinear and ill-posed nature of the super-resolution problem can cause strong variations in the output even for very subtle changes in the input.
Revised: NN L S trunc pad pad Y ~ n ~ r k b) a) f s ( x ) f s ( x ) When not taking temporal coherence explicitly into account, the highly nonlinear and ill-posed nature of the super-resolution problem can cause strong variations in the output even for very subtle changes in the input.

Original: This results in signiﬁcant temporal artifacts that manifest themselves as ﬂickering.
Revised: This results in signiﬁcant temporal artifacts that manifest themselves as ﬂickering.

Original: In order to stabilize the output while at the same time keeping the network structure as small and simple as possible, we propose the following training setup.
Revised: In order to stabilize the output while at the same time keeping the network structure as small and simple as possible, we propose the following training setup.

Original: Given a sequence of high resolution point clouds Y t , with t indicating time, we can compute a velocity
Revised: Given a sequence of high resolution point clouds Y t , with t indicating time, we can compute a velocity

Original: V t = { v t 1 , v t 2 , ..., v tk } , where v ti ∈ R 3 .
Revised: V t = { v t 1 , v t 2 , ..., v tk } , where v ti ∈ R 3 .

Original: For this we use a ﬁnite difference ( y t
Revised: For this we use a ﬁnite difference ( y t

Original: +1 i − y ti ) , where we assume, without loss of generality, ∆ t = 1 , i.e. the time step is normalized to one.
Revised: +1 i − y t i ) , where we assume, without loss of generality, ∆ t = 1 , i.e. the time step is normalized to one.

Original: For training, the low resolution inputs X can now be generated from Y via down-sampling by a factor of r , which yields a subset of points with velocities.
Revised: For training, the low resolution inputs X can now be generated from Y via down-sampling by a factor of r , which yields a subset of points with velocities.

Original: Details of our data generation process will be given below.
Revised: Details of our data generation process will be given below.

Original: To train a temporally coherent network with the Y t sequences, we employ a Siamese setup shown in
Revised: To train a temporally coherent network with the Y t sequences, we employ a Siamese setup shown in

Original: Figure 2b.
Revised: Figure 2b.

Original: We evaluate the network several times (3 times in practice) with the same set of weights, and moving inputs, in order to enforce the output to behave consistently.
Revised: We evaluate the network several times (3 times in practice) with the same set of weights, and moving inputs, in order to enforce the output to behave consistently.

Original: In this way we avoid recurrent architectures that would have to process the high resolution outputs multiple times.
Revised: In this way we avoid recurrent architectures that would have to process the high resolution outputs multiple times.

Original: In addition, we can compute temporal derivatives from the input points, and use them to control the behavior of the generated output.[Figure]
Revised: In addition, we can compute temporal derivatives from the input points, and use them to control the behavior of the generated output.

Original: Sheet [Figure] Under the assumption of slowly moving inputs, which theoretically could be ensured for training, a straightforward way to enforce temporal coherence would be to minimize the generated positions over consecutive time steps in terms of an L 2 norm:
Revised: Under the assumption of slowly moving inputs, which theoretically could be ensured for training, a straightforward way to enforce temporal coherence would be to minimize the generated positions over consecutive time steps in terms of an L 2 norm:

Original: [Equation]
Revised: [Equation]

Original: While this reduces ﬂickering, it does not constrain the change of velocities, i.e., the acceleration.
Revised: While this reduces ﬂickering, it does not constrain the change of velocities, i.e., the acceleration.

Original: This results in a high frequency jittering of the generated point positions.
Revised: This results in a high frequency jittering of the generated point positions.

Original: The jitter can be reduced by also including the previous state at time step t − 1 to constrain the acceleration in terms of its L 2 norm:
Revised: The jitter can be reduced by also including the previous state at time step t − 1 to constrain the acceleration in terms of its L 2 norm:

Original: [Equation]
Revised: [Equation]

Original: However, a central problem of a direct temporal constraint via Equations (2) and (3) is that it consistently leads to a highly undesirable clustering of generated points around the center point.
Revised: However, a central problem of a direct temporal constraint via Equations (2) and (3) is that it consistently leads to a highly undesirable clustering of generated points around the center point.

Original: This is caused by the fact that the training procedure as described so far is unbalanced, as it only encourages minimizing changes.
Revised: This is caused by the fact that the training procedure as described so far is unbalanced, as it only encourages minimizing changes.

Original: The network cannot learn to reconstruct realistic, larger motions in this way, but rather.
Revised: The network cannot learn to reconstruct realistic, larger motions in this way, but rather.

Original: can trivially minimize the loss by contracting all outputs to a single point.
Revised: can trivially minimize the loss by contracting all outputs to a single point.

Original: For this reason, we instead use the estimated velocity of the ground truth point cloud sequence with a forward difference in time, to provide the network with a reference.
Revised: For this reason, we instead use the estimated velocity of the ground truth point cloud sequence with a forward difference in time, to provide the network with a reference.

Original: By using the EMD-based mapping φ established for the spatial loss in Equation (1), we can formulate the temporal constraint in a permutation invariant manner as
Revised: By using the EMD-based mapping φ established for the spatial loss in Equation (1), we can formulate the temporal constraint in a permutation invariant manner as

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this means the generated outputs should mimic the motion of the closest ground truth points.
Revised: Intuitively, this means the generated outputs should mimic the motion of the closest ground truth points.

Original: As detailed for the L 2 -based approaches above, it makes sense to also take the ground truth acceleration into account to minimize rapid changes of velocity over time.
Revised: As detailed for the L 2 -based approaches above, it makes sense to also take the ground truth acceleration into account to minimize rapid changes of velocity over time.

Original: We can likewise formulate this in a permutation invariant way w.r.t.
Revised: We can likewise formulate this in a permutation invariant way w.r.t.

Original: ground truth points via:
Revised: ground truth points via:

Original: [Equation]
Revised: [Equation]

Original: We found that a combination of L EV and L EA together with the spatial loss L S from Eq.
Revised: We found that a combination of L EV and L EA together with the spatial loss L S from Eq.

Original: 1 provides the best results, as we will demonstrate below.
Revised: 1 provides the best results, as we will demonstrate below.

Original: First, we will introduce the additional loss terms of our algorithm.
Revised: First, we will introduce the additional loss terms of our algorithm.

Original: 3. V ARIABLE P OINT C LOUD S IZES
Revised: 3. V ARIABLE P OINT C LOUD S IZES

Original: Existing network architectures are typically designed for processing a ﬁxed amount of input and output points.
Revised: Existing network architectures are typically designed for processing a ﬁxed amount of input and output points.

Original: However, in many cases, and especially for a localized inference of super-resolution, the number of input and output points varies signiﬁcantly.
Revised: However, in many cases, and especially for a localized inference of super-resolution, the number of input and output points varies signiﬁcantly.

Original: While we can safely assume that no patch exceeds the maximal number of inputs k max (this can be ensured by working on a subset), it can easily happen that a certain spatial region has fewer points.
Revised: While we can safely assume that no patch exceeds the maximal number of inputs k max (this can be ensured by working on a subset), it can easily happen that a certain spatial region has fewer points.

Original: Simply including more distant points could guarantee that we have a full set of samples, but this would mean the network has to be invariant to scaling, and to produce features at different spatial scales.
Revised: Simply including more distant points could guarantee that we have a full set of samples, but this would mean the network has to be invariant to scaling, and to produce features at different spatial scales.

Original: Instead, we train our ﬁle:///Users/thuerey/LRZ Sync+Share/NeuralParticles/2019_01...
Revised: Instead, we train our [Figure] NeuralParticles [Figure]

Revised: Sheet

Original: 1 of22/01/2019, 11.
Revised: [Figure] Frequency [Figure] the network output, i.e. r times larger than the input.

Revised: The latter deviates from the in (c), but follows its overall structure.

Revised: This is conﬁrmed in (b), which shows a heat map visualization of input vs. ground output size.

Revised: The diagonal structure of the peak conﬁrms the approximately linear relationship.

Original: ﬁle:///Users/thuerey/LRZ Sync+Share/NeuralParticles/2019_01...
Revised: ﬁle:///Users/thuerey/LRZ Sync+Share/NeuralParticles/2019_01...

Original: 1 of22/01/2019, 11.
Revised: 1 of22/01/2019, 11.

Original: ﬁle:///Users/thuerey/LRZ Sync+Share/NeuralParticles/2019_01...
Revised: ﬁle:///Users/thuerey/LRZ Sync+Share/NeuralParticles/2019_01...

Original: 1 of22/01/2019, 11.
Revised: 1 of22/01/2019, 11.

Original: ﬁle:///Users/thuerey/LRZ Sync+Share/NeuralParticles/2019_01...
Revised: ﬁle:///Users/thuerey/LRZ Sync+Share/NeuralParticles/2019_01...

Original: [Figure] [Figure]
Revised: 1 of

Original: [Figure]
Revised: [Figure]

Original: network for a ﬁxed spatial size, and ensure that it can process varying numbers of inputs.
Revised: network for a ﬁxed spatial size, and ensure that it can process varying numbers of inputs.

Original: For inputs with fewer than k max points, we pad the input vector to have a ﬁxed size.
Revised: For inputs with fewer than k max points, we pad the input vector to have a ﬁxed size.

Original: Here, we ensure that the padding values are not misinterpreted by the network as being point data.
Revised: Here, we ensure that the padding values are not misinterpreted by the network as being point data.

Original: Therefore, we pad
Revised: Therefore, we pad

Original: X with p ∈ {− 2 } d , which represents a value outside the regular patch coordinate range [ − 1 , 1] : ¯
Revised: X with p ∈ {− 2 } d , which represents a value outside the regular patch coordinate range [ − 1 , 1] : ¯

Original: X = { x 1 , x 2 , ..., x k , p, p, ..., p (cid:124) (cid:123)(cid:122) (cid:125)
Revised: X = { x 1 , x 2 , ..., x k , p, p, ..., p (cid:124) (cid:123)(cid:122) (cid:125)

Original: k max − k } .
Revised: k max − k } .

Original: The ﬁrst convolutional layer in our network now ﬁlters out the padded entries using the following mask: M in = { m i ∈ [0 ,k ] } = { 1 , 1 , ..., 1 (cid:124) (cid:123)(cid:122) (cid:125) k , 0 , 0 , ..,(cid:124) (cid:123)(cid:122) (cid:125)
Revised: The ﬁrst convolutional layer in our network now ﬁlters out the padded entries using the following mask: M in = { m i ∈ [0 ,k ] } = { 1 , 1 , ..., 1 (cid:124) (cid:123)(cid:122) (cid:125) k , 0 , 0 , ..,(cid:124) (cid:123)(cid:122) (cid:125)

Original: k max − k } .
Revised: k max − k } .

Original: The entries of p allow us to compute the mask on the ﬂy throughout the whole network, without having to pass through k .
Revised: The entries of p allow us to compute the mask on the ﬂy throughout the whole network, without having to pass through k .

Original: For an input of size k , our network has the task to generate ˜ n = rk points.
Revised: For an input of size k , our network has the task to generate ˜ n = rk points.

Original: As the size of the network output is constant with rk max , the outputs are likewise masked with M out to truncate it to length ˜ n for all loss calculations, e.g., the EMD mappings.
Revised: As the size of the network output is constant with rk max , the outputs are likewise masked with M out to truncate it to length ˜ n for all loss calculations, e.g., the EMD mappings.

Original: Thus, as shown in Figure 2a, ˜ n is used to truncate the point cloud ¯ Y =
Revised: Thus, as shown in Figure 2a, ˜ n is used to truncate the point cloud ¯ Y =

Original: { ¯ y 1 , ¯ y 2 , ..., ¯ y n max } via a mask M out to form the ﬁnal output ˜
Revised: { ¯ y 1 , ¯ y 2 , ..., ¯ y n max } via a mask M out to form the ﬁnal output ˜

Original: Y =
Revised: Y =

Original: { ¯ y i | i ∈ [1 , ˜ n ] } .
Revised: { ¯ y i | i ∈ [1 , ˜ n ] } .

Original: Note that in Sec. 3.1, we have for simplicity assumed that n = rk , however, in practice the number of ground truth points n varies.
Revised: Note that in Sec. 3.1, we have for simplicity assumed that n = rk , however, in practice the number of ground truth points n varies.

Original: As such, ˜ n only provides an approximation of the true number of target points in the ground truth data.
Revised: As such, ˜ n only provides an approximation of the true number of target points in the ground truth data.

Original: While the approximation is accurate for planar surfaces and volumes, it is less accurate in the presence of detailed surface structures that are smaller than the spatial frequency of the low-resolution data.
Revised: While the approximation is accurate for planar surfaces and volumes, it is less accurate in the presence of detailed surface structures that are smaller than the spatial frequency of the low-resolution data.

Original: We have analyzed the effect of this approximation in Fig.
Revised: We have analyzed the effect of this approximation in Fig.

Original: The histograms show that the strongly varying output counts are an important factor in practice, and Fig.
Revised: The histograms show that the strongly varying output counts are an important factor in practice, and Fig.

Original: 4 additionally shows the improvement in terms of target shape that results from incorporating variable output sizes.
Revised: 4 additionally shows the improvement in terms of target shape that results from incorporating variable output sizes.

Original: In general, ˜ n provides a good approximation for our data sets.
Revised: In general, ˜ n provides a good approximation for our data sets.

Original: However, as there is a chance to infer an improved estimate of the correct output size based on the input points, we have experimented with training a second network to predict ˜ n in conjunction with a differentiable M out .
Revised: However, as there is a chance to infer an improved estimate of the correct output size based on the input points, we have experimented with training a second network to predict ˜ n in conjunction with a differentiable M out .

Original: While this could be an interesting feature for future applications, we have not found it to signiﬁcantly improve results.
Revised: While this could be an interesting feature for future applications, we have not found it to signiﬁcantly improve results.

Original: As such, the evaluations and results below will use the analytic calculation, i.e., ˜ n = rk .
Revised: As such, the evaluations and results below will use the analytic calculation, i.e., ˜ n = rk .

Original: 3. P REVENTING H ALO A RTIFACTS
Revised: 3. P REVENTING H ALO A RTIFACTS

Original: For each input point the network generates r output points, which can be seen as individual groups g : ψ ( g ) =
Revised: For each input point the network generates r output points, which can be seen as individual groups g : ψ ( g ) =

Original: { ˜ Y i | i ∈
Revised: { ˜ Y i | i ∈

Original: [ rg + 1 , ( r + 1) g ] } .
Revised: [ rg + 1 , ( r + 1) g ] } .

Original: These groups of size r in the output are strongly related to the input points they originate from.
Revised: These groups of size r in the output are strongly related to the input points they originate from.

Original: Networks that focus on maintaining temporal coherence for the dynamically changing output tends to slide into local minima where r output points are attached as a  ﬁxed structure to the input point location.
Revised: Networks that focus on maintaining temporal coherence for the dynamically changing output tends to slide into local minima where r output points are attached as a [Figure] [Table] ﬁxed structure to the input point location.

Original: This manifests itself as visible halo-like structures that move statically with the input.
Revised: This manifests itself as visible halo-like structures that move statically with the input.

Original: Although temporal coherence is good in this case, these cluster-like structures lead to gaps and suboptimal point distributions in the output, particularly over time.
Revised: Although temporal coherence is good in this case, these cluster-like structures lead to gaps and suboptimal point distributions in the output, particularly over time.

Original: These structures can be seen as a form of temporal mode collapse that can be observed in other areas deep learning, such as GANs.
Revised: These structures can be seen as a form of temporal mode collapse that can be observed in other areas deep learning, such as GANs.

Original: To counteract this effect, we introduce an additional mingling loss term to prevent the formation of clusters by pushing the individual points of a group apart:
Revised: To counteract this effect, we introduce an additional mingling loss term to prevent the formation of clusters by pushing the individual points of a group apart:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: L S

Original: L N L M
Revised: [Equation]

Original: L [Table] V
Revised: [Figure]

Original: L [Table]
Revised: [Equation]

Original: L EV

Original: [Table]
Revised: [Figure]

Original: EA 2D Previous

Original: [Table]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Note that in contrast to previously used repulsion losses (Yu et al., 2018a), L M encourages points to globally mix rather than just locally repelling each other.
Revised: Note that in contrast to previously used repulsion losses (Yu et al., 2018a), L M encourages points to globally mix rather than just locally repelling each other.

Original: While a repulsion term can lead to a deterioration of the generated outputs, our formulation preserves spatial structure and temporal coherence while leading to well distributed points, as is illustrated in Fig.
Revised: While a repulsion term can lead to a deterioration of the generated outputs, our formulation preserves spatial structure and temporal coherence while leading to well distributed points, as is illustrated in Fig.

Original: In combination with the spatial and temporal terms from above, this leads to our ﬁnal loss function
Revised: In combination with the spatial and temporal terms from above, this leads to our ﬁnal loss function

Original: L final =
Revised: L final =

Original: L S + γ L EV + µ L EA + ν L M , with weighting terms γ, µ, ν .
Revised: L S + γ L EV + µ L EA + ν L M , with weighting terms γ, µ, ν .

Original:  E VALUATION AND R ESULTS
Revised:  E VALUATION AND R ESULTS

Original: We train our network in a fully supervised manner with simulated data.
Revised: We train our network in a fully supervised manner with simulated data.

Original: To illustrate the effect of our temporal loss functions, we employ it in conjunction with established network architectures from previous work (Qi Charles et al., 2017; Yu et al., 2018a).
Revised: To illustrate the effect of our temporal loss functions, we employ it in conjunction with established network architectures from previous work (Qi Charles et al., 2017; Yu et al., 2018a).

Original: Details of the data generation and network architectures are given in the appendix.
Revised: Details of the data generation and network architectures are given in the appendix.

Original: We ﬁrst discuss our data generation and training setup, then illustrate the effects of the different terms of our loss function, before showing results for more complex 3D data sets.
Revised: We ﬁrst discuss our data generation and training setup, then illustrate the effects of the different terms of our loss function, before showing results for more complex 3D data sets.

Original: As our results focus on temporal coherence, which is best seen
Revised: As our results focus on temporal coherence, which is best seen

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: l a t e n t s p ace
Revised: l a t e n t s p ace

Original: [Figure]
Revised: [Figure]

Revised: →

Revised: [Figure]

Original: a m p lit ud 
Revised: a m p lit ud e

Original: [Figure]
Revised: [Figure]

Revised: a m p lit ud e frequency

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original:  r i v a ti 
Revised: 1 s t d e r i v a ti v e

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Figure]

Original: in motion
Revised: e rr o r time

Original: [Table]
Revised: [Table]

Original:  we refer readers to the supplemental video at https://www.dropbox.com/sh/ btrzxavn34qftfe/AAADpIBME0eguA4ew4ylvCX_a?dl=0 in order to fully evaluate the resulting quality.
Revised: in motion we refer readers to the supplemental video at https://www.dropbox.com/sh/ btrzxavn34qftfe/AAADpIBME0eguA4ew4ylvCX_a?dl=0 in order to fully evaluate the resulting quality.

Original: Ablation Study We evaluate the effectiveness of our loss formulation with a two dimensional ablation study.
Revised: Ablation Study We evaluate the effectiveness of our loss formulation with a two dimensional ablation study.

Original: An exemplary patch of this study is shown in Fig.
Revised: An exemplary patch of this study is shown in Fig.

Original: In order to compare our method to previous work [Table] we have trained a previously proposed method for point-cloud super-resolution, the
Revised: In order to compare our method to previous work, we have trained a previously proposed method for point-cloud super-resolution, the

Original: PU-Net (Yu et al., 2018a) which internally uses a PointNet++ (Qi et al., 2017), with our data set, the only difference being that we use zero-padding here.
Revised: PU-Net (Yu et al., 2018a) which internally uses a PointNet++ (Qi et al., 2017), with our data set, the only difference being that we use zero-padding here.

Original: This architecture will be used in the following comparisons to previous work.
Revised: This architecture will be used in the following comparisons to previous work.

Original: Fig.
Revised: Fig.

Original: 6a) shows a result generated with this network.
Revised: 6a) shows a result generated with this network.

Original: As this ﬁgure contains an average of multiple frames to indicate temporal stability, the blurred regions, esp.
Revised: As this ﬁgure contains an average of multiple frames to indicate temporal stability, the blurred regions, esp.

Original: visible on the right side of Fig.
Revised: visible on the right side of Fig.

Original: 6a), indicate erroneous motions in the output.
Revised: 6a), indicate erroneous motions in the output.

Original: For this network the difﬁculties of temporally changing data and varying output sizes additionally lead to a suboptimal approximation of the target points, that is also visible in terms of an increased L S loss in Table 1.
Revised: For this network the difﬁculties of temporally changing data and varying output sizes additionally lead to a suboptimal approximation of the target points, that is also visible in terms of an increased L S loss in Table 1.

Original: While Fig.
Revised: While Fig.

Original: 6b) signiﬁcantly reduces motions, and leads to an improved shape as well as L S loss, its motions are overly constrained.
Revised: 6b) signiﬁcantly reduces motions, and leads to an improved shape as well as L S loss, its motions are overly constrained.

Original: E.g., at the bottom of the shown patch, the generated points should follow the black input points, but in (b) the generated points stay in place.
Revised: E.g., at the bottom of the shown patch, the generated points should follow the black input points, but in (b) the generated points stay in place.

Original: In addition, the lack of permutation invariance leads to an undesirable clustering of generated points in the patch center.
Revised: In addition, the lack of permutation invariance leads to an undesirable clustering of generated points in the patch center.

Original: Both problems are removed with L EV in Fig.
Revised: Both problems are removed with L EV in Fig.

Original: 6c), which still contains small scale jittering motions, unfortunately.
Revised: 6c), which still contains small scale jittering motions, unfortunately.

Original: These are removed by L EA in Fig.
Revised: These are removed by L EA in Fig.

Original: 6d), which shows the result of our full algorithm.
Revised: 6d), which shows the result of our full algorithm.

Original: The success of our approach for dynamic output sizes is also shown in in the L N column of Table 1, which contains an L 2 error w.r.t.
Revised: The success of our approach for dynamic output sizes is also shown in in the L N column of Table 1, which contains an L 2 error w.r.t.

Original: ground truth size of the outputs.
Revised: ground truth size of the outputs.

Original: Temporally Coherent Features A central goal of our work is to enable the learning of features that remain stable over time.
Revised: Temporally Coherent Features A central goal of our work is to enable the learning of features that remain stable over time.

Original: To shed light on how our approach inﬂuences the established latent space, we analyze its content for different inputs.
Revised: To shed light on how our approach inﬂuences the established latent space, we analyze its content for different inputs.

Original: The latent space in our case consists of a 256-dimensional vector that contains the features extracted by the ﬁrst set of layers of our network.
Revised: The latent space in our case consists of a 256-dimensional vector that contains the features extracted by the ﬁrst set of layers of our network.

Original: Fig.
Revised: Fig.

Original: 8 contains a qualitative example for 100 randomly selected patch sequences from our test data set, where we collect input data by following the trajectory of each patch center for 50 time steps to extract coherent data sets.
Revised: 8 contains a qualitative example for 100 randomly selected patch sequences from our test data set, where we collect input data by following the trajectory of each patch center for 50 time steps to extract coherent data sets.

Original: Fig.
Revised: Fig.

Original: 8a) shows the averaged latent space content over time for these sequences.
Revised: 8a) shows the averaged latent space content over time for these sequences.

Original: While the model trained with temporal coherence (green curve) is also visually smoother, the difference becomes clearer when considering temporal frequencies.
Revised: While the model trained with temporal coherence (green curve) is also visually smoother, the difference becomes clearer when considering temporal frequencies.

Original: We measure averaged frequencies of the latent space dimensions over time, as shown in Fig.
Revised: We measure averaged frequencies of the latent space dimensions over time, as shown in Fig.

Original: 8b,c).
Revised: 8b,c).

Original: We quantify the differences by calculating the integral of the frequency spectrum ˜ f , weighted by the frequency x to emphasize high frequencies, i.e, (cid:82) x x · ˜ f
Revised: We quantify the differences by calculating the integral of the frequency spectrum ˜ f , weighted by the frequency x to emphasize high frequencies, i.e, (cid:82) x x · ˜ f

Original: ( x ) dx .
Revised: ( x ) dx .

Original: Hence, small values are better.
Revised: Hence, small values are better.

Original: As shown in Fig. 8b), the version trained without our loss formulations contains signiﬁcantly more high frequency content.
Revised: As shown in Fig. 8b), the version trained without our loss formulations contains signiﬁcantly more high frequency content.

Original: This is also reﬂected in the weighted integrals, which are 36.56 for the method without temporal loss, and 16.98 for the method with temporal loss.
Revised: This is also reﬂected in the weighted integrals, which are 36.56 for the method without temporal loss, and 16.98 for the method with temporal loss.

Original: To verify that our temporal model actually establishes a stable temporal latent space instead of ignoring temporal information altogether, we evaluate the temporal frequencies for the same 100 inputs as above, but with a randomized order over time.
Revised: To verify that our temporal model actually establishes a stable temporal latent space instead of ignoring temporal information altogether, we evaluate the temporal frequencies for the same 100 inputs as above, but with a randomized order over time.

Original: In this case, our model correctly identiﬁes the incoherent inputs, and yields similarly high frequencies as the regular model with 28.44 and 35.24, respectively.
Revised: In this case, our model correctly identiﬁes the incoherent inputs, and yields similarly high frequencies as the regular model with 28.44 and 35.24, respectively.

Original: More details in Appendix C.
Revised: More details in Appendix C.

Original: In addition, we evaluated the changes of generated outputs over time w.r.t.
Revised: In addition, we evaluated the changes of generated outputs over time w.r.t.

Original: ground truth motion.
Revised: ground truth motion.

Original: For this we mapped the generated point clouds ˜
Revised: For this we mapped the generated point clouds ˜

Original: Y t =
Revised: Y t =

Original: { y t 1 , y t 2 , ..., y t ˜ n } for 100 frames to evenly and dense sampled ground-truth points on the original mesh Y t = { y t 1 , y t 2 , ..., y tn } (the moving man shown in
Revised: { y t 1 , y t 2 , ..., y t ˜ n } for 100 frames to evenly and dense sampled ground-truth points on the original mesh Y t = { y t 1 , y t 2 , ..., y tn } (the moving man shown in

Original: Fig. 1).
Revised: Fig. 1).

Original: This gives us a dense correlation between the data and the generated point clouds.
Revised: This gives us a dense correlation between the data and the generated point clouds.

Original: For the mapping we used an assignment based on nearest neighbors.
Revised: For the mapping we used an assignment based on nearest neighbors.

Original: γ : ˜ y
Revised: γ : ˜ y

Original: → y .
Revised: → y .

Original: Using γ we divide ˜
Revised: Using γ we divide ˜

Original: Y t into n subsets ˆ Y i =
Revised: Y t into n subsets ˆ Y i =

Original: { ˜ y
Revised: { ˜ y

Original: j | γ (˜ y j ) =
Revised: j | γ (˜ y j ) =

Original: y i } which correlate with the corresponding ground-truth points.
Revised: y i } which correlate with the corresponding ground-truth points.

Original: For each subset we can now compute the mean position 1 | ˆ Y i | (cid:80) ˆ
Revised: For each subset we can now compute the mean position 1 | ˆ Y i | (cid:80) ˆ

Original: y ∈
Revised: y ∈

Original: ˆ Y
Revised: ˆ Y

Original: i ˆ y and the sample density
Revised: i ˆ y and the sample density

Original: | ˆ Y
Revised: | ˆ Y

Original: i | measured by the number of points assigned to a ground-truth sample position.
Revised: i | measured by the number of points assigned to a ground-truth sample position.

Original: The temporal change of these values are of particular interest.
Revised: The temporal change of these values are of particular interest.

Original: The change of the mean positions should correspond to the ground-truth changes, while the change of the density should be as small as possible.
Revised: The change of the mean positions should correspond to the ground-truth changes, while the change of the density should be as small as possible.

Original: We have evaluated the error of the ﬁrst and second derivative of positions, as well as the ﬁrst and second derivative of density (see Fig.
Revised: We have evaluated the error of the ﬁrst and second derivative of positions, as well as the ﬁrst and second derivative of density (see Fig.

Original: 9 and Table 3).
Revised: 9 and Table 3).

Original: As can be seen from the plots, our method leads to clear improvements for all measured quantities.
Revised: As can be seen from the plots, our method leads to clear improvements for all measured quantities.

Original: The individual spikes that are visible for both versions in the position errors (c,d) most likely correspond to sudden changes of the input motions for which our networks undershoots by producing a smooth version of the motion.
Revised: The individual spikes that are visible for both versions in the position errors (c,d) most likely correspond to sudden changes of the input motions for which our networks undershoots by producing a smooth version of the motion.

Original: [Table]
Revised: [Table]

Original: 3D Results Our patch-based approach currently relies on a decomposition of the input volumes into patches over time, as outlined in Appendix A.
Revised: 3D Results Our patch-based approach currently relies on a decomposition of the input volumes into patches over time, as outlined in Appendix A.

Original: As all of the following results involve temporal data, full sequences are provided in the accompanying video.
Revised: As all of the following results involve temporal data, full sequences are provided in the accompanying video.

Original: We apply our method to several complex 3D models to illustrate its performance.
Revised: We apply our method to several complex 3D models to illustrate its performance.

Original: Fig.
Revised: Fig.

Original: 7 shows the input as well as several frames generated with our method for an animation of a spider.
Revised: 7 shows the input as well as several frames generated with our method for an animation of a spider.

Original: Our method produces an even and temporally stable reconstruction of the object.
Revised: Our method produces an even and temporally stable reconstruction of the object.

Original: In comparison, Fig.
Revised: In comparison, Fig.

Original: 7b) shows the output from the previous work architecture (Yu et al., 2018a).
Revised: 7b) shows the output from the previous work architecture (Yu et al., 2018a).

Original: It exhibits uneven point distributions and outliers, e.g., above the legs of the spider, in addition to uneven motions.
Revised: It exhibits uneven point distributions and outliers, e.g., above the legs of the spider, in addition to uneven motions.

Original: A second example for a moving human ﬁgure is shown in Fig.
Revised: A second example for a moving human ﬁgure is shown in Fig.

Original: In both examples, our network covers the target shape much more evenly despite using much fewer points, as shown in Table 2.
Revised: In both examples, our network covers the target shape much more evenly despite using much fewer points, as shown in Table 2.

Original: Thanks to the ﬂexible output size of our network, it can adapt to sparsely covered regions by generating correspondingly fewer outputs.
Revised: Thanks to the ﬂexible output size of our network, it can adapt to sparsely covered regions by generating correspondingly fewer outputs.

Original: The previous work architecture, with its ﬁxed output size, needs to concentrate the ﬁxed number of output points within the target shape, leading to an unnecessarily large point count.
Revised: The previous work architecture, with its ﬁxed output size, needs to concentrate the ﬁxed number of output points within the target shape, leading to an unnecessarily large point count.

Original: In order to demonstrate the ﬂexibility of our method, we also apply it to a volumetric moving point cloud obtained from a liquid simulation.
Revised: In order to demonstrate the ﬂexibility of our method, we also apply it to a volumetric moving point cloud obtained from a liquid simulation.

Original: Thanks to the patch-based evaluation of our network, it is agnostic to the overall size of the input volume.
Revised: Thanks to the patch-based evaluation of our network, it is agnostic to the overall size of the input volume.

Original: In this way, it can be used to generate coherent sets with millions of points.
Revised: In this way, it can be used to generate coherent sets with millions of points.

Original: These examples also highlight our method’s capabilities for generalization.
Revised: These examples also highlight our method’s capabilities for generalization.

Original: While the 3D model was only trained on data from physics simulations, as outlined above, it learns stable features that can be ﬂexibly applied to volumetric as well as to surface-based data.
Revised: While the 3D model was only trained on data from physics simulations, as outlined above, it learns stable features that can be ﬂexibly applied to volumetric as well as to surface-based data.

Original: The metrics in Table 1 show that for both 2D and 3D cases, our method leads to signiﬁcantly improved quality, visible in lower loss values for spatial as well as temporal terms.
Revised: The metrics in Table 1 show that for both 2D and 3D cases, our method leads to signiﬁcantly improved quality, visible in lower loss values for spatial as well as temporal terms.

Original: Another interesting ﬁeld of application for our algorithm are physical simulations.
Revised: Another interesting ﬁeld of application for our algorithm are physical simulations.

Original: Complex simulations such as ﬂuids, often employ particle-based representations.
Revised: Complex simulations such as ﬂuids, often employ particle-based representations.

Original: On the one hand, the volume data is much larger than surface-based data, which additionally motivates our dynamic output.
Revised: On the one hand, the volume data is much larger than surface-based data, which additionally motivates our dynamic output.

Original: On the other hand, time stability plays a very important role for physical phenomena.
Revised: On the other hand, time stability plays a very important role for physical phenomena.

Original: Our method produces detailed outputs for liquids, as can be seen in our supplemental video.
Revised: Our method produces detailed outputs for liquids, as can be seen in our supplemental video.

Original: Convergence graphs for the different versions are shown in Fig.
Revised: Convergence graphs for the different versions are shown in Fig.

Original: 12 of the supplemental material.
Revised: 12 of the supplemental material.

Original: These graphs show that our method not only successfully leads to very low errors in terms of temporal coherence, but also improves spatial accuracy.
Revised: These graphs show that our method not only successfully leads to very low errors in terms of temporal coherence, but also improves spatial accuracy.

Original: The ﬁnal values of L S for the 2D case are below 0.
Revised: The ﬁnal values of L S for the 2D case are below 0.

Original: for our algorithm, compared to almost 0.08 for previous work.
Revised: for our algorithm, compared to almost 0.08 for previous work.

Original: For 3D, our approach yields 0.04 on average, in contrast to ca.
Revised: For 3D, our approach yields 0.04 on average, in contrast to ca.

Original: 0.1 for previous work.
Revised: 0.1 for previous work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed a ﬁrst method to infer temporally coherent features for point clouds.
Revised: We have proposed a ﬁrst method to infer temporally coherent features for point clouds.

Original: This is made possible by a combination of a novel loss function for temporal coherence in combination with enabling ﬂexible truncation of the results.
Revised: This is made possible by a combination of a novel loss function for temporal coherence in combination with enabling ﬂexible truncation of the results.

Original: In addition we have shown that it is crucial to prevent static patterns as easy-to-reach local minima for the network, which we avoid with the proposed a mingling loss term.
Revised: In addition we have shown that it is crucial to prevent static patterns as easy-to-reach local minima for the network, which we avoid with the proposed a mingling loss term.

Original: Our super-resolution results above demonstrate that our approach takes an important ﬁrst step towards ﬂexible deep learning methods for dynamic point clouds.
Revised: Our super-resolution results above demonstrate that our approach takes an important ﬁrst step towards ﬂexible deep learning methods for dynamic point clouds.

Original: Looking ahead, our method could also be ﬂexibly combined with other network architectures or could be adopted for other applications.
Revised: Looking ahead, our method could also be ﬂexibly combined with other network architectures or could be adopted for other applications.

Original: Speciﬁcally, a combination with PSGN (Fan et al., 2016) could be used to generate point clouds from image sequences instead of single images.
Revised: Speciﬁcally, a combination with PSGN (Fan et al., 2016) could be used to generate point clouds from image sequences instead of single images.

Original: Other conceivable applications could employ methods like Dahnert et al.
Revised: Other conceivable applications could employ methods like Dahnert et al.

Original: with our approach for generating animated meshes.
Revised: with our approach for generating animated meshes.

Original: Due to the growing popularity and ubiquity of scanning devices it will, e.g., be interesting to investigate classiﬁcation tasks of 3D scans over time as future work.
Revised: Due to the growing popularity and ubiquity of scanning devices it will, e.g., be interesting to investigate classiﬁcation tasks of 3D scans over time as future work.

Original: Apart from that, physical phenomena such as elastic bodies and ﬂuids (Li et al., 2019) can likewise be represented in a
Revised: Apart from that, physical phenomena such as elastic bodies and ﬂuids (Li et al., 2019) can likewise be represented in a

Original: Lagrangian manner, and pose interesting challenges and complex spatio-temporal changes
Revised: Lagrangian manner, and pose interesting challenges and complex spatio-temporal changes.

Original: A T RAINING AND E VALUATION M ODALITIES
Revised: A T RAINING AND E VALUATION M ODALITIES

Original: Data Generation We employ a physical simulation to generate our input and output pairs for training.
Revised: Data Generation We employ a physical simulation to generate our input and output pairs for training.

Original: This has the advantage that it leads a large variety of complex motions, and gives full control of the generation process.
Revised: This has the advantage that it leads a large variety of complex motions, and gives full control of the generation process.

Original: More speciﬁcally, we employ the IISPH (Ihmsen et al., 2014) algorithm, a form of Lagrangian ﬂuid simulator that efﬁciently generates incompressible liquid volumes.
Revised: More speciﬁcally, we employ the IISPH (Ihmsen et al., 2014) algorithm, a form of Lagrangian ﬂuid simulator that efﬁciently generates incompressible liquid volumes.

Original: These simulations also have the advantage that they inherently control the density of the point sampling thanks to their volume conserving properties.
Revised: These simulations also have the advantage that they inherently control the density of the point sampling thanks to their volume conserving properties.

Original: In order to generate input pairs for training, we randomly sample regions near the surface and extract points with a given radius around a central point.
Revised: In order to generate input pairs for training, we randomly sample regions near the surface and extract points with a given radius around a central point.

Original: This represents the high-resolution target.
Revised: This represents the high-resolution target.

Original: To compute the low-resolution input, we downsample the points with a Poisson-disk sampling to compute a point set with the desired larger spacing.
Revised: To compute the low-resolution input, we downsample the points with a Poisson-disk sampling to compute a point set with the desired larger spacing.

Original: In order to prevent aliasing from features below the coarse resolution, we perform a pass of surface fairing and smoothing before downsampling.
Revised: In order to prevent aliasing from features below the coarse resolution, we perform a pass of surface fairing and smoothing before downsampling.

Original: Due to the large number of patches that can be extracted from these simulations, we did not ﬁnd it necessary to additionally augment the generated data sets.
Revised: Due to the large number of patches that can be extracted from these simulations, we did not ﬁnd it necessary to additionally augment the generated data sets.

Original: Examples of the low- and high-resolution pairs are shown in the supplemental material.
Revised: Examples of the low- and high-resolution pairs are shown in the supplemental material.

Original: Below we will demonstrate that models trained with this data can be ﬂexibly applied to moving surface data as well as new liquid conﬁgurations.
Revised: Below we will demonstrate that models trained with this data can be ﬂexibly applied to moving surface data as well as new liquid conﬁgurations.

Original: The surface data is generated from animated triangle meshes that were resampled with bicubic interpolation in order to match a chosen average per-point area.
Revised: The surface data is generated from animated triangle meshes that were resampled with bicubic interpolation in order to match a chosen average per-point area.

Original: This pattern was generated once and then propagated over time with the animation.
Revised: This pattern was generated once and then propagated over time with the animation.

Original: When applying our method to new liquid simulations, we do not perform any downsampling, but rather use all points of a low-resolution simulation directly, as a volumetric re-sampling over time is typically error prone, and gives incoherent low resolution inputs.
Revised: When applying our method to new liquid simulations, we do not perform any downsampling, but rather use all points of a low-resolution simulation directly, as a volumetric re-sampling over time is typically error prone, and gives incoherent low resolution inputs.

Original: Given a moving point cloud, we decompose it into temporally coherent patches in the following manner: We start by sampling points via a Poisson-disk sampling in a narrow band around the surface, e.g., based on a signed distance function computed from the input cloud.
Revised: Given a moving point cloud, we decompose it into temporally coherent patches in the following manner: We start by sampling points via a Poisson-disk sampling in a narrow band around the surface, e.g., based on a signed distance function computed from the input cloud.

Original: These points will persist as patch centers over time, unless they move too close to others, or too far away from the surface, which triggers their deletion.
Revised: These points will persist as patch centers over time, unless they move too close to others, or too far away from the surface, which triggers their deletion.

Original: In addition, we perform several iterations for every new frame to sample new patches for points in the input cloud that are outside all existing patches.
Revised: In addition, we perform several iterations for every new frame to sample new patches for points in the input cloud that are outside all existing patches.

Original: Note that this resampling of patches over time happens instantaneously in our implementation.
Revised: Note that this resampling of patches over time happens instantaneously in our implementation.

Original: While a temporal fading could easily be added, we have opted for employing transitions without fading, in order to show as much of the patch content as possible.
Revised: While a temporal fading could easily be added, we have opted for employing transitions without fading, in order to show as much of the patch content as possible.

Original: Network Architecture and Training Our architecture heavily relies on a form of hierarchical point-based convolutions.
Revised: Network Architecture and Training Our architecture heavily relies on a form of hierarchical point-based convolutions.

Original: I.e., the network extracts features for a subset of the points and their nearest neighbors.
Revised: I.e., the network extracts features for a subset of the points and their nearest neighbors.

Original: For the point convolution, we ﬁrst select a given number of group centers that are evenly distributed points from a given input cloud.
Revised: For the point convolution, we ﬁrst select a given number of group centers that are evenly distributed points from a given input cloud.

Original: For each group center, we then search for a certain number of points within a chosen radius (a fraction of the [-1,1] range).
Revised: For each group center, we then search for a certain number of points within a chosen radius (a fraction of the [-1,1] range).

Original: This motivates our choice for a coordinate far outside the regular range for the padded points from Sec. 3.2.
Revised: This motivates our choice for a coordinate far outside the regular range for the padded points from Sec. 3.2.

Original: They are too far away from all groups by construction, so they are ﬁltered out without any additional overhead.
Revised: They are too far away from all groups by construction, so they are ﬁltered out without any additional overhead.

Original: In this way, both feature extraction and grouping operations work ﬂexibly with the varying input sizes.
Revised: In this way, both feature extraction and grouping operations work ﬂexibly with the varying input sizes.

Original: Each group is then processed by a PointNet-like sub-structure (Qi Charles et al., 2017), yielding one feature vector per group.
Revised: Each group is then processed by a PointNet-like sub-structure (Qi Charles et al., 2017), yielding one feature vector per group.

Original: The result is a set of feature vectors and the associated group position, which can be interpreted as a new point cloud to repeatedly apply a point convolution.
Revised: The result is a set of feature vectors and the associated group position, which can be interpreted as a new point cloud to repeatedly apply a point convolution.

Original: In this way, the network extracts increasingly abstract and global features.
Revised: In this way, the network extracts increasingly abstract and global features.

Original: The last set of features is then interpolated back to the original points of the input.
Revised: The last set of features is then interpolated back to the original points of the input.

Original: Afterwards a sub-pixel convolution layer is used to scale up the point cloud extended with features and ﬁnally the ﬁnal position vectors are generated with the help of two additional shared, fully connected layers.
Revised: Afterwards a sub-pixel convolution layer is used to scale up the point cloud extended with features and ﬁnally the ﬁnal position vectors are generated with the help of two additional shared, fully connected layers.

Original: While we keep the core network architecture unmodiﬁed to allow for comparisons with previous work, an important distinction of our approach is the input and output masking, as described in Sec. 3.2.
Revised: While we keep the core network architecture unmodiﬁed to allow for comparisons with previous work, an important distinction of our approach is the input and output masking, as described in Sec. 3.2.

Original: Our point data was generated with a mean point spacing, i.e., Poisson disk radius, of 0 .
Revised: Our point data was generated with a mean point spacing, i.e., Poisson disk radius, of 0 .

Original: 5 units.
Revised: 5 units.

Original: For the 2D tests, an upscaling factor of r = 9 was used.
Revised: For the 2D tests, an upscaling factor of r = 9 was used.

Original: For this purpose, patches with a diameter of 5 were extracted from the low-resolution data and patches with a diameter of 15 from the high-resolution data.
Revised: For this purpose, patches with a diameter of 5 were extracted from the low-resolution data and patches with a diameter of 15 from the high-resolution data.

Original: We used the thresholds k max = 100 and n max = 900 .
Revised: We used the thresholds k max = 100 and n max = 900 .

Original: For the loss, we used γ = 10 , µ = 10 , and ν = 0 . 001 .
Revised: For the loss, we used γ = 10 , µ = 10 , and ν = 0 . 001 .

Original: The network was trained with 5 epochs for a data set with 185k pairs, and a batch size of 16, the learning rate was 0.001 with a decay of 0.003.
Revised: The network was trained with 5 epochs for a data set with 185k pairs, and a batch size of 16, the learning rate was 0.001 with a decay of 0.003.

Original: For the 3D results below, the scaling factor r was set to 8.
Revised: For the 3D results below, the scaling factor r was set to 8.

Original: The diameter of the patches was 6 for the low-resolution data and 12 for the high-resolution data, with k max = 1280 and n max = 10240 .
Revised: The diameter of the patches was 6 for the low-resolution data and 12 for the high-resolution data, with k max = 1280 and n max = 10240 .

Original: The loss parameters were γ = µ = 5 , with ν = 0 .
Revised: The loss parameters were γ = µ = 5 , with ν = 0 .

Original: Learning rate and decay were the same for training, but instead we used 10 epochs with 54k patches in 3D, and a batch size of 4.
Revised: Learning rate and decay were the same for training, but instead we used 10 epochs with 54k patches in 3D, and a batch size of 4.

Original: B N ETWORK A RCHITECTURE D ETAILS
Revised: B N ETWORK A RCHITECTURE D ETAILS

Original: The input feature vector is processed in the ﬁrst part of our network, which consists of four point convolutions.
Revised: The input feature vector is processed in the ﬁrst part of our network, which consists of four point convolutions.

Original: We use ( n g , r g , [ l 1 , ..., l d ]) to represent a level with n g groups of radius r g and [ l 1 , ..., l d ] the d fully connected layers with the width l i ( i = 1 , ..., d ) .
Revised: We use ( n g , r g , [ l 1 , ..., l d ]) to represent a level with n g groups of radius r g and [ l 1 , ..., l d ] the d fully connected layers with the width l i ( i = 1 , ..., d ) .

Original: The parameters we use are ( k max , 0 . 25 , [32 , 32 , 64]) , ( k max / 2 , 0 . 5 , [64 , 64 , 128]) , ( k max / 4 , 0 . 6 , [128 , 128 , 256]) and ( k max / 8 , 0 .
Revised: The parameters we use are ( k max , 0 . 25 , [32 , 32 , 64]) , ( k max / 2 , 0 . 5 , [64 , 64 , 128]) , ( k max / 4 , 0 . 6 , [128 , 128 , 256]) and ( k max / 8 , 0 .

Original: We then use interpolation layers to distribute the features of each convolution level among the input points.
Revised: We then use interpolation layers to distribute the features of each convolution level among the input points.

Original: In this step, we reduce the output of each convolution layer with one shared, fully-connected layer per level, to a size of 64 and then distribute the features to all points of the input point cloud depending on their position.
Revised: In this step, we reduce the output of each convolution layer with one shared, fully-connected layer per level, to a size of 64 and then distribute the features to all points of the input point cloud depending on their position.

Original: This extends the points of our original point cloud with 256 features.
Revised: This extends the points of our original point cloud with 256 features.

Original: Fig.
Revised: Fig.

Original: 11 shows a visual overview of the data ﬂow in our network.
Revised: 11 shows a visual overview of the data ﬂow in our network.

Original: Afterwards, we process the data in r separate branches consisting of two shared, fully interconnected layers with 256 and 128 nodes.
Revised: Afterwards, we process the data in r separate branches consisting of two shared, fully interconnected layers with 256 and 128 nodes.

Original: The output is then processed with two shared fully connected layers of 64 and 3 nodes.
Revised: The output is then processed with two shared fully connected layers of 64 and 3 nodes.

Original: Finally, we add our resulting data to the input positions that have been repeated r times.
Revised: Finally, we add our resulting data to the input positions that have been repeated r times.

Original: This provides an additional skip connection which leads to slightly more stable results.
Revised: This provides an additional skip connection which leads to slightly more stable results.

Original: All convolution layers and fully interconnected layers use a tanh() activation function.
Revised: All convolution layers and fully interconnected layers use a tanh() activation function.

Original: For the input feature vector, we make use of additional data ﬁelds in conjunction with the point positions.
Revised: For the input feature vector, we make use of additional data ﬁelds in conjunction with the point positions.

Original: Our network also accepts additional features such as velocity, density and pressure of the
Revised: Our network also accepts additional features such as velocity, density and pressure of the

Original: SPH simulations used for data generation.
Revised: SPH simulations used for data generation.

Original: For inputs from other sources, those values could be easily computed with suitable SPH interpolation kernels.
Revised: For inputs from other sources, those values could be easily computed with suitable SPH interpolation kernels.

Original: In practice, we use position, velocity and pressure ﬁelds.
Revised: In practice, we use position, velocity and pressure ﬁelds.

Original: Whereas the ﬁrst two are important (as mentioned in Sec. 3.1), the pressure ﬁelds turned out to have negligible inﬂuence.
Revised: Whereas the ﬁrst two are important (as mentioned in Sec. 3.1), the pressure ﬁelds turned out to have negligible inﬂuence.

Original: C F REQUENCY E VALUATION OF L ATENT S PACE
Revised: C F REQUENCY E VALUATION OF L ATENT S PACE

Original: In this section we give details for the frequency evaluation of Sec. 4.
Revised: In this section we give details for the frequency evaluation of Sec. 4.

Original: In order to measure the stability of the latent space against temporal changes, we evaluated the latent space of our network with and without temporal loss, once for 100 ordered patch sequences and once for 100 un-ordered ones.
Revised: In order to measure the stability of the latent space against temporal changes, we evaluated the latent space of our network with and without temporal loss, once for 100 ordered patch sequences and once for 100 un-ordered ones.

Original: The central latent space of our network consists of the features generated by the point-convolution layers in the ﬁrst part of the network and is 256 dimensional (see Fig. 11).
Revised: The central latent space of our network consists of the features generated by the point-convolution layers in the ﬁrst part of the network and is 256 dimensional (see Fig. 11).

Original: To obtain information about its general behavior, we average the latent space components over all 100 patch sequences, subtract the mean, and normalize the resulting vector w.r.t.
Revised: To obtain information about its general behavior, we average the latent space components over all 100 patch sequences, subtract the mean, and normalize the resulting vector w.r.t.

Original: maximum value for each data set.
Revised: maximum value for each data set.

Original: The result is a time sequence of scalar values representing the mean deviations of the latent space.
Revised: The result is a time sequence of scalar values representing the mean deviations of the latent space.

Original: The Fourier transform of these vectors ˜ f , are shown in Fig. 8, and were used to compute the weighted frequency content (cid:82) x x · ˜ f ( x ) dx .
Revised: The Fourier transform of these vectors ˜ f , are shown in Fig. 8, and were used to compute the weighted frequency content (cid:82) x x · ˜ f ( x ) dx .

Original: Here, large values indicate strong temporal changes of the latent space dimensions.
Revised: Here, large values indicate strong temporal changes of the latent space dimensions.

Original: The resulting values are given in the main document, and highlight the stability of the latent space learned by our method.
Revised: The resulting values are given in the main document, and highlight the stability of the latent space learned by our method.

Original: D T RAINING D ATA AND G RAPHS
Revised: D T RAINING D ATA AND G RAPHS

Original: Two examples with ground truth points and down-sampled input versions are shown in Fig.
Revised: Two examples with ground truth points and down-sampled input versions are shown in Fig.

Original: Additionally, Fig.
Revised: Additionally, Fig.

Original: 12 shows loss graphs for the different versions shown in the main text: 2D previous work, our full algorithm in 2D, as well as both cases for 3D.

Original: The mingling loss L M is only shown as reference for the previous work versions, but indicates the strong halo-like patterns forming for the architectures based on previous work.

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: interpolate

Original: [Figure]
Revised: [Figure]

Original: 128 ×  128 ×  128 ×  128 × r branches
Revised: 128 ×  128 ×  128 ×  128 × r branches

Original: [Figure]
Revised: [Figure]

Original: PointNet PointNet
Revised: PointNet PointNet

Original: PointNet
Revised: PointNet

Original: MLP MLP
Revised: MLP MLP

Original: MLP k max) ( k max) ( k max) ( k max) ( k max) ( k max) ( k max) ( , 64 ) ( ,,,,, 64 , 64 ,
Revised: MLP k max) ( k max) ( k max) ( k max) ( k max) ( k max) ( k max) ( , 64 ) ( ,,,,, 64 , 64 ,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Table]
Revised: [Table]