Original: PSTN ET : P OINT S PATIO -T EMPORAL C ONVOLUTION ON P OINT C LOUD S EQUENCES
Revised: PSTN ET : P OINT S PATIO -T EMPORAL C ONVOLUTION ON P OINT C LOUD S EQUENCES

Original: A BSTRACT
Revised: A BSTRACT

Original: Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension.
Revised: Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension.

Original: Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences.
Revised: Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences.

Original: In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences.
Revised: In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences.

Original: The proposed PST convolution ﬁrst disentangles space and time in point cloud sequences.
Revised: The proposed PST convolution ﬁrst disentangles space and time in point cloud sequences.

Original: Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension.
Revised: Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension.

Original: Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner.
Revised: Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner.

Original: Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.
Revised: Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern robotic and automatic driving systems usually employ real-time depth sensors, such as
Revised: Modern robotic and automatic driving systems usually employ real-time depth sensors, such as

Original: LiDAR, to capture the geometric information of scenes accurately while being robust to different lighting conditions.
Revised: LiDAR, to capture the geometric information of scenes accurately while being robust to different lighting conditions.

Original: A scene geometry is thus represented by a 3D point cloud, i.e ., a set of measured point coordinates { ( x, y, z ) } .
Revised: A scene geometry is thus represented by a 3D point cloud, i.e ., a set of measured point coordinates { ( x, y, z ) } .

Original: Moreover, when RGB images are available, they are often used as additional features associated with the 3D points to enhance the discriminativeness of point clouds.
Revised: Moreover, when RGB images are available, they are often used as additional features associated with the 3D points to enhance the discriminativeness of point clouds.

Original: However, unlike conventional grid based videos, dynamic point clouds are irregular and unordered in the spatial dimension while points are not consistent and even ﬂow in and out over time.
Revised: However, unlike conventional grid based videos, dynamic point clouds are irregular and unordered in the spatial dimension while points are not consistent and even ﬂow in and out over time.

Original: Therefore, existing 3D convolutions on grid based videos (Tran et al., 2015; Carreira & Zisserman, 2017; Hara et al., 2018) are not suitable to model raw point cloud sequences, as shown in Fig.
Revised: Therefore, existing 3D convolutions on grid based videos (Tran et al., 2015; Carreira & Zisserman, 2017; Hara et al., 2018) are not suitable to model raw point cloud sequences, as shown in Fig.

Original: To model the dynamics of point clouds, one solution is converting point clouds to a sequence of 3D voxels, and then applying 4D convolutions (Choy et al., 2019) to the voxel sequence.
Revised: To model the dynamics of point clouds, one solution is converting point clouds to a sequence of 3D voxels, and then applying 4D convolutions (Choy et al., 2019) to the voxel sequence.

Original: However, directly performing convolutions on voxel sequences require a large amount of computation.
Revised: However, directly performing convolutions on voxel sequences require a large amount of computation.

Original: Furthermore, quantization errors are inevitable during voxelization, which may restrict applications that require precise measurement of scene geometry.
Revised: Furthermore, quantization errors are inevitable during voxelization, which may restrict applications that require precise measurement of scene geometry.

Original: Another solution MeteorNet (Liu et al., 2019e) is extending the static point cloud method PointNet++ (Qi et al., 2017b) to process raw point cloud sequences by appending 1D temporal dimension to 3D points.
Revised: Another solution MeteorNet (Liu et al., 2019e) is extending the static point cloud method PointNet++ (Qi et al., 2017b) to process raw point cloud sequences by appending 1D temporal dimension to 3D points.

Original: However, simply concatenating coordinates and time together and treating point cloud sequences as unordered 4D point sets neglect the temporal order of timestamps, which may not properly exploit the temporal information and lead to inferior performance.
Revised: However, simply concatenating coordinates and time together and treating point cloud sequences as unordered 4D point sets neglect the temporal order of timestamps, which may not properly exploit the temporal information and lead to inferior performance.

Original: Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible.
Revised: Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible.

Original: Treating them equally is not in favor of network optimization.
Revised: Treating them equally is not conducive for network optimization.

Original: Besides,
Revised: Besides,

Original: MeteorNet only considers spatial neighbors and neglects the local dependencies of neighboring frames.
Revised: MeteorNet only considers spatial neighbors and neglects the local dependencies of neighboring frames.

Original: With the use of whole sequence length as its temporal receptive ﬁeld, MeteorNet cannot construct temporal hierarchy.
Revised: With the use of whole sequence length as its temporal receptive ﬁeld, MeteorNet cannot construct temporal hierarchy.

Original: As points are not consistent and even ﬂow in and out of the region, especially for long sequences and fast motion embedding points in a spatially local area along an entire sequence handicaps capturing accurate local dynamics of point clouds.
Revised: As points are not consistent and even ﬂow in and out of the region, especially for long sequences and fast motion embedding points in a spatially local area along an entire sequence handicaps capturing accurate local dynamics of point clouds.

Original: In this paper, we propose a point spatio-temporal (PST) convolution to directly process raw point cloud sequences.
Revised: In this paper, we propose a point spatio-temporal (PST) convolution to directly process raw point cloud sequences.

Original: As dynamic point clouds are spatially irregular but ordered in the temporal dimension, we
Revised: As dynamic point clouds are spatially irregular but ordered in the temporal dimension, we

Original: 𝐶 feature pixel 𝐶 𝐶 ′
Revised: 𝐶 feature pixel 𝐶 𝐶 ′

Original: 𝐻 𝐿
Revised: 𝐻 𝐿

Original: 𝑊 𝐿 ′ 𝐻 ′
Revised: 𝑊 𝐿 ′ 𝐻 ′

Original: 𝑊 ′ 𝐿
Revised: 𝑊 ′ 𝐿

Original: 𝑁 coordinate 𝑁 ′ 𝐿 ′ 𝐶 ′
Revised: 𝑁 coordinate 𝑁 ′ 𝐿 ′ 𝐶 ′

Original: (a)Conventional Grid based Video Encoding (b)Point Cloud Sequence Encoding 𝑎 𝑏 𝑐 𝑑
Revised: (a)Conventional Grid based Video Encoding (b)Point Cloud Sequence Encoding 𝑎 𝑏 𝑐 𝑑

Original: 𝑑 𝑎 𝑏 𝑐 𝑏 𝑑 𝑐 𝑒 𝑒 𝑐 𝑓 𝑑 𝑎 𝑐 𝑑 𝑓 𝑒 𝑑 decouple the spatial and temporal information to model point cloud sequences.
Revised: 𝑑 𝑎 𝑏 𝑐 𝑏 𝑑 𝑐 𝑒 𝑒 𝑐 𝑓 𝑑 𝑎 𝑐 𝑑 𝑓 𝑒 𝑑 decouple the spatial and temporal information to model point cloud sequences.

Original: Speciﬁcally, PST convolution consists of (i) a point based spatial convolution that models the spatial structure of 3D points and (ii) a temporal convolution that captures the temporal dynamics of point cloud sequences.
Revised: Speciﬁcally, PST convolution consists of (i) a point based spatial convolution that models the spatial structure of 3D points and (ii) a temporal convolution that captures the temporal dynamics of point cloud sequences.

Original: In this fashion
Revised: In this fashion

Original: PST convolution signiﬁcantly facilitates the modeling of dynamic point clouds and reduces the impact of the spatial irregularity of points on temporal modeling.
Revised: PST convolution signiﬁcantly facilitates the modeling of dynamic point clouds and reduces the impact of the spatial irregularity of points on temporal modeling.

Original: Because point cloud sequences emerge inconsistently across frames, it is challenging to perform convolution on them.
Revised: Because point cloud sequences emerge inconsistently across frames, it is challenging to perform convolution on them.

Original: To address this problem, we introduce a point tube to preserve the spatio-temporal local structure.
Revised: To address this problem, we introduce a point tube to preserve the spatio-temporal local structure.

Original: To enhance the feature extraction ability, we incorporate the proposed PST convolution into a spatiotemporally hierarchical network, namely PSTNet.
Revised: To enhance the feature extraction ability, we incorporate the proposed PST convolution into a spatiotemporally hierarchical network, namely PSTNet.

Original: Moreover, we extend our PST convolution to a transposed version to address point-level prediction tasks.
Revised: Moreover, we extend our PST convolution to a transposed version to address point-level prediction tasks.

Original: Different from the convolutional version, the PST transposed convolution is designed to interpolate temporal dynamics and spatial features.
Revised: Different from the convolutional version, the PST transposed convolution is designed to interpolate temporal dynamics and spatial features.

Original: Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of the proposed PST convolution and the superiority of PSTNet in modeling point clouds sequences.
Revised: Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of the proposed PST convolution and the superiority of PSTNet in modeling point clouds sequences.

Original: The contributions of this paper are fourfold:
Revised: The contributions of this paper are fourfold:

Original: • To the best of our knowledge, we are the ﬁrst attempt to decompose spatial and temporal information in modeling raw point cloud sequences, and propose a generic point based convolutional operation, named PST convolution, to encode raw point cloud sequences.
Revised: • To the best of our knowledge, we are the ﬁrst attempt to decompose spatial and temporal information in modeling raw point cloud sequences, and propose a generic point based convolutional operation, named PST convolution, to encode raw point cloud sequences.

Original: • We propose a PST transposed convolution to decode raw point cloud sequences via interpolating the temporal dynamics and spatial feature for point-level prediction tasks.
Revised: • We propose a PST transposed convolution to decode raw point cloud sequences via interpolating the temporal dynamics and spatial feature for point-level prediction tasks.

Original: • We construct convolutional neural networks based on the PST convolutions and transposed convolutions, dubbed PSTNet, to tackle sequence-level classiﬁcation and point-level prediction tasks.
Revised: • We construct convolutional neural networks based on the PST convolutions and transposed convolutions, dubbed PSTNet, to tackle sequence-level classiﬁcation and point-level prediction tasks.

Original: To the best of our knowledge, our PSTNet is the ﬁrst deep neural network to model raw point cloud sequences in a both spatially and temporally hierarchical manner.
Revised: To the best of our knowledge, our PSTNet is the ﬁrst deep neural network to model raw point cloud sequences in a both spatially and temporally hierarchical manner.

Original: • Extensive experiments on four datasets indicate that our method improves the accuracy of 3D action recognition and 4D semantic segmentation.
Revised: • Extensive experiments on four datasets indicate that our method improves the accuracy of 3D action recognition and 4D semantic segmentation.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Learning Representations on Grid based Videos.
Revised: Learning Representations on Grid based Videos.

Original: Impressive progress has been made on generating compact and discriminative representations for RGB/RGBD videos due to the success of deep neural networks.
Revised: Impressive progress has been made on generating compact and discriminative representations for RGB/RGBD videos due to the success of deep neural networks.

Original: For example, two-stream convolutional neural networks (Simonyan & Zisserman, 2014; Wang et al., 2016) utilize a spatial stream and an optical ﬂow stream for video modeling.
Revised: For example, two-stream convolutional neural networks (Simonyan & Zisserman, 2014; Wang et al., 2016) utilize a spatial stream and an optical ﬂow stream for video modeling.

Original: To summarize the temporal dependencies of videos, recurrent neural networks (Ng et al., 2015; Fan et al., 2018) and pooling techniques (Fan et al., 2017) are employed.
Revised: To summarize the temporal dependencies of videos, recurrent neural networks (Ng et al., 2015; Fan et al., 2018) and pooling techniques (Fan et al., 2017) are employed.

Original: In addition, by stacking multiple 2D frames into a 3D tensor, 3D convolutional neural networks (Tran et al., 2015; Carreira & Zisserman, 2017; Tran et al., 2018; Hara et al., 2018) are widely used to learn spatio-temporal representations for videos, and achieve promising performance.
Revised: In addition, by stacking multiple 2D frames into a 3D tensor, 3D convolutional neural networks (Tran et al., 2015; Carreira & Zisserman, 2017; Tran et al., 2018; Hara et al., 2018) are widely used to learn spatio-temporal representations for videos, and achieve promising performance.

Original: Besides, interpretable video or action reasoning methods (Zhuo et al., 2019; Fan et al., 2021b) are proposed by explicitly parsing changes in videos.
Revised: Besides, interpretable video or action reasoning methods (Zhuo et al., 2019; Fan et al., 2021b) are proposed by explicitly parsing changes in videos.

Original: Static Point Cloud Processing.
Revised: Static Point Cloud Processing.

Original: Static point cloud analysis has been widely investigated for many problems, such as classiﬁcation, object part segmentation,
Revised: Static point cloud analysis has been widely investigated for many problems, such as classiﬁcation, object part segmentation,

Original: scene semantic segmentation (Qi et al., 2017a;b; Li et al., 2018b; Wu et al., 2019; Thomas et al., 2019; Wang et al., 2019), reconstruction (Dai et al., 2017; Yu et al., 2018) and object detection (Chen et al., 2017; Qi et al., 2019).
Revised: scene semantic segmentation (Qi et al., 2017a;b; Li et al., 2018b; Wu et al., 2019; Thomas et al., 2019; Wang et al., 2019), reconstruction (Dai et al., 2017; Yu et al., 2018) and object detection (Chen et al., 2017; Qi et al., 2019).

Original: Most recent works aim to directly manipulate point sets without transforming coordinates into regular voxel grids.
Revised: Most recent works aim to directly manipulate point sets without transforming coordinates into regular voxel grids.

Original: Since a point cloud is essentially a set of unordered points and invariant to permutations of its points, static point cloud processing methods mainly focus on designing effective point based spatial correlation operations that do not rely on point orderings.
Revised: Since a point cloud is essentially a set of unordered points and invariant to permutations of its points, static point cloud processing methods mainly focus on designing effective point based spatial correlation operations that do not rely on point orderings.

Original: Dynamic Point Cloud Modeling.
Revised: Dynamic Point Cloud Modeling.

Original: Compared with static point cloud processing, dynamic point cloud modeling is a fairly new task.
Revised: Compared with static point cloud processing, dynamic point cloud modeling is a fairly new task.

Original: Fast and Furious (FaF)
Revised: Fast and Furious (FaF)

Original: (Luo et al., 2018) converts a point cloud frame into a bird’s view voxel and then extracts features via 3D convolutions.
Revised: (Luo et al., 2018) converts a point cloud frame into a bird’s view voxel and then extracts features via 3D convolutions.

Original: MinkowskiNet (Choy et al., 2019) converts a point cloud sequence into a 4D occupancy grid and then applies 4D Spatio-Temporal
Revised: MinkowskiNet (Choy et al., 2019) converts a point cloud sequence into a 4D occupancy grid and then applies 4D Spatio-Temporal

Original: ConvNets.
Revised: ConvNets.

Original: PointRNN (Fan & Yang, 2019) leverages point based recurrent neural networks for raw point cloud sequence forecasting.
Revised: PointRNN (Fan & Yang, 2019) leverages point based recurrent neural networks for raw point cloud sequence forecasting.

Original: MeteorNet (Liu et al., 2019e) extends 3D points to 4D points and then appends a temporal dimension to PointNet++ to process these 4D points.
Revised: MeteorNet (Liu et al., 2019e) extends 3D points to 4D points and then appends a temporal dimension to PointNet++ to process these 4D points.

Original: 3DV (Wang et al., 2020)
Revised: 3DV (Wang et al., 2020)

Original: ﬁrst integrates 3D motion information into a regular compact voxel set and then applies
Revised: ﬁrst integrates 3D motion information into a regular compact voxel set and then applies

Original: PointNet++ to extract representations from the set for 3D action recognition.
Revised: PointNet++ to extract representations from the set for 3D action recognition.

Original: P4Transformer
Revised: P4Transformer

Original: (Fan et al., 2021a) employs transformer to avoid point tracking for raw point cloud sequence modeling.
Revised: (Fan et al., 2021a) employs transformer to avoid point tracking for raw point cloud sequence modeling.

Original: Niemeyer et al.
Revised: Niemeyer et al.

Original: (2019) learned a temporal and spatial vector ﬁeld for 4D reconstruction.
Revised: (2019) learned a temporal and spatial vector ﬁeld for 4D reconstruction.

Original: Prantl et al.
Revised: Prantl et al.

Original: (2020) learned stable and temporal coherent feature spaces for point based super-resolution.
Revised: (2020) learned stable and temporal coherent feature spaces for point based super-resolution.

Original: CaSPR (Rempe et al., 2020) learns to encode spatio-temporal changes in object shape from point clouds for reconstruction and camera pose estimation.
Revised: CaSPR (Rempe et al., 2020) learns to encode spatio-temporal changes in object shape from point clouds for reconstruction and camera pose estimation.

Original: In this paper, we propose a point based convolution to model raw point cloud sequences in a spatio-temporally hierarchical manner.
Revised: In this paper, we propose a point based convolution to model raw point cloud sequences in a spatio-temporally hierarchical manner.

Original:  P ROPOSED P OINT S PATIO -T EMPORAL C ONVOLUTIONAL N ETWORK
Revised:  P ROPOSED P OINT S PATIO -T EMPORAL C ONVOLUTIONAL N ETWORK

Original: In this section, we ﬁrst brieﬂy review the grid based 3D convolution operation as our point spatiotemporal (PST) convolution is motivated by it.
Revised: In this section, we ﬁrst brieﬂy review the grid based 3D convolution operation as our point spatiotemporal (PST) convolution is motivated by it.

Original: Then, we introduce how the proposed PST convolution extracts features from dynamic point clouds in detail.
Revised: Then, we introduce how the proposed PST convolution extracts features from dynamic point clouds in detail.

Original: In order to address dense point prediction tasks, i.e ., semantic segmentation, we develop a PST transposed convolution.
Revised: In order to address dense point prediction tasks, i.e ., semantic segmentation, we develop a PST transposed convolution.

Original: Finally, we incorporate our operations into deep hierarchical networks to address different dynamic point cloud tasks.
Revised: Finally, we incorporate our operations into deep hierarchical networks to address different dynamic point cloud tasks.

Original: 3. M OTIVATION FROM G RID BASED 3D C ONVOLUTION
Revised: 3. M OTIVATION FROM G RID BASED 3D C ONVOLUTION

Original: The power of convolutional neural network (CNNs) comes from local structure modeling by convolutions and global representation learning via hierarchical architectures.
Revised: The power of convolutional neural network (CNNs) comes from local structure modeling by convolutions and global representation learning via hierarchical architectures.

Original: Given an input feature map F ∈ R C × L × H × W , where C , L , H and W denote the input feature dimension, length, height and width, 3D convolution is designed to capture the spatio-temporal local structure of F , written as:
Revised: Given an input feature map F ∈ R C × L × H × W , where C , L , H and W denote the input feature dimension, length, height and width, 3D convolution is designed to capture the spatio-temporal local structure of F , written as:

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R C (cid:48)
Revised: where W ∈ R C (cid:48)

Original: × C × l × h × w is the convolution kernel, ( l, h, w ) is the kernel size, C (cid:48) represents the output feature dimension and · is matrix multiplication.
Revised: × C × l × h × w is the convolution kernel, ( l, h, w ) is the kernel size, C (cid:48) represents the output feature dimension and · is matrix multiplication.

Original: The W ( i,j ) k ∈ R C (cid:48)
Revised: The W ( i,j ) k ∈ R C (cid:48)

Original: × C denotes the weight at kernel position ( k, i, j ) and F ( x + i,y + j ) t + k ∈ R C × 1 denotes the feature of pixel at input position ( t + k, x
Revised: × C denotes the weight at kernel position ( k, i, j ) and F ( x + i,y + j ) t + k ∈ R C × 1 denotes the feature of pixel at input position ( t + k, x

Original: + i, y + j ) .
Revised: + i, y + j ) .

Original: Usually, CNNs employ small kernel sizes, e.g ., (3 , 3 , 3) , which are much smaller than the input size, to effectively model the local structure.
Revised: Usually, CNNs employ small kernel sizes, e.g ., (3 , 3 , 3) , which are much smaller than the input size, to effectively model the local structure.

Original: To construct hierarchical CNNs, stride ( > 1 ) is used to subsample input feature maps during the convolution operation.
Revised: To construct hierarchical CNNs, stride ( > 1 ) is used to subsample input feature maps during the convolution operation.

Original: In this fashion, following convolutional layers will have relatively larger receptive ﬁelds.
Revised: In this fashion, following convolutional layers will have relatively larger receptive ﬁelds.

Original: 3. P OINT S PATIO -T EMPORAL (PST) C ONVOLUTION
Revised: 3. P OINT S PATIO -T EMPORAL (PST) C ONVOLUTION

Original: Let P t ∈ R 3 × N and F t ∈ R C × N denote the point coordinates and features of the t -th frame in a point cloud sequence, where N and C denote the number of points and feature channels, respectively.
Revised: Let P t ∈ R 3 × N and F t ∈ R C × N denote the point coordinates and features of the t -th frame in a point cloud sequence, where N and C denote the number of points and feature channels, respectively.

Original: Given a point cloud sequence (cid:0) [ P 1 ; F 1 ] , [ P 2 ; F 2 ] , · · · , [ P L ; F L ] (cid:1) , the proposed PST convolution will encode the sequence to (cid:0) [ P (cid:48) 1 ; F (cid:48) 1 ] , [ P (cid:48) 2 ; F (cid:48) 2 ] , · · · , [ P (cid:48)
Revised: Given a point cloud sequence (cid:0) [ P 1 ; F 1 ] , [ P 2 ; F 2 ] , · · · , [ P L ; F L ] (cid:1) , the proposed PST convolution will encode the sequence to (cid:0) [ P (cid:48) 1 ; F (cid:48) 1 ] , [ P (cid:48) 2 ; F (cid:48) 2 ] , · · · , [ P (cid:48)

Original: L (cid:48) ; F (cid:48)
Revised: L (cid:48) ; F (cid:48)

Original: L (cid:48) ] (cid:1) , where L and L (cid:48) indicate the number of frames, and P (cid:48)
Revised: L (cid:48) ] (cid:1) , where L and L (cid:48) indicate the number of frames, and P (cid:48)

Original: t ∈ R 3 × N (cid:48) and F (cid:48) t ∈ R C (cid:48)
Revised: t ∈ R 3 × N (cid:48) and F (cid:48) t ∈ R C (cid:48)

Original: × N (cid:48) represent the encoded coordinates and features.
Revised: × N (cid:48) represent the encoded coordinates and features.

Original: 3.2. D ECOMPOSING S PACE AND T IME IN P OINT C LOUD S EQUENCE M ODELING
Revised: 3.2. D ECOMPOSING S PACE AND T IME IN P OINT C LOUD S EQUENCE M ODELING

Original: Because point clouds are irregular and unordered [Equation] grid based 3D convolution cannot be directly applied to point cloud sequences.
Revised: Because point clouds are irregular and unordered, grid based 3D convolution cannot be directly applied to point cloud sequences.

Original: As point cloud sequences are spatially irregular and unordered but temporally ordered [Equation] this motivates us to decouple these two dimensions in order to reduce the impacts of the spatial irregularity of points on temporal modeling.
Revised: As point cloud sequences are spatially irregular and unordered but temporally ordered, this motivates us to decouple these two dimensions in order to reduce the impacts of the spatial irregularity of points on temporal modeling.

Original: Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible.
Revised: Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible.

Original: Treating them equally is not in favor of network optimization.
Revised: Treating them equally is not conducive for network optimization.

Original: By decoupling the spatio-temporal information in point cloud sequences, we not only make the spatio-temporal modeling easier but also signiﬁcantly improve the ability to capture the temporal information.
Revised: By decoupling the spatio-temporal information in point cloud sequences, we not only make the spatio-temporal modeling easier but also signiﬁcantly improve the ability to capture the temporal information.

Original: Therefore, our PST convolution is formulated as:
Revised: Therefore, our PST convolution is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where ( x, y, z ) ∈ P t , ( δ x , δ y , δ z ) represents displacement and r is spatial search radius.
Revised: where ( x, y, z ) ∈ P t , ( δ x , δ y , δ z ) represents displacement and r is spatial search radius.

Original: The convolution kernel 
Revised: The convolution kernel W ∈

Original: [Equation]
Revised: [Equation]

Original:  l × R is decomposed into a spatial convolution kernel S ∈ R C m × C × l × R and a temporal convolution kernel T ∈ R C (cid:48)
Revised: C × l × R is decomposed into a spatial convolution kernel S ∈ R C m × C × l × R and a temporal convolution kernel T ∈ R C (cid:48)

Original: × C m × l × R and C m is the dimension of the intermediate feature.
Revised: × C m × l × R and C m is the dimension of the intermediate feature.

Original: Moreover, because space and time are orthogonal and independent of each other, we further decompose spatial and temporal modeling as:
Revised: Moreover, because space and time are orthogonal and independent of each other, we further decompose spatial and temporal modeling as:

Original: [Equation]
Revised: [Equation]

Original: where S ∈ R C m × C × R and T ∈ R C (cid:48)
Revised: where S ∈ R C m × C × R and T ∈ R C (cid:48)

Original: × C m × l .
Revised: × C m × l .

Original: The above decomposition can also be expressed as applying temporal convolution ﬁrst and then spatial convolution to input features.
Revised: The above decomposition can also be expressed as applying temporal convolution ﬁrst and then spatial convolution to input features.

Original: However, doing so requires point tracking to capture point motion.
Revised: However, doing so requires point tracking to capture point motion.

Original: Because it is difﬁcult to achieve accurate point trajectory and tracking points usually relies on point colors and may fail to handle colorless point clouds, we opt to model the spatial structure of irregular points ﬁrst, and then capture the temporal information from the spatial regions.
Revised: Because it is difﬁcult to achieve accurate point trajectory and tracking points usually relies on point colors and may fail to handle colorless point clouds, we opt to model the spatial structure of irregular points ﬁrst, and then capture the temporal information from the spatial regions.

Original: In this fashion, Eq.
Revised: In this fashion, Eq.

Original: (3) can be rewritten as:
Revised: (3) can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: The spatial convolution captures spatial local structures of point clouds in 3D space, while the temporal convolution aims to describe the temporal local dynamics of point cloud sequences along the time dimension.
Revised: The spatial convolution captures spatial local structures of point clouds in 3D space, while the temporal convolution aims to describe the temporal local dynamics of point cloud sequences along the time dimension.

Original: In order to capture the distributions of neighboring points, it is required to learn a convolution kernel S for all displacements.
Revised: In order to capture the distributions of neighboring points, it is required to learn a convolution kernel S for all displacements.

Original: However, this is impossible because point displacements are not discrete.
Revised: However, this is impossible because point displacements are not discrete.

Original: To address this problem, we convert the kernel to a function of displacements, deﬁned as:
Revised: To address this problem, we convert the kernel to a function of displacements, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where f : R 1 × 3 → R C m × C is a function of ( δ x , δ y , δ z ) parametrized by θ to generate different
Revised: where f : R 1 × 3 → R C m × C is a function of ( δ x , δ y , δ z ) parametrized by θ to generate different

Original: R C m × C according to different displacements.
Revised: R C m × C according to different displacements.

Original: There can be many ways to implement the f function.
Revised: There can be many ways to implement the f function.

Original: In principle, the f function should be both computation-efﬁcient and memory-efﬁcient so that PST convolution is able to encode long sequences.
Revised: In principle, the f function should be both computation-efﬁcient and memory-efﬁcient so that PST convolution is able to encode long sequences.

Original: Existing static point cloud convolution methods (Li et al., 2018b; Wu et al., 2019; Wang et al., 2019) usually contains multilayer perceptrons (MLPs), which are computation and memory consuming.
Revised: Existing static point cloud convolution methods (Li et al., 2018b; Wu et al., 2019; Wang et al., 2019) usually contains multilayer perceptrons (MLPs), which are computation and memory consuming.

Original: In this paper, we design a lightweight implementation for f , which contains only a few parameters.
Revised: In this paper, we design a lightweight implementation for f , which contains only a few parameters.

Original: Speciﬁcally, we further decompose the f function as f (cid:0) ( δ x , δ y , δ z ); θ (cid:1)
Revised: Speciﬁcally, we further decompose the f function as f (cid:0) ( δ x , δ y , δ z ); θ (cid:1)

Original: =
Revised: =

Original: θ d · ( δ x , δ y , δ z ) T · 1 (cid:12) θ s , where θ =
Revised: θ d · ( δ x , δ y , δ z ) T · 1 (cid:12) θ s , where θ =

Original: [ θ d , θ s ] , θ d ∈ R C m × 3 is a displacement transform kernel, θ s ∈ R C m × C is a sharing kernel, 1 = (1 , · · · , 1) ∈ R 1 × C is for broadcasting, and (cid:12) is element-wise product.
Revised: [ θ d , θ s ] , θ d ∈ R C m × 3 is a displacement transform kernel, θ s ∈ R C m × C is a sharing kernel, 1 = (1 , · · · , 1) ∈ R 1 × C is for broadcasting, and (cid:12) is element-wise product.

Original: The sharing kernel θ s is to increase point feature dimension to improve the feature representation ability.
Revised: The sharing kernel θ s is to increase point feature dimension to improve the feature representation ability.

Original: The displacement kernel θ d is to capture spatial local structure based on displacements.
Revised: The displacement kernel θ d is to capture spatial local structure based on displacements.

Original: In this fashion, f generates a unique spatial kernel for each displacement so that the lightweight spatial convolution is able to increase feature dimension while capturing spatial local structure like conventional convolutions.
Revised: In this fashion, f generates a unique spatial kernel for each displacement so that the lightweight spatial convolution is able to increase feature dimension while capturing spatial local structure like conventional convolutions.

Original: temporal st r ide =temporal st r ide =[𝑃 1 ;𝐹 1 ] [𝑃 2 ;𝐹 2 ] [𝑃 3 ;𝐹 3 ] [𝑃 4 ;𝐹 4 ] [𝑃 5 ;𝐹 5 ] anchors anchors anchors (c
Revised: temporal st r ide =temporal st r ide =[𝑃 1 ;𝐹 1 ] [𝑃 2 ;𝐹 2 ] [𝑃 3 ;𝐹 3 ] [𝑃 4 ;𝐹 4 ] [𝑃 5 ;𝐹 5 ] anchors anchors anchors (c

Original: [Figure]
Revised: [Figure]

Original: 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑀 1𝑎 𝑀 2𝑎 𝑀 1𝑏 𝑀 2𝑏 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝐹 ′1𝑎 𝐹 ′1𝑏 𝑀 2𝑐
Revised: 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑀 1𝑎 𝑀 2𝑎 𝑀 1𝑏 𝑀 2𝑏 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝐹 ′1𝑎 𝐹 ′1𝑏 𝑀 2𝑐

Original: 𝑀 3𝑐
Revised: 𝑀 3𝑐

Original: 𝑀 4𝑐 𝑀 2𝑑 𝑀 3𝑑 𝑀 4𝑑
Revised: 𝑀 4𝑐 𝑀 2𝑑 𝑀 3𝑑 𝑀 4𝑑

Original: 𝐹 ′2𝑐 𝐹 ′2𝑑 𝑀 4𝑒 𝑀 𝟓𝑒 𝑀 4𝑓
Revised: 𝐹 ′2𝑐 𝐹 ′2𝑑 𝑀 4𝑒 𝑀 𝟓𝑒 𝑀 4𝑓

Original: 𝑀 5𝑓 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝐹 ′3𝑒 𝐹 ′3𝑓
Revised: 𝑀 5𝑓 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝑝𝑎𝑑𝑑𝑖𝑛𝑔 𝐹 ′3𝑒 𝐹 ′3𝑓

Original: [Figure]
Revised: [Figure]

Original: 𝑎 )
Revised: 𝑎 )

Original: 3.2. P OINT T UBE
Revised: 3.2. P OINT T UBE

Original: Grid based 3D convolutions can be easily performed on regular conventional videos by sliding along the length, height and width.
Revised: Grid based 3D convolutions can be easily performed on regular conventional videos by sliding along the length, height and width.

Original: Because point cloud sequences are irregular and unordered in 3D space and emerge inconsistently across different frames, it is challenging to perform convolution on them.
Revised: Because point cloud sequences are irregular and unordered in 3D space and emerge inconsistently across different frames, it is challenging to perform convolution on them.

Original: To address this problem, we introduce a point tube to preserve the spatio-temporal local structure.
Revised: To address this problem, we introduce a point tube to preserve the spatio-temporal local structure.

Original: In contrast to pixel cubes in 3D convolution, in which pixels are distributed regularly, point tubes are dynamically generated based on the input sequences so that dense areas have more tubes than sparse ones.
Revised: In contrast to pixel cubes in 3D convolution, in which pixels are distributed regularly, point tubes are dynamically generated based on the input sequences so that dense areas have more tubes than sparse ones.

Original: Speciﬁcally, the point tube is constructed as follows:
Revised: Speciﬁcally, the point tube is constructed as follows:

Original: Temporal anchor frame selection.
Revised: Temporal anchor frame selection.

Original: For an entire sequence of point clouds, we need to select some anchor frames to generate our tubes.
Revised: For an entire sequence of point clouds, we need to select some anchor frames to generate our tubes.

Original: Temporal anchor frames in a point cloud sequence are automatically selected based on temporal kernel size ( l ), temporal stride ( s t ) and temporal padding ( p ), where l is set to an odd number so that an anchor frame is located in the middle of a point tube.
Revised: Temporal anchor frames in a point cloud sequence are automatically selected based on temporal kernel size ( l ), temporal stride ( s t ) and temporal padding ( p ), where l is set to an odd number so that an anchor frame is located in the middle of a point tube.

Original: Moreover, we set (cid:98) l 2 (cid:99) ≥ p to avoid selecting a padding frame as an anchor frame.
Revised: Moreover, we set (cid:98) l 2 (cid:99) ≥ p to avoid selecting a padding frame as an anchor frame.

Original: Spatial anchor point sampling.
Revised: Spatial anchor point sampling.

Original: Once a temporal anchor frame is selected, we need to choose spatial anchor points that can represent the distribution of all the points in the frame.
Revised: Once a temporal anchor frame is selected, we need to choose spatial anchor points that can represent the distribution of all the points in the frame.

Original: Given a spatial subsampling rate s s , this operation aims to subsample N points to N (cid:48)
Revised: Given a spatial subsampling rate s s , this operation aims to subsample N points to N (cid:48)

Original: = (cid:98)
Revised: = (cid:98)

Original: Ns s (cid:99) points.
Revised: Ns s (cid:99) points.

Original: We employ the farthest point sampling (FPS) (Qi et al., 2017b) to sample points in each anchor frame.
Revised: We employ the farthest point sampling (FPS) (Qi et al., 2017b) to sample points in each anchor frame.

Original: Point tubes are generated according to sampled anchor points.
Revised: Point tubes are generated according to sampled anchor points.

Original: Transferring spatial anchor points.
Revised: Transferring spatial anchor points.

Original: 3D convolutions can effectively capture local changes within a cube of a kernel size ( l, h, w ) rather than tracking a speciﬁc pixel.
Revised: 3D convolutions can effectively capture local changes within a cube of a kernel size ( l, h, w ) rather than tracking a speciﬁc pixel.

Original: Following this idea, we propagate the positions of sampled anchor points to the neighboring frames without tracking, and they are regarded as the anchor points in those frames.
Revised: Following this idea, we propagate the positions of sampled anchor points to the neighboring frames without tracking, and they are regarded as the anchor points in those frames.

Original: Speciﬁcally, each anchor point is transferred to the (cid:98) l 2 (cid:99) nearest frames.
Revised: Speciﬁcally, each anchor point is transferred to the (cid:98) l 2 (cid:99) nearest frames.

Original: The original and transferred anchor points form the central axis of a tube.
Revised: The original and transferred anchor points form the central axis of a tube.

Original: Spatial neighbor.
Revised: Spatial neighbor.

Original: This step aims to ﬁnd the spatial neighbors of every anchor point in each frame for performing spatial convolution.
Revised: This step aims to ﬁnd the spatial neighbors of every anchor point in each frame for performing spatial convolution.

Original: As indicated in Eq.
Revised: As indicated in Eq.

Original: (5), radius neighborhood r is used to search neighbors within the tube slice, where local structure of points is depicted.
Revised: (5), radius neighborhood r is used to search neighbors within the tube slice, where local structure of points is depicted.

Original: Note that, padding is usually used in grid based convolution to align feature maps.
Revised: Note that, padding is usually used in grid based convolution to align feature maps.

Original: However, our point based spatial convolution is not conducted on grids and thus spatial padding is not employed.
Revised: However, our point based spatial convolution is not conducted on grids and thus spatial padding is not employed.

Original: By performing PST convolution on point tubes, our network is able to capture the dynamic changes in local areas.
Revised: By performing PST convolution on point tubes, our network is able to capture the dynamic changes in local areas.

Original: The temporal kernel size l and spatial search radius r allow our PST convolution to capture temporal and spatial local structure, respectively.
Revised: The temporal kernel size l and spatial search radius r allow our PST convolution to capture temporal and spatial local structure, respectively.

Original: Frame subsampling (according to s t ) and point subsampling (according to s s ) make our PSTNet be both temporally and spatially hierarchical.
Revised: Frame subsampling (according to s t ) and point subsampling (according to s s ) make our PSTNet be both temporally and spatially hierarchical.

Original: Global movement can be summarized by merging the information from these tubes in a spatio-temporally hierarchical manner.
Revised: Global movement can be summarized by merging the information from these tubes in a spatio-temporally hierarchical manner.

Original: An illustration of PST convolution is shown in Fig.
Revised: An illustration of PST convolution is shown in Fig.

Original: 3. P OINT S PATIO -T EMPORAL T RANSPOSED C ONVOLUTION
Revised: 3. P OINT S PATIO -T EMPORAL T RANSPOSED C ONVOLUTION

Original: After PST convolution, the original point cloud sequence is both spatially and temporally subsampled.
Revised: After PST convolution, the original point cloud sequence is both spatially and temporally subsampled.

Original: However, for point-level prediction tasks, we need to provide point features for all the original points.
Revised: However, for point-level prediction tasks, we need to provide point features for all the original points.

Original: Thus, we develop a PST transposed convolution.
Revised: Thus, we develop a PST transposed convolution.

Original: Suppose (cid:0) [ P (cid:48) 1 ; F (cid:48) 1 ] , [ P (cid:48) 2 ; F (cid:48) 2 ] , · · · , [ P (cid:48)
Revised: Suppose (cid:0) [ P (cid:48) 1 ; F (cid:48) 1 ] , [ P (cid:48) 2 ; F (cid:48) 2 ] , · · · , [ P (cid:48)

Original: L (cid:48) ; F (cid:48)
Revised: L (cid:48) ; F (cid:48)

Original: L (cid:48) ] (cid:1) is the encoded sequence of the original one (cid:0)
Revised: L (cid:48) ] (cid:1) is the encoded sequence of the original one (cid:0)

Original: [ P 1 ; F 1 ] , [ P 2 ; F 2 ] , · · · , [ P L ; F L ] (cid:1) .
Revised: [ P 1 ; F 1 ] , [ P 2 ; F 2 ] , · · · , [ P L ; F L ] (cid:1) .

Original: PST transposed convolution propagates features ( F (cid:48) 1 , F (cid:48) 2 , · · · , F (cid:48) L (cid:48) ) to the original point coordinates ( P 1 , P 2 , · · · , P L ) , thus outputting new features ( F (cid:48)(cid:48) 1 , F (cid:48)(cid:48) 2 , · · · , F (cid:48)(cid:48) L ) , where F (cid:48)(cid:48) t ∈ R C (cid:48)(cid:48) × N and C (cid:48)(cid:48) denotes the new feature dimension.
Revised: PST transposed convolution propagates features ( F (cid:48) 1 , F (cid:48) 2 , · · · , F (cid:48) L (cid:48) ) to the original point coordinates ( P 1 , P 2 , · · · , P L ) , thus outputting new features ( F (cid:48)(cid:48) 1 , F (cid:48)(cid:48) 2 , · · · , F (cid:48)(cid:48) L ) , where F (cid:48)(cid:48) t ∈ R C (cid:48)(cid:48) × N and C (cid:48)(cid:48) denotes the new feature dimension.

Original: To this end, PST transposed convolution ﬁrst recovers the temporal length by a temporal transposed convolution:
Revised: To this end, PST transposed convolution ﬁrst recovers the temporal length by a temporal transposed convolution:

Original: [Equation]
Revised: [Equation]

Original: where T (cid:48) ∈ R l × C (cid:48) m × C (cid:48) is the temporal transposed convolution kernel, and then increases the number of points by assigning temporal features to original points.
Revised: where T (cid:48) ∈ R l × C (cid:48) m × C (cid:48) is the temporal transposed convolution kernel, and then increases the number of points by assigning temporal features to original points.

Original: Inspired by (Qi et al., 2017b), the interpolated features are weighted by inverse distance between an original point and neighboring anchor points:
Revised: Inspired by (Qi et al., 2017b), the interpolated features are weighted by inverse distance between an original point and neighboring anchor points:

Original: [Equation]
Revised: [Equation]

Original: where S (cid:48) ∈ R C (cid:48)(cid:48) × C (cid:48) m is a sharing kernel to enhance the interpolated features and change the feature dimension from C (cid:48) m to C (cid:48)(cid:48) .
Revised: where S (cid:48) ∈ R C (cid:48)(cid:48) × C (cid:48) m is a sharing kernel to enhance the interpolated features and change the feature dimension from C (cid:48) m to C (cid:48)(cid:48) .

Original: An illustration of PST transposed convolution is shown in Appendix A.
Revised: An illustration of PST transposed convolution is shown in Appendix A.

Original: 3. PSTN ET A RCHITECTURES
Revised: 3. PSTN ET A RCHITECTURES

Original: PSTNet for 3D action recognition.
Revised: PSTNet for 3D action recognition.

Original: We employ 6 PST convolution layers and a fully-connected (FC) layer for 3D action recognition.
Revised: We employ 6 PST convolution layers and a fully-connected (FC) layer for 3D action recognition.

Original: In the 1st, 2nd, 4th, 6th layers, the spatial subsampling rate is set to 2 to halve the spatial resolution.
Revised: In the 1st, 2nd, 4th, 6th layers, the spatial subsampling rate is set to 2 to halve the spatial resolution.

Original: Because point clouds are irregular and unordered, spatial receptive ﬁelds cannot be relatively enlarged by spatial subsampling.
Revised: Because point clouds are irregular and unordered, spatial receptive ﬁelds cannot be relatively enlarged by spatial subsampling.

Original: To address this problem, we progressively increase the spatial search radius.
Revised: To address this problem, we progressively increase the spatial search radius.

Original: In the 2nd and 4th layers, the temporal stride is set to 2 to halve the temporal resolution.
Revised: In the 2nd and 4th layers, the temporal stride is set to 2 to halve the temporal resolution.

Original: In the 2-4 layers, the temporal kernel size is set to 3 to capture temporal correlation.
Revised: In the 2-4 layers, the temporal kernel size is set to 3 to capture temporal correlation.

Original: Temporal paddings are added when necessary.
Revised: Temporal paddings are added when necessary.

Original: After the convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively.
Revised: After the convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively.

Original: Finally, the FC layer maps the global feature to action predictions.
Revised: Finally, the FC layer maps the global feature to action predictions.

Original: PSTNet for 4D semantic segmentation.
Revised: PSTNet for 4D semantic segmentation.

Original: We use 4 PST convolution layers and 4 PST transposed convolution layers for 4D semantic segmentation.
Revised: We use 4 PST convolution layers and 4 PST transposed convolution layers for 4D semantic segmentation.

Original: The spatial subsampling rate is set to 4 to reduce the spatial resolution in the 1-3 PST convolutions and 2 in the fourth PST convolution.
Revised: The spatial subsampling rate is set to 4 to reduce the spatial resolution in the 1-3 PST convolutions and 2 in the fourth PST convolution.

Original: Similar to 3D action recognition, the spatial search radius progressively increases to grow spatial receptive ﬁelds along the network layers.
Revised: Similar to 3D action recognition, the spatial search radius progressively increases to grow spatial receptive ﬁelds along the network layers.

Original: In the 3rd PST convolution and 2nd transposed convolution, the temporal kernel size is set to 3 to reduce and increase temporal dimension, respectively.
Revised: In the 3rd PST convolution and 2nd transposed convolution, the temporal kernel size is set to 3 to reduce and increase temporal dimension, respectively.

Original: Skip connections are added between the corresponding convolution layers and transposed convolution layers.
Revised: Skip connections are added between the corresponding convolution layers and transposed convolution layers.

Original: After the last PST transposed convolution layer, a 1D convolution layer is appended for semantic predictions.
Revised: After the last PST transposed convolution layer, a 1D convolution layer is appended for semantic predictions.

Original: In addition, batch normalization and ReLU activation are inserted between PST convolution and transposed convolution layers.
Revised: In addition, batch normalization and ReLU activation are inserted between PST convolution and transposed convolution layers.

Original: Our PSTNet architectures are illustrated in Appendix B.
Revised: Our PSTNet architectures are illustrated in Appendix B.

Original:  [Figure] [Table] [Figure]
Revised:  E XPERIMENTS 4. 3D A CTION R ECOGNITION

Original: [Table] [Figure]
Revised: To show the effectiveness in sequence-level classiﬁcation, we apply PSTNet to 3D action recognition.

Original: [Table] [Figure] [Table] [Figure]
Revised: Following (Liu et al., 2019e; Wang et al., 2020), we sample 2,048 points for each frame.

Original: [Table] [Figure] [Table] [Figure]
Revised: Point

Original: [Figure]
Revised: [Figure]

Original: Interestingly, PST convolution outputs high activation to salient motion. Best viewed in color.  E XPERIMENTS 4. 3D A CTION R ECOGNITION To show the effectiveness in sequence-level classiﬁcation
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: we apply PSTNet to 3D action recognition. Following (Liu et al., 2019e; Wang et al., 2020), we sample 2,048 points for each frame.
Revised: [Table]

Original: Point cloud sequences are split into multiple clips (with a ﬁxed number of frames) as inputs.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: Interestingly, PST convolution outputs high activation to salient motion.

Revised: Best viewed in color.

Revised: cloud sequences are split into multiple clips (with a ﬁxed number of frames) as inputs.

Original: For training, sequence-level labels are used as clip-level labels.
Revised: For training, sequence-level labels are used as clip-level labels.

Original: For evaluation, the mean of the clip-level predicted probabilities is used as the sequence-level prediction.
Revised: For evaluation, the mean of the clip-level predicted probabilities is used as the sequence-level prediction.

Original: Point colors are not used.
Revised: Point colors are not used.

Original: 4.1. MSR-A CTION 3D
Revised: 4.1. MSR-A CTION 3D

Original: The MSR-Action3D (Li et al., 2010) dataset consists of 567 Kinect depth videos, with 20 action categories and 23K frames in total.
Revised: The MSR-Action3D (Li et al., 2010) dataset consists of 567 Kinect depth videos, with 20 action categories and 23K frames in total.

Original: We use the same training/test split as previous works (Wang et al., 2012; Liu et al., 2019e).
Revised: We use the same training/test split as previous works (Wang et al., 2012; Liu et al., 2019e).

Original: We conduct experiments with 10 times and report the mean.
Revised: We conduct experiments with 10 times and report the mean.

Original: Comparison with the state-of-the-art.
Revised: Comparison with the state-of-the-art.

Original: We compare our PSTNet with skeleton-based, depth-based and point-based 3D action recognition methods on this dataset.
Revised: We compare our PSTNet with skeleton-based, depth-based and point-based 3D action recognition methods on this dataset.

Original: As shown in Table 1, the proposed
Revised: As shown in Table 1, the proposed

Original: PSTNet signiﬁcantly outperforms all the state-of-the-art methods, demonstrating the superiority of our PST convolution on feature extraction.
Revised: PSTNet signiﬁcantly outperforms all the state-of-the-art methods, demonstrating the superiority of our PST convolution on feature extraction.

Original: Moreover, when encoding long point sequences, our spatio-temporally hierarchical PSTNet is more effective than the spatially hierarchical MeteorNet.
Revised: Moreover, when encoding long point sequences, our spatio-temporally hierarchical PSTNet is more effective than the spatially hierarchical MeteorNet.

Original: Speciﬁcally, from 16 frames to 24 frames, MeteorNet only achieves a slight improvement of 0.29%, while our method increases the accuracy by 1.30%.
Revised: Speciﬁcally, from 16 frames to 24 frames, MeteorNet only achieves a slight improvement of 0.29%, while our method increases the accuracy by 1.30%.

Original: What does PST convolution learn?
Revised: What does PST convolution learn?

Original: To investigate what PST convolution learns, we visualize the output of the middle layer ( i.e ., the 3rd layer) in Fig.
Revised: To investigate what PST convolution learns, we visualize the output of the middle layer ( i.e ., the 3rd layer) in Fig.

Original: As ReLU is employed as the activation[Table] function, larger values indicate higher activation.
Revised: As ReLU is employed as the activation function, larger values indicate higher activation.

Original: As expected, PSTNet outputs higher activation on moving areas, demonstrating that our PSTNet captures the most informative clues in action reasoning.
Revised: As expected, PSTNet outputs higher activation on moving areas, demonstrating that our PSTNet captures the most informative clues in action reasoning.

Original: 4.1. NTU RGB+D 60 AND NTU RGB+D
Revised: 4.1. NTU RGB+D 60 AND NTU RGB+D

Original: The NTU RGB+D 60 (Shahroudy et al., 2016) is the second largest dataset for 3D action recognition.
Revised: The NTU RGB+D 60 (Shahroudy et al., 2016) is the second largest dataset for 3D action recognition.

Original: It consists of 56K videos, with 60 action categories and 4M frames in total.
Revised: It consists of 56K videos, with 60 action categories and 4M frames in total.

Original: The videos are captured  using Kinect v2, with 3 cameras and 40 subjects (performers).
Revised: The videos are captured [Table] using Kinect v2, with 3 cameras and 40 subjects (performers).

Original: The dataset deﬁnes two types of evaluation, i.e ., cross-subject and cross-view.
Revised: The dataset deﬁnes two types of evaluation, i.e ., cross-subject and cross-view.

Original: The cross-subject evaluation splits the 40 performers into training and test groups.
Revised: The cross-subject evaluation splits the 40 performers into training and test groups.

Original: Each group consists of 20 performers.
Revised: Each group consists of 20 performers.

Original: The cross-view evaluation uses all the samples from camera 1 for testing and samples from cameras 2 and 3 for training.
Revised: The cross-view evaluation uses all the samples from camera 1 for testing and samples from cameras 2 and 3 for training.

Original: The NTU RGB+D 120 (Liu et al., 2019a) dataset, the largest dataset for 3D action recognition, is an extension of NTU RGB+D 60.
Revised: The NTU RGB+D 120 (Liu et al., 2019a) dataset, the largest dataset for 3D action recognition, is an extension of NTU RGB+D 60.

Original: It consists of 114K videos, with 120 action categories and 8M frames in total.
Revised: It consists of 114K videos, with 120 action categories and 8M frames in total.

Original: The videos are captured with 106 performers and 32 collection setups (locations and backgrounds).
Revised: The videos are captured with 106 performers and 32 collection setups (locations and backgrounds).

Original: Besides cross-subject evaluation, the dataset deﬁnes a new evaluation setting, i.e ., cross-setup, where 16 setups are used for training, and the others are used for testing.
Revised: Besides cross-subject evaluation, the dataset deﬁnes a new evaluation setting, i.e ., cross-setup, where 16 setups are used for training, and the others are used for testing.

Original: Comparison with state-of-the-art methods.
Revised: Comparison with state-of-the-art methods.

Original: As indicated in Table 2, PSTNet outperforms all the other approaches in all evaluation settings.
Revised: As indicated in Table 2, PSTNet outperforms all the other approaches in all evaluation settings.

Revised: Particularly, as indicated by the cross-setup evaluation on

Revised: NTU RGB+D 120, PSTNet outperforms the second best 3DV-PointNet++ (Wang et al., 2020) by 4.6%.

Original: Particularly, to  that extracts motion from voxels, PSTNet directly models the dynamic information of raw point cloud sequences and thus is efﬁcient.
Revised: Moreover, compared to 3DV that extracts motion from voxels, PSTNet directly models the dynamic information of raw point cloud sequences and thus is efﬁcient.

Original: Computational efﬁciency.
Revised: Computational efﬁciency.

Original: We provide a running time comparison with the second best 3DVPointNet++ (Wang et al., 2020).
Revised: We provide a running time comparison with the second best 3DVPointNet++ (Wang et al., 2020).

Original: The average running time per video is shown in Fig.
Revised: The average running time per video is shown in Fig.

Original: Experiments are conducted using 1 Nvidia RTX 2080Ti GPU on NTU RGB+D 60.
Revised: Experiments are conducted using 1 Nvidia RTX 2080Ti GPU on NTU RGB+D 60.

Original: Impressively, compared to 3DV-PointNet++, PSTNet reduces running time by about 2s, showing that PSTNet is efﬁcient.
Revised: Impressively, compared to 3DV-PointNet++, PSTNet reduces running time by about 2s, showing that PSTNet is efﬁcient.

Original: 4. 4D S EMANTIC S EGMENTATION
Revised: 4. 4D S EMANTIC S EGMENTATION

Original: To demonstrate that our PST convolution can be generalized to point-level dense prediction tasks, we apply PSTNet to 4D semantic segmentation.
Revised: To demonstrate that our PST convolution can be generalized to point-level dense prediction tasks, we apply PSTNet to 4D semantic segmentation.

Original: Following the works (Choy et al., 2019; Liu et al.,

Original: [Table] [Figure] 2019e) , we conduct experiments on video clips with length of 3 frames.
Revised: Following the works (Choy et al., 2019; Liu et al., 2019e), we conduct experiments on video clips with length of 3 frames.

Original: Note that, although semantic segmentation can be achieved from a single frame, exploring temporal consistency would facilitate exploring the structure of scenes, thus improving segmentation accuracy and robustness to noise.
Revised: Note that, although semantic segmentation can be achieved from a single frame, exploring temporal consistency would facilitate exploring the structure of scenes, thus improving segmentation accuracy and robustness to noise.

Original: The widely-used standard mean Intersection over Union (mIoU) is adopted as the evaluation metric.
Revised: The widely-used standard mean Intersection over Union (mIoU) is adopted as the evaluation metric.

Original: Synthia 4D (Choy et al., 2019) uses the Synthia dataset (Ros et al., 2016) to create 3D video sequences.
Revised: Synthia 4D (Choy et al., 2019) uses the Synthia dataset (Ros et al., 2016) to create 3D video sequences.

Original: The Synthia 4D dataset includes 6 sequences of driving scenarios.
Revised: The Synthia 4D dataset includes 6 sequences of driving scenarios.

Original: Each sequence consists of 4 stereo
Revised: Each sequence consists of 4 stereo

Revised: [Table]

Revised: [Figure]

Original: RGB-D images taken from the top of a moving car.
Revised: RGB-D images taken from the top of a moving car.

Original: Following (Liu et al., 2019e), we use the same training/validation/test split, with 19,888/815/1,886 frames, respectively.
Revised: Following (Liu et al., 2019e), we use the same training/validation/test split, with 19,888/815/1,886 frames, respectively.

Original: As seen in Table 3, PSTNet ( l = 3 ) exploits the temporal information and outperforms the state-ofthe-art.
Revised: As seen in Table 3, PSTNet ( l = 3 ) exploits the temporal information and outperforms the state-ofthe-art.

Original: Moreover,due to decoupling spatio-temporal modeling, our method saves 0.11M, relative 6% of parameters compared to the second best method MeteorNet (Liu et al., 2019e).
Revised: Moreover, our method saves 0.11M, relative 6% of parameters compared to the second best method MeteorNet (Liu et al., 2019e).

Original: We visualize a few segmentation examples in Appendix N.
Revised: We visualize a few segmentation examples in Appendix N.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Clip length.
Revised: Clip length.

Original: Usually, information is not equally distributed in sequences along time.
Revised: Usually, information is not equally distributed in sequences along time.

Original: Short point cloud clips may miss key frames and thus confuse the models as noise.
Revised: Short point cloud clips may miss key frames and thus confuse the models as noise.

Original: Therefore, as shown in
Revised: Therefore, as shown in

Original: Table 1, increasing clip length ( i.e ., the number of frames) beneﬁts models for action recognition.
Revised: Table 1, increasing clip length ( i.e ., the number of frames) beneﬁts models for action recognition.

Original: Temporal kernel size.
Revised: Temporal kernel size.

Original: The temporal kernel size l controls the temporal dynamics modeling of point cloud sequences.
Revised: The temporal kernel size l controls the temporal dynamics modeling of point cloud sequences.

Original: Fig.
Revised: Fig.

Original: 3(a) shows the accuracy on MSR-Action3D with different l .
Revised: 3(a) shows the accuracy on MSR-Action3D with different l .

Original: (a) When l is set to 1, temporal correlation is not captured.
Revised: (a) When l is set to 1, temporal correlation is not captured.

Original: However, PSTNet can still observe 24 frames and the pooling operation allows PSTNet to capture the pose information of an entire clip.
Revised: However, PSTNet can still observe 24 frames and the pooling operation allows PSTNet to capture the pose information of an entire clip.

Original: Moreover, some actions ( e.g ., “golf swing”) can be easily recognized by a certain pose, and thus PSTNet with l =can still achieve satisﬁed accuracy.
Revised: Moreover, some actions ( e.g ., “golf swing”) can be easily recognized by a certain pose, and thus PSTNet with l =can still achieve satisﬁed accuracy.

Original: (b) When l is greater than 1, PSTNet models temporal dynamics and therefore improves accuracy on actions that rely on motion or trajectory reasoning ( e.g ., “draw x”, “draw tick” and “draw circle”).
Revised: (b) When l is greater than 1, PSTNet models temporal dynamics and therefore improves accuracy on actions that rely on motion or trajectory reasoning ( e.g ., “draw x”, “draw tick” and “draw circle”).

Original: (c) When l is greater than 3, the accuracy decreases.
Revised: (c) When l is greater than 3, the accuracy decreases.

Original: This mainly depends on motion in sequences.
Revised: This mainly depends on motion in sequences.

Original: Because most actions in MSR-Action3D are fast ( e.g ., “high arm wave”), using smaller temporal kernel size facilitates capturing fast motion, and long-range temporal dependencies will be captured in high-level layers.
Revised: Because most actions in MSR-Action3D are fast ( e.g ., “high arm wave”), using smaller temporal kernel size facilitates capturing fast motion, and long-range temporal dependencies will be captured in high-level layers.

Original: Since we aim to present generic point based convolution operations, we do not tune the kernel size for each action but use the same size.
Revised: Since we aim to present generic point based convolution operations, we do not tune the kernel size for each action but use the same size.

Original: When
Revised: When

Original: PSTNet is used in 4D semantic segmentation, we observe that PSTNet ( l = 3 ) improves mIoU by 1.45% compared to PSTNet ( l = 1 ) that neglects temporal structure (shown in Table 3).
Revised: PSTNet is used in 4D semantic segmentation, we observe that PSTNet ( l = 3 ) improves mIoU by 1.45% compared to PSTNet ( l = 1 ) that neglects temporal structure (shown in Table 3).

Original: Spatial search radius.
Revised: Spatial search radius.

Original: The spatial search radius r controls the range of the spatial structure to be modeled.
Revised: The spatial search radius r controls the range of the spatial structure to be modeled.

Original: As shown in Fig. 3(b), using too small r cannot capture sufﬁcient structure information while using large r will decrease the discriminativeness of spatial local structure for modeling.
Revised: As shown in Fig. 3(b), using too small r cannot capture sufﬁcient structure information while using large r will decrease the discriminativeness of spatial local structure for modeling.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose an point spatio-temporal (PST) convolution to learn informative representations from raw point cloud sequences and a PST transposed convolution for point-level dense prediction tasks.
Revised: In this paper, we propose an point spatio-temporal (PST) convolution to learn informative representations from raw point cloud sequences and a PST transposed convolution for point-level dense prediction tasks.

Original: We demonstrate that, by incorporating the proposed convolutions into deep networks, dubbed PSTNets, our method is competent to address various point-based tasks.
Revised: We demonstrate that, by incorporating the proposed convolutions into deep networks, dubbed PSTNets, our method is competent to address various point-based tasks.

Original: Extensive experiments demonstrate that our PSTNet signiﬁcantly improves the 3D action recognition and 4D semantic segmentation performance by effectively modeling point cloud sequences.
Revised: Extensive experiments demonstrate that our PSTNet signiﬁcantly improves the 3D action recognition and 4D semantic segmentation performance by effectively modeling point cloud sequences.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This research is supported by the Agency for Science, Technology and Research (A*STAR) under its
Revised: This research is supported by the Agency for Science, Technology and Research (A*STAR) under its

Original: AME Programmatic Funding Scheme (#A18A2b0046)
Revised: AME Programmatic Funding Scheme (#A18A2b0046).

Original: A PST T RANSPOSED C ONVOLUTION
Revised: A PST T RANSPOSED C ONVOLUTION

Original: temporal st r ide =temporal st r ide =(𝑐) 𝑃𝑃𝑃𝑀′ 1𝑐 𝑀′ 2𝑐 𝑀′ 3𝑐 𝑀′ 𝟏𝒅 𝑀′ 2𝑑 𝑀′ 3𝑑 𝐹′ 2𝑐 𝐹 2′𝑑 𝑀′
Revised: temporal st r ide =temporal st r ide =(𝑐) 𝑃𝑃𝑃𝑀′ 1𝑐 𝑀′ 2𝑐 𝑀′ 3𝑐 𝑀′ 𝟏𝒅 𝑀′ 2𝑑 𝑀′ 3𝑑 𝐹′ 2𝑐 𝐹 2′𝑑 𝑀′

Original: 1𝑎 𝑀′ 2𝑎 𝑀′ 1𝑏 𝑀′ 2𝑏 𝐹′ 1𝑎 𝐹′ 1𝑏 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 𝑀′ 2𝑒 𝑀′ 3𝑒 𝑀′ 2𝑓 𝑀′ 3𝑓 𝐹′ 3𝑒 𝐹′ 3𝑓 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 (𝑎) (𝑏) 𝑃𝑃𝑃 1′ ;[𝑀 ′1𝑎 ,𝑀 ′1𝑏 ] 𝑃 2′ ;
Revised: 1𝑎 𝑀′ 2𝑎 𝑀′ 1𝑏 𝑀′ 2𝑏 𝐹′ 1𝑎 𝐹′ 1𝑏 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 𝑀′ 2𝑒 𝑀′ 3𝑒 𝑀′ 2𝑓 𝑀′ 3𝑓 𝐹′ 3𝑒 𝐹′ 3𝑓 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 𝑑𝑖𝑠𝑐𝑎𝑟𝑑 (𝑎) (𝑏) 𝑃𝑃𝑃 1′ ;[𝑀 ′1𝑎 ,𝑀 ′1𝑏 ] 𝑃 2′ ;

Original: [𝑀 ′1𝑐 ,𝑀 ′1𝑑 ] 𝑃 1′ ;[𝑀 ′2𝑎 ,𝑀 ′2𝑏 ] 𝑃 2′ ;
Revised: [𝑀 ′1𝑐 ,𝑀 ′1𝑑 ] 𝑃 1′ ;[𝑀 ′2𝑎 ,𝑀 ′2𝑏 ] 𝑃 2′ ;

Original: [𝑀 ′2𝑐 ,𝑀′ 2𝑑 ] 𝑃 2′ ;[𝑀 ′3𝑐 ,𝑀 ′3𝑑 ] 𝑃 3′ ;[𝑀 ′2𝑒 ,𝑀 ′2𝑓 ] 𝑃 3′ ;[𝑀 ′3𝑒 ,𝑀 ′3𝑓 ] [𝑃 1 ;𝐹 1′′ ]
Revised: [𝑀 ′2𝑐 ,𝑀′ 2𝑑 ] 𝑃 2′ ;[𝑀 ′3𝑐 ,𝑀 ′3𝑑 ] 𝑃 3′ ;[𝑀 ′2𝑒 ,𝑀 ′2𝑓 ] 𝑃 3′ ;[𝑀 ′3𝑒 ,𝑀 ′3𝑓 ] [𝑃 1 ;𝐹 1′′ ]

Original: [𝑃 2 ;𝐹 2′′ ]
Revised: [𝑃 2 ;𝐹 2′′ ]

Original: [𝑃 3 ;𝐹 3′′ ] [𝑃 4 ;𝐹 4′′ ]
Revised: [𝑃 3 ;𝐹 3′′ ] [𝑃 4 ;𝐹 4′′ ]

Original: [𝑃 5 ;𝐹 5′′ ] temporal kernelsize
Revised: [𝑃 5 ;𝐹 5′′ ] temporal kernelsize

Original: [Figure]
Revised: [Figure]

Original: We illustrate an example of PST transposed convolution in Fig. 6.
Revised: We illustrate an example of PST transposed convolution in Fig. 6.

Original: Given a convolved sequence (cid:0) [ P (cid:48) 1 ; F (cid:48) 1 ] , [ P (cid:48) 2 ; F (cid:48) 2 ] , [ P (cid:48) 3 (cid:48) ; F (cid:48) 3 (cid:48) ] (cid:1) , and its original coordinate sequence (cid:0) P 1 , P 2 , P 3 , P 4 , P 5 (cid:1) , PST transposed convolution aims to generate new features ( F (cid:48)(cid:48) 1 , F (cid:48)(cid:48) 2 , F (cid:48)(cid:48) 3 , F (cid:48)(cid:48) 4 , F (cid:48)(cid:48) 5 ) according to the original point coordinates.
Revised: Given a convolved sequence (cid:0) [ P (cid:48) 1 ; F (cid:48) 1 ] , [ P (cid:48) 2 ; F (cid:48) 2 ] , [ P (cid:48) 3 (cid:48) ; F (cid:48) 3 (cid:48) ] (cid:1) , and its original coordinate sequence (cid:0) P 1 , P 2 , P 3 , P 4 , P 5 (cid:1) , PST transposed convolution aims to generate new features ( F (cid:48)(cid:48) 1 , F (cid:48)(cid:48) 2 , F (cid:48)(cid:48) 3 , F (cid:48)(cid:48) 4 , F (cid:48)(cid:48) 5 ) according to the original point coordinates.

Original: First, with the temporal kernel size l = 3 , each input point feature is decoded to 3 features by a temporal transposed convolution.
Revised: First, with the temporal kernel size l = 3 , each input point feature is decoded to 3 features by a temporal transposed convolution.

Original: Second, 5 frames are interpolated in accordance with a temporal stride s t = 2 , and the input points with the decoded features are assigned to interpolated frames.
Revised: Second, 5 frames are interpolated in accordance with a temporal stride s t = 2 , and the input points with the decoded features are assigned to interpolated frames.

Original: In this way, the temporal cross-correlation is constructed.
Revised: In this way, the temporal cross-correlation is constructed.

Original: Third, given the original coordinates of a point cloud frame ( e.g ., P 1 ) and the corresponding interpolated frame ( i.e ., (cid:2)
Revised: Third, given the original coordinates of a point cloud frame ( e.g ., P 1 ) and the corresponding interpolated frame ( i.e ., (cid:2)

Original: P (cid:48) 1 ; [ M (cid:48) a 1 , M (cid:48)
Revised: P (cid:48) 1 ; [ M (cid:48) a 1 , M (cid:48)

Original: b 1 ] (cid:3) ), we generate features ( i.e ., F (cid:48)(cid:48) 1 ) for all the original point coordinates.
Revised: b 1 ] (cid:3) ), we generate features ( i.e ., F (cid:48)(cid:48) 1 ) for all the original point coordinates.

Original: B PSTN ET A RCHITECTURES B. PSTN ET FOR 3D A CTION R ECOGNITION
Revised: B PSTN ET A RCHITECTURES B. PSTN ET FOR 3D A CTION R ECOGNITION

Original: [Figure]
Revised: [Figure]

Original: As shown in Fig.
Revised: As shown in Fig.

Original: [Figure]
Revised: [Figure]

Original: the architecture of PSTNet for 3D action recognition consists of 6 PST convolutions,
Revised: the architecture of PSTNet for 3D action recognition consists of 6 PST convolutions,

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e ., PSTConv1, PSTConv2a, PSTConv2b, PSTConv3a, PSTConv3b, PSTConv
Revised: e ., PSTConv1, PSTConv2a, PSTConv2b, PSTConv3a, PSTConv3b, PSTConv

Original: [Figure]
Revised: [Figure]

Original: and a fully-connected (FC) layer.
Revised: and a fully-connected (FC) layer.

Original: In PSTConv1, PSTConv2a, PSTConv3a and PSTConv4, the spatial subsampling rate is set to 2 to halve the spatial resolution.
Revised: In PSTConv1, PSTConv2a, PSTConv3a and PSTConv4, the spatial subsampling rate is set to 2 to halve the spatial resolution.

Original: In PSTConv2a and PSTConv3a, the temporal stride is set to 2 to halve the temporal resolution.
Revised: In PSTConv2a and PSTConv3a, the temporal stride is set to 2 to halve the temporal resolution.

Original: The r o denotes the initial spatial search radius.
Revised: The r o denotes the initial spatial search radius.

Original: The temporal padding p is split as [ p 1 , p 2 ] , where p 1 and p 2 denote the left padding and the right padding, respectively.
Revised: The temporal padding p is split as [ p 1 , p 2 ] , where p 1 and p 2 denote the left padding and the right padding, respectively.

Original: After these convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively.
Revised: After these convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively.

Original: The FC layer is then used as a classiﬁer.
Revised: The FC layer is then used as a classiﬁer.

Original: [Figure]
Revised: [Figure]

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 8, we employ 4 PST convolutional layers, i.e ., PSTConv1, PSTConv2, PSTConv3, and PSTConv4, and 4 PST transposed convolutional layers, i.e ., PSTConvTrans1, PSTConvTrans2,
Revised: 8, we employ 4 PST convolutional layers, i.e ., PSTConv1, PSTConv2, PSTConv3, and PSTConv4, and 4 PST transposed convolutional layers, i.e ., PSTConvTrans1, PSTConvTrans2,

Original: PSTConvTrans3, and PSTConvTrans4 for 4D Semantic Segmentation.
Revised: PSTConvTrans3, and PSTConvTrans4 for 4D Semantic Segmentation.

Original: The spatial subsampling rate is set to 4 to reduce the spatial resolution in each PST convolution layer.
Revised: The spatial subsampling rate is set to 4 to reduce the spatial resolution in each PST convolution layer.

Original: Following (Choy et al., 2019; Liu et al., 2019e), we conduct 4D semantic segmentation on clips with the length of 3 frames, i.e ., L is ﬁxed to 3.
Revised: Following (Choy et al., 2019; Liu et al., 2019e), we conduct 4D semantic segmentation on clips with the length of 3 frames, i.e ., L is ﬁxed to 3.

Original: In PSTConv3 and PSTConvTrans2, the temporal kernel size is set to 3.
Revised: In PSTConv3 and PSTConvTrans2, the temporal kernel size is set to 3.

Original: Skip connections are added from input to PSTConvTrans1, PSTConv1 to PSTConvTrans2, PSTConv2 to
Revised: Skip connections are added from input to PSTConvTrans1, PSTConv1 to PSTConvTrans2, PSTConv2 to

Original: PSTConvTrans3, and PSTConv3 to PSTConvTrans4.
Revised: PSTConvTrans3, and PSTConv3 to PSTConvTrans4.

Original: After PSTConvTrans4, a 1D convolution layer is appended for semantic predictions.
Revised: After PSTConvTrans4, a 1D convolution layer is appended for semantic predictions.

Original: C I MPLEMENTATION D ETAILS
Revised: C I MPLEMENTATION D ETAILS

Original: To build a uniﬁed network and train the network with mini-batch, the number of spatial neighbors needs to be ﬁxed.
Revised: To build a uniﬁed network and train the network with mini-batch, the number of spatial neighbors needs to be ﬁxed.

Original: We follow existing point based works (Qi et al., 2017b; Liu et al., 2019e) to randomly sample a ﬁxed number of neighbors.
Revised: We follow existing point based works (Qi et al., 2017b; Liu et al., 2019e) to randomly sample a ﬁxed number of neighbors.

Original: For 3D action recognition, the number is set to 9.
Revised: For 3D action recognition, the number is set to 9.

Original: For 4D semantic segmentation, we follow (Liu et al., 2019e) to set the number to 32.
Revised: For 4D semantic segmentation, we follow (Liu et al., 2019e) to set the number to 32.

Original: If the actual number of neighbors of a point is less than the set one ( e.g ., 9), we randomly repeat some neighbors.
Revised: If the actual number of neighbors of a point is less than the set one ( e.g ., 9), we randomly repeat some neighbors.

Original: We train our models for 35 epochs with the SGD optimizer.
Revised: We train our models for 35 epochs with the SGD optimizer.

Original: Learning rate is set to 0.01, and decays with a rate of 0.1 at the 10th epoch and the 20th epoch, respectively.
Revised: Learning rate is set to 0.01, and decays with a rate of 0.1 at the 10th epoch and the 20th epoch, respectively.

Original: MSR-Action3D (Li et al., 2010):
Revised: MSR-Action3D (Li et al., 2010):

Original: Following (Liu et al., 2019e), batch size is set to 16 and frame sampling stride are set to and 1, respectively.
Revised: Following (Liu et al., 2019e), batch size is set to 16 and frame sampling stride are set to and 1, respectively.

Original: We set the initial spatial search radius r o to 0.5.
Revised: We set the initial spatial search radius r o to 0.5.

Original: NTU RGB+D 60 /

Revised: NTU RGB+D 60 /

Original: (Shahroudy et al., 2016; Liu et al., 2019a):
Revised: (Shahroudy et al., 2016; Liu et al., 2019a):

Original: Following (Wang et al., 2020), batch size is set to 32.
Revised: Following (Wang et al., 2020), batch size is set to 32.

Original: We set the clip length, frame sampling stride and the initial spatial search radius r o to 23, 2 and 0.1, respectively.
Revised: We set the clip length, frame sampling stride and the initial spatial search radius r o to 23, 2 and 0.1, respectively.

Original: Synthia 4D (Choy et al., 2019):
Revised: Synthia 4D (Choy et al., 2019):

Original: Following (Liu et al., 2019e), batch size and frame sampling stride are set to 12 and 1, respectively.
Revised: Following (Liu et al., 2019e), batch size and frame sampling stride are set to 12 and 1, respectively.

Original: We set the spatial search radius r o to 0.9.
Revised: We set the spatial search radius r o to 0.9.

Original: We have implemented our PSTNets using both PyTorch and PaddlePaddle, which have shown similar performance.
Revised: We have implemented our PSTNets using both PyTorch and PaddlePaddle, which have shown similar performance.

Original: D I NFLUENCE OF T EMPORAL K ERNEL S IZE ON D IFFERENT A CTIONS
Revised: D I NFLUENCE OF T EMPORAL K ERNEL S IZE ON D IFFERENT A CTIONS

Original: We show the inﬂuence of temporal kernel size on different actions in MSR-Action3D in Fig.
Revised: We show the inﬂuence of temporal kernel size on different actions in MSR-Action3D in Fig.

Original: Clip length is 24.
Revised: Clip length is 24.

Original: When temporal kernel size l = 1 , temporal correlation is not exploited.
Revised: When temporal kernel size l = 1 , temporal correlation is not exploited.

Original: However, PSTNet can leverage pose information for recognition, especially for actions whose poses are unique in the dataset.
Revised: However, PSTNet can leverage pose information for recognition, especially for actions whose poses are unique in the dataset.

Original: For example, PSTNet with l = 1 can correctly recognize the “jogging” action because the action’s pose is discriminative in the MSR-Action3D dataset.
Revised: For example, PSTNet with l = 1 can correctly recognize the “jogging” action because the action’s pose is discriminative in the MSR-Action3D dataset.

Original: However, PSTNet with motion modeling ( l ≥ 3 ) can distinguish similar actions such as “draw x”, “draw tick” and “draw circle”.
Revised: However, PSTNet with motion modeling ( l ≥ 3 ) can distinguish similar actions such as “draw x”, “draw tick” and “draw circle”.

Original: The results support our intuition that PSTNet effectively captures dynamics in point cloud sequences.
Revised: The results support our intuition that PSTNet effectively captures dynamics in point cloud sequences.

Original: [Table]
Revised: [Table]

Original: (l) jogging    9 temporal kernel size  a cc u r a c y ( % ) (m) tennis serve    9 temporal kernel size  a cc u r a c y ( % )
Revised: (l) jogging    9 temporal kernel size  a cc u r a c y ( % ) (m) tennis serve    9 temporal kernel size  a cc u r a c y ( % )

Original: (n) two hand wave    9 temporal kernel size  a cc u r a c y ( % ) (o) golf swing    9 temporal kernel
Revised: (n) two hand wave    9 temporal kernel size  a cc u r a c y ( % ) (o) golf swing    9 temporal kernel

Original: [Figure]
Revised: [Figure]

Original: 0102030405060708090100 a cc u r a c y ( % ) (p) pick up & throw    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (q) horizont arm wave    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (r) forward punch    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (s) hand clap    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (t) side-boxing
Revised: 0102030405060708090100 a cc u r a c y ( % ) (p) pick up & throw    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (q) horizont arm wave    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (r) forward punch    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (s) hand clap    9 temporal kernel size 0102030405060708090100 a cc u r a c y ( % ) (t) side-boxing

Original: E F EATURE V ISUALIZATION
Revised: E F EATURE V ISUALIZATION

Original: [Figure]
Revised: [Figure]

Original: We qualitatively evaluate PSTNet’s ability to encode point cloud sequence by visualizing the learned features.
Revised: We qualitatively evaluate PSTNet’s ability to encode point cloud sequence by visualizing the learned features.

Original: We compare PSTNet with the sate-of-the-art MeteorNet (Liu et al., 2019e).
Revised: We compare PSTNet with the sate-of-the-art MeteorNet (Liu et al., 2019e).

Original: Features are extracted from the layer before classiﬁer (FC).
Revised: Features are extracted from the layer before classiﬁer (FC).

Original: These features are then projected to 2-dimensional space using t-SNE.
Revised: These features are then projected to 2-dimensional space using t-SNE.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 10, PSTNet feature is more compact and discriminative than
Revised: 10, PSTNet feature is more compact and discriminative than

Original: MeteorNet.
Revised: MeteorNet.

Original: F S YNTHIA 4D S EMANTIC S EGMENTATION R ESULT D ETAILS
Revised: F S YNTHIA 4D S EMANTIC S EGMENTATION R ESULT D ETAILS

Original: We list the segmentation result for each class in Table 4.
Revised: We list the segmentation result for each class in Table 4.

Original: The Synthia 4D dataset contains 12 categories.
Revised: The Synthia 4D dataset contains 12 categories.

Original: Our PSTNet achieves ﬁve best accuracies among them, demonstrating the effectiveness of PSTNet.
Revised: Our PSTNet achieves ﬁve best accuracies among them, demonstrating the effectiveness of PSTNet.

Original: [Table]
Revised: [Table]

Original: G PST C ONVOLUTION WITHOUT D ISENTANGLING S PACE AND T IME
Revised: G PST C ONVOLUTION WITHOUT D ISENTANGLING S PACE AND T IME

Original: To conﬁrm the effectiveness of our disentangling structure for spatio-temporal modeling, we perform 3D action recognition on MSR-Action3D with non-decomposing convolution.
Revised: To conﬁrm the effectiveness of our disentangling structure for spatio-temporal modeling, we perform 3D action recognition on MSR-Action3D with non-decomposing convolution.

Original: To combine space and time, 3D displacements and time differences are encoded together to generate convolution kernel.
Revised: To combine space and time, 3D displacements and time differences are encoded together to generate convolution kernel.

Original: We evaluate our PSTNet with different clip lengths.
Revised: We evaluate our PSTNet with different clip lengths.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 11, the disentangling
Revised: 11, the disentangling

Original: PST convolution achieves better accuracy than the non-decomposing structure, demonstrating the effectiveness of disentangling space and time for point cloud sequence modeling.
Revised: PST convolution achieves better accuracy than the non-decomposing structure, demonstrating the effectiveness of disentangling space and time for point cloud sequence modeling.

Original: [Figure]
Revised: [Figure]

Original: H I MPACT OF D ISPLACEMENT K ERNEL AND S HARING K ERNEL
Revised: H I MPACT OF D ISPLACEMENT K ERNEL AND S HARING K ERNEL

Original: In this paper, we design a lightweight implementation for the f function in Eq.
Revised: In this paper, we design a lightweight implementation for the f function in Eq.

Original: (5), i.e ., f (cid:0) ( δ x , δ y , δ z ); θ (cid:1)
Revised: (5), i.e ., f (cid:0) ( δ x , δ y , δ z ); θ (cid:1)

Original: =
Revised: =

Original: θ d · ( δ x , δ y , δ z ) T · 1 (cid:12) θ s , where θ d is a displacement transform kernel, θ s is a sharing kernel.
Revised: θ d · ( δ x , δ y , δ z ) T · 1 (cid:12) θ s , where θ d is a displacement transform kernel, θ s is a sharing kernel.

Original: In this section, we investigate the inﬂuence of these two kernels.
Revised: In this section, we investigate the inﬂuence of these two kernels.

Original: We conduct 3D action recognition on MSR-Action3D.
Revised: We conduct 3D action recognition on MSR-Action3D.

Original: The experimental results are shown in Table 5.
Revised: The experimental results are shown in Table 5.

Original: The MSR-Action3D dataset is used.
Revised: The MSR-Action3D dataset is used.

Original: [Table]
Revised: [Table]

Original: Without θ s , the spatial convolution degenerates into M ( x,y,z ) t
Revised: Without θ s , the spatial convolution degenerates into M ( x,y,z ) t

Original: [Equation]
Revised: [Equation]

Original: In this way, only the points in the ﬁnal layer of our PSTNet are used.
Revised: In this way, only the points in the ﬁnal layer of our PSTNet are used.

Original: Therefore, the accuracy decreases dramatically.
Revised: Therefore, the accuracy decreases dramatically.

Original: Without θ d , the spatial convolution becomes M
Revised: Without θ d , the spatial convolution becomes M

Original: [Equation]
Revised: [Equation]

Original: In this fashion, the points in a neighborhood are treated equally.
Revised: In this fashion, the points in a neighborhood are treated equally.

Original: Point features are leveraged but their positions are ignored.
Revised: Point features are leveraged but their positions are ignored.

Original: Because spatial structure is not well captured, the accuracy decreases compared to the case where both θ d and θ s are used.
Revised: Because spatial structure is not well captured, the accuracy decreases compared to the case where both θ d and θ s are used.

Original: I C OMPUTATIONAL E FFICIENCY AND M EMORY U SAGE
Revised: I C OMPUTATIONAL E FFICIENCY AND M EMORY U SAGE

Original: In this section, we evaluate the computational efﬁciency and memory usage, i.e ., the running time and number of parameters, of our method.
Revised: In this section, we evaluate the computational efﬁciency and memory usage, i.e ., the running time and number of parameters, of our method.

Original: We conduct 3D action recognition with a clip length ofon MSR-Action3D using 1 Nvidia Quadro RTX 6000 GPU.
Revised: We conduct 3D action recognition with a clip length ofon MSR-Action3D using 1 Nvidia Quadro RTX 6000 GPU.

Original: As shown in Table 6, the proposed PSTNet is 41.5% faster than MeteorNet (Liu et al., 2019e).
Revised: As shown in Table 6, the proposed PSTNet is 41.5% faster than MeteorNet (Liu et al., 2019e).

Original: This is because that MeteorNet is based on MLPs, which are less efﬁcient than convolutions.
Revised: This is because that MeteorNet is based on MLPs, which are less efﬁcient than convolutions.

Original: [Table]
Revised: [Table]

Original: Second, we study the impact of the number of PSTNet layers on running time and parameter.
Revised: Second, we study the impact of the number of PSTNet layers on running time and parameter.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 12, when PSTNet becomes deep, the number of parameters signiﬁcantly grows.
Revised: 12, when PSTNet becomes deep, the number of parameters signiﬁcantly grows.

Original: This is because, like most deep neural networks, the point feature dimension exponentially increases to improve the feature representation ability in PSTNet, which needs more parameters.
Revised: This is because, like most deep neural networks, the point feature dimension exponentially increases to improve the feature representation ability in PSTNet, which needs more parameters.

Original: However, the running time does not increase signiﬁcantly.
Revised: However, the running time does not increase signiﬁcantly.

Original: This is because our PSTNet is spatio-temporally hierarchical, points are exponentially reduced along both spatial and temporal dimensions, thus saving running time.
Revised: This is because our PSTNet is spatio-temporally hierarchical, points are exponentially reduced along both spatial and temporal dimensions, thus saving running time.

Original: [Figure]
Revised: [Figure]

Original: m e t
Revised: m e t

Original: [Figure]
Revised: [Figure]

Original: r s ( M )
Revised: r s ( M )

Original: [Figure]
Revised: [Figure]

Original: unn i
Revised: unn i

Original: [Figure]
Revised: [Figure]

Original: g t i
Revised: t i

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Third, we investigate the running time of main operations in PSTNet, including convolution, farthest point sampling and spatial neighbor search.
Revised: Third, we investigate the running time of main operations in PSTNet, including convolution, farthest point sampling and spatial neighbor search.

Original: We show the running time proportion of these operations in Fig. 13.
Revised: We show the running time proportion of these operations in Fig. 13.

Original: Finally, because the f function in Eq.
Revised: Finally, because the f function in Eq.

Original: (5) can be implemented in different ways, we replace our lightweight spatial convolution with PointConv (Wu et al., 2019) to investigate the inﬂuence of the spatial modeling in our PSTNet.
Revised: (5) can be implemented in different ways, we replace our lightweight spatial convolution with PointConv (Wu et al., 2019) to investigate the inﬂuence of the spatial modeling in our PSTNet.

Original: As shown in Table 6, using PointConv only improves accraucy slightly compared to our lightweight spatial convolutions, but signiﬁcantly increases parameters and running time.
Revised: As shown in Table 6, using PointConv only improves accraucy slightly compared to our lightweight spatial convolutions, but signiﬁcantly increases parameters and running time.

Original: This is because PointConv utilizes MLP based operations to process points and their features, which are less efﬁcient than our lightweight convolutions.
Revised: This is because PointConv utilizes MLP based operations to process points and their features, which are less efﬁcient than our lightweight convolutions.

Original: J I RREGULAR F RAME S AMPLING
Revised: J I RREGULAR F RAME S AMPLING

Original: Our PSTNet can also process irregularly sampled point cloud sequences by frame interpolation.
Revised: Our PSTNet can also process irregularly sampled point cloud sequences by frame interpolation.

Original: In this section, we randomly remove some frames in sequences.
Revised: In this section, we randomly remove some frames in sequences.

Original: Speciﬁcally, we conduct 3D action recognition on MSR-Action with 24 frames.
Revised: Speciﬁcally, we conduct 3D action recognition on MSR-Action with 24 frames.

Original: We randomly remove 8 frames from each clip.
Revised: We randomly remove 8 frames from each clip.

Original: Then, we use replication and Iterative Closest Point (ICP)
Revised: Then, we use replication and Iterative Closest Point (ICP)

Original: (Besl & McKay, 1992) to interpolate missing frames, respectively.
Revised: (Besl & McKay, 1992) to interpolate missing frames, respectively.

Original: We compare our PSTNet with MeteorNet (Liu et al., 2019e), which explicitly encodes time and does not need interpolation.
Revised: We compare our PSTNet with MeteorNet (Liu et al., 2019e), which explicitly encodes time and does not need interpolation.

Original: [Table]
Revised: [Table]

Original: As shown in Table 7, our PSTNet still achieves the best accuracy, indicating that PSTNet is able to model point cloud sequences with irregular frame sampling.
Revised: As shown in Table 7, our PSTNet still achieves the best accuracy, indicating that PSTNet is able to model point cloud sequences with irregular frame sampling.

Original: K S CENE F LOW E STIMATION
Revised: K S CENE F LOW E STIMATION

Original: To evaluate our method on the LiDAR data where points in a frame can be highly irregular, we apply our method to scene ﬂow estimation.
Revised: To evaluate our method on the LiDAR data where points in a frame can be highly irregular, we apply our method to scene ﬂow estimation.

Original: We follow the setting proposed by MeteorNet (Liu et al., 2019e), i.e ., given a point cloud sequence, estimating a ﬂow vector for every point in the last frame.
Revised: We follow the setting proposed by MeteorNet (Liu et al., 2019e), i.e ., given a point cloud sequence, estimating a ﬂow vector for every point in the last frame.

Original: However, because our point tube is constructed according to the middle frame, which is not applicable to the last-frame scene ﬂow estimation, we adapt temporal anchor frame selection and spatial anchor point transferring.
Revised: However, because our point tube is constructed according to the middle frame, which is not applicable to the last-frame scene ﬂow estimation, we adapt temporal anchor frame selection and spatial anchor point transferring.

Original: Speciﬁcally, we select the last frame in each tube as the anchor frame.
Revised: Speciﬁcally, we select the last frame in each tube as the anchor frame.

Original: Then, after spatial sampling, each anchor point is transferred to its previous nearest l frames.
Revised: Then, after spatial sampling, each anchor point is transferred to its previous nearest l frames.

Original: Following MeteorNet, we ﬁrst train our model on a FlyingThings3D dataset according to the synthetic method in (Liu et al., 2019e), and then ﬁne-tune the model on a KITTI scene ﬂow dataset (Liu et al., 2019e).
Revised: Following MeteorNet, we ﬁrst train our model on a FlyingThings3D dataset according to the synthetic method in (Liu et al., 2019e), and then ﬁne-tune the model on a KITTI scene ﬂow dataset (Liu et al., 2019e).

Original: Point tracking is not used.
Revised: Point tracking is not used.

Original: [Table]
Revised: [Table]

Original: As shown in Table 8, our PSTNet achieves promising accuracy on scene ﬂow estimation.
Revised: As shown in Table 8, our PSTNet achieves promising accuracy on scene ﬂow estimation.

Original: L M OTION R ECOGNITION ON THE S YNTHETIC M OVING MNIST P OINT C LOUD D ATASET
Revised: L M OTION R ECOGNITION ON THE S YNTHETIC M OVING MNIST P OINT C LOUD D ATASET

Original: In the real world, it is impossible to obtain point IDs.
Revised: In the real world, it is impossible to obtain point IDs.

Original: To evaluate our PSTNet in the scenario where points are correlated and can be accurately tracked across frames, we conduct motion recognition on a synthetic Moving MNIST Point Cloud dataset.
Revised: To evaluate our PSTNet in the scenario where points are correlated and can be accurately tracked across frames, we conduct motion recognition on a synthetic Moving MNIST Point Cloud dataset.

Original: To synthesize moving MNIST digit sequences, we use a generation process similar to that described in (Fan & Yang, 2019).
Revised: To synthesize moving MNIST digit sequences, we use a generation process similar to that described in (Fan & Yang, 2019).

Original: Each synthetic sequence consists of 16 consecutive point cloud frames.
Revised: Each synthetic sequence consists of 16 consecutive point cloud frames.

Original: Each frame contains one handwritten digit moving and bouncing inside a 64 × 64 area.
Revised: Each frame contains one handwritten digit moving and bouncing inside a 64 × 64 area.

Original: The digits are chosen randomly from the MNIST dataset.
Revised: The digits are chosen randomly from the MNIST dataset.

Original: We use the same training/test split as the original MNIST dataset.
Revised: We use the same training/test split as the original MNIST dataset.

Original: For each digit, we sample 128 points.
Revised: For each digit, we sample 128 points.

Original: Point order is maintained across frames so that tracking can be employed.
Revised: Point order is maintained across frames so that tracking can be employed.

Original: We design 144 motions, including
Revised: We design 144 motions, including

Original: • nine initial locations: (1 , 1) , (1 , 3) , (1 , 5) , (3 , 1) , (3 , 3) , (3 , 5) , (5 , 1) , (5 , 3) , (5 , 5) .
Revised: • nine initial locations: (1 , 1) , (1 , 3) , (1 , 5) , (3 , 1) , (3 , 3) , (3 , 5) , (5 , 1) , (5 , 3) , (5 , 5) .

Original: • eight velocities: (1 , 1) , ( − 1 , 1) , ( − 1 , − 1) , (1 , − 1) , (2 , 2) , ( − 2 , 2) , ( − 2 , − 2) , (2 , − 2) .
Revised: • eight velocities: (1 , 1) , ( − 1 , 1) , ( − 1 , − 1) , (1 , − 1) , (2 , 2) , ( − 2 , 2) , ( − 2 , − 2) , (2 , − 2) .

Original: • two kinds of digit distortion: scaling digits horizontally or vertically, i.e ., ( | 0 .
Revised: • two kinds of digit distortion: scaling digits horizontally or vertically, i.e ., ( | 0 .

Original: 4 − 0 .
Revised: 4 − 0 .

Original: 05 t | + 0 . 6) × ( x − c x ) + c x or ( | 0 .
Revised: 05 t | + 0 . 6) × ( x − c x ) + c x or ( | 0 .

Original: 4 − 0 .
Revised: 4 − 0 .

Original: 05 t | + 0 . 6) ×
Revised: 05 t | + 0 . 6) ×

Original: ( y − c y )
Revised: ( y − c y )

Original: + c y , where ( c x , c y ) is the digit center and t ∈ [1 , 16] .
Revised: + c y , where ( c x , c y ) is the digit center and t ∈ [1 , 16] .

Original: A few motion examples are shown in Fig.
Revised: A few motion examples are shown in Fig.

Original: Because motion and appearance are independent of each other, it is challenging to recognize motion while avoiding interference from digit appearance.
Revised: Because motion and appearance are independent of each other, it is challenging to recognize motion while avoiding interference from digit appearance.

Original: [Figure]
Revised: [Figure]

Original: To exploit point correlation, anchor points and their neighbors are selected in the ﬁrst frame, which are then propagated to other frames.
Revised: To exploit point correlation, anchor points and their neighbors are selected in the ﬁrst frame, which are then propagated to other frames.

Original: In this fashion, anchors and their neighbors are tracked across frames.
Revised: In this fashion, anchors and their neighbors are tracked across frames.

Original: [Table]
Revised: [Table]

Original: As shown in Table 9, both the original PSTNet and the tracking based PSTNet achieve promising accuracy on the Moving MNIST Point Cloud dataset.
Revised: As shown in Table 9, both the original PSTNet and the tracking based PSTNet achieve promising accuracy on the Moving MNIST Point Cloud dataset.

Original: Our original PSTNet achieves similar accuracy with the tracking based PSTNet in this simulated case, demonstrating that our PSTNet does not heavily rely on point IDs or tracking.
Revised: Our original PSTNet achieves similar accuracy with the tracking based PSTNet in this simulated case, demonstrating that our PSTNet does not heavily rely on point IDs or tracking.

Original: M V ISUALIZATION OF THE O UTPUT OF E ACH PST C ONVOLUTION L AYER IN PSTN ET
Revised: M V ISUALIZATION OF THE O UTPUT OF E ACH PST C ONVOLUTION L AYER IN PSTN ET

Original: We visualize the output of each layer in the PSTNet architecture trained on MSR-Action3D in Fig.
Revised: We visualize the output of each layer in the PSTNet architecture trained on MSR-Action3D in Fig.

Original: Because we use ReLU as the activation function, all outputs are greater than zero and large output indicate high activation.
Revised: Because we use ReLU as the activation function, all outputs are greater than zero and large outputs indicate high activation.

Original: To visualize outputs, we squeeze output vectors to scalars via l 1 norm.
Revised: To visualize outputs, we squeeze output vectors to scalars via l 1 norm.

Original: We observe that:
Revised: We observe that:

Original: • For PSTConv1, because temporal kernel size l = 1 , it does not capture the temporal correlation.
Revised: • For PSTConv1, because temporal kernel size l = 1 , it does not capture the temporal correlation.

Original: In this case, PSTConv1 focuses on the appearance and therefore outputs high activation to performer contours.
Revised: In this case, PSTConv1 focuses on the appearance and therefore outputs high activation to performer contours.

Original: • PSTConv2a to PSTConv4.
Revised: • PSTConv2a to PSTConv4.

Original: When modeling the spatio-temporal structure of point cloud sequence, PST convolutions mainly focus on moving areas so as to achieve informative representations of actions.
Revised: When modeling the spatio-temporal structure of point cloud sequence, PST convolutions mainly focus on moving areas so as to achieve informative representations of actions.

Original: The visualization results support our intuition that the proposed PST convolution effectively captures dynamics of point cloud sequences.
Revised: The visualization results support our intuition that the proposed PST convolution effectively captures dynamics of point cloud sequences.

Original: N V ISUALIZATION OF 4D S EMANTIC S EGMENTATION
Revised: N V ISUALIZATION OF 4D S EMANTIC S EGMENTATION

Original: We visualize segmentation results from the Synthia 4D dataset in Fig.
Revised: We visualize segmentation results from the Synthia 4D dataset in Fig.

Original: 16 and Fig.
Revised: 16 and Fig.

Original: Our PSTNet can accurately segment most objects.
Revised: Our PSTNet can accurately segment most objects.

Original: O L IMITATION
Revised: O L IMITATION

Original: Like most deep neural networks, the proposed PSTNet relies on large-scale labeled datasets for training.
Revised: Like most deep neural networks, the proposed PSTNet relies on large-scale labeled datasets for training.

Original: A potential improvement is to integrate some learning methods, e.g ., few-shot learning (Liu et al., 2019c;f; Zhu & Yang, 2020; Liu et al., 2021), into point cloud sequences modeling to reduce the reliance on human-annotated data.
Revised: A potential improvement is to integrate some learning methods, e.g ., few-shot learning (Liu et al., 2019c;f; Zhu & Yang, 2020; Liu et al., 2021), into point cloud sequences modeling to reduce the reliance on human-annotated data.

Original: I npu t : 𝑁 =
Revised: I npu t : 𝑁 =

Original: P S T C o n v: 𝑁 = w / o t e m p o r a l m o d e li n g
Revised: P S T C o n v: 𝑁 = w / o t e m p o r a l m o d e li n g

Original: P S T C o n va : 𝑁 =
Revised: P S T C o n va : 𝑁 =

Original: P S T C o n vb : 𝑁 =
Revised: P S T C o n vb : 𝑁 =

Original: P S T C o n va : 𝑁 =
Revised: P S T C o n va : 𝑁 =

Original: P S T C o n vb : 𝑁 = P S T C o n v: 𝑁 =
Revised: P S T C o n vb : 𝑁 = P S T C o n v: 𝑁 =

Original: I npu t : 𝑁 =
Revised: I npu t : 𝑁 =

Original: P S T C o n v: 𝑁 = w / o t e m p o r a l m o d e li n g
Revised: P S T C o n v: 𝑁 = w / o t e m p o r a l m o d e li n g

Original: P S T C o n va : 𝑁 =
Revised: P S T C o n va : 𝑁 =

Original: P S T C o n vb : 𝑁 =
Revised: P S T C o n vb : 𝑁 =

Original: P S T C o n va : 𝑁 =
Revised: P S T C o n va : 𝑁 =

Original: P S T C o n vb : 𝑁 =
Revised: P S T C o n vb : 𝑁 =

Original: P S T C o n v: 𝑁 =
Revised: P S T C o n v: 𝑁 =

Original: F i gu r e 15 :
Revised: F i gu r e 15 :

Original: V i s u a li za ti on o f t h e ou t pu t
Revised: V i s u a li za ti on o f t h e ou t pu t

Original: o f eac h l a y e
Revised: o f eac h l a y e

Original: r
Revised: r

Original: i n
Revised: i n

Original: PS T
Revised: PS T

Original: N e t .
Revised: N e t .

Original: F o r t h e i npu
Revised: F o r t h e i npu

Original: t po i n t c l oud s e qu e n ce , c
Revised: t po i n t c l oud s e qu e n ce , c

Original: o l o r e n c
Revised: o l o r e n c

Original: od e s d e p
Revised: od e s d e p

Original: t
Revised: t

Original: h .
Revised: h .

Original: F o r t
Revised: F o r t

Original: h e ou t pu t s , b r i gh t e r c
Revised: h e ou t pu t s , b r i gh t e r c

Original: o l o r i nd
Revised: o l o r i nd

Original: i ca t e s h
Revised: i ca t e s h

Original: i gh e r ac ti v a ti on .
Revised: i gh e r ac ti v a ti on .

Original: I npu
Revised: I npu

Original: t s e qu e n ce s c on s i s t
Revised: t s e qu e n ce s c on s i s t

Original: o f fr a m e s .
Revised: o f fr a m e s .

Original: D u e t o t h e s p a ti a l s ub s a m p
Revised: D u e t o t h e s p a ti a l s ub s a m p

Original: li ng s s a nd t
Revised: li ng s s a nd t

Original: h e t e m po r a l s t r i d e s t , po i n t s a nd fr a m
Revised: h e t e m po r a l s t r i d e s t , po i n t s a nd fr a m

Original: e s p r og r e ss
Revised: e s p r og r e ss

Original: i v e l y d ec r
Revised: i v e l y d ec r

Original: ea s e a l ong t h e n e t w o r k d e p
Revised: ea s e a l ong t h e n e t w o r k d e p

Original: t
Revised: t

Original: h .
Revised: h .