S PATIO -T EMPORAL G RAPH S CATTERING T RANSFORM
A BSTRACT
Although spatio-temporal graph neural networks have achieved great empirical success in handling multiple correlated time series, they may be impractical in some real-world scenarios due to a lack of sufﬁcient high-quality training data.
Furthermore, spatio-temporal graph neural networks lack theoretical interpretation.
To address these issues, we put forth a novel mathematically designed framework to analyze spatio-temporal data.
Our proposed spatio-temporal graph scattering transform (ST-GST) extends traditional scattering transforms to the spatiotemporal domain.
It performs iterative applications of spatio-temporal graph wavelets and nonlinear activation functions, which can be viewed as a forward pass of spatio-temporal graph convolutional networks without training.
Since all the ﬁlter coefﬁcients in ST-GST are mathematically designed, it is promising for the real-world scenarios with limited training data, and also allows for a theoretical analysis, which shows that the proposed ST-GST is stable to small perturbations of input signals and structures.
Finally, our experiments show that i) ST-GST outperforms spatio-temporal graph convolutional networks by an increase of 35% in accuracy for MSR Action3D dataset; ii) it is better and computationally more efﬁcient to design the transform based on separable spatio-temporal graphs than the joint ones; and iii) the nonlinearity in ST-GST is critical to empirical performance.
I NTRODUCTION
Processing and learning from spatio-temporal data have received increasing attention recently.
Examples include: i) skeleton-based human action recognition based on a sequence of human poses (Liu et al.
(2019)), which is critical to human behavior understanding (Borges et al.
(2013)), and ii) multi-agent trajectory prediction (Hu et al.
(2020)), which is critical to robotics and autonomous driving (Shalev-Shwartz et al.
A common pattern across these applications is that data evolves in both spatial and temporal domains.
This paper aims to analyze this type of data by developing novel spatio-temporal graph-based data modeling and operations .
Spatio-temporal graph-based data modeling.
Graphs are often used to model data where irregularly spaced samples are observed over time.
Good spatio-temporal graphs can provide informative priors that reﬂect the internal relationships within data.
For example, in skeleton-based human action recognition, we can model a sequence of 3D joint locations as data supported on skeleton graphs across time, which reﬂects both the human physical constraints and temporal consistency (Yan et al.
Recent studies on modeling spatio-temporal graphs have followed either joint or separable processing frameworks.
Joint processing is based on constructing a single spatio-temporal graph and processing (e.g., ﬁltering) via operations on this graph (Kao et al.
Liu et al.
In contrast, a separable processing approach works separately, and possibly with different operators, across the space and time dimension.
In this case, independent graphs are used for space and time (Yan et al.
; Cheng et al.
However, no previous work thoroughly analyzes and compares these two constructions.
In this work, we mathematically study these two types of graphs and justify the beneﬁts of separable processing from both theoretical and empirical aspects.
Spatio-temporal graph-based operations.
Graph operations can be performed once the graph structure is given.
Some commonly used graph operations include the graph Fourier transform (Shuman et al.
and graph wavelets (Hammond et al. (2011)).
It is possible to extend those operations to the spatio-temporal graph domain.
For example, Grassi et al.
(2017) developed the short time-vertex Fourier transform and spectrum-based time-vertex wavelet transform.
However, those mathematically designed, linear operations show some limitations in terms of empirical performances.
In comparison, many recent deep neural networks adopt trainable graph convolution operations to analyze spatio-temporal data (Yan et al.
Liu et al.
However, most networks are designed through trial and error.
It is thus hard to explain the rationale behind empirical success and further improve the designs (Monga et al.
In this work, to ﬁll in the gap between mathematically designed linear transforms and trainable spatio temporal graph neural networks, we propose a novel spatio-temporal graph scattering transform (ST-GST), which is a mathematically designed, nonlinear operation.
Speciﬁcally, to characterize the spatial and temporal dependencies, we present two types of graphs corresponding to joint and separable designs.
We then construct spatio-temporal graph wavelets based on each of these types of graphs.
We next propose the framework of ST-GST, which adopts spatio-temporal graph wavelets followed by a nonlinear activation function as a single scattering layer.
All the ﬁlter coefﬁcients in ST-GST are mathematically designed beforehand and no training is required.
We further show that i) a design based on separable spatio-temporal graph is more ﬂexible and computationally efﬁcient than a joint design; and ii) ST-GST is stable to small perturbations on both input spatio-temporal graph signals and structures
Finally, our experiments on skeletonbased human action recognition
show
that
the proposed ST-
GST
outperforms spatio
temporal graph convolutional
networks by 35% accuracy in MSR Action3D dataset
We summarize the main contributions of this work as follows:
• We propose wavelets for both separable and joint spatio-temporal graphs.
We show that it is more ﬂexible and computationally efﬁcient to design wavelets based on separable spatio-temporal graphs; • We propose a novel spatio-temporal graph scattering transform (ST-GST), which is a non-trainable counterpart of spatio-temporal graph convolutional networks and a nonlinear version of spatiotemporal graph wavelets.
We also theoretically show that ST-GST is robust and stable in the presence of small perturbations on both input spatio-temporal graph signals and structures; • For skeleton-based human action recognition, our experiments show that: i) ST-GST can achieve similar or better performances than spatio-temporal graph convolutional networks and other nondeep-learning approaches in small-scale datasets; ii) separable spatio-temporal scattering works signiﬁcantly better than joint spatio-temporal scattering; and iii) ST-GST signiﬁcantly outperforms spatio-temporal graph wavelets because of the nonlinear activation function.
R ELATED W ORK
Scattering transforms.
Convolutional neural networks (CNNs) use nonlinearities coupled with trained ﬁlter coefﬁcients and are well known to be hard to analyze theoretically (Anthony & Bartlett (2009)).
As an alternative, Mallat (2012); Bruna & Mallat (2013) propose scattering transforms, which are non-trainable versions of CNNs.
Under admissible conditions, the resulting transform enjoys both great performance in image classiﬁcation and appealing theoretical properties.
These ideas have been extended to the graph domain (Gama et al.
(2019a) ; Zou & Lerman (2020); Gao et al. ; Ioannidis et al.
Speciﬁcally, the graph scattering transform (GST) proposed in (Gama et al. (2019a)) iteratively applies predeﬁned graph ﬁlter banks and element-wise nonlinear activation function.
In this work, we extend classical scattering transform to the spatio-temporal domain and provide a new mathematically designed transform to handle spatio-temporal data.
The key difference between GST and our proposed ST-GST lies in the graph ﬁlter bank design, where ST-GST needs to handle both spatial and temporal domains.
Spatio-temporal neural networks.
Deep neural networks have been adapted to operate on spatiotemporal domain.
For example, Liu et al.
(2019) uses LSTM to process time series information, while ST-GCN (Yan et al.
(2018)) combines a graph convolution layer and a temporal convolution layer as a unit computational block in the network architecture.
However, those networks all require a huge amount of high-quality labeled data, and training them is computationally expensive, which may make them impractical for many real-world scenarios.
Furthermore, many architectures are designed through trial and error, making it hard to justify the design choices and further improve them.
In this work, the proposed ST-GST is a nonlinear transform with a forward procedure similar to that of ST-GCN.
However, ST-GST does not require any training, which is useful in many applications where only limited training data is available.
Furthermore, since all ﬁlter coefﬁcients in ST-GST are predeﬁned, it allows us to perform theoretical analysis and the related conclusions potentially shed some light on the design of spatio-temporal networks.
Skeleton-based human action recognition.
Conventional skeleton-based action recognition models learn semantics based on hand-crafted features (Wang et al.
To handle time series information, some recurrent-neural-network-based models are proposed to capture the temporal dependencies between consecutive frames (Kim & Reiter (2017)).
Recently, graph-based approaches have gained in popularity while achieving excellent performance (Yan et al., 2018; Li et al., 2019).
In this work, our experiments focus on this task and show that ST-GST outperforms the state-of-the-art spatio-temporal graph neural networks, like MS-G3D (Liu et al., 2020), on small-scale datasets.
S PATIO -T EMPORAL G RAPH S CATTERING T RANSFORM
In this section, we ﬁrst deﬁne spatio-temporal graph structures and signals.
We next design our spatio-temporal graph wavelets.
Finally, we present ST-GST.
S PATIO
[Figure] STRUCTURES AND SIGNALS
Spatio-temporal data can be represented as a matrix X ∈ R N × T , where N is the number of the spatial positions and T is the number of time stamps.
In this matrix, each row is a time-series for a spatial node, and each column is a spatial signal at a certain time stamp.
Note that the index of spatial information can be arbitrary: we will associate each spatial location to a vertex on the spatial graph, and the edges will provide information about the relative position of the nodes.
We can reshape the matrix to form a vector x of length NT , where the element x ( s,t ) :
= x ( s − 1) T + t is the feature value corresponding to the s -th vertex at time t .
To construct a spatio-temporal graph, we create connections based on physical constraints.
For example, for skeleton-based action recognition, the spatial graph is the human skeleton graph, reﬂecting bone connections; see Fig. 1(a); and the temporal graph is a line graph connecting consecutive time stamps; see Fig. 1(b).
As a starting point, we choose a spatial graph G s =
( V s , E s , A s ) with |V s | = N , reﬂecting the graph structure of each column in X and a temporal graph G t =
( V t , E t , A t ) with |V t | =
T , reﬂecting the graph structure of each row in X .
The separable spatio-temporal design is achieved by processing the columns and rows of X separately based on their respective graphs.
As an alternative, a product graph, denoted as G = G s (cid:5) G t =
( V , E , A ) can be constructed to unify the relations in both the spatial and temporal domains, allowing us to process data jointly across space and time.
The product graph G s (cid:5) G t has |V| = NT nodes and an appropriately deﬁned NT × NT adjacency matrix A .
The operation (cid:5) interweaves two graphs to form a unifying graph structure.
The edge weight A ( s 1 ,t 1 ) , ( s 2 ,t 2 ) :
= A ( s 1 − 1) T + t 1 , ( s 2 − 1) T + t 2 characterizes the relation, such as similarity or dependency, between the s 1 -th spatial node at the t 1 -th time stamp and the s 2 -th spatial node at the t 2 -th time stamp.
There are three commonly used product graphs (Sandryhaila & Moura, 2014): i) Kronecker product: G = G s ⊗G t with graph adjacency matrix as A =
A s ⊗ A t and ⊗ represents the Kronecker product of matrices; see Fig 1(c); ii) Cartesian product: G = G s × G t with A
=
A s ⊗ I T
+ I N ⊗ A t ; see Fig 1(d); and iii) strong product: G = G s (cid:2) G t with A =
A s ⊗ A t
+ A s ⊗ I T
+ I N ⊗ A t , which can be viewed as a combination of Kronecker and Cartesian products; see Fig 1(e).
The joint spatio-temporal design is achieved based on a product graph.
In this paper, we consider designs based on both separable graphs and product graphs.
3. S PATIO - TEMPORAL GRAPH FILTERING
We now show two graph ﬁlter designs, separable and joint ﬁltering, based on the corresponding spatio-temporal graphs we just described.
For each design, we ﬁrst deﬁne the spatio-temporal graph shift, which is the most elementary graph ﬁlter and deﬁnes how information should propagate in a spatio-temporal graph.
We then propose spatio-temporal graph ﬁltering in both graph vertex and graph spectral domains.
Separable graph ﬁlters.
Given the spatial graph G s =
( V s , E s , A s ) and the temporal graph G t =
( V t , E t , A t ) , let the spatial graph shift be S s =
A s and the temporal graph shift be S t =
A t 1 .
For simplicity, we focus on the symmetric graph shifts.
For a spatio-temporal graph signal, spatial and temporal graph ﬁltering work as, H ( S s )
X = (cid:80) P [-− 1 p =
-][Equation]
{+p =
+} h p S ps X and XG (cid:62)
[-X ( (cid:80) Q − 1 q =
-][Equation]
, where h p are g q are spatial and temporal ﬁlter coefﬁcients, respectively.
In each modality, the graph ﬁlter is a polynomial of the graph shift.
The polynomial orders P and Q control the length of ﬁlters in the spatial and temporal modalities, respectively.
Note that these two values can be chosen to be different, which provides additional design ﬂexibility.
Then, a separable spatio-temporal graph ﬁltering operation can be deﬁned as
[Equation]
where the second equality follows from the property: M 1 XM T {+
+}
= ( M 1 ⊗ M 2 ) x .
We can also represent the ﬁltering process in the graph spectral domain.
Let the eigen-decomposition of the spatial and temporal graphs be S s =
V s Λ s V s (cid:62) and S t =
V t Λ t V t (cid:62) , respectively, where
V s ∈ R N × N , V t ∈ R T × T form the spatial and temporal graph Fourier bases.
The elements along the diagonals of Λ s , Λ t represent the spatial and temporal graph frequencies.
We have
[Equation]
XV t G ( Λ t ) V t (cid:62) .
Letting V
= V s ⊗ V t , the spectral representation of the separable spatio-temporal graph ﬁltering is then ( V s H ( Λ s ) V (cid:62)
s )
X ( V t G ( Λ t ) V (cid:62) t ) (cid:62) =
V ( H ( Λ s ) ⊗ G ( Λ t ))
V (cid:62) x .
Joint graph ﬁlters.
Given the joint graph structure G = G s (cid:5)G t =
( V , E , A ) , let the spatio-temporal graph shift be S
= A .
Then, a joint spatio-temporal graph ﬁltering operation can be deﬁned as:
[Equation]
where h k is the ﬁlter coefﬁcient.
The kernel function h ( λ )
=
(cid:80) K − 1 k =
h k λ k is applied to each diagonal element of Λ to obtain H ( Λ ) .
Here h ( λ ) is independent of any speciﬁc graph structure, and characterizes the ﬁlter response in the graph spectral domain.
Note that
h =
( h 0 , · · · , h K − 1 ) , h ( λ ) and H ( · ) are essentially the same thing, and are used interchangeably in this paper.
It is worth pointing out that these three product graphs share the same form of joint spatio-temporal graph ﬁltering (2) as well as the same graph Fourier bases V = V s ⊗ V t .
Following from (2), the spectral representation of the joint spatio-temporal graph ﬁltering can be formulated as
[Equation]
Comparison between separable and joint graph ﬁlters.
First of all, we stress that both separable and joint spatio-temporal graph ﬁltering share the same Fourier bases, meaning that they share the same frequency space and their difference only comes from the frequency responses.
Second, designing ﬁlters based on separable spatio-temporal graphs provides additional ﬂexibility.
Although it is conceptually simple to design graph ﬁlters directly on product graphs, the eigenvalues along the spatial and temporal domains are tied together, making it difﬁcult to adjust the frequency responses independently for the two modalities.
Moreover, two domains are forced to share the same set of ﬁlter coefﬁcients and length.
Take a ﬁlter deﬁned on Kronecker product graph as an example.
By expanding the term H ( Λ s ⊗ Λ t ) we can have that H ( Λ s ⊗ Λ t )
= (cid:80) K − 1 k =
h k ( Λ s ⊗ Λ t ) k
= (cid:80) K − 1 k =
h k ( Λ ks ⊗ Λ kt ) .
This shows that the ﬁlter coefﬁcients are applied on the product of spatial and temporal eigenvalues, making it hard to decompose and interpret the functionality of the ﬁlter in single modality.
Such limitations make them less practical for spatio-temporal signals which might have distinct patterns in each of the two modalities.
This problem is overcome by separable graph ﬁltering, where different ﬁlters are applied to each modality.
The ﬂexibility of separable graph ﬁlters means that one can design different ﬁlters ( h and g ) with independent ﬁlter lengths ( P and Q ) in the spatial and temporal domains.
However, it is worth pointing out that the representation power of these two formulations does not have a clear relationship that one is a subset of the other.
Consider a joint graph ﬁlter designed on a strong product graph with length K = 3 .
The ﬁlter kernel is deﬁned as
[Equation]
k .
Similarly, the kernel of a separable graph ﬁlter with P = Q
= 3 can be written as H ( Λ s ) ⊗ G ( Λ t )
=
{+( (cid:80) 2 p =
+}
h p Λ ps ) ⊗ {+( (cid:80) 2 q =
+}
g q Λ qt ) .
By expanding the expression and rearranging the coefﬁcients, one can obtain the coefﬁcient matrices for the joint graph ﬁlter and the separable graph ﬁlter, C 1 and C 2 , respectively; that is,
[Equation]
where ( [-(cid:80) 2 p =
 h p Λ ps-]i, j ) -th element means the coefﬁcient of term Λ i − 1 s ⊗[-( (cid:80) 2 q =
 g q-] Λ j − 1 t .
On one hand, it is obvious that C 2 is always a rank 1 matrix, while C 1 could have rank 1, 2, or 3.
So C 1 is not a special case of C 2 .
On the other hand, C 1 is always a symmetric matrix, but C 2 can be either symmetric or non-symmetric, depending on the choices of h and g .
So C 2 is also not a special case of C 1 .
Therefore, the families spanned by two designs do not have any simple relationship that one is a subset of the other.
Similar conclusions hold for the Kronecker and Cartesian products.
Third, designing based on separable spatio-temporal graphs is computationally more efﬁcient.
In a separable graph ﬁltering process, we only need to deal with two small matrix multiplications (1), instead of one large matrix-vector multiplication (2), reducing the computational cost from
O ( N 2 T 2 ) to O ( NT ( N + T )) .
In short, the joint and separable graph ﬁlters are two different design methods for spatio-temporal graphs.
Though the representation power of separable graph ﬁlters is not necessarily much stronger than joint ones, separable design enjoys the ﬂexibility, computation efﬁciency and straightforward interpretation.
Empirical performances also show that the separable design outperforms the joint one; see Section 5. Note that this separable design coincides with the basic module widely used in spatio-temporal graph convolutional networks Li et al. (2019), which consists of one graph convolution layer followed by one temporal convolution layer.
3. S PATIO - TEMPORAL GRAPH WAVELETS
In time-series analysis and image processing, wavelets are one of the best tools to design ﬁlter banks, allowing us to trade-off between the time-frequency resolutions and touching the lower bound of uncertainty principle of the time-frequency representations (Akansu & Haddad, 2000).
Inspired by this, we propose spatio-temporal graph wavelets, which include a series of mathematically designed graph ﬁlters to provide multiresolution analysis for spatio-temporal graph signals.
The proposed spatio-temporal graph wavelets are later used at each layer in the proposed ST-GST framework.
Based on two types of graph structures, we consider two designs: separable and joint wavelets.
Separable graph wavelets.
Based on separable spatio-temporal graph ﬁltering (1), we are able to design spatial graph wavelets, { H j 1 ( S s ) =
(cid:80) P − 1 p =
h ( j 1 ) p S ps } J s j
=
1 , and temporal graph wavelets, { G
[-= (cid:80) Q − 1 q =
-][Equation]
=
[Equation]
S qt } J t j 2 =1 , separately.
For each modality, the ﬁlter at scale j is deﬁned as H j ( S ) =
S 2 j − 1 − S 2 j = S 2 j − 1 ( I − S 2 j − 1 ) .
There are also many other off-the-shelf graph wavelets we can choose from.
More discussion about wavelets and their properties can be found in
Appendix A. Since two modalities are designed individually, the number of wavelet scales for each modality could be different.
This is important in practice because the number of time samples T is normally larger than the number of spatial nodes N .
For each node in spatio-temporal graph, using different wavelet scales in the two domains allows for more ﬂexibility to diffuse the signal with its neighbors.
Based on this construction, when we choose J s and J t scales for spatial and temporal domains, respectively, the overall number of scales for spatio-temporal wavelets is then J
= J s × J t .
Joint graph wavelets.
When the joint ﬁltering (2) is chosen, we can directly apply existing graph wavelet designs, such as the spectral graph wavelet transform (Hammond et al., 2011).
3. S PATIO - TEMPORAL GRAPH SCATTERING TRANSFORM
The proposed ST-GST is a nonlinear version of spatio-temporal graph wavelets, which iteratively uses wavelets followed by a nonlinearity activation function.
ST-GST includes three components: (i) spatio-temporal graph wavelets, (ii) a pointwise nonlinearity activation function σ ( · ) , and (iii) a low-pass pooling operator U .
These operations are performed sequentially to extract representative features from input spatio-temporal graph signal X .
The main difference between ST-GST and spatio-temporal graph wavelets is the application of nonlinear activation at each layer.
The nonlinear transformation disperses signals through the graph spectrum, producing more patterns in spectrum.
Separable ST-GST.
Let Z ∈ R N × T be a spatio-temporal graph signal.
At each scattering layer, we sequentially use spatial graph wavelets { H j 1 } J s j 1 =1 and temporal wavelets { G j 2 } J t j 2 =1 to convolve with Z .
Since each graph ﬁlter generates a new spatio-temporal graph signal, separable spatiotemporal graph ﬁltering generates J
=
J s × J t spatio-temporal graph signals.
Then, the nonlinear activation is applied for each spatio-temporal graph signal.
For example, the ( j 1 , j 2 ) -th signal is
Z ( j 1 ,j 2 ) =
σ ( H j 1 ( S s ) ZG (cid:62) j 2 ( S t )) .
We can treat each ﬁltered spatio-temporal graph signal as one tree node.
Given Z as the parent node, a scattering layer produces J children nodes.
To construct ST-GST, we ﬁrst initialize the input data Z
=
X be the root of the scattering tree; and then, we recursively apply scattering layers at each node to produce children nodes, growing a scattering tree; see Fig.
We can index all the nodes in this scattering tree by a unique path from the root to each node.
For example, p ( (cid:96) )
=
(( j (1)1 , j (1)2 ) , . . .
, ( j ( (cid:96) ) 1 , j ( (cid:96) ) 2 )) is the path from root to one tree node in the (cid:96) -th layer, and the signal associated with it is Z ( p ( (cid:96) ) ) .
Data matrix Z ( p ( (cid:96) ) ) is then summarized by an pooling operator U ( · ) to obtain a lower-dimensional vector φ ( p ( (cid:96) ) )
=
U (cid:0) Z ( p ( (cid:96) ) ) (cid:1) .
Various pooling methods can lead to different dimensions of scattering features.
Common choices for U ( · ) include average in the spatial domain ( U = 1 N 1 1 × N , φ = UZ ∈ R T ), average in the temporal domain ( U = 1 T 1 T × 1 , φ = ZU ∈ R N ) and average in both modalities ( U = 1 NT 1 N × T , φ =
U ◦ Z ∈ R ), where ◦ represent Hadamard product.
Finally, all scattering features φ ( p ( (cid:96) ) ) are concatenated to construct a scattering feature map Φ( S s , S t , X ) :
= {{ φ ( p ( (cid:96) ) ) } all p ( (cid:96) ) }
L − 1 (cid:96) =0 ,
Joint ST-GST.
Since we deal with a unifying graph, we can use the spatio-temporal product graph directly in combination with the ordinary graph scattering transform (Gama et al.
(2019b)).
Comparison with ST-GCNs.
One distinct difference between ST-GST and ST-GCNs lies in the fact that the trainable graph convolution in each layer of ST-GCN performs the multiplication between a spatial graph shift and the feature matrix, which only extracts low-frequency information over the graph; while ST-GST leverages multiple spatio-temporal graph ﬁlters to cover multiple frequency bands.
Furthermore, predeﬁned ﬁlter coefﬁcients conform a frame (3) in each layer of ST-GST, which is crucial for establishing the stability of ST-GST as shown in next section.
T HEORETICAL A NALYSIS
Stability is the key to designing robust and reliable algorithms.
However, since the training process of ST-GCNs is data-driven, it might be vulnerable to small perturbations added to training data, which may lead to signiﬁcant degradation in practice.
Trainable parameters make it hard to develop a theoretical analysis for ST-GCNs.
In contrast, here we show that the proposed separable ST-GST is stable to perturbations on both spatio-temporal graph signals and structures.
All proofs of statements in this section are explained thoroughly in Appendix B. Unless speciﬁed, (cid:107) x (cid:107) is the (cid:96) 2 norm for vector x , while (cid:107) X (cid:107) and (cid:107) X (cid:107) 2 are the Frobenius and spectral norm for matrix X , respectively.
Here we show the results for separable spatio-temporal graph scattering transform.
But all the results can be extended to the joint version.
Before introducing perturbations, we ﬁrst show that separable spatio-temporal graph wavelets also satisfy certain frame bounds.
Thus, with a separable construction, we can control bound constants for spatio-temporal wavelet and build tight frames.
Lemma 1.
Let { H j 1 } J s j 1 =1 and { G j 2 } J t j 2 =
be the wavelet ﬁlter bank for spatial domain and for temporal domain, respectively.
Both satisfy frame properties such that for any x ∈ R N and y ∈ R T ,
[Equation]
Then, for any Z ∈ R N × T and its corresponding reshaped vector z ∈ R NT , it holds that
[Equation]
Lemma 1 guarantees that separable design can also lead to valid wavelets.
Furthermore, when we choose both spatial { H j 1 } J s j 1 =1 and temporal { G j 2 } J t j 2 =1 to be tight frames with A 1 = B
=
A 2 = B 2 = 1 (Shuman et al., 2015), the resulting separable wavelet also conforms a tight frame.
In later context, denote B = B 1 × B 2 as the frame bound constant for separable spatio-temporal graph wavelet, and separable ST-GST is conﬁgured with L layers and J =
J s × J t scales at each layer.
4. S TABILITY TO PERTURBATION OF SPATIO - TEMPORAL GRAPH SIGNALS
Consider the perturbed spatio-temporal graph signal (cid:101)
X
= X + ∆ ∈ R N × T , where ∆ ∈ R N × T is the perturbation matrix.
Such an additive model can represent measurement noise caused by devices or adversarial perturbations added manually.
Theorem 1 shows that the feature map for perturbed signal will not deviate much from original feature map under small input perturbations.
Theorem 1.
Consider the additive noise model for input data X , it then holds that
[Equation]
The difference of output is normalized by the squared root of dimension of the ﬁnal feature map.
Note that we can construct spatio-temporal wavelet easily with B = 1 when spatial and temporal wavelets are both tight frames, then the normalized bound presented in (4) indicates that the transform is insensitive to perturbations on input signals as the factor is much smaller than 1. [Table] [Table] [Figure]
4. S TABILITY TO PERTURBATION OF SPATIO - TEMPORAL GRAPH STRUCTURES
Perturbations on the underlying graph usually happen when the graph is unknown or when the graph changes over time (Segarra et al., 2017).
Since such kind of perturbations usually happen in spatial domain, here we simply consider the structure perturbations on the spatial graph only.
Speciﬁcally, we consider the perturbed spatial graph (cid:98) S s =
S s + E (cid:62) S s + S s E , where E is the perturbation matrix and temporal graph S t is not changed.
Detailed descriptions see Appendix B.
Theorem 2.
Suppose eigenvalues { m i } Ni =1 of E ∈ R N × N are organized in order such that | m 1 | ≤ | m 2 | ≤ · · · ≤ | m N | , satisfying | m N | ≤ (cid:15)/ 2 and | m i /m
N − 1 | ≤ (cid:15) for (cid:15) > 0 and all i ’s. Suppose spatial ﬁlter bank { H j 1 } J s j 1 =1 satisﬁes max i | λh (cid:48)
i ( λ ) | ≤ C and temporal ﬁlter bank { G j 2 } J t j 2 = satisﬁes limited spectral response max i |
g i ( λ ) | ≤ D .
It then holds that
[Equation]
where (cid:15) characterizes the perturbation level.
Theorem 2 shows that ST-GST is a stable transform also for structure deformation, as the norm of change of feature map is linear in (cid:15) .
It is worth pointing out that the upper bound in both Theorems 1 and 2 only depend on the choice of ﬁlter banks and structure of scattering tree, instead of quantities related with speciﬁc graph support S s and S t that are shown in previous works (Gama et al., 2019b; Zou & Lerman, 2020; Levie et al., 2019).
E XPERIMENTAL R ESULTS
We now evaluate the performance of proposed ST-GST in skeleton-based action recognition task.
Experimental setup.
The number of layers, L , the number of spatial wavelet scales, J s , and the number of temporal wavelet scales, J t , are represented by ( J s , J t , L ) for separable ST-GST, and ( J, L ) for joint ST-GST.
Training ratio means the fraction of data used for training in the training set.
For the spatial domain, we use the skeleton graph; and for the temporal domain, we use a line graph connecting consecutive time stamps, see Fig. 1(a)(b).
Geometric scattering wavelets are used in both domain, and the nonlinear activation σ ( · ) is absolute value function which has the property of energy-preserving.
Features output by ST-GST are then utilized by random forest classiﬁer with 300 decision trees for classiﬁcation.
Comparison with state-of-the-art methods.
We consider two datasets, MSR Action3D and NTURGB+D (cross-subject).
For MSR Action3D, the proposed ST-GST is compared with GFT facilitated by temporal pyramid matching (GFT+TPM) (Kao et al., 2019), Bayesian hierarchical dynamic model (HDM) (Zhao et al., 2019), and a few deep learning approaches, including temporal convolution neural networks (Kim & Reiter, 2017), ST-GCN (Yan et al., 2018), and MS-G3D (Liu et al., 2020).
For NTU-RGB+D, Deep LSTM (Liu et al., 2019), part-aware LSTM (PA-LSTM)
(Liu et al., 2019) and spatio-temporal LSTM with trust gates (ST-LSTM+TG) (Liu et al., 2016) are included in comparison.
Methods labeled “ﬁxed topology” are modiﬁed
so as not to use adaptive training of the adjacency matrix in order for the comparison with ST-GST to be fair.
Tables 1 and 2 compares the classiﬁcation accuracies on MSR Action3D and NTU-RGB+D, respectively.
We see that even without any training, the performance of ST-GST is better than other non-deep-learning and LSTMbased methods, and is comparable with state-of-the-art GCN-based methods in large-scale dataset.
Further, ST-GST outperforms all other methods when the size of training set is small.
Fig.
3(a) shows the classiﬁcation accuracy as a function of the training ratio.
When training ratio is less than 20%, ST-GST signiﬁcantly outperforms ST-GCN.
Fig. 3(b) shows the accuracy-running time plot, reﬂecting that ST-GST is much faster than ST-GCN with similar classiﬁcation performance.
ST-GST works well in small-scale-data regime.
Table 1 and Fig. 3(a) show that ST-GST outperforms other deep learning methods in the small-scale-data regime, which can be explained as follows.
The good performance of spatio-temporal graph neural networks highly relies on the assumption that the training data is abundant.
When the size of training set is limited, most of them can be easily trapped into bad local optima due to overﬁtting, resulting in a signiﬁcant drop of classiﬁcation accuracy.
But in practice, obtaining a huge amount of training data with high-quality labels could be extremely expensive.
On the other hand, since ST-GST is a non-trainable framework, ﬁlter coefﬁcients in ST-GST are mathematically designed rather than trained by data, which avoids the problem of overﬁtting when the training ratio is low.
Another advantage of ST-GST compared to ST-GCN is that it requires less computation because no training process is involved in ST-GST.
Separable design is better than joint design.
Tables 1 and 2 also show that separable spatiotemporal graph wavelets work much better than joint ones, achieving 25% increase in classiﬁcation accuracy for MSR Action3D dataset.
The result is consistent with our analysis in Section 3.2.
The intuition is that when dealing with spatio-temporal data generated from complex structures like skeleton sequences, the ﬁxed dependencies generated by product graphs highly restrict the way how signals can be diffused in spatio-temporal graphs and thus limit the efﬁciency of feature extraction.
Nonlinearity is beneﬁcial.
Fig. 3(c) compares ST-GST with and without nonlinearity and shows that it is critical to ST-GST, also reﬂecting the potential effect of nonlinearity in ST-GCNs.
C ONCLUSIONS
In this work we propose a novel spatio-temporal graph scattering transform (ST-GST), which can be viewed as one forward pass of spatio-temporal graph convolutional networks (ST-GCNs) without any training.
ST-GST is stable to small perturbations on both input signals and structures.
Our experiments show that: i) ST-GST can achieve better performance than both non-deep-learning and
ST-GCNs based methods when the size of training samples is limited; ii) designing spatial and temporal graph ﬁlters separately is more ﬂexible and computationally efﬁcient than designing them jointly; and iii) the nonlinearity is critical to the performance.  A CKNOWLEDGEMENT
A D IFFERENT D ESIGN OF G RAPH W AVELETS
There are many off-the-shelf, well-developed graph wavelets we can choose.
They mainly focus on extracting features from multiple frequency bands of input signal spectrum.
Some of them are shown as follows.
Monic Cubic wavelets.
Monic Cubic wavelets (Hammond et al., 2011) deﬁne the kernel function h ( λ ) as h [Equation] =  [Equation]
for λ < 1; −
[Equation]
+
[Equation] λ [Equation] + λfor 1 ≤ λ ≤ 2; 2 /λ for λ > 2 .
Different scales of ﬁlters are implemented by scaling and translation of above kernel function.
Itersine wavelets.
Itersine wavelets deﬁne the kernel function at scale j as
[Equation]
Itersine wavelets form tight frames.
Geometric scattering wavelets.
Geometric scattering wavelet ﬁlter bank (Gao et al., 2019) contains a set of ﬁlters based on lazy random walk matrix.
The ﬁlter at scale j is deﬁned as H j ( S ) =
S 2 j
[Equation]
S
[Equation]
S
[Equation]
1 ( I − S 2 j − 1 ) , where S = 12 ( I + AD − 1 ) is the lazy random walk matrix and D is the degree matrix.
Note that one is also allowed to customize either spatial or temporal graph wavelets, once they conform a frame and satisfy integral Lipschitz constraint shown as follows
[Equation]
where A, B are scalar constants and h (cid:48) ( · ) is the gradient of the kernel function.
B P ROOFS B. P ROOF OF L EMMA
By reshaping the signal from Z to z with Z s,t = z
( s − 1) T + t , we can have that
[Equation]
Since S s and S t do not change over computation process, we just use H j 1 and G j 2 to represent
H j 1 ( S s ) and G j 2 ( S t ) , respectively.
Suppose H
j 1 =   hh 1 N .
h Nh NN   ∈ R N × N , then we have the Kronecker product as H j 1 ⊗ G j 2 = 

[Equation]
G
[Equation]
N G j 2 .
h
[Equation]
G
[Equation]
h NN G j  .
Apply it to vector z and we can have a ﬁltered signal y j 1 ,j
= ( H j 1 ⊗ G j 2 )
z ∈ R NT .
The ﬁrst T elements of y can also be written as
[Equation]
Therefore we have
A 22 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i
=
h 1 i  
Z i, 1 Z i,...
Z i,T   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:88) j(cid:107)
y j 1 ,j 2 (1 : T ) (cid:107) 2 ≤ B 22 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i
=
h 1 i  
Z i, 1 Z i,...
Z i,T   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Thus (cid:80) j 2 (cid:107) y
j 1 ,j 2 (cid:107) 2 can be sandwiched as
A
N (cid:88) k
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i = h ki  
Z i, 1 Z i,...
Z i,T   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:88) j(cid:107)
y j 1 ,j 2 (cid:107) 2 ≤ B
N (cid:88) k
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i = h ki  
Z i, 1 Z i,...
Z i,T   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
By deﬁnition of vector (cid:96) 2 norm, we can rewrite the upper and lower bound in Eq.
(6) as
A
T (cid:88) i = (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
H j 
Z 1 ,i Z 2 ,i ...
Z N,i   (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:88) j(cid:107)
y j 1 ,j 2 (cid:107) 2 ≤ B
T (cid:88) i = (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
H j 
Z 1 ,i Z 2 ,i ...
Z N,i   (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Summing above quantity over j 1 gives us that
A 21 A 22 (cid:107) Z (cid:107)
=
A 21 A
T (cid:88) i = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Z 1 ,i Z 2 ,i ...
Z N,i   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:88)
j 1 ,j(cid:107)
y j 1 ,j 2 (cid:107) 2 ≤ B 21 B
T (cid:88) i = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Z 1 ,i Z 2 ,i ...
Z N,i   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
B 21 B 22 (cid:107) Z (cid:107) 2 , which completes the proof.
Lemma 1 is a very handful result.
It shows that we can easily construct new spatio-temporal wavelets just by combining spatio and temporal ones.
Moreover, the constants for new frame bound can be easily obtained once we know the characteristics of the wavelets in each domain.
In particular, it also provides us a convenient way to build tight frames for spatio-temporal data analysis with A = B , because we just need to choose tight frames for spatial and temporal domain separately without considering possible correlations.
B. P ROOF OF T HEOREM
We are considering pooling operator U ( · ) as average in spatial domain in this proof, so U = 1 N 1 1 × N and φ = UZ ∈ R T .
The proof techniques can be easily generalized to any form of U ( · ) .
When reshaping Z ∈ R N × T to z ∈ R NT , the new pooling operator can be simply represented as
[Equation]
that (cid:107) U (cid:48) (cid:107) 2 = 1 √ N .
Consider scattering tree nodes at the last layer L − 1 .
Suppose they are indexed from 1 to J L − 1 associated with signal a 1 , · · · , a J L − 1 , and their parent nodes are indexed from 1 to J L − 2 associated with signal b 1 , · · · , b J L − 2 .
When the input data X is perturbed, all signals in scattering tree will change correspondingly.
Here we simply denote them as (cid:101) a , (cid:101) b .
Then for the change of feature vector located at node with a 1 , it holds that
[Equation]
[Equation]
where j 1 = j 2 = 1 .
The last inequality holds because we are using absolute value function as nonlinear activation, which is non-expansive.
Summing above quantity over j 1 , j 2 and by the frame bound proved in Lemma 1, we can have that
[Equation]
Note that for sum of square norm of change at layer L − 2 it is
[Equation]
Compare Eq.
(7) and (8).
The upper bound only differs with a factor B 2 .
Then by induction we can have that
[Equation]
Normalize it with the dimension of ﬁnal feature map, we have
[Equation]
[Equation]
B. P ROOF OF T HEOREM
Perturbations on the underlying graph usually happen when the graph is unknown or when the graph changes over time (Segarra et al., 2017).
Take skeleton-based action recognition as an example.
Some joints may be misrecognized with others due to measurement noise of devices during certain frames, thus the location signals of those joints are interchanged.
This leads to different spatial graph structures at those time stamps.
Since such kind of perturbations usually happen in spatial domain, here we simply consider the structure perturbations on the spatial graph only.
But the results can be extended to more general cases.
Consider the original spatio-temporal graph as G with spatial graph shift matrix S s and temporal one S t , and the perturbed graph as (cid:98) G with (cid:98) S s and S t .
We ﬁrst show that ST-GST is invariant to node permutations in spatial domain, where the set of permutation matrices is deﬁned as
P = (cid:8) P ∈ { 0 , 1 } N × N : P 1 = 1 , P (cid:62) 1 = 1 , PP (cid:62) =
I N (cid:9) .
Note that we are considering average in spatial domain for U ( · ) , so U = 1 N 1 1 × N and φ = UZ ∈ R T , (cid:98) U = UP .
Lemma 2.
Consider the spatial permutation (cid:98) S s
= P (cid:62) S s P and input data (cid:98) X
= P (cid:62) X are also permuted in spatial domain correspondingly.
Then, it holds that
[Equation]
Proof.
Note that the permutation holds for all signals computed in scattering tree; that is to say,
(cid:98) Z ( p ( (cid:96) ) )
=
P (cid:62) Z ( p ( (cid:96) ) ) .
Suppose for path p
( (cid:96) )
the last two ﬁlter are chosen as H ( (cid:98) S s ) and G ( S t ) , then the feature vector after pooling with respect to new graph support and data can be written as
[Equation]
The last equation holds due to deﬁnition of H ( S ) .
Since nonlinear activation is applied elementwise, we can rewrite it as
[Equation]
This conclusion holds independently of speciﬁc path p ( (cid:96) ) , so it holds for all feature vector after pooling in scattering tree.
Since ﬁnal feature map is just a concatenation of all feature vectors, the proof is complete.
Lemma 2 shows that the output of ST-GST is essentially independent of the node ordering in spatial domain, as long as the permutation is consistent across all time stamps.
This result is intuitive because the output of graph convolution should only depend on relative neighborhood structure of each node.
Since node reordering will not alter neighborhood topology, the output should remain unchanged.
Based on Lemma 2, we use a relative perturbation model for structure modiﬁcations (Gama et al., 2019b), which focuses more on the change of neighborhood topology compared to absolute perturbations adopted in Levie et al.
Deﬁne the set of permutations that make S s and (cid:98) S the closet as P s :=
arg min P ∈P (cid:107) P (cid:62) (cid:98) S
s P − S s (cid:107) 2 .
Consider the set of perturbation matrices
E ( S , (cid:98) S ) =
{ E | P (cid:62) (cid:98) S s P = S
s + E (cid:62) S s + S
s E , P ∈ P s , E ∈ R N × N } .
Then the relative distance to measure structure perturbations can be deﬁned as
[Equation]
Note that if (cid:98) S s = P (cid:62) S s P , meaning that the structure perturbation is purely permutation, then the relative distance d ( S s , (cid:98) S s ) = 0 , which is consistent with result shown in Lemma 2.
Therefore, without loss of generality, we can assume that P = I N and (cid:98) S s =
S s + E (cid:62) S s + S
s E in later context.
With this formulation, we are ready to prove Lemma 3.
Lemma 3.
Suppose eigenvalues { m i } Ni =1 of E are organized in order such that | m 1 | ≤ | m 2 | ≤ · · · ≤ | m N | , satisfying | m N | ≤ (cid:15)/ 2 and | m i /m
N − 1 | ≤ (cid:15) for (cid:15) > 0 .
For spatial graph ﬁlter H ( S s ) and temporal graph ﬁlter G ( S t ) , denote their kernel functions as h ( λ ) and g ( λ ) , respectively.
If for all λ , h ( λ ) is chosen to satisfy integral Lipschitz constraint | λh (cid:48) ( λ ) | ≤ C and g ( λ ) has bounded spectral response | g ( λ ) | ≤ D .
Then it holds that
[Equation]
Proof.
From Proposition 2 in Gama et al.
(2019b)
we can have that when E satisﬁes above conditions, (cid:107) H ( S s ) − H ( (cid:98) S s ) (cid:107) 2 ≤ (cid:15)C + O ( (cid:15) 2 ) .
So
[Equation]
H
[Equation]
S
[Equation]
The second line holds because H ( S s ) − H ( (cid:98) S s ) is a symmetric matrix, which can be written as eigen-decomposition as FΩF (cid:62) .
And ( FΩF (cid:62) ) ⊗ ( VΛV T ) =
( F ⊗ V )
( Ω ⊗ Λ )( F ⊗ V ) (cid:62) holds, which ﬁnishes the proof.
As for general structural perturbations, where we want to ﬁnd (cid:107) H ( S s ) ⊗
G ( S t ) − H ( (cid:98) S s ) ⊗ G ( (cid:98) S t ) (cid:107) 2 , we can add and subtract term H ( (cid:98) S s ) ⊗ G ( (cid:98) S t ) , use triangle inequality and further bound those two terms with more assumptions on h ( λ ) and g ( λ ) .
The bound shown in Lemma 3 indicates that the difference of output caused by changing spatial graph support from S s to (cid:98) S s is proportional to (cid:15) , which is a scalar characterizing the level of the perturbation.
Constraints on eigenvalues of E limits the change of graph structure.
A more detailed description explaining the necessity of such constraints can be found in Gama et al.
(2019b).
With
Lemma 3 in hand, we are ready to show the change of feature vector after pooling at each node in scattering tree when such structure perturbations happen.
Lemma 4.
Consider a ST-GST with L layers and J
= J s × J t scales at each layer.
Suppose that the graph ﬁlter bank forms a frame with upper bound B = B 1 × B 2 , where B 1 , B 2 are frame bounds for spatial and temporal domain, respectively.
Suppose for all λ , spatial wavelet ﬁlter bank { H j 1 } J s j 1 = satisﬁes max i | λh (cid:48)
i ( λ ) | ≤ C and temporal wavelet ﬁlter bank { G j 2 } J t j 2 =1 satisﬁes max i | g i ( λ ) | ≤
D , and other conditions the same as Lemma 3.
Then for the change of feature vector φ p ( (cid:96) ) associated with path p ( (cid:96) )
it holds that
[Equation]
Proof.
Expand (cid:107) φ p (
[Equation]
) ( S s , S t , X ) − φ
[Equation]
[Equation]
S s , S t , X )
[Equation]
as
[Equation]
where (cid:107) U (cid:48) (cid:107) 2 = 1 / √ N
and ( σ ( H j ( (cid:96)
[Equation]
1 ( S s ) ⊗ G j ( (cid:96)
[Equation]
2 ( S t ))) p ( [Equation] ) is a shorthand for applying spatiotemporal ﬁlters and nonlinear activation in order to input data (cid:96) times according to the path p ( (cid:96) ) .
Add and subtract term σ ( H
[Equation]
t ))
x and apply triangle inequality, we can have that
[Equation]
Recursive quantities can be observed above and the bound can be solved explicitly (Gama et al., 2019b).
By induction and conclusion from Lemma
[Equation]
we can get that
[Equation]
[Table]
Multiplying the coefﬁcient 1 / √ N caused by pooling gets us the ﬁnal result.
Note that the upper bound in Lemma 4 holds for all path of length (cid:96) .
Thus the square norm of change in ﬁnal feature map can be summarized by the sum of square norm of change at each layer, which ﬁnishes the proof of Theorem 2.
C A DDITIONAL E XPERIMENTS C. D ATASET
MSR Action3D dataset (Li et al., 2010) is a small dataset capturing indoor human actions.
It covers 20 action types and 10 subjects, with each subject repeating each action 2 or 3 times.
The dataset contains 567 action clips with maximum number of frames 76; however, 10 of them are discarded because the skeleton information are either missing or too noisy (Wang et al., 2012).
For each clip, locations of 20 joints are recorded, and only one subject is present.
Training and testing set is decided by cross-subject split for this dataset, with 288 samples for training and 269 for testing.
NTU-RGB+D (Liu et al., 2019) is currently the largest dataset with 3D joints annotations for human action recognition task.
It covers 60 action types and 40 subjects.
The dataset contains 56,880 action clips with maximum number of frames 300, and there are 25 joints for each subject in one clip.
Each clip is guaranteed to have at most 2 subjects.
The cross-subject benchmark of NTU-RGB+D includes 40,320 clips for training and 16,560 for testing.
Full table of performance on MSR Action3D dataset .
The table contains performance comparison for different algorithms with different set of parameters on MSR Action3D dataset.
Note that the triple shown after ST-GST represents the value for ( J s , J t , L ) .
Methods labeled “ﬁxed topology” are modiﬁed
so as not to use adaptive training of the adjacency matrix in order for the comparison with ST-GST to be fair.
Methods labeled “learnable topology” means that we use adaptive training for adjacency matrix to further validate our claim.
Other conﬁgurations of compared methods are then set by default.
From the table we can see that ST-GST outperforms all other methods even when the graph topology can be learned by neural networks.
The intuition behind this is that deep learning methods need large amount of training data due to the complex structures, and it can easily
[Table] [Table]
be trapped into bad local optima due to overﬁtting when the size of training set is limited, which is common in practice.
Also the good performance of ST-GST in sparse label regime could potentially inspire active learning for processing spatio-temporal data (Bilgic et al., 2010).
Performance on MSR Action3D dataset with standard deviations.
We repeat part of our experiments 20 times on MSR Action3D dataset, especially for joint approaches, to obtain the standard deviations of classiﬁcation accuracy.
The results are shown in Table 4.
Note that since ST-GST is a mathematically designed transform, the output features should be the same for different trails, and the randomness comes from classiﬁers used later (random forest in this case).
It can be seen that the standard deviations are comparable in all these methods, and therefore the conclusion that separable
ST-GST consistently outperforms joint ST-GST still holds.
Comparison between different choices of wavelets.
In practice we ﬁnd that using graph geometric scattering wavelets (Gao et al., 2019) for both spatial and temporal domain can achieve the best performance, which is reported in main text.
Classiﬁcation accuracy using other type of wavelets is shown here.
All experiments performed here are separable ST-GST with J s = 5 , J t = 15 , L = 3 on
MSR Action3D dataset.
An interesting observation is that there is a signiﬁcant reduction in accuracy when we change temporal wavelet from diffusion based one (Geometric) to spectrum based one (MonicCubic or Itersine).
This may caused by the design of different wavelets.
Stability of ST-GST.
We also show the classiﬁcation accuracy under different level of perturbations on spatio-temporal signals and spatial graph structures in Fig.
The experiments are con-
[Table]
[Figure]
ducted on MSR Action3D dataset.
For signal perturbation, signal-to-noise ratio (SNR) is deﬁned as 10 log (cid:107) X (cid:107) 2 (cid:107) ∆ (cid:107) 2 .
For structure perturbation, E is set to be a diagonal matrix, whose diagonal elements satisfy corresponding constraints on (cid:15) .
From both Fig. 4(a) and (b) we can see that ST-GST is stable and will not deviate much from original output when the perturbations are small.