Graph Scattering beyond Wavelet Shackles
Abstract
This work develops a ﬂexible and mathematically sound framework for the design and analysis of graph scattering networks with variable branching ratios and generic functional calculus ﬁlters.
Spectrally-agnostic stability guarantees for node- and graph-level perturbations are derived; the vertex-set non-preserving case is treated by utilizing recently developed mathematical-physics based tools.
Energy propagation through the network layers is investigated and related to truncation stability.
New methods of graph-level feature aggregation are introduced and stability of the resulting composite scattering architectures is established.
Finally, scattering transforms are extended to edge- and higher order tensorial input.
Theoretical results are complemented by numerical investigations: Suitably chosen scattering networks conforming to the developed theory perform better than traditional graphwavelet based scattering approaches in social network graph classiﬁcation tasks and signiﬁcantly outperform other graph-based learning approaches to regression of quantum-chemical energies on QM 7 .
Introduction
Euclidean wavelet scattering networks [22, 4] are deep convolutional architectures where outputfeatures are generated in each layer.
Employed ﬁlters are designed rather than learned and derive from a ﬁxed (tight) wavelet frame, resulting in a tree structured network with constant branching ratio.
Such networks provide state of the art methods in settings with limited data availability and serve as a mathematically tractable model of standard convolutional neural networks (CNNs).
Rigorous investigations — establishing remarkable invariance- and stability properties of wavelet scattering networks — were initially carried out in [22].
The extensive mathematical analysis [38] generalized the term ’scattering network’ to include tree structured networks with varying branching rations and frames of convolutional ﬁlters, thus signiﬁcantly narrowing the conceptual gap to general CNNs.
With increasing interest in data on graph-structured domains, well performing networks generalizing
Euclidean CNNs to this geometric setting emerged [18, 5, 9].
If efﬁciently implemented, such graph convolutional networks (GCNs) replace Euclidean convolutional ﬁlters by functional calculus ﬁlters; i.e. scalar functions applied to a suitably chosen graph-shift-oprator capturing the geometry of the underlying graph [18, 14, 9].
Almost immediately, proposals aimed at extending the success story of
Euclidean scattering networks to the graph convolutional setting began appearing:
In [48], the authors utilize dyadic graph wavelets (see e.g. [14]) based on the non-normalized graph Laplacian resulting in a norm preserving graph wavelet scattering transform.
In [10], diffusion wavelets (see e.g. [8]) are used to construct a graph scattering transform enjoying spectrum-dependent stability guarantees to graph level perturbations.
For scattering transforms with N layers and K distinct functional calculus ﬁlters, the work [11] derives node-level stability bounds of O p K N { 2 q and conducts corresponding numerical experiments choosing diffusion wavelets, monic cubic wavelets [14] and tight Hann wavelets [35] as ﬁlters.
In [12] the authors, following [8], construct so called geometric wavelets and establish the expressivity of a scattering transform based on such a frame through extensive numerical experiments.
A theoretical analysis of this and a closely related wavelet based scattering transform is the main focus of [28].
Additionally, graph-wavelet based scattering transforms have been extended to the spatio-temporal domain [27], utilized to overcome the problem of oversmoothing in GCNs [25] and pruned to deal with their exponential (in network depth) increase in needed resources [15].
Common among all these contributions is the focus on graph wavelets, which are generically understood to derive in a scale-sampling procedure from a common wavelet generating kernel function g : R Ñ R satisfying various properties [14].
Established stability or expressivity properties — especially to structural perturbations — are then generally linked to the speciﬁc choice of the wavelet kernel g and utilized graph shift operator [10, 28].
This severely limits the diversity of available ﬁlter banks in the design of scattering networks and draws into question their validity as models for more general GCNs whose ﬁlters generically do not derive from a wavelet kernel.
A primary focus of this work is to provide alleviation in this situation: After reviewing the graph signal processing setting in Section 2, we introduce a general framework for the construction of (generalized) graph scattering transforms beyond the wavelet setting in Section 3.
Section 4 establishes spectrumagnostic stability guarantees on the node signal level and for the ﬁrst time also for graph-level perturbations.
To handle the vertex-set non-preserving case, a new ’distance measure’ for operators capturing the geometry of varying graphs is utilized.
After providing conditions for energy decay (with the layers) and relating it to truncation stability, we consider graph level feature aggregation and higher order inputs in Sections 5 and 6 respectively.
In Section 7 we then provide numerical results indicating that general functional calculus ﬁlter based scattering is at least as expressive as standard wavelet based scattering in graph classiﬁcation tasks and outperforms leading graph neural network approaches to regression of quantum chemical energies on QM 7 .
Graph Signal Processing
Taking a signal processing approach, we consider signals on graphs as opposed to graph embeddings:
Node-Signals: Given a graph p G, E q , we are primarily interested in node-signals, which are functions from the node-set G to the complex numbers, modelled as elements of C | G | .
We equip this space with an inner product according to x f, g y
“ ř | G | i “ 1 f i g i µ i (with all vertex weights µ i ě 1 ) and denote the resulting inner product space by (cid:96) 2 p G q .
We forego considering arbitrary inner products on
C | G | solely in the interest of increased readability.
Functional Calculus Filters: Our fundamental objects in investigating node-signals will be functional calculus ﬁlters based on a normal operator ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p G q .
Prominent examples include the adjacency matrix W , the degree matrix D , normalized p
´ D ´ 12 WD ´ 12 q or un-normalized ( L : “ D ´ W ) graph Laplacians Writing normalized eigenvalue-eigenvector pairs of ∆ as p
λ
i , φ i q | G | i “ 1 , the ﬁlter obtained from applying g : C Ñ C is given by g p ∆ q f “ ř | G | i “ 1 g p λ
i qx φ
i , f y (cid:96) 2 p V q φ i .
The operator we utilize in our numerical investigations of Section 6, is given by L : “ L { λ max p L q .
We divide by the largest eigenvalue to ensure that the spectrum σ p L q is contained in the interval r 0 , 1 s , which aids in the choice of functions from which ﬁlters are derived.
Generalized Frames: We are most interested in ﬁlters that arise from a collection of functions adequately covering the spectrum of the operator to which they are applied.
To this end we call a collection t g i p¨qu i P I of functions a generalized frame if it satisﬁes the generalized frame condition
A ď ř
i P I
| g i p c q| 2 ď B for any c in C for constants A ; B ą 0 .
As proved in Appendix B, this condition is sufﬁcient to guarantee that the associated operators form a frame:
Theorem 2.1.
Let ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p
G q be normal.
If the family t g i p¨qu i P I of bounded functions satisﬁes A ď
ř i P I
| g i p c q| 2 ď B for all c in the spectrum σ p ∆ q , we have ( @ f P (cid:96) 2 p G q )
[Equation]
Notably, the functions t g i u i P
I need not be continuous:
In fact, in our numerical implementations, we will – among other mappings – utilize the function δ 0 p¨q , deﬁned by δ 0 p 0 q “ 1 and δ
p c q “ 0 for c ‰ 0 as well as a modiﬁed cosine, deﬁned by cos p 0 q “ 0 and cos p c q “ cos p c q for c ‰ 0 .
The Generalized Graph Scattering Transform
A generalized graph scattering transform is a non-linear map Φ based on a tree structured multilayer graph convolutional network with constant branching factor in each layer.
For an input signal f P (cid:96) 2 p G q , outputs are generated in each layer of such a scattering network, and then concatenated to form a feature vector in a feature space F .
The network is built up from three ingredients:
Connecting Operators: To allow intermediate signal representations in the ’hidden’ network layers to be further processed with functional calculus ﬁlters based on varying operators, which might not all be normal for the same choice of node-weights, we allow these intermediate representations to live in varying graph signal spaces.
In fact, we do not even assume that these signal spaces are based on a common vertex set.
This is done to allow for modelling of recently proposed networks where input- and ’processing’ graphs are decoupled (see e.g. [1, 36]), as well as architectures incorporating graph pooling [20].
Instead, we associate one signal space (cid:96) 2 p G n q to each layer n .
Connecting operators are then (not necessarily linear) operators P n : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n q connecting the signal spaces of subsequent layers.
We assume them to be Lipschitz continuous ( } P p f q ´ P p g q} (cid:96) 2 p G n ´ 1 q ď R ` } f ´ g } (cid:96) 2 p G n q q and triviality preserving ( P p 0 q “ 0 ).
For our original node-signal space we also write (cid:96) 2 p G q ” (cid:96) 2 p G 0 q .
Non-Linearities:
To each layer, we also associate a (possibly) non-linear function ρ n : C Ñ C acting poinwise on signals in (cid:96) 2 p G n q .
Similar to connecting operators, we assume ρ n preserves zero and is Lipschitz-continuous with Lipschitz constant denoted by L ` n .
This deﬁnition allows for the absolute value non-linearity, but also ReLu or – trivially – the identity function.
Operator Frames: Beyond these ingredients, the central building block of our scattering architecture is comprised of a family of functional calculus ﬁlters in each layer.
That is, we assume that in each layer, the node signal space (cid:96) 2 p G n q carries a normal operator ∆ n and an associated collection of functions comprised of an output generating function χ n p¨q as well as a ﬁlter bank t g γ n p¨qu γ n P Γ n indexed by an index set Γ n .
As the network layer n varies (and in contrast to wavelet-scattering networks) we allow the index set Γ n as well as the collection t χ n p¨qu Ť t g γ n p¨qu γ n P Γ n of functions to vary.
We only demand that in each layer the functions in the ﬁlter bank together with the output generating function constitute a generalized frame with frame constants A n , B n ě 0 .
We refer to the collection of functions Ω N : “ p ρ n , t χ n p¨qu Ť t g γ n p¨qu γ n P Γ n q Nn “ 1 as a module sequence and call D N : “ p P n , ∆ n q Nn “ 1 our operator collection .
The generalized scattering transform is then constructed iteratively:
To our initial signal f P (cid:96) 2 p G q we ﬁrst apply the connecting operator P 1 , yielding a signal representation in (cid:96) 2 p G 1 q .
Subsequently, we apply the pointwise non-linearity ρ 1 .
Then we apply our graph ﬁlters t χ 1 p ∆ 1 qu
[Equation]
to ρ 1 p P 1 p f qq yielding the output V 1 p f q : “ χ 1 p ∆ 1 q ρ 1 p P 1 p f qq as well as the intermediate hidden representations t U 1 r γ 1 sp f q : “ g γ 1 p ∆ 1 q ρ 1 p P 1 p f qqu γ 1 P Γ 1 obtained in the ﬁrst layer.
Here we have introduced the one-step scattering propagator U n r γ n s : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n q mapping f ÞÑ g γ n p ∆ n q ρ n p P n p f qq as well as the output generating operator
V n :
[Equation]
p G n
[Equation]
q Ñ (cid:96) 2 p G n q mapping f to χ n p ∆ n q ρ n p P n p f qq .
Upon deﬁning the set Γ N ´ 1 : “ Γ N ´ 1 ˆ ... ˆ Γ 1 of paths of length p N ´ 1 q terminating in layer N ´ 1 (with Γtaken to be the one-element set) and iterating the above procedure, we see that the outputs generated in the N th -layer are indexed by paths Γ N ´terminating in the previous layer.
Outputs generated in the N th layer are thus given by t V N ˝
U r γ N ´ 1 s˝ ...
˝
U r γ 1 sp f qu p γ N ´ 1 ,...,γ 1 qP Γ N ´ 1 .
Concatenating the features obtained in the various layers of a network with depth N , our full feature vectors thus live in the feature space
[Equation]
The associated canonical norm is denoted } ¨ } F N .
For convenience, a brief review of direct sums of spaces, their associated norms and a discussion of corresponding direct sums of maps is provided in
Appendix A. We denote the hence constructed generalized scattering transform of length N , based on a module sequence Ω N and operator collection D N by Φ N .
In our numerical experiments in Section 7, we consider two particular instantiations of the above general architecture.
In both cases the utilized shift-operator is L : “ L { λ max p L q , node weights satisfy µ i “ 1 , the branching ratio in each layer is chosen as 4 and the depth is set to N “ 4 as well.
The connecting operators are set to the identity and non-linearities are set to the modulus ( |¨| ).
The two architectures differ in the utilized ﬁlters, which are repeated in each layer and depicted in Fig.
Postponing a discussion of other parameterchoices, we note here that the ﬁlters t sin p π { 2 ¨q , cos p π { 2 ¨qu provide a high and a low pass ﬁlter on the spectrum σ p L q Ď r 0 , 1 s , while t sin p π ¨q , cos p π ¨qu provides a spectral reﬁnement of the former two ﬁlters.
The inner two elements of the ﬁlter bank in Architecture II thus separate an input signal into high- and low-lying spectral components.
The outer two act similarly at a higher spectral scale.
Additionally Architecture I – utilizing cos and δ 0 as introduced Section 2 – prevents the lowest lying spectral information from propagating.
Instead it is extracted via δ
p¨q in each layer.
Note that Id arises from applying the constant- 1 function to L .
Normalizations are chosen to generate frames with upper bounds B ž 1 .
Stability Guarantees
In order to produce meaningful signal representations, a small change in input signal should produce a small change in the output of our generalized scattering transforms.
This property is captured in the result below, which is proved in Appendix C.
Theorem 4.1.
With the notation of Section 3, we have for all f, h P (cid:96) 2 p G q :
[Equation]
In the case where upper frame bounds B n and Lipschitz constants L ` n and R ` n are all smaller than or equal one, this statement reduces to the much nicer inequality:
[Equation]
Below, we always assume R ` n , L ` n ď 1 as this easily achievable through rescaling.
We will keep B n variable to demonstrate how ﬁlter size inﬂuences stability results.
As for our experimentally tested architectures (cf.
Fig. 2), we note for Architecture I that B n “ 1 { 2 for all n , so that (2) applies.
For
Architecture II we have B n “ 3 , which yields a stability constant of ?
1 ` 2 ¨ 3 ` 2 ¨
3 2 ` 2 ¨ 3 3 “ 9 .
Similar to other constants derived in this section, this bound is however not necessarily tight.
Operators capturing graph geometries might only be known approximately in real world tasks; e.g. if edge weights are only known to a certain level of precision.
Hence it is important that our scattering representation be insensitive to small perturbations in the underlying normal operators in each layer, which is captured by our next result, proved in Appendix D. Smallness here is measured in Frobenius norm } ¨ } F , which for convenience is brieﬂy reviewed in Appendix A).
Theorem 4.2.
Let Φ N and r
Φ N be two scattering transforms based on the same module sequence Ω N and operator sequences D N , r D N with the same connecting operators ( P n “ r P n ) in each layer.
Assume R ` n , L ` n ď 1 and B n ď B for some B and n ď N .
Assume that the respective normal operators satisfy } ∆ n ´ r ∆ n } F ď δ for some δ ą 0 .
Further assume that the functions t g γ n u γ n P Γ n and χ n in each layer are Lipschitz continuous with associated Lipschitz constants satisfying L 2 χ n `
[Equation]
2 for all n ď N and some D ą 0 .
Then we have
[Equation]
for all f P (cid:96) 2 p G q .
If B ď 1 { 2 , the stability constant improves to
[Equation]
The condition B ď 12 is e.g. satisﬁed by our Architecture I, but –strictly speaking– we may not apply Theorem 4.2, since not all utilized ﬁlters are Lipschitz continuous.
Remark D.3 in Appendix D however shows, that the above stability result remains applicable for this architecture as long as we demand that ∆ and r ∆ are (potentially rescaled) graph Laplacians.
For Architecture II we note that
D “ π ? 10 { 2 and thus the stability constant is given by
[Equation]
We are also interested in perturbations that change the vertex set of the graphs in our architecture.
This is important for example in the context of social networks, when passing from nodes representing individuals to nodes representing (close knit) groups of individuals.
To investigate this setting, we utilize tools originally developed within the mathematical physics community [29]:
Deﬁnition 4.3.
Let H and r H be two ﬁnite dimensional Hilbert spaces.
Let ∆ and r ∆ be normal operators on these spaces.
Let J : H Ñ r H and r J : r H Ñ H be linear maps — called identiﬁcation operators .
We call the two spaces δ -quasi-unitarily-equivalent (with δ ě 0 ) if for any f P H and u P r H we have
[Equation]
} r H ď δ } f } H , } f ´ r JJf } H ď δ b } f } 2 H ` x f, | ∆ | f y H , } u ´ J r Ju } r H ď δ b } u } 2 r H ` x u, | r ∆ | u y r H .
If, for some w P C the resolvent R : “ p ∆ ´ ω q ´ 1 satisﬁes }p r RJ ´ JR q f } r H ď δ } f } H for all f P H , we say that ∆ and r ∆ are ω - δ -close with identiﬁcation operator J .
Absolute value | ∆ | and adjoint r J ˚ of operators are brieﬂy reviewed in
Appendix A. While the above deﬁnition might seem fairly abstract at ﬁrst, it is in fact a natural setting to investigate structural perturbations as Figure 3 exempliﬁes.
In our current setting, the Hilbert spaces in Deﬁnition 4.3 are node-signal spaces H “ (cid:96) 2 p G q , r H “ (cid:96) 2 p r G q of different graphs.
The notion of ω - δ -closeness is then useful, as it allows to compare ﬁlters deﬁned on different graphs but obtained from applying the same function to the respective graph-operators:
Lemma 4.4.
In the setting of Deﬁnition 4.3 let ∆ and r ∆ be ω - δclose and satisfy } ∆ } op , } r ∆ } op ď K for some K ą 0 .
If g : C Ñ C is holomorphic on the disk B K ` 1 p 0 q of radius p K ` 1 q , there is a constant C g ě
so that
[Equation]
with C g depending on g , ω and K .
An explicit characterization of C g together with a proof of this result is presented in Appendix F. Lemma 4.4 is our main tool in establishing our next result, proved in Appendix G, which captures stability under vertex-set non-preserving perturbations:
[Figure]
Spaces with p´ 1 q - 12 δ -close
Laplacians.
Details in Appendix E.
Theorem 4.5.
Let Φ N , r Φ N be scattering transforms based on a common module sequence Ω N and differing operator sequences D N , r D N .
Assume R ` n , L ` n ď 1 and B n ď B for some B and n ě 0 .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q , r J n : (cid:96) 2 p r G n q Ñ (cid:96) 2 p G n q ( 0 ď n ď N ) so that the respective signal spaces are δ -unitarily equivalent, the respective normal operators ∆ n , r ∆ n are ω - δ -close as well as bounded (in norm) by K ą 0 and the connecting operators satisfy } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q “ 0 .
For the common module sequence Ω N assume that the non-linearities satisfy } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q “
and that the constants C χ n and t C g γn u γ n P Γ N associated through Lemma 4.4 to the functions of the generalized frames in each layer satisfy C 2 χ n ` ř
[Equation]
D 2 for some D ą 0 .
Denote the operator that the family t J n u n of identiﬁcation operators induce on F N through concatenation by J N : F N Ñ Ă F N .
Then, with
K N “
[Equation]
and
[Equation]
The stability result persists with slightly altered stability constants, if identiﬁcation operators only almost commute with non-linearities and/or connecting operators, as Appendix G further elucidates.
Theorem 4.5 is not applicable to Architecture I, where ﬁlters are not all holomorphic, but is directly applicable to Architecture II.
Stability constants can be calculated in terms of D and B as before.
Beyond these results, stability under truncation of the scattering transform is equally desirable:
Given the energy W N : “ ř p
[Equation]
N
[Equation]
N q stored in the network at layer N , it is not hard to see that after extending Φ N p f q by zero to match dimensions with Φ N ` 1 p f q we have
[Equation]
(see Appendix H for more details).
A bound for W N is then given as follows:
Theorem 4.6.
Let Φ 8 be a generalized graph scattering transform based on a an operator sequence D
[Equation]
“ 1 and a module sequence Ω 8 with each ρ n p¨q ě 0 .
Assume in each layer n ěthat there is an eigenvector ψ n of ∆ n with solely positive entries; denote the smallest entry by m n : “ min i P G n ψ n r
i s and the eigenvalue corresponding to ψ
n by λ n .
Quantify the ’spectral-gap’ opened up at this eigenvalue through neglecting the output-generating function by η n : “ ř γ n P Γ n | g γ n p λ n q|and assume B n m n ě η n .
We then have (with C ` N : “ ś Ni “ 1 max
[Equation]
The product in (3) decays if C ` N Ñ C ` converges and ř Nn “ 1 p m n ´ η n { B n q Ñ 8 diverges as
N Ñ 8 .
The positivity-assumptions on the eigenvectors ψ n can e.g. always be ensured if they are chosen to lie in the lowest lying eigenspace of a graph Laplacian or normalized graph Laplacian (irrespective of the connectedness of the underlying graphs).
As an example, we note that if we extend our Architecture I to inﬁnite depth (recall from Section 3 that we are using the same ﬁlters, operators, etc.
in each layer) we have upon choosing λ n “ 0 and ψ n to be the constant normalized vector that η n “ 0 , C N “ 1 and m n “ 1 { a | G | , for a graph with | G | vertices.
On a graph with 16 vertices, we then e.g. have W N ď p 3 { 4 q N } f } 2 (cid:96) 2 p G q and thus } Φ N p f q ´ Φ N ` 1 p f q
[Equation]
ď
[Equation]
p G q { 2 .
As detailed in Appendix H, Theorem 4.6 also implies that under the given assumptions the scattering transform has trivial ’kernel’ for N Ñ 8 , mapping only 0 to 0 .
Graph-Level Feature Aggregation
To solve tasks such as graph classiﬁcation or regression over multiple graphs, we need to represent graphs of varying sizes in a common feature space.
Given a scattering transform Φ N , we thus need to ﬁnd a stability preserving map from the feature space F N to some Euclidean space that is independent of any vertex set cardinalities.
Since F N is a large direct sum of smaller spaces (cf.
(1)), we simply construct such maps on each summand independently and then concatenate them.
General non-linear feature aggregation: Our main tool in passing to graph-level features is a non-linear map N Gp : (cid:96) 2 p G q Ñ R p given as
[Equation]
with µ G : “ ř i P G µ i and } f } (cid:96) q p G q : “ p ř i P G | f i | q µ i q 1 { q .
Our inspiration to use this map stems from the standard case where all µ i “ 1 : For p ě | G | , the vector | f | “ pp| f 1 | , ..., | f G |q J can then be recovered from N Gp p f q up to permutation of indices [23].
Hence, employing N Gp (with p ě | G | ) to aggregate node-information into graph-level information, we lose the minimal necessary information about node permutation (clearly N Gp p f q “ N Gp p Π f q for any permutation matrix Π ) and beyond that only information about the complex phase (respectively the sign in the real case) in each entry of f .
Given a scattering transform Φ N mapping from (cid:96) 2 p G q to the feature space F N “ ‘ Nn “` (cid:96) 2 p G n q ˘ | Γ n ´ 1 | , we obtain a corresponding map Ψ N mapping from (cid:96) 2 p G q to
R N “ ‘ Nn “ 1 p R p n q | Γ n ´ 1 | by concatenating the feature map Φ N with the operator that the family of non-linear maps t N p n G n u Nn “ 1 induces on F N by concatenation.
Similarly we obtain the map r Ψ N : (cid:96) 2 p r G q Ñ R N by concatenating the map r Φ N : (cid:96) 2 p r G q Ñ ‘ Nn “´ (cid:96) 2 p r G n q ¯ | Γ n ´ 1 | with the operator induced by the family t N p n r G n u Nn “ 1 .
The feature space R N is completely determined by path-sets Γ N and used maximal p -norm indices p n .
It no longer depends on cardinalities of vertex sets of any graphs, allowing to compare (signals on) varying graphs with each other.
Most of the results of the previous sections then readily transfer to the graph-level-feature setting (c.f. Appendix I.1).
Low-pass feature aggregation:
The spectrum-free aggregation scheme of the previous paragraph is especially adapted to settings where there are no high-level spectral properties remaining constant under graph perturbations.
However, many commonly utilized operators, such as normalized and un-normalized graph Laplacians, have a somewhat ’stable’ spectral theory: Eigenvalues are always real, non-negative, the lowest-lying eigenvalue equals zero and simple (if the graph is connected).
In this section we shall thus assume that each mentioned normal operator ∆ n ( r ∆ n ) has these spectral properties.
We denote the lowest lying normalized eigenvector (which is generically determined up to a complex phase) by ψ ∆ n and denote by M |x¨ , ¨y| G n : (cid:96) 2 p G n q Ñ C the map given by M |x¨ , ¨y| G n p f q “ |x ψ ∆ n , f y (cid:96) 2 p G n q | .
The absolute value around the inner product is introduced to absorb the phaseambiguity in the choice of ψ ∆ n .
Given a scattering transform Φ N mapping from (cid:96) 2 p G q to the feature space F N , we obtain a corresponding map Ψ |x¨ , ¨y| N mapping from (cid:96) 2 p G q to C N “ ‘ Nn “ 1 C | Γ n ´ 1 | by concatenating the feature map Φ N with the operator that the family of maps t M |x¨ , ¨y| G n u Nn “ 1 induces on F
N by concatenation.
As detailed in Appendix I.2, this map inherits stability properties in complete analogy to the discussion of Section 4.
Higher Order Scattering
Node signals capture information about nodes in isolation.
However, one might be interested in binary, ternary or even higher order relations between nodes such as distances or angles in graphs representing molecules.
In this section we focus on binary relations – i.e. edge level input – as this is the instantiation we also test in our regression experiment in Section 7.
Appendix J.2 provides more details and extends these considerations beyond the binary setting.
We equip the space of edge inputs with an inner product according to x f, g y
“ ř | G | i,j “ 1 f ij g ij µ ij and denote the resulting inner-product space by (cid:96) 2 p E q with E “ G ˆ G the set of edges.
Setting e.g. node-weights µ i and edge weights µ ik to one, the adjacency matrix W as well as normalized or un-normalized graph Laplacians constitute selfadjoint operators on (cid:96) 2 p E q , where they act by matrix multiplication.
Replacing the G n of Sectionby E n , we can then follow the recipe laid out there in constructing 2 nd -order scattering transforms; all that we need are a module sequence Ω N and an operator sequence D 2 N : “ p P 2 n , ∆ 2 n q Nn “ 1 , where now
P 2 n : (cid:96) 2 p E n ´ 1 q Ñ (cid:96) 2 p E n q and ∆ 2 n : (cid:96) 2 p E n q Ñ (cid:96) 2 p E n q .
We denote the resulting feature map by Φ 2 N and write F 2 N for the corresponding feature space.
The map N G p introduced in (4) can also be adapted to aggregate higher-order features into graph level features: With } f } q : “ p ř ij P G | f ij | q µ ij q 1 { q and µ E : “ ř | G | ij “ 1 µ ij , we deﬁne N Ep p f q “ p} f } (cid:96) 1 p E q {? µ E , } f } (cid:96) 2 p E q , } f } (cid:96) 3 p E q , ..., } f } (cid:96) p p E q q J {? p .
Given a feature map Φ 2 N with feature space F 2 N “ ‘ Nn “` (cid:96) 2 p E n q ˘ | Γ n ´ 1 | , we obtain a corresponding map Ψ 2 N mapping from (cid:96) 2 p E q to R N “ ‘ Nn “ 1 p R p n q | Γ n ´ 1 | by concatenating Φ EN with the map that the family of non-linear maps t N E n p n u Nn “ 1 induces on F N by concatenation.
The stability results of the preceding sections then readily translate to Φ 2 N and Ψ 2 N (c.f. Appendix J).
Experimental Results
We showcase that even upon selecting the fairly simple Architectures I and II introduced in Section 3 (c.f. also Fig. 2), our generalized graph scattering networks are able to outperform both waveletbased scattering transforms and leading graph-networks under different circumstances.
To aid visual clarity when comparing results, we colour-code the best-performing method in green, the second-best performing in yellow and the third-best performing method in orange respectively.
Social Network Graph Classiﬁcation: To facilitate contact between our generalized graph scattering networks, and the wider literature, we combine a network conforming to our general theory namely Architecture I in Fig.
2 (as discussed in Section 3 with depth N “ 4 , identity as connecting operators and | ¨ | -non-linearities) with the low pass aggregation scheme of Section 5 and a
Euclidean support vector machine with RBF-kernel (GGSN+EK).
The choice N “ 4 was made to keep computation-time palatable, while aggregation scheme and non-linearities were chosen to facilitate comparison with standard wavelet -scattering approaches.
For this hybrid architecture (GGSN+EK), classiﬁcation accuracies under the standard choice of 10 -fold cross validation on ﬁve common social network graph datasets are compared with performances of popular graph kernel approaches, leading deep learning methods as well as geometric wavelet scattering (GS-SVM)
More details are provided in Appendix K.
As evident from Table 1, our network consistently achieves higher accuracies than the geometric wavelet scattering transform of [12], with the performance gap becoming signiﬁcant on the more complex REDDIT datasets, reaching a relative mean performance increase of more than 10% on REDDIT- 12 K. This indicates the liberating power of transcending the graph wavelet setting.
While on comparatively smaller and somewhat simpler datasets there is a performance gap between our static architecture and fully trainable networks, this gap closes on more complex datasets: While P-Poinc e.g. outperforms our method on IMDB datasets, the roles are reversed on REDDIT datasets.
On REDDIT-B our approach trails only GIN; with difference in accuracies insigniﬁcant.
On REDDIT- 5 K our method comes in third, with the gap to the second best method (GIN) being statistically insigniﬁcant.
On REDDIT- 12 K we generate state of the art results.
[Table]
Regression of Quantum Chemical Energies: In order to showcase the prowess of both our higher order scattering scheme and our spectrum-agnostic aggregation method of Section 5, we combine these building blocks into a hybrid architecture which we then apply in combination with kernel methods ( 2 GGST + EK) to the task of atomization energy regression on QM 7 .
This is a comparatively small dataset of 7165 molecular graphs, taken from the 970 million strong molecular database GDB13 [2].
Each graph in QM 7 represents an organic molecule, with nodes corresponding to individual atoms.
Beyond the node-level information of atomic charge, there is also edge level information characterising interaction strengths between individual nodes/atoms available.
This is encoded into so called Coulomb matrices (see e.g. [31] or Appendix K) of molecular graphs, which for us serve a dual purpose: On the one hand we consider a Coulomb matrix as an edge-level input signal on a given graph.
On the other hand, we also treat it as an adjacency matrix from which we build up a graph Laplacian
L .
Our normal operator is then chosen as L “ L { λ max p L q again.
Connecting operators are set to the identity, while non-linearities are ﬁxed to ρ n ě 1 p¨q “ | ¨ | .
Filters are chosen as p sin p π { 2 ¨ L q , cos p π { 2 ¨ L q , sin p π ¨ L q , cos p π ¨ L qq acting through matrix multiplication.
Output generating functions are set to the identity and depth is N “ 4 , so that we essentially recover Architecture II of now to
Graph level features are aggregated via the map N E 5 of Section 6.
We chose p “(and not p " 5 ) for N Ep to avoid overﬁtting.
Generated feature vectors are combined with node level scattering features obtained from applying Architecture II of Fig.
2 to the input signal of atomic charge into composite feature vectors; plotted in Figure 5.
As is visually evident, even when reduced to the low-dimensional subspace of their ﬁrst three principal components, the generated scattering features are able to aptly resolve the atomization energy of the molecules.
This aptitude is also reﬂected in Table 2, comparing our approach with leading graph-based learning methods trained with ten-fold cross validation on node and (depending on the model) edge level information.
Our method is the best performing.
We signiﬁcantly outperform the next best model (DTNN), producing less than half of its mean absolute error (MAE).
Errors of other methods are at least one — sometimes two — orders of magnitude greater.
In part, this performance discrepancy might be explained by the hightened suitability of our scattering transform for environments with somewhat limited training-data availability.
Here we speculate that the additional performance gap might be explained by the fact that our graph shift operator ∆ carries the same information as the Coulomb matrix (a proven molecular graph descriptor in itself [31]).
Additionally, our ﬁlters being inﬁnite series’ in powers of the underlying normal operator allows for rapid dispersion of information across underlying molecular graphs, as opposed to e.g. the ﬁlters in GraphConv
[Table]
or SchNet, which do not incorporate such higher powers.
To quantify the effect of including second order scattering coefﬁcients, we also include the result of performing kernel-regression solely on ﬁrst order features generated through Architecture II of Fig.
2 (GGST + EK).
While results are still better than those of all but one leading approach, incorporating higher order scattering improves performance signiﬁcantly.
Discussion
Leaving behind the traditional reliance on graph wavelets, we developed a theoretically well founded framework for the design and analysis of (generalized) graph scattering networks; allowing for varying branching rations, non-linearities and ﬁlter banks.
We provided spectrum independent stability guarantees, covering changes in input signals and for the ﬁrst time also arbitrary normal perturbations in the underlying graph-shift-operators.
After introducing a new framework to quantify vertex-set non-preserving changes in graph domains, we obtained spectrum-independent stability guarantees for this setting too.
We provided conditions for energy decay and discussed implications for truncation stability.
Then we introduced a new method of graph-level feature aggregation and extended scattering networks to higher order input data.
Our numerical experiments showed that a simple scattering transform conforming to our framework is able to outperform the traditional graph-wavelet based approach to graph scattering in social network graph classiﬁcation tasks.
On complex datasets our method is also competitive with current fully trainable methods, ouperforming all competitors on REDDIT- 12 K. Additionally, higher order graph scattering transforms signiﬁcantly outperform current leading graph-based learning methods in predicting atomization energies on QM 7 .
A reasonable critique of scattering networks as tractable models for general graph convolutional networks is their inability to emulate non-tree-structured network topologies.
While transcending the wavelet setting has arguably diminished the conceptual gap between the two architectures, this structural difference persists.
Additionally we note that despite a provided promising example, it is not yet clear whether the newly introduced graph-perturbation framework can aptly provide stability guarantees to all reasonable coarse-graining procedures.
Exploring this question is the subject of ongoing work.
Broader Impact
We caution against an over-interpretation of established mathematical guarantees: Such guarantees do not negate biases that may be inherent to utilized datasets.
Disclosure of Funding
Christian Koke acknowledges support from the German Research Foundation through the MIMO
II-project (DFG SPP 1798, KU 1446/21-2).
[-If g : C Ñ C is holomorphic on the disk B K ` 1 p 0 q of radius p K ` 1 q , there is a constant C g ě 
-]
Gitta Kutyniok acknowledges support from the ONE
Munich Strategy Forum (LMU Munich, TU Munich, and the Bavarian Ministery for Science and
Art), the Konrad Zuse School of Excellence in Reliable AI (DAAD), the Munich Center for Machine
[-|| f || 
-]
[-´ q n q with 0 ď q n ă 
-]
Learning (BMBF) as well as the German Research Foundation under Grants DFG-SPP-2298, KU 1446/31-1 and KU 1446/32-1 and under Grant DFG-SFB/TR 109, Project C09 and the Federal
Ministry of Education and Research under Grant MaGriDo.