Graph Scattering beyond Wavelet Shackles
Anonymous Author(s)
AfﬁliationAddress email
Abstract
This work develops a ﬂexible and mathematically sound framework for the design and analysis of graph scattering networks with variable branching ratios andgeneric functional calculus ﬁlters.
Spectrally-agnostic stability guarantees fornode- and graph-level perturbations are derived; the vertex-set non-preservingcase is treated by utilizing recently developed mathematical-physics based tools.
Additionally, conditions for energy decay are presented and subsequently utilizedto prove asymptotic expressivity of corresponding scattering transforms.
Newmethods of graph-level feature aggregation are introduced and stability of theresulting composite scattering architectures is established.
Finally, scatteringtransforms are extended to edge- and higher order tensorial input.
Theoreticalresults are complemented by numerical investigations: Suitably chosen scatteringnetworks conforming to the developed theory perform better than traditional graphwavelet based scattering approaches in social network graph classiﬁcation tasksand signiﬁcantly outperform other graph-based learning approaches to regressionof quantum-chemical energies on QM 7 .
Introduction
Euclidean wavelet scattering networks [23, 4] are deep convolutional architectures where outputfeatures are generated in each layer.
Employed ﬁlters are designed rather than learned and derivefrom a ﬁxed (tight) wavelet frame, resulting in a tree structured network with constant branching ratio.
Such networks provide state of the art methods in settings with limited data availability and serveas a mathematically tractable model of standard convolutional neural networks (CNNs).
Rigorousinvestigations — establishing remarkable invariance- and stability properties of wavelet scatteringnetworks — were initially carried out in [23].
The extensive mathematical analysis [40] generalizedthe term ’scattering network’ to include tree structured networks with varying branching rations andframes of convolutional ﬁlters, thus signiﬁcantly narrowing the conceptual gap to general CNNs.
With increasing interest in data on graph-structured domains, well performing networks generalizing
Euclidean CNNs to this geometric setting emerged [18, 5, 9].
If efﬁciently implemented, such graphconvolutional networks (GCNs) replace Euclidean convolutional ﬁlters by functional calculus ﬁlters;i.e.
scalar functions applied to a suitably chosen graph-shift-oprator capturing the geometry of theunderlying graph [18, 14, 9].
Almost immediately, proposals aimed at extending the success story of
Euclidean scattering networks to the graph convolutional setting began appearing: In [50], the authorsutilize dyadic graph wavelets (see e.g. [14]) based on the non-normalized graph Laplacian resultingin a norm preserving graph wavelet scattering transform.
In [10], diffusion wavelets (see e.g. [8]) areused to construct a graph scattering transform enjoying spectrum-dependent stability guarantees tograph level perturbations.
For scattering transforms with N layers and K distinct functional calculus ﬁlters, the work [11] derives node-level stability bounds of O p K N { 2 q and conducts correspondingnumerical experiments choosing diffusion wavelets, monic cubic wavelets [14] and tight Hannwavelets [36] as ﬁlters.
In [12] the authors, following [8], construct so called geometric wavelets andestablish the expressivity of a scattering transform based on such a frame through extensive numericalexperiments.
A theoretical analysis of this and a closely related wavelet based scattering transform isthe main focus of [29].
Additionally, graph-wavelet based scattering transforms have been extendedto the spatio-temporal domain [28], utilized to overcome the problem of oversmoothing in GCNs [26]and pruned to deal with their exponential (in network depth) increase in needed computing resources[15].
Common among all these contributions is the focus on graph wavelets, which are genericallyunderstood to derive in a scale-sampling procedure from a common wavelet generating kernelfunction g : R Ñ R satisfying various properties [14].
Established stability or expressivity properties— especially to structural perturbations — are then generally linked to the speciﬁc choice of thewavelet kernel g and utilized graph shift operator [10, 29].
This severely limits the diversity ofavailable ﬁlter banks in the design of scattering networks and draws into question their validity asmodels for more general GCNs whose ﬁlters generically do not derive from a wavelet kernel.
A primary focus of this work is to provide alleviation in this situation: After reviewing the graph signalprocessing setting in Section 2, we introduce a general framework for the construction of (generalized)graph scattering transforms beyond the wavelet setting in Section 3.
Section 4 establishes spectrumagnostic stability guarantees on the node signal level and for the ﬁrst time also for graph-levelperturbations.
To handle the vertex-set non-preserving case, a new ’distance measure’ for operatorscapturing the geometry of varying graphs is utilized.
After providing conditions for energy decay(with the layers) and deriving a related asymptotic expressivity result in Section 5, we consider graphlevel feature aggregation and higher order inputs in Sections 6 and 7 respectively.
In Section 8 wethen provide numerical results indicating that general functional calculus ﬁlter based scattering is atleast as expressive as standard wavelet based scattering in graph classiﬁcation tasks and outperformsleading graph neural network approaches to regression of quantum chemical energies on QM 7 .
Graph Signal Processing
Taking a signal processing approach, we consider signals on graphs as opposed to graph embeddings:
Node-Signals: Given a graph p G, E q , we are primarily interested in node-signals, which arefunctions from the node-set G to the complex numbers, modelled as elements of C | G | .
We equip thisspace with an inner product according to x f, g y
“ ř | G | i “ 1 f i g i µ i (with all vertex weights µ i ě 1 ) anddenote the resulting inner product space by (cid:96) 2 p G q .
We could consider arbitrary inner products on
C | G | without fundamentally changing any of the results below, but will forego this in the interest ofbetter readability.
Functional Calculus Filters: Our fundamental objects in investigating node-signals will be functional calculus ﬁlters based on a normal operator ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p G q .
Prominent examplesinclude the adjacency matrix W , the degree matrix D , [-normalized p 
-]un-normalized ( D ´ W ) or normalizedp 1 ´ D
´ 12 WD ´ 12 q graph Laplacians and the lazy random walk matrix p 1 ` WD ´ 1 q (normal for nodeweights 9 inverse degree [29]).
Writing normalized eigenvalue-eigenvector pairs of ∆ as p λ i , φ i q | G | i “ 1 ,the ﬁlter obtained from applying g : C Ñ C is given by g p ∆ q f “ ř | G | i “ 1 g p λ
i qx φ
i , f y (cid:96) 2 p V q φ i .
Generalized Frames: We are most interested in ﬁlters that arise from a collection of functionsadequately covering the spectrum of the operator to which they are applied.
To this end we call acollection t g i p¨qu i P I of functions a generalized frame if it satisﬁes the generalized frame condition
A ď ř
i P I
| g i p c q| 2 ď B for any c in C for constants A ; B ą 0 .
As proved in Appendix B, thiscondition is sufﬁcient to guarantee that the associated operators form a frame:
Theorem 2.1.
Let ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p
G q be normal.
If the family t g i p¨qu i P I of bounded continuousfunctions satisﬁes A ď
ř i P I
| g i p c q| 2 ď B for all c in the spectrum σ p ∆ q , we have ( @ f P (cid:96) 2 p G q )
[Equation]
[-In fact, in our numerical implementations, wewill – among other mappings – utilize the function δ 0 p¨q , deﬁned by δ 0 p 0 q “ 1 and δ 
-]
The Generalized Graph Scattering Transform
A generalized graph scattering transform is a non-linear map Φ based on a tree structured multilayergraph convolutional network with constant branching factor in each layer.
For an input signalf P (cid:96) 2 p G q , outputs are generated in each layer of such a scattering network, and then concatenated toform a feature vector in a feature space F .
The network is built up from three ingredients:
Connecting Operators: To allow intermediate signal representations in the ’hidden’ networklayers to be further processed with functional calculus ﬁlters based on varying operators, which mightnot all be normal for the same choice of node-weights, we allow these intermediate representationsto live in varying graph signal spaces.
In fact, we do not even assume that these signal spaces arebased on a common vertex set.
This is done to allow for modelling of recently proposed networkswhere input- and ’processing’ graphs are decoupled (see e.g. [1, 37]), as well as architecturesincorporating graph pooling [21].
Instead, we associate one signal space (cid:96) 2 p G n q to each layern .
Connecting operators are then (not necessarily linear) operators P n : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n qconnecting the signal spaces of subsequent layers.
We assume them to be Lipschitz continuous( } P p f q ´ P p g q} (cid:96) 2 p G n ´ 1 q ď R ` } f ´ g } (cid:96) 2 p G n q q and triviality preserving ( P p 0 q “ 0 ).
For our originalnode-signal space we also write (cid:96) 2 p G q ” (cid:96) 2 p G 0 q .
Non-Linearities:
To each layer, we also associate a (possibly) non-linear function ρ n : C Ñ Cacting poinwise on signals in (cid:96) 2 p G n q .
Similar to connecting operators, we assume ρ n preserves zeroand is Lipschitz-continuous with Lipschitz constant denoted by L ` n .
This deﬁnition allows for theabsolute value non-linearity, but also ReLu or, trivially, the identity function.
Operator Frames: Beyond these ingredients, the central building block of our scatteringarchitecture is comprised of a family of functional calculus ﬁlters in each layer.
That is, we assumethat in each layer, the node signal space (cid:96) 2 p G n q carries a normal operator ∆ n and an associatedcollection of continuous functions comprised of an output generating function χ n p¨q as wellas a ﬁlter bank t g γ n p¨qu γ n P Γ n indexed by an index set Γ n .
As the network layer n varies (andin contrast to wavelet-scattering networks)
we allow the index set Γ n as well as the collectiont χ n p¨qu Ť t g γ n p¨qu γ n P Γ n of functions to vary.
We only demand that in each layer the functions in theﬁlter bank together with the output generating function constitute a generalized frame with frameconstants A n , B n ą 0 .
We refer to the collection of functions Ω N : “ p ρ n , t χ n p¨qu Ť t g γ n p¨qu γ n P Γ n q Nn “ 1 as a module sequence and call D N : “ p P n , ∆ n q Nn “ 1 our operator collection .
The generalized scatteringtransform is then constructed iteratively:
To our initial signal f P (cid:96) 2 p G q we ﬁrst apply the connecting operator P 1 , yielding a signal representation in (cid:96) 2 p G 1 q .
Subsequently, we apply the pointwise non-linearity ρ 1 .
Then we apply our graph ﬁlters t χ 1 p ∆ 1 qu Ť
[Equation]
1 to ρ 1 p P 1 p
f
qq yielding the output V 1 p f q : “ χ 1 p ∆ 1 q ρ 1 p P 1 p f qq as well as the intermediate hidden representations t U 1 r γ 1 sp f q : “ g γ 1 p ∆ 1 q ρ 1 p P 1 p f qqu γ 1 P Γ 1 obtained in the ﬁrst layer.
Here we have introduced the one-step scattering propagator U n r γ n s : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n q mapping f ÞÑ g γ n p ∆ n q ρ n p P n p f qq as well as the output generating operator
V n : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n q mapping f to χ n p ∆ n q ρ n p P n p f qq .
Upon deﬁning the set Γ N ´ 1 : “ Γ N ´ 1 ˆ ... ˆ Γ 1 of paths of length p N ´ 1 q terminating in layer N ´ 1 (with Γtaken to be the one-element set) and iterating the above procedure, we see that the outputs generated in the N th -layer are indexed by paths Γ N ´terminating in the previous layer.
Outputs generated in the N th layer are thus given by t V N ˝
U r γ N ´ 1 s˝ ...
˝
U r γ 1 sp f qu p γ N ´ 1 ,...,γ 1 qP Γ N ´ 1 .
Concatenating the features obtained in the various layers of a network with depth N , our full featurevectors thus live in the feature space
[Equation]
The associated canonical norm is denoted } ¨ } F N .
For convenience, a brief review of direct sums ofspaces, their associated norms and a discussion of corresponding direct sums of maps is provided in
Appendix A. We denote the hence constructed generalized scattering transform of length N , basedon a module sequence Ω N and operator collection D N by Φ N .[-
-]
[-Instead it is extracted via δ 
-]
Stability Guarantees
In order to produce meaningful signal representations, a small change in input signal should producea small change in the output of our generalized scattering transforms.
This property is captured in theresult below, which is proved in Appendix C.
Theorem 4.1.
With the notation of Section 3, we have for all f, h P (cid:96) 2 p G q :} Φ N p f q
[Equation]
q [Equation] ´ [Equation]
0 u n ´ 1 ź k “
[Equation]
¸ [Equation] We call a scattering transform Φ N based on a module sequence Ω N and operator collection D Nadmissible , if the upper frame bounds B n and Lipschitz constants L ` n and R ` n are all smaller than orequal one in each layer n ď N .
Admissible scattering transforms are 1 -Lipschitz,}
Φ N p
f
q ´ Φ N p h q} F N ď } f ´ h } (cid:96) 2 p G q , which greatly simpliﬁes the presentation.
We shall henceforth assume admissibility, as is not a severelimitation since it can always be achieved through a rescaling operation.
Operators capturing graph geometries might only be known approximately in real world tasks; e.g. ifedge weights are only known to a certain level of precision.
Hence it is important that our scatteringrepresentation be insensitive to small perturbations in the underlying normal operators in each layer,which is captured by our next result, proved in Appendix D. Smallness here is measured in Frobeniusnorm } ¨ } F , which for convenience is brieﬂy reviewed in Appendix A).
Theorem 4.2.
Let Φ N and r Φ N be two admissible scattering transforms based on the same modulesequence Ω N and operator sequences D N , r D N with the same connecting operators ( P n “ r P n )
ineach layer.
Assume that the respective normal operators satisfy } ∆ n ´ r ∆ n } F ď δ for some δ ą 0 .
Further assume that the functions t g γ n u γ n P Γ n and χ n in each layer are Lipschitz continuous withassociated Lipschitz constants satisfying L 2 χ n `
[Equation]
ď D
[Equation]
for all n ď N and some D ą 0 . Then we have for all f P (cid:96) 2 p G q
[Equation]
We are also interested in perturbations that change the vertex set of the graphs in our architecture.
This is important for example in the context of social networks, when passing from nodes representingindividuals to nodes representing (close knit) groups of individuals.
To investigate this setting, weutilize tools originally developed within the mathematical physics community [30]:
Deﬁnition 4.3.
Let H and r H be two ﬁnite dimensional Hilbert spaces.
Let ∆ and r ∆ be normaloperators on these spaces.
Let J : H Ñ r H and r J : r H Ñ H be linear maps — called identiﬁcationoperators .
We call the two spaces δ -quasi-unitarily-equivalent (with δ ě 0 ) if for any f P H andu P r H we have
[Equation]
b [Equation] H .
If, for some w P C the resolvent R : “ p ∆ ´ ω q ´ 1 satisﬁes }p r RJ ´ JR q f } r H ď δ } f } H for all f P H ,we say that ∆ and r ∆ are ω - δ -close with identiﬁcation operator J .
Absolute value | ∆ | and adjoint r J ˚ of operators are brieﬂy reviewed in
Appendix A. While the above deﬁnition might seem fairly abstract at ﬁrst, it is in fact a natural setting to investigate structural perturbations as Figure 2 exempliﬁes.
In our current setting, the Hilbert spaces in Deﬁnition 4.3 are node-signal spaces H “ (cid:96) 2 p G q , r H “ (cid:96) 2 p r G q of different graphs.
The notion of ω - δ -closeness is then useful, as it allows to compare ﬁlters deﬁned on different graphs but obtained from applying the same function to the respective graph-operators:
Lemma 4.4.
In the setting of Deﬁnition 4.3 let ∆ and r ∆ be ω - δclose and satisfy } ∆ } op , } r ∆ } op ď K for some K ą 0 .
If g : C Ñ C is holomorphic on the disk B K ` 1 p 0 q of radius p K ` 1 q , there is a constant C g ě
so that
[Equation]
with C g depending on g , ω and K .
An explicit characterization of C g together with a proof of this result is presented in Appendix F. Lemma 4.4 is our main tool in establishing our next result, proved in Appendix G, which captures stability under vertex-set non-preserving perturbations:
Theorem 4.5.
Let Φ N , r Φ N be admissible scattering transforms based on a common module sequence Ω N and differing operator sequences D N , r D N .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q , r J n : (cid:96) 2 p r G n q Ñ (cid:96) 2 p G n q ( 0 ď n ď N ) so that the respective signalspaces are δ -unitarily equivalent, the respective normal operators ∆ n , r ∆ n are ω - δ -close as well asbounded (in norm) by K ą 0 and the connecting operators satisfy } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q ďδ } f } (cid:96) 2 p G n ´ 1 q .
For the common module sequence Ω N assume that the non-linearities satisfy} ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q ď δ } f } (cid:96) 2 p G n q and that the constants C χ n and t C g γn u γ n P Γ N associated through Lemma 4.4 to the functions of the generalized frames in each layer satisfy C 2 χ n ` ř γ n P Γ N C 2 g γn ď D 2 for some D ą 0 .
Denote the operator that the family t J n u n ofidentiﬁcation operators induce on F N through concatenation by J N : F N Ñ Ă F N .
Then we havewith K N “
[Equation]
`
[Equation]
q{ 7 that
[Equation]
If additionally } r P n J n ´ 1 f ´ J n P n
[Equation]
[Equation]
p
[Equation]
[Equation]
or } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q “ 0 holds in eachlayer, then we have K N “ O p 4 N { 2 q ; if both additional equations hold, we have K N “ O p 2 N { 2 q .  Energy Decay and Expressivity
Since stability alone does not ensure usefulness (e.g. a constant all-zero map would be both perfectlystable and perfectly useless), we are also interested in guarantees about the expressivity of scatteringtransforms.
We analyse this question in the limit of inﬁnite network depth and provide conditionsunder which } f } (cid:96) 2 p G q À } Φ 8 p f q} F 8 . This then implies that the only signal mapped to the zerofeature in F 8 is in fact the zero signal in (cid:96) 2 p G q . To achieve this, we consider the energy W N
[Equation]
stored in the network at layer N . We assume that in each layer L ´ n | ¨ | ď | ρ n p¨q| ď L ` n | ¨ | and R ´ n } f } (cid:96) 2 p G n ´ 1 q ď } P n p f q} (cid:96) 2 p G n q ď R ` n } f } (cid:96) 2 p G n ´ 1 q with all multiplicative constants positive.
In 180 Appendix H we then prove the following result, generalizing corresponding Euclidean results [39]:
Lemma 5.1. The energy W N stored in layer N satisﬁes [Equation] with constants C ´ N
[Equation]
Using this, our next result (proved in Appendix I) ensures decay of the energy W N as we progressthrough the layers.
Theorem 5.2. Let Φ 8 be a generalized admissible graph scattering transform based on a an operatorsequence D 8 “ p P n , ∆ n q 8 n “ 1 and a module sequence Ω 8 with each ρ n p¨q ě 0 .
Assume in eachlayer n ě 1 that there is an eigenvector ψ n of ∆ n with solely positive entries; denote the smallestentry by m n : “ min i P G n ψ n r
i s and the eigenvalue corresponding to ψ
n by λ n .
Quantify the’spectral-gap’ opened up at this eigenvalue through neglecting the output-generating function byη n : “ ř γ n P Γ n | g γ n p λ n q| 2 and assume B n m n ě η n .
We then have
[Equation]
The positivity-assumptions on the eigenvectors ψ n can e.g. be ensured if they are chosen to lie in thelowest lying eigenspace of a graph Laplacian.
With this and Lemma 5.1 we immediately ﬁnd:
Corollary 5.3.
In the setting of Theorem 5.2, the generalized scattering transform satisﬁes Φ ´ 1 8 p 0 q “t 0 u if C ˘ N Ñ C ˘ for some positive constants C ˘ and ř Nn “ 1 p m n ´ η n { B n q Ñ 8 as N Ñ 8 .
Graph-Level Feature Aggregation
To solve tasks such as graph classiﬁcation or regression over multiple graphs, we need to representgraphs of varying sizes and connectivity in a common feature space.
Given a scattering transformΦ N , we thus need to ﬁnd a stability preserving map from the feature space F N to some Euclideanspace that is independent of any vertex set cardinalities.
Since the feature space F N is a large directsum of smaller spaces (cf.
(1)), we simply construct such maps on each summand independently andthen
concatenate them into a map on F N .
Our main tool in passing to graph-level features is a non-linear map N Gp : (cid:96) 2 p G q Ñ R p given as
[Equation]
with µ G : “ ř i P G µ i and } f } (cid:96) q p G q : “ p ř i P G | f i | q µ i q 1 { q .
Our inspiration to use this map stems fromthe standard case where all µ i “ 1 : For p ě | G | , the vector | f | “ pp| f 1 | , ..., | f G |q J can then berecovered from N Gp p f q up to permutation of indices [24].
Hence, employing N Gp (with p ě | G | ) toaggregate node-information into graph-level information, we lose the minimal necessary informationabout node permutation (clearly N Gp p f q “ N Gp p Π f q for any permutation matrix Π ) and beyond thatonly information about the complex phase (respectively the sign in the real case) in each entry of f .
Given a scattering transform Φ N mapping from (cid:96) 2 p G q to the feature space F N “ ‘ N n “` (cid:96) 2 p G n q ˘ | Γ n ´ 1 | , we obtain a corresponding map Ψ N mapping from (cid:96) 2 p G q to
R N “ ‘ Nn “ 1 p R p n q | Γ n ´ 1 | by concatenating the feature map Φ N with the operator that the family of non-linear maps t N p n G n u Nn “ 1 induces on F N by concatenation.
Similarly we obtain the map r Ψ N : (cid:96) 2 p r G q Ñ R N by concatenating the map r Φ N : (cid:96) 2 p r G q Ñ ‘ Nn “´ (cid:96) 2 p r G n q ¯ | Γ n ´ 1 | with the operator induced by the family t N p n r G n u Nn “ 1 .
The feature space R N is completely determined by path-sets Γ N and used maximal p -norm indices p n .
It no longer depends on cardinalities of vertex sets of any graphs, allowing to compare (signals on) varying graphs with each other.
Most of the results of the previous sections are then readily transferred to the setting of graph level features (c.f. Appendix J Theorem 6.1.
Assuming admissibility, we have } Ψ N p f q ´ Ψ N p h q} R N ď } f ´ h } (cid:96) 2 p G q for allf, h P (cid:96) 2 p G q .
With the conditions and notation of Theorem 4.2 we have
[Equation] Additionally, in the setting of Theorem 4.5, assuming that for each n ď N the identiﬁcation operator [Equation] satisﬁes
[Equation]
for 2 ď k ď p n implies ( @ f P (cid:96) 2 p G q )
[Equation]
Furhermore, under the assumptions of Corollary 5.3 Ψ 8 p f q “ 0 implies f “ 0 . 6.2 Low-pass feature aggregation While the spectrum-free aggregation scheme of the previous section is especially adapted to settingswhere there are no high-level spectral properties remaining constant under graph perturbations (e.g.if we choose ∆ n to be the ajacency matrix W n ), there are also settings in which graph scatteringtransforms amend themselves to spectral aggregation schemes: Many commonly utilized operators,such as normalized and un-normalized graph Laplacians, have a somewhat ’stable’ spectral theory: Eigenvalues are always real, non-negative, the lowest-lying eigenvalue is equal zero and simple (ifthe graph is connected).
In this section we shall thus assume that each mentioned normal operator ∆ n ( r ∆ n ) has these spectralproperties.
We denote the lowest lying normalized eigenvector (which is generically determined upto a complex phase) by ψ ∆ n and denote by M |x¨ , ¨y| G n : (cid:96) 2 p G n q Ñ C the map given by M |x¨ , ¨y| G n p f q “|x ψ ∆ n , f y (cid:96) 2 p G n q | .
The absolute value around the inner product is introduced to absorb the phaseambiguity in the choice of ψ ∆ n .
Given a scattering transform Φ N mapping from (cid:96) 2 p G q to the featurespace F N , we obtain a corresponding map Ψ |x¨ , ¨y| N mapping from (cid:96) 2 p G q to C N “ ‘ Nn “ 1 C | Γ n ´ 1 | byconcatenating the feature map Φ N with the operator that the family of maps t M |x¨ , ¨y| G n u Nn “ 1 induces on 232 F
N by concatenation.
As detailed in Appendix J.2, this map inherits stability properties in completeanalogy to Theorem 6.1.
Higher Order Scattering
Node signals capture information about nodes in isolation.
However, one might also want to analyseor incorporate information about binary, ternary or even higher order relations between nodes, suchas distances or angles between nodes representing atoms in a molecule. This can be formalizedby considering tensorial input signals:
A 2 -tensor on a graph G is simply an element of C | G |ˆ| G | ,a 3 -tensor an element of C | G |ˆ| G |ˆ| G | and so forth.
For ﬁxed k , we equip the space of k -tensorswith an inner product according to x f, g y
“ ř | G | i 1 ,...,i k “ 1 f i 1 ,...,i k g i 1 ,...,i k µ i 1 ,...,i k and denote theresulting inner-product space by (cid:96) 2 p G k q .
Setting e.g. node-weights µ i and edge weights µ ik toone, the adjacency matrix W as well as normalized or un-normalized graph Laplacians constituteself-adjoint operators on (cid:96) 2 p G 2 q , where they act by matrix multiplication.
We can then follow therecipe laid out Section 3 in constructing k th -order scattering transforms; all that we need are a modulesequence Ω N and an operator sequence D kN : “ p P kn , ∆ kn q Nn “ 1 , where now P kn : (cid:96) 2 p G kn ´
1 q Ñ (cid:96) 2 p G kn qand ∆ kn : (cid:96) 2 p G kn q Ñ (cid:96) 2 p G kn q .
We denote the resulting feature map by Φ kN and write F kN for thecorresponding feature space.
The node-level stability results of the preceding sections then readilytranslate to higher order scattering transforms; details are given in Appendix K.
The map N Gpintroduced in (2) can also be adapted to aggregate higher-order tensorial features into graph levelfeatures: With } f } q : “ p ř i 1 ,...,i k P G | f i 1 ,...,i k | q µ i 1 ,...,i k q 1 { q and µ G k : “ ř | G | i 1 ...i k “ 1 µ i 1 ,...,i k , wedeﬁne N G k p p
f q “ p} f } (cid:96) 1 p G k q {? µ G k , } f } (cid:96) 2 p G k q , } f } (cid:96) 3 p G k q , ..., } f } (cid:96)
p p G k q q J {? p .
Given a featuremap Φ kN with feature space F N “ ‘ Nn “` (cid:96) 2 p G kn q ˘ | Γ n ´ 1 | , we obtain a corresponding map Ψ kNmapping from (cid:96) 2 p G k q to R N “ ‘ Nn “ 1 p R p n q | Γ n ´ 1 | by concatenating Φ kN with the map that thefamily of non-linear maps t N p n G kn u Nn “ 1 induces on F N by concatenation. The resulting map Ψ kNagain has stability properties analogous to the node level case, as detailed in Appendix K.
Experimental Results
We showcase that even upon selecting a fairly simple architecture conforming to our general theory,our generalized graph scattering networks are able to outperform both wavelet-based scatteringtransforms and leading graph-networks under different circumstances.
To aid visual clarity whencomparing results, we colour-code the best-performing method in green, the second-best performingin yellow and the third-best performing method in orange respectively.
Social Network Graph Classiﬁcation: To facilitate contact between our generalized graph scattering networks, and the wider literature, we combine a network conforming to our general theory with the low pass aggregation scheme of Section 6.2 and a Euclidean support vector machine with RBF-kernel (GGSN+EK). For this hybrid architecture, classiﬁcation accuracies under 10 -fold cross validation on ﬁve common social network graph datasets are compared withperformances of popular graph kernel approaches, leading deep learning methods as well as geometric wavelet scattering (GS-SVM)
We chose a generalized scattering architecture ofdepth N “ 4 . As normal-operators, we utilize in each layer the un-normalized graph Laplacian L
“ D ´ W scaled by its largest eigenvalue ( ∆ “ L { λ max p L q ). Filters are chosen as12 p [Equation]
q , r cos p [Equation] q [Equation]
∆ s , sin p π ¨ ∆ q , r cos p π ¨ ∆ q ´ ψ ∆ ψ J ∆ sq , which allows to specifythe output generating function solely by demanding χ p 0 q “ 1 and χ p λ q “ 0 on all other eigenvaluesof ∆ .
Here ψ ∆ is the normalized vector of all ones (satisfying ∆ ψ ∆ “ 0 ).
Non-linearities are chosenas modulus.
Appendix L provides more details and explains how this architecture conforms to ourgeneral framework.
As evident from Table 1, our network consistently achieves higher accuracies thanthe geometric wavelet scattering transform of [12], with the performance gap becoming signiﬁcant onthe more complex REDDIT datasets, ﬁnally reaching a relative mean performance increase of morethan 10% on REDDIT- 12 K. This indicates the liberating power of transcending the graph waveletsetting.
While on comparatively smaller and somewhat simpler datasets there is a performance gapbetween our static architecture and fully trainable networks, this gap closes on more complex datasets: While P-Poinc outperforms our method on IMDB datasets for example, the roles are reversed on REDDIT datasets.
On REDDIT-B our approach trails only GIN, with the difference in accuraciesnot being signiﬁcant.
On REDDIT- 5 K our method comes in third, with the gap to the second bestmethod (GIN) being statistically insigniﬁcant.
On REDDIT- 12 K we generate state of the art results.
[Table]
Regression of Quantum Chemical Energies: In order to showcase the prowess of both our higherorder scattering scheme and our spectrum-agnostic aggregation method of Section 6.1, we combinethese building blocks into a hybrid architecture which we then apply in combination with kernelmethods ( 2 GGST + EK) to the task of atomization energy regression on QM 7 .
This is a comparativelysmall dataset of 7165 molecular graphs, taken from the 970 million strong molecular database GDB13 [2].
Each graph in QM 7 represents an organic molecule, with nodes corresponding to individualatoms.
Beyond the node-level information of atomic charge, there is also edge level informationcharacterising interaction strengths between individual nodes/atoms available.
This is encoded intoso called Coulomb matrices (see e.g. [32] or Appendix L) of molecular graphs, which for us serve adual purpose: On the one hand we consider a Coulomb matrix as an edge-level input signal on a givengraph.
On the other hand, we also treat it as an adjacency matrix from which we build up a graph
Laplacian L .
Our normal operator is then chosen as ∆ “ L { λ max p L q again.
Connecting operators are set to the identity, while non-linearities are ﬁxed to ρ n ě 1 p¨q “ | ¨ | .
Filters are chosen as p sin p π { 2 ¨ ∆ q , cos p π { 2 ¨ ∆ q , sin p π ¨ ∆ q , cos p π ¨ ∆ qq acting through matrix multiplication.
Output generating functions are set to the identity and depth is N “ 4 .
Graph level features are aggregated via the map N G 2of Section 7.
Generated feature vectors are combined with node node level scattering features (based on atomic charge) into composite feature vectors; plotted in Figure 4.
As is visually evident, even when reduced to the low-dimensional subspace of their ﬁrst three principal components, the generated scattering features are able to aptly resolve the atomization energy of the molecules.
This aptitude is also reﬂected in Table 2, comparing our approach with leading graph-based learning methodstrained with ten-fold cross validation on node and (depending on the model) edge level information.
300 Our method is the best performing.
We signiﬁcantly outperform the next best model (DTNN), producing less than half of its mean absolute error (MAE).
Errors of other methods are at least one — sometimes two — orders of magnitude greater.
In part, this performance discrepancy might be explained by the hightened suitability of our scattering transform for environments with somewhat limited training-data availability.
Here we speculate that the additional performance gap might be explained by the fact that our graph shift operator ∆ carries the same information as the Coulomb matrix (a proven molecular graph descriptor in itself [32]).
Additionally, our ﬁlters being inﬁnite series’ in powers of the underlying normal operator allows for rapid dispersion of information across underlying molecular graphs, as opposed to e.g. the ﬁlters in GraphConv or SchNet, which do not incorporate such higher powers.
[Table]
Discussion of Results in Context
Leaving behind the traditional reliance on graph wavelets, we developed a theoretically well foundedframework for the design and analysis of (generalized) graph scattering networks; allowing for varyingbranching rations, non-linearities and ﬁlter banks.
We provided spectrum independent stabilityguarantees, covering changes in input signals and for the ﬁrst time also arbitrary normal perturbationsin the underlying graph-shift-operators.
After introducing a new framework to quantify vertex-setnon-preserving changes in graph domains, we were able to obtain spectrum-independent stabilityguarantees for this setting too.
We provided conditions for energy decay in scattering networks anddiscussed their relation to asymptotic expressivity.
Then we introduced a new method of graph-levelfeature aggregation, discussed stability properties of the resulting graph-level scattering transformand extended scattering networks to higher order input data.
Our numerical experiments showedthat a simple scattering transform conforming to our framework is able to outperform the traditionalgraph-wavelet based approach to graph scattering in social network graph classiﬁcation tasks.
Oncomplex datasets our method is also competitive with current fully trainable methods, ouperformingall competitors on REDDIT- 12 K. Additionally, higher order graph scattering transforms signiﬁcantlyoutperform current leading graph-based learning methods in predicting atomization energies on QM 7 .
A reasonable critique of scattering networks as tractable models for general graph convolutionalnetworks is their inability to emulate non-tree-structured network topologies.
While transcendingthe wavelet setting has arguably diminished the conceptual gap between the two architectures, thisstructural difference persists in our work.
Additionally we note that despite a provided promisingexample, it is not yet clear whether the newly introduced graph-perturbation framework can aptlyprovide stability guarantees to all reasonable coarse-graining procedures.
Exploring this questionis ongoing work.
Finally, we caution against an over-interpretation of established mathematicalguarantees: Such guarantees certainly do not negate biases that may be inherent to utilized datasets.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] A discussion of how and in which order the main claims
made in Abstract and Introduction were substantiated within the paper is a main focus
of Section
(b) Did you describe the limitations of your work?
[Yes] This is a second focus of Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] This is
part of Section 9.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Every Theorem
and Lemma is stated in a mathematically precise way, with all underlying assumptions
included.
Additionally, Appendix A brieﬂy reviews some terminology that might not
be immediately present in every readers mind, but is utilized in order to be able to state
theoretical results in a precise and concise manner.
(b) Did you include complete proofs of all theoretical results?
[Yes] This is the Focus of
Appendices B, C, D, F, G, H, I and J. Results in Appendix K are merely stated, as
statements and corresponding proofs are in complete analogy (in fact almost verbatim
the same) to previously discussed statements and proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Yes; please see
the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] This is the main focus of Appendix L.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] Errors for results of conducted experiments are includedin Table and Table respectively.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] This is described at the beginning
of Appendix L
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] All utilized datasets
were matched to the papers that introduced them (cf.
Section L).
Additionally, we
partially built on code corresponding to [12] which we mentioned in Appendix L.
(b) Did you mention the license of the assets?
[Yes] We mentioned in Appendix L that the
code corresponding to [12] is freely available under an Apache License.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Please see supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] For the utilized social network datasets,
we
mention in Appendix L that neither personally identifyable data nor content that might
be considered offensive is utilised.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Some Concepts in Linear Algebra
In the interest of self-containedness, we provide a brief review of some concepts from linear algebrautilized in this work that might potentially be considered more advanced.
Presented results are allstandard; a very thorough reference is [25].
Hilbert Spaces: To us, a Hilbert space — often denoted by H — is a vector space over the complexnumbers which also has an inner product — often denoted by x¨ , ¨y H .
Prototypical examples aregiven by the Euclidean spaces C d with inner product x x, y y C d : “ ř di “ 1 x i y i .
Associated to an innerproduct is a norm, denoted by } ¨ } H and deﬁned by } x } H : “ a x x, x y H for x P H .
Direct Sums of Spaces: Given two potentially different Hilbert spaces H and p H , one can formtheir direct sum H ‘ p H .
Elements of H ‘ p H are vectors of the form p a, b q , with a P H and b P p H .
Addition and scalar multiplication are deﬁned in the obvious way byp a, b q ` λ p c, d q : “ p a ` λc, b ` λd q for a, c P H , b, d P p H and λ P C .
The inner product on the direct sum is deﬁned byxp a, b q , p c, d qy H ‘ p H : “ x a, c y H ` x b, d y p H .
As is readily checked, this implies that the norm } ¨ } H ‘ p H on the direct sum is given by}p
[Equation]
Standard examples of direct sums are again the Euclidean spaces, where one has C d “ C n ‘ C m ifm ` n “ d , as is easily checked.
One might also consider direct sums with more than two summands,writing C d “ ‘ di “ 1 C for example.
In fact, one might also consider inﬁnite sums of Hilbert spaces:
The space ‘ 8 i “ 1 H i is made up of those elements a “ p a 1 , a 2 , a 3 , ...
q with a i P H i for which thenorm
[Equation]
is ﬁnite.
This means for example that the vector p 1 , 0 , 0 , 0 , ...
q is in ‘ 8 i “ 1 C , while p 1 , 1 , 1 , 1 , ...
q isnot.
Direct Sums of Maps: Suppose we have two collections of Hilbert spaces t H
i u Γ i “ 1 , t r H
i u Γ
i “ 1 withΓ P N or Γ “ 8 .
Suppose further that for each i ď Γ (resp.
i ă Γ ) we have a (not necessarily linear)map J i : H
i Ñ r H i .
Then the collection t J i u Γ i “ 1 of these ’component’ maps induce a ’composite’map
[Equation]
between the direct sums.
Its value on an element a “ p a 1 , a 2 , a 3 , ...
q P ‘ Γ i “ 1 H i is deﬁned by
[Equation]
Strictly speaking, one has to be a bit more careful in the case where Γ “ 8 to ensure that} J p a q} ‘ 8 i “ 1 r H i ‰ 8 .
This can however be ensured if we have } J i p
Operator Norm: Let J : H Ñ r H be a linear operator between Hilbert spaces.
We measure its’size’ by what is called the operator norm, denoted by } ¨ } op and deﬁned by
[Equation]
Adjoint Operators Let J : H Ñ r H be a linear operator from the Hilbert space H to the Hilbertspace r H .
Its adjoint J ˚ : r H Ñ H is an operator mapping in the opposite direction.
It is uniquelydetermined by demanding that
[Equation]
holds true for arbitrary f P H and u P r H .
Normal Operators:
If a linear operator ∆ : H Ñ H maps from and to the same Hilbert space,we can compare it directly with its adjoint.
If ∆∆ ˚ “ ∆ ˚ ∆ , we say that the operator ∆ is normal.
Special instances of normal operators are self-adjoint operators, for which we have the strongerproperty ∆ “ ∆ ˚ .
If an operator is normal, there are unitary maps U : H Ñ H diagonalizing ∆ as
[Equation]
with eigenvalues in C .
We call the collection of eigenvalues the spectrum σ p ∆ q of ∆ .
If dim H “ d ,we may write σ p ∆ q “ t λ u di “ 1 .
It is a standard exercise to verify that each eigenvalue satisﬁes|
λ i | ď } ∆ } op .
Associated to each eigenvalue is an eigenvector φ i .
The collection of all (normalized)eigenvectors forms an orthonormal basis of H .
We may then write
[Equation]
Resolvent of a (normal) Operator:
Given a normal operator ∆ on some Hilbert space H , we havethat the operator p ∆ ´
z q : H Ñ H is invertible precisely if z ‰ σ p ∆ q .
In this case we write
[Equation]
and call this operator the resolvent of ∆ at z .
It can be proved that the norm of the resolvent satisﬁes
[Equation]
where dist p z, σ p ∆ qq denotes the minimal distance between z and any eigenvalue of ∆ .
Functional Calculus:
Given a normal operator ∆ : H Ñ H on a Hilbert space of dimension d anda complex function g : C Ñ C , we can deﬁne another normal operator obtained from applying thefunction g to ∆ by
[Equation]
For example if g p¨q “ | ¨ | , we obtain the absolute value | ∆ | of ∆ by specifying for all f P H that
[Equation]
Similarly we ﬁnd (if z R σ p ∆ q and for f P H )
[Equation]
where we think of the left-hand-side as applying a function to ∆ , while we think of the right-hand-sideas inverting the operator p ∆ ´ z q .
This now allows us to apply tools from complex analysis also tooperators: If a function g is analytic (i.e. can be expanded into a power series), we have
[Equation]
for any circle S Ď C encircling λ by Cauchy’s integral formula.
Thus, if we chose S large enough toencircle the entire spectrum σ p ∆ q , we have
[Equation]
Frobenius Norm:
Given a ﬁnite dimensional Hilbert space H with inner product x¨ , ¨y H , and anorthonormal basis t φ i u di “ 1 , we deﬁne the trace of an operator A : H Ñ H as
[Equation]
It is a standard exercise to show that this is independent of the choice of orthonormal basis.
Theassociated Frobenius inner product on the space of operators is then given as
[Equation]
Hence the Frobenius norm of an operator is determined by
[Equation]
It is a standard exercise to verify that we have } A } op ď } A } F .
Since the trace is independent of thechoice of orthonormal basis, the Frobenius norm is invariant under unitary transformations.
Moreprecisely, if U, V : H Ñ H are unitary, we have
[Equation]
Frobenius norms can be used to transfer Lipschitz continuity properties of complex functions to thesetting of functions applied to normal operators:
Lemma A.1.
Let g : C Ñ C be Lipschitz continuous with Lipschitz constant D g .
This implies
[Equation]
for normal operators X, Y on H .
Proof.
This proof is taken (almost) verbatim from [38].
For an operator A : H Ñ H denote by A ijits matrix representation with respect to the orthonormal basis t φ i u di “ 1 :
[Equation]
We then have
[Equation]
as a quick calculation shows.
Let now U, W be unitary (with respect to the inner product x¨ , ¨y H )operators diagonalizing the normal operators X and Y as
[Equation]
Since the Frobenius norm is invariant under unitary transformations we ﬁnd
[Equation]
B Proof of Theorem 2.
Theorem B.1.
Let ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p
G q be normal.
If the family t g i p¨qu i P I of bounded continuousfunctions satisﬁes A ď
ř i P I
| g i p c q| 2 ď B for all c in the spectrum σ p ∆ q , we have ( @ f P (cid:96) 2 p G q )
A } f } 2 (cid:96) 2 p G q ď ÿ i P I } g i p ∆ q f } 2 (cid:96) 2 p G q ď B } f }
[Equation]
Proof.
Writing the normalized eigenvalue-eigenvector sequence of ∆ as p λ i , φ i q | G | i “ 1 , we simply noteÿ
[Equation]
Now under the assumption, we can estimate the sum in brackets by A from below and by B fromabove.
Then we need only use Bessel’s (in)equality to prove
[Equation]
C Proof of Theorem 4.
Theorem C.1.
With the notation of Section 3 and setting B 0 “ 1 , we have:
[Equation]
To streamline the argumentation let us ﬁrst introduce some notation:
Notation C.2.
Let us denote paths in Γ N as q : “ p γ N , ..., γ 1 q .
For f P (cid:96) 2 p G q let us write
[Equation]
Proof.
By Deﬁnition, we have
[Equation]
[Equation]
We proceed in two steps:
Our initial goal is to upper bound a n as
[Equation]
for b n : “ ř q P
[Equation]
n } f q ´ h q } [Equation] 2 p G n q with b 0 “ } f ´ h } 2 (cid:96) 2 p G q .
To achieve this we note that (3) isequivalent to
[Equation]
which upon unraveling deﬁnitions may be written asÿ
[Equation]
To establish (4)
[Equation]
we note
[Equation]
that in the sum over paths of length n
[Equation]
any p q P Γ n can uniquely be writtenas p q “ p γ n , q q , with the path q P Γ n ´ 1 of length p n ´ 1 q determined by
[Equation]
With this we ﬁndÿ
[Equation]
Thus we can rewrite the left hand side of (4) asÿ
[Equation]
The fact that in each layer the function t χ n p¨qu Ť t g γ n p¨qu γ n P Γ n form a generalized frame with upperframe constant B n implies by Theorem 2.1, that we can further bound this as
[Equation]
Using the Lipschitz continuity of ρ n and P n , we arrive at the desired expression (4).
Having established that
[Equation]
holds true, we note that we can establish
[Equation]
arguing similarly as in the case of (4) by using (for f P (cid:96) 2 p G n ´ 1 q )ÿ
[Equation]
together with the frame property and Lipschitz continuities.
We then iterate this inequality and recallthat b 0 “ } f ´ h } 2 (cid:96) 2 p G q .
Using the fact that
[Equation]
we ﬁnally ﬁnd
[Equation]
D Proof or Theorem 4.
Theorem D.1.
Let Φ N and r Φ N be two admissible scattering transforms based on the same modulesequence Ω N and operator sequences D N , r D N with the same connecting operators ( P n “ r P n )
ineach layer.
Assume that the respective normal operators satisfy } ∆ n ´ r ∆ n } F ď δ for some δ ą 0 .
Further assume that the functions t g γ n u γ n P Γ n and χ n in each layer are Lipschitz continuous withassociated Lipschitz constants satisfying L 2 χ n ` ř
[Equation]
ď D 2 for all n ď N and some D ą 0 . Then we have for all f P (cid:96) 2 p G q
[Equation]
Notation D.2.
Let us denote scattering propagators based on operators ∆ n and connecting operators
P n by U n and scattering propagators based on operators r ∆ n by r U n .
Similarly, to Notation C.2, let usthen write (with q “ p γ N , ..., γ 1 q )
[Equation]
Proof.
By deﬁnition we have
[Equation]
˛ ‚ loooooooooooooooooooooooooooooooooooooooomoooooooooooooooooooooooooooooooooooooooon
[Equation]
We deﬁne b n : “ ř q P Γ n } f q ´ r f q } 2 (cid:96) 2 p G n q , with b 0 “ } f ´ h } 2 (cid:96) 2 p G q “ 0 and note
[Equation]
ÿ
[Equation]
n
[Equation]
¸ .
Using (with | a ` b | 2 ď 2 p| a | 2 ` | b | 2 q )
[Equation]
and
[Equation]
(c.f. Lemma A.1 ), we ﬁnd
[Equation]
Using L 2 χ n ` ř γ n P Γ n L 2 γ n ď D 2 , we then infer (using the admissibility assumption L ` n , R ` n ,
[Equation]
[Equation]
Using similar arguments, we ﬁnd
[Equation]
Thus we now know
[Equation]
In total we ﬁnd
[Equation]
where we have estimated the sum over p b n ´ 1 ´ b n q by zero from above again.
This establishes theclaim.
E Prototypical Example illustrating ω - δ Closeness and δ -Unitary
Equivalence
To investigate the example of Figure 2, we label the vertices of the respective graphs as depicted in Figure 5.
We denote the left graph by G and the right graph by r G .
The node-weights on r G are given as r µ i “ 1 for 1 ď i ď 7 , while on G the weights are given as µ i “ 1 for 1 ď i ď 5 while µ 6 “ 2 .
We then consider the respective un-normalized graph Laplacians ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p G q and r ∆ : (cid:96) 2 p r G q Ñ (cid:96) 2 p r G q , which for a given adjacency matrix W on a graph signal space (cid:96) 2 p G q with node weights t µ i u i is given as
[Equation]
Such operators are positive and hence | ∆ | “ ∆ (similarly for r ∆ ).
We now need to ﬁnd operators J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q and r J : (cid:96)
2 p r G q Ñ (cid:96) 2 p G q satisfying the conditions of Deﬁnition 4.3.
To construct J , we deﬁne a family t ψ i u 6 i “ 1 of vectors on (cid:96) 2 p r G q as
[Equation]
The map J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q is then deﬁned as
Jf
[Equation]
[Equation]
[Equation]
f i ψ i , for any f P (cid:96) 2 p G q .
We take r J : (cid:96)
2 p r G q Ñ (cid:96) 2 p G q to be its adjoint ( r J : “ J ˚ ), which determinedexplicitly by
[Equation]
for any u P (cid:96) 2 p r G q We shall now ﬁrst check the conditions for δ -quasi unitary equivalence, which welist again for convenience; now adapted to our current setting:
[Equation]
We ﬁrst note that since r J “ J ˚ , we have }p J ´ r J ˚ q f } (cid:96) 2 p r G q “ 0 .
Next we note
[Equation]
Furthermore we note
[Equation]
p r G q loooooooomoooooooon “ δ ik “ f i and hence } f ´ r JJf } 2 (cid:96) 2 p G q “ 0 .
It remains to control } u ´ J r Ju } 2 (cid:96) 2 p r G q .
We noter
Ju “ p u 1 , u 2 , u 3 , u 4 , u 5 , p u
[Equation]
` u 6 q
[Equation]
2 q J and thus
[Equation]
Which implies
[Equation]
and thus
[Equation]
We have
[Equation]
Since Ă W 67 “ 1 { δ 2 by assumption, we have
[Equation]
Thus we have proven δ -unitary-equivalence and it remains to establish p´ 1 q - 12 δ closeness.
Combining Proposition 4.4.12.
and Theorem 4.4.15 of [30], instead of bounding }p r RJ ´ JR q f } (cid:96) 2 p r G q ď12 δ } f } (cid:96) 2 p G q directly, we may instead establish that there are operators J 1 : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q ,
[Equation]
satisfying
[Equation]
[Equation]
and
[Equation]
We chose J 1 “ J and determine Ă J 1 by setting (for ( 1 ď i ď 6 ))
[Equation]
Thus (5) is clearly satisﬁed.
For (6) we note that we have
[Equation]
Thus we have
[Equation]
¯ as before.
It remains to establish (7).
We have
[Equation]
while we have
[Equation]
We have (with all node-weights on (cid:96) 2 p G q equal to unity)
[Equation]
˜ÿ
[Equation]
˜ÿ
[Equation]
¸
And thus
[Equation]
which proves the claim.
F Proof of Lemma 4.4
Lemma F.1.
In the setting of Deﬁnition 4.3 let ∆ and r ∆ be ω - δ -close and satisfy } ∆ } op , } r ∆ } op ď
Kfor some K ą 0 .
If g : C Ñ C is holomorphic on the disk B K ` 1 p 0 q of radius p K ` 1 q , there is aconstant C g ě 0 so that
[Equation]
with C g depending on g , ω and K .
Proof.
Without loss of generality, let us assume that K ą | ω | .
Let us denote the circle of radius r in 694 C by S r .
For any holomorphic function g and (normal) operator ∆ whose spectrum is enclosed bythe circle S r , we can express the operator g p ∆ q as
[Equation]
as discussed in Appendix A (see also [7] for more details).
Note that in our case the resolvent
R p z, ∆ q “ p ∆ ´ z q ´ 1 is well deﬁned for | z | ě K , since with our assumptions all eigenvalues arewithin the circle of radius K .
Additionally note that we have
[Equation]
if | z | ě K .
The same holds true after replacing ∆ with r ∆ .
Since for any normal operator ∆ we have
[Equation]
R p z, ∆ q
[Equation]
we ﬁnd
[Equation]
To quantify the difference } R p z, r ∆ q J ´ JR p z, ∆ q} op in terms of the difference } r R p ω q J ´
JR p ω q} op ď δ , we deﬁne the function
[Equation]
for which
[Equation]
holds, as proved (in more general form) in Lemma 4.5.9 in [30].
Since on S K ` 1 we have and| z ´ ω | ď 2 K ` 1 hence γ 0 p z q ď 2 p K ` 1 q , we ﬁnd
[Equation]
Thus we may set
[Equation]
G Proof of Theorem 4.
Theorem G.1.
Let Φ N , r Φ N be admissible scattering transforms based on a common modulesequence Ω N and differing operator sequences D N , r D N .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q , r J n : (cid:96) 2 p r G n q Ñ (cid:96) 2 p G n q ( 0 ď n ď N ) so thatthe respective signal spaces are δ -unitarily equivalent, the respective normal operators ∆ n , r ∆ nare ω - δ -close as well as bounded (in norm) by K ą 0 and the connecting operators satisfy} r P n J n
[Equation]
1 f ´ J n P n f } (cid:96) 2 p r G n q ď δ } f } (cid:96) 2 p G n ´ 1 q .
For the common module sequence Ω N assume thatthe non-linearities satisfy } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q ď δ } f } (cid:96) 2 p G n q and that the constants C χ n andt C g γn u γ n P Γ N associated through Lemma 4.4 to the functions of the generalized frames in each layersatisfy C 2 χ n ` ř γ n P Γ N C 2 g γn ď D 2 for some D ą 0 .
Denote the operator that the family t J n u n ofidentiﬁcation operators induce on F N through concatenation by J N : F N Ñ Ă F N .
Then we havewith K N “ a [-p 8 N ´ 
-][Equation]
2 ` 12 q{ 7 that
[Equation]
P
[Equation]
G
[Equation]
If additionally } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q “ 0 or } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q “ 0 holds in eachlayer, then we have K N “ O p 4 N[-´ 
 qp 2 D 2 ` 4 B q{ 3 ¨ B N ´ 1 if B ą 1-] { 2 q ; if both additional equations hold, we have K N “ O p 2 N { [-
-]2 q .
Notation G.2.
Let us denote scattering propagators based on operators ∆ n and P n by U n andscattering propagators based on operators r ∆ n and r P n by r U n .
Similarly, to Notation D.2 and , let usthen write (with q “ p γ N , ..., γ 1 q )
[Equation]
Proof.
By deﬁnition we have
[Equation]
We deﬁne b n : “ ř q P Γ n } J n f q ´ r f q } 2 (cid:96) 2 p r G n q , with b 0 “ } J 0 f
´ J 0 f } 2 (cid:96) 2 p r G q “ 0 and notea n ` b n “ ÿ q P Γ n ´
[Equation]
¸ .
Using
[Equation]
and }r g γ n p ∆ n q ´ g γ n p r ∆ n qs} 8 ď C 2 g γ ¨ δ 2 (c.f. Lemma 4.4), we ﬁnd
[Equation]
where the second inequality arises from permuting the identiﬁcation operator J n through non-linearityand connecting operator.
Using C 2 χ n ` ř
[Equation]
n ď D 2 , we then infer
[Equation]
Using similar arguments, we ﬁnd
[Equation]
In total we ﬁnd
N ÿ
[Equation]
If one of the additional equations holds, we ﬁnd
[Equation]
and
[Equation]
Thus
N ÿ
[Equation]
If both additional equations hold, we ﬁnd
[Equation]
[-U r γ 1 sp f q} 
 (cid:96) 2 p G N q stored inthe network at layer N , we have after extending Φ N p f q by zero to match dimensions with Φ N ` 1 p f qthat-]and
[Equation]
which then implies
[Equation]
{+H Proof of Lemma 5.
+}
Lemma H.1.
The energy W N stored in layer N satisﬁes
[Equation]
with constants C
[Equation]
Proof.
min
[Equation]
and similarly
[Equation]
This yields the starting point for our induction.
Now for the inductive step assume the claim holds upuntil layer N ´ 1 .
Then we have
[Equation]
using Notation C.2.
We note
[Equation]
Every path r q P Γ N may be written as q “ p γ n , q q , for some γ n P Γ n and q P Γ N ´ 1 .
Thus we have
[Equation]
Inserting this in the above equation yields
[Equation]
˛
[Equation]
¸ looooooooooooooooooooooooooooooooooooooooooooooooomooooooooooooooooooooooooooooooooooooooooooooooooon
[Equation]
We have
[Equation]
by the operator frame property.
With this we ﬁnd:
[Equation]
u
[Equation]
after unravelling the deﬁnition
[Equation]
The induction hypothesis together with the deﬁnition of C ˘ N now yields the claim.
I Proof of Lemma 5.2 and Corollary 5.3
Theorem I.1.
Let Φ 8 be a generalized admissible graph scattering transform based on a an operatorsequence D 8 “ p P n , ∆ n q 8 n “ 1 and a module sequence Ω 8 with each ρ n p¨q ě 0 .
Assume in eachlayer n ě 1 that there is an eigenvector ψ n of ∆ n with solely positive entries; denote the smallestentry by m n : “ min i P G n ψ n r i s .
Denote the eigenvalue corresponding to ψ n by λ n .
Quantify the’spectral-gap’ opened up at this eigenvalue through neglecting the output-generating function byη n : “ ř γ n P Γ n | g γ n p λ n q| 2 and assume B n m n ě η n .
We then have
[Equation]
Proof.
Denote the spectral projection (i.e. the orthogonal projection projecting to the space ofeigenvectors) onto the eigenspace corresponding to λ n by P nc .
Then we have
[Equation]
By orthogonality of the spectral projection [Equation] we then have
[Equation]
Furthermore, we have
[Equation]
with equality if the multiplicity of λ N is exactly one.
With this we ﬁnd
[Equation]
Since the image of ρ N is contained in R ` by assumption, we have
[Equation]
Here the second to last inequality follows since in any ﬁnite dimensional vector space, the 1 -norm islarger than the 2 -norm (
|| f ||
ě
||
f
|| 2 ) and all weights are assumed to satisfy µ i ě 1 .
Thus we nowknow
[Equation]
Inserting this in our estimate for W N p f q we ﬁnd
W N p f q ď ˆ 1 ´ ˆ m 2 N ´ η n B n ˙˙
L ` N R ` N B N ¨ W N ´ 1 p f q ď C ` N
N ź n “ˆ 1 ´ ˆ m 2 N ´ η n B n ˙˙ || f || 2 (cid:96) 2 p G q .
Taking N to inﬁnity, we know that C ` N converges to something larger than zero by assumption.
For products of the form N ś n “ 0 p 1 ´ q n q with 0 ď q n ă 1 it is a standard exercise to prove that the limitis non-zero precisely if the sum over the q n converges.
But this is excluded by our assumption in
Corollary 5.3.
J Stability of Graph Level Feature AggregationJ.1 Proof of Theorem 6.1
Theorem J.1.
Assuming admissibility, we have } Ψ N p f q ´ Ψ
N p g q} R N ď } f ´ g } (cid:96) 2 p G q .
With theconditions and notation of Theorem 4.2 we have
[Equation]
Additionally, in the setting of Theorem 4.5, assuming that for each n ď N the identiﬁcation operator
J n satisﬁes
[Equation]
[Equation]
2 ď
[Equation]
ď p [Equation] ) implies ( @ f P (cid:96) 2 p G q )
[Equation]
δ ¨ } f }
[Equation]
Furhermore, under the assumptions of Corollary 5.3 Ψ 8 p f q “ 0 implies f “ 0 .
Proof.
Let f, h P (cid:96) 2 p G q .
To prove the ﬁrst two claims, it sufﬁces to prove
[Equation]
and
[Equation]
Both statements follow immediately, as soon as we have proved
[Equation]
for arbitrary choices of p and G .
To this end we note that for p
ě 2 we have } f } (cid:96)
p p G q ď } f } (cid:96) 2 p G qby the monotonicity of p -norms, while we have } f } (cid:96) 1 p G q ď ?
µ G ¨ } f } (cid:96) 2 p G q by Hölder’s inequality.
With this we ﬁnd
[Equation]
where we have employed the reverse triangle inequality in the ﬁrst step.
To prove the second claim, we note that we have
[Equation]
Thus it remains to bound the last expression.
We have
[Equation]
By our results of Appendix C and since we assume admissibility, we have
[Equation]
Thus in total
[Equation]
from which our stability claim follows.
It remains to prove that the assumptions of Corollary 5.3 Ψ 8 p f q “ 0 imply f “ 0 .
But since
N Gp p f q “ 0 implies f “ 0 , this is clear.
J.2 Statement and Proof of the Analogue of Theorem 6.1 for Low-Pass feature Aggregation
The main assumption we have in this section is that each operator ∆ n (and r ∆ n ) has a simple lowestlying eigenvalue equal to zero.
We denote the associated eigenvector (determined up to a complexphase) by ψ ∆ n and the associated spectral projection to the lowest lying eigenvalue by P ∆ n .
It actsas
[Equation]
Now we are ready to state our main stability result under these circumstances:
Theorem J.2.
Assuming admissibility, we have } Ψ |x¨ , ¨y| N p f q ´ Ψ |x¨ ,
[Equation]
| N p g q} C N ď } f ´ g } (cid:96) 2 p G q .
Withthe conditions and notation of Theorem 4.2 and under the additional assumption }p P ∆ n ´ P r ∆ n q} op ď
K ¨ δ for n ď N and some K ě 0 , we have
[Equation]
N
[Equation]
2 p
[Equation]
q .
In the setting of Theorem 4.5 and under the additional assumption |} P ∆ n f } (cid:96) 2 p G n q ´} P r ∆ n J n f } (cid:96) 2 p r G n q | ď
Kδ ||
f || (cid:96) 2 p
G n q for all f P (cid:96) 2 p G n q ( n ď N ), we have
[Equation]
Proof.
Let f, h P (cid:96) 2 p G q .
To prove the ﬁrst claim, it sufﬁces to prove
[Equation]
This immediately follows from the fact that for all f P (cid:96)
[Equation]
G n q
[Equation]
by Hölder’s inequality
[Equation]
The next claim we want to prove is that we have for all f P (cid:96) 2 p G q
[Equation]
We note
[Equation]
ď
[Equation]
Hence we need to bound the expression " }p P ∆ n ´ P r ∆ n q x q } 2 (cid:96) 2 p G n q ".
We note
[Equation]
and thus
[Equation]
and the claim follows.
Finally we want to prove
[Equation]
We note
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
which proves the claim.
In establishing expressivity, we have to be a tiny bit more careful:
Theorem J.3.
In the setting of of Corollary 5.3, assume that in each layer n , the output generatingfunction χ n of the underlying scattering transform satisﬁes χ n p 0 q ‰ 0 and χ n p λ i q “ 0 for orderednon-zero eigenvalues λ 2 ď ... ď λ | G n | of the operator ∆ n .
Then Ψ |x¨ , ¨y|p f q “ 0 implies f “ 0 .
Proof.
Under these assumptions, we do not lose any information by projecting to ψ ∆ n in each(cid:96) 2 p G n q , since the image of χ n p ∆ n q is already contained in the one-dimensional space generated bythe lowest lying eigenvector ψ ∆ n .
K Stability of Higher Order Scattering
In this section, we list stability theorems for higher order scattering transforms.
As the respectiveproofs are identical to the corresponding results for the node setting, we do not repeat them here.
Theorem K.1.
With the notation of Section 7, we have for all f, h P (cid:96) 2 p G k q :
[Equation]
n ´ 1 ź (cid:96) “
[Equation]
(cid:96) ¸
[Equation]
(cid:96)
[Equation]
q
Theorem K.2.
Let Φ N and r Φ N be two admissible scattering transforms based on the same modulesequence Ω N and operator sequences D kN , r D kN with the same connecting operators ( P kn “ r P kn ) ineach layer.
Assume that the respective normal operators satisfy } ∆ kn ´ r ∆ kn } F ď δ for some δ ą 0 .
Further assume that the functions t g γ n u γ n P Γ n and χ n in each layer are Lipschitz continuous withassociated Lipschitz constants satisfying L 2 χ n `
[Equation]
2 for all n ď N and some D ą 0 .
Then we have for all f P (cid:96) 2 p G k q
[Equation]
Theorem K.3.
Let Φ kN , r Φ kN be admissible scattering transforms based on a common modulesequence Ω N and differing operator sequences D kN , r D kN .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G kn q Ñ (cid:96) 2 p r G kn q , r J n : (cid:96) 2 p r G n q Ñ (cid:96) 2 p G kn q ( 0 ď n ď N ) so thatthe respective signal spaces are δ -unitarily equivalent, the respective normal operators ∆ kn , r ∆ knare ω - δ -close as well as bounded (in norm) by K ą 0 and the connecting operators satisfy} r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q ď δ } f } (cid:96) 2 p G n ´ 1 q .
For the common module sequence Ω N assume thatthe non-linearities satisfy } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G kn q ď δ } f } (cid:96) 2 p G kn q and that the constants C χ n andt C g γn u γ n P Γ N associated through Lemma 4.4 to the functions of the generalized frames in each layersatisfy C 2 χ n ` ř γ n P Γ N C 2 g γn ď D 2 for some D ą 0 .
Denote the operator that the family t J n u n ofidentiﬁcation operators induce on F kN through concatenation by J N : F kN Ñ Ă F kN .
Then we have} r Φ kN p J 0 f q ´ [-
 qp 2 D 2 ` 12 B q{ 7 ¨ B-]J N Φ kN p f q} Ă F kN ď K N ¨ δ ¨ } f } (cid:96) 2 p G k q , @
f P (cid:96) 2 p G k q , with K N “ a p 8 N ´ 1 qp 2 D 2 ` 12 q{ 7 .
If additionally } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G kn q “ 0 or} ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G kn q “ 0 holds in each layer, then we have K N “ O p 4 N { 2 q ; if bothadditional equations hold, we have K N “ O p 2 N { 2 q .
Theorem K.4.
Assuming admissibility, we have } Ψ kN p
f q ´ Ψ kN p
h q} R N ď } f ´ h } (cid:96) 2 p
G k q for allf, h P (cid:96) 2 p G q .
With the conditions and notation of Theorem K.2 we have
[Equation]
kN p f q} R N ď b 2 p 2 N ´ 1 q ¨ D ¨ δ ¨ } f } (cid:96) 2 p G k q .
Additionally, in the setting of Theorem K.3, assuming that for each n ď N the identiﬁcation operator
J n satisﬁes ˇˇ } J n f } (cid:96) 1 p r G kn q { a µ r G kn ´} f } (cid:96) 1 p G kn q {? µ G kn ˇˇ , ˇˇ } J n f } (cid:96) r p r G kn q ´} f } (cid:96) r p G kn q ˇˇ ď δ
¨ K ¨} f } (cid:96) 2 p G kn qfor 2 ď r ď p n implies ( @ f P (cid:96) 2 p G k q )} r Ψ N p J 0 f q ´ Ψ N p f q} R N ď ?
2 ¨ b
K 2 N ` K 2 ¨ δ ¨ } f } (cid:96) 2 p G k q .
L Additional Details on Experiments
Here we provide additional details on utilized scattering architectures, training procedures, datasetsand (performance of) other methods our approach is being compared to.
Irrespective of task, ourmodels are trained on an NVIDIA DGX A100 architecture utilizing between two and eight NVIDIA
Tesla A100 GPUs with 80GB memory each.
Running 10 -fold cross validation for the respectiveexperiments took at most 71 hours (which was needed for social network graph classiﬁcation on
REDDIT- 12 K).
L.1 Social Network Graph Classiﬁcation
Datasets: The data we are working with is taken from [45].
In particular this work introduced sixsocial network datasets extracted from from scientiﬁc collaborations (COLLAB), movie collaborations (IMDB-B, IMDB-M) and Reddit discussion threads (REDDIT-B, REDDIT- 5 K, REDDIT- 12 K).
Data is anonymised and contains no content that might be considered offensive.
Each graph carries aclass label, and the goal is to predict this label.
Some basic properties of these datasets are listed in
Table 3 below.
[Table]
871 These datasets contain graph structures , however they don’t contain associated weights or graphsignals.
Having unspeciﬁed weights simply means that the adjacency matrix W from which weconstruct the graph Laplacian
[Equation]
on which our operator ∆ is based simply has each entry corresponding to an edge set to unity.
Ifno edge is present between vertices i and j , the entry W ij is set to zero.
It remains to solve theproblem of the missing input signals.
Our strategy is to generate signals reﬂecting the geometryof the underlying graph.
We do this by utilizing features that associate to each node a number thatcharacterizes its role or importance within its local environment or within the entire graph.
We brieﬂydescribe them here:
Degree : The degree of a node is the number of edges incident at this node.
Eccentricity : For a connected graph, the eccentricity of a node is the maximum distancefrom this node to all other nodes.
On a disconnected graph it is not deﬁned.
Clustering :
For unweighted graphs the clustering c p u q of a node u is the fraction of possible
triangles through that node that actually exist
It is calculated as
[Equation]
Number of triangles : The number of triangles containing the given node as a vertex.
Core number : A k-core is a maximal subgraph that only contains nodes of degree k ormore.
The core number of a node is the largest value k of a k-core containing that node.
Clique number :
A clique is a subset of vertices of an undirected graph such that every two
distinct vertices in the clique are adjacent.
This input assigns the number of cliques the nodes participates in to each node.
Pagerank :
This returns the PageRank of the respective nodes in the graph.
PageRank
computes a ranking of the nodes in the graph based on the structure of the edges.
Originally
it was designed as an algorithm to rank web pages.
For the ﬁrst three datasets listed in Table 3 we utilize all listed input features.
For the latter threedatasets we have to refrain from using eccentricity as an input signal, as these datasets contain graphsthat have multiple non-connected graph components.
Scattering Architecture: We chose a generalized scattering architecture of depth N “ 4 .
Asnormal-operators, we utilize in each layer the un-normalized graph Laplacian L “ D ´ W scaledby its largest eigenvalue ( ∆ “ L { λ max p L q ).
Filters are chosen as 12 p sin p π { 2 ¨ ∆ q , r cos p π { 2 ¨ ∆ q ´ψ ∆ ψ J ∆ s , sin p π ¨ ∆ q , r cos p π ¨ ∆ q ´ ψ ∆ ψ J ∆ sq , which allows to specify the output generating functionsolely by demanding χ p 0 q “ 1 and χ p λ q “ 0 on all other eigenvalues of ∆ .
Here ψ ∆ is thenormalized vector of all ones (satisfying ∆ ψ ∆ “ 0 ).
Connecting operators are chosen as the identity,while we set ρ n ě 1 p¨q “ | ¨ | .
This scattering architecture is then applied to each of these input signal individually.
For each input signal, this returns a feature vector with 1 ` 4 ` 16 ` 64 “ 85 entries.
These individual feature vectors are then concatenated into one ﬁnal composite feature vector foreach graph.
Concerning applicable theoretical results, we note the following:
Let us ﬁrst ﬁx a single graph and let us assume the graph is connected. The associated graph Laplacianhas zero as lowest lying eigenvalue. Denote the second lowest eigenvalue by λ 2 p L q . The secondlowest eigenvalue of our normal operator ∆ is then given by λ 2 p
∆ q “ λ 2 p L q{ λ max p L qq .
Let χbe any function satisfying χ p λ q ď 1 , χ p 0 q “ 1 and χ p λ q “ 0 if λ ě λ 2 p ∆ q{ 2 .
Our architecturethen yields the same result as an architecture with χ as output generating function and ﬁlter bankgiven as 12 p sin p π {
[Equation]
∆ q ,
[Equation]
q ´ χ p ∆ qq , where weinterpret a cos 2 p¨q
[Equation]
cos p¨q (whenever χ “ 0 ).
This is an architecture based on a generalized framewith frame constants A “ B “ 1 (as deﬁned in Section 2).
Checking conditions of the varioustheorems, we see that the resulting scattering transform is admissible and if it were extended to depth
N “ 8 instead of being truncated at depth N “ 4 would satisfy the conditions of Theorem 5.2 withη n “ 0 .
As connecting operators are chosen as the identity and we have ∆ n “ ∆ for all n ď N , thismeans that the conditions for Corollary 5.3 are also fulﬁlled.
To check for stability under operatorperturbations, we note that
[Equation]
since ψ ∆ “ ψ r ∆ .
Thus we also have
[Equation]
Since the output generating ﬁlter is χ ∆ chosen to evaluate to one only on the lowest lying eigenvalue(i.e.
zero, which is stable under perturbations) and to zero on higher lying eigenvalues, this introducesno additional instability.
In total this implies that in the setting of Theorem 4.2 our architecture enjoysthe same stability guarantees as one that is based on a ﬁlterbank given by 12 p sin p π { 2 ¨ ∆ q , cos p π { 2 ¨∆ q , sin p π ¨ ∆ q , cos p π ¨ ∆ qq and an output generating function with negligible Lipschitz constant.
Similar considerations apply in the vertex-set non-conserving setting.
Training Procedure: We train RBF kernel support vector classiﬁers on our composite scatteringfeatures.
We ﬁx (cid:15) “ 0 . 1 .
The hyperparameter γ scaling the exponent is chosen from
[Equation]
while we pick the C that controls the error our slack variables introduce among [Equation]
[Equation]
[Equation] 1 , 1 , 10 , 25 , 50 , 100 , 1000 u .
We could simply implement the training of the RBF-classiﬁer on our composite scattering featuresby dividing each social-network dataset into 10 folds, then iteratively choosing one fold for testingand among the remaining 9 folds randomly choosing one for validation (i.e. for tuning the hyperparameters).
Instead, following [12] (whose code is released under an Apache license and on whichwe partially built), we take a slightly different approach: We still randomly split our dataset into 10folds.
Among the 10 folds, we iteratively pick one for testing.
Say we have picked the n t h fold fortesting.
Then there are 9 remaining folds.
We iteratively pick the m t hn (with 1 ď m n ď 9 ) of theremaining 9 folds for choosing hyperparameters.
This leaves 8 folds on which we train our modelfor each choice of hyper parameter in C pool ˆ G pool .
The resulting classiﬁers are all evaluated on them t hn fold.
The one that performs best is retained as classiﬁer m n .
As m n varies between 1 and 9(still for ﬁxed n ), this yields a set t f m n : 1 ď m n ď 9 u of nine classiﬁers.
From these we build theclassiﬁer f n , whose classiﬁcation result is obtained from a majority vote among the nine classiﬁersin t f m n : 1 ď m n ď 9 u .
Then we evaluate the performance of f n on the n t h fold to obtain then t h estimation of how well our model performs.
As n varies from one to ten, we built the meanand variance of the performances of the classiﬁers f n on the n t h fold expressed as the percentage ofcorrect classiﬁcations.
Reference Methods: To allow for a comparison of our results to the literature, typical classiﬁcationaccuracies for graph algorithms on social network datasets are displayed in Table 1.
Following thestandard format of reporting classiﬁcation accuracies, they are presented in the format (Accuracy ˘ standard deviation).
If results are not reported for a dataset, we denote this as not available (N/A).
The ﬁrst three rows of Table 1 display results for graph kernel methods; namely Weisfeiler-Lehmangraph kernels (WL, [34]), Graphlet kernels (Graphlet, [35]) and deep graph kernels (DGK, [44]).
Thesubsequent rows display results for geometric deep learning algorithms: Deep graph convolutionalneural networks (DGCNN,[48]), Patchy-san (PSCN (with k=10), [27]), recurrent neural networkautoencoders (S2S-N2N-PP, [16]) and graph isomorphism networks (GIN [43]).
These results aretaken from [12].
Additionally we compare with P-Poinc [20], which embeds nodes into a hyperbolicspace (the Poincare ball, to be precise), GSN-e [3] which combines message passing with structuralfeatures extracted via subgraph isomorphism and WKPI-kC [49] which utilizes a weighted kernelwithin its metric learning framework.
The second to last row (GS-SVM [12]) provides a result that isalso based on a method that combines a static scattering architecture with a support vector machine.
Its ﬁlters are based on graph wavelets built from differences between lazy random walks that havepropagated at different time scales.
L.2 Regression of Quantum Chemical Energies
Dataset: The dataset we consider is the QM 7 dataset, introduced in [2, 32].
This dataset containsdescriptions of 7165 organic molecules, each with up to seven heavy atoms, with all non-hydrogenatoms being considered heavy.
A molecule is represented by its Coulomb matrix C Clmb , whoseoff-diagonal elements
[Equation]
(8) correspond to the Coulomb-repulsion between atoms i and j , while diagonal elements encode apolynomial ﬁt of atomic energies to nuclear charge [32]:
[Equation]
For each atom in any given molecular graph, the individual Cartesian coordinates R i and the atomiccharge Z i are also accessible individually.
To each molecule an atomization energy - calculated viadensity functional theory - is associated.
The objective is to predict this quantity, the performancemetric is mean absolute error.
Numerically, atomization energies are negative numbers in the range´ 600 to ´ 2200 .
The associated unit is r kcal/mol s .
Scattering Architecture: Off-diagonal entries in the Coulomb Matrix clearly represent an inversedistance.
A weight of zero can then heuristically be thought of as the inverse distance between twoinﬁnitely separated atoms.
After calculating the degree matrix D associated to C , we obtain thegraph Laplacian once more as L “ D ´ C and set our normal operator to
[Equation]
If we continuously vary the distances in (8), staying clear of zero, then the adjacency matrix andhence the graph Laplacian L varies continuously.
As long as we avoid complete degeneracy, thelargest eigenvalue λ max p L q will remain positive.
This implies that our normal operator ∆ variescontinuously under changes of the inter-atomic distances, which implies that our feature vector alsovaries continuously, as distances are changed.
Connecting operators are set to the identity, whilenon-linearities are ﬁxed to ρ n ě 1 p¨q “ | ¨ | .
Filters are chosen as p sin p π { 2 ¨ ∆ q , cos p π { 2 ¨ ∆ q , sin p π ¨∆ q , cos p π ¨ ∆ qq acting through matrix multiplication.
The output generating functions are set tothe identity as well.
Graph level features are aggregated via the map N G 2 5 p¨q of Section 7; slightlymodiﬁed to neglect the normalizing factor in the ﬁrst entry for improved convenience in numericalimplementability.
As weights µ ij for our second-order feature space are set to unity and moleculargraphs in QM 7 contain at most 23 -molecules, we note that ?
µ G 2 ď ?
Going throughthe proofs of our graph-level stability results, we see that they remain valid after multiplying eachstability constant by 23 .
The Coulomb matrix (divided by a factor of 10 as this empirically improvedperformance) is then also utilized as an edge level input signal.
Node level features are obtainedby applying the above architecture to the node level information provided by the respective atomiccharges t Z i u on each graph.
We aggregate to graph level features using N G 5 (cf.
Section 6.1), again neglecting the normalizing factor in the ﬁrst entry for improved convenience in implementing.
Thenetwork depth is set to N “ 4 in both cases.
We then concatenate graph level features obtained fromnode- and edge level input into a composite scattering feature vector.
Training Procedure: The QM 7 dataset comes with a precomputed partition into ﬁve subsets; eachcontaining a representative amount of heavy and light molecules covering the entire complexity rangeof QM 7 .
To allow for 10 -fold cross validation, we further dissect each of these subsets into twosmaller datasets, one containing graphs indexed by an even number, one containing graphs indexedby an odd number.
On these 10 -subsets, we then perform 10-fold cross validation.
Among the 10folds, we iteratively pick one for testing.
Say we have picked the n t h fold for testing.
Then thereare 9 remaining folds.
We iteratively pick the m t hn (with 1 ď m n ď 9 ) of the remaining 9 foldsfor choosing hyperparameters.
This leaves 8 folds on which we train our model for each choice ofhyper parameter in C pool ˆ G pool .
This yields 8 regression models, which we average to built ourﬁnal predictor for the n t h run.
This mean absolute error of this predictor is then evaluated on the n t hfold which was retained for testing.
As n varies from one to ten, we built the mean and variance ofthe performances of the generated regression models.
To reduce training and evaluation time, wesimpliﬁed our sets of hyperparametes to only consist of single values.
We have set G pool “ t 0 .
00003 uand G pool “ t 400000 u which were found to perform well enough.
Reference Methods: We comprehensively evaluate our method against 11 popular baselines andstate of the art approaches.
Among these methods are graph convolutional methods such as GraphConv[18], Weave [17] or SchNet [33].
MPNN [13] and its variant DMPNN [46] are models consideringedge features during message passing.
AttentiveFP [42] is an extension of the graph attentionframework, while N-Gram [22] is a pretrained method.
Results for these methods as well as for
GROVER are taken from [31].
PhysChem [47] learns molecular representations via fusing physicaland chemical information.
Deep Tensor Neural Networks (DTNN [41]) are adaptable extensions of the
Coulomb Matrix featurizer mapping atom numbers to trainable embeddings which are then updatedbased on distance information and other (node-level) atomic features.
Finally Path-Augmented Graph
Transformer Networks (PATGN, [6]) exploit the connectivity structure of the data in a global attentionmechanism.